# Positioning is All You Need

Xin Li

Department of Computer Science, University at Albany, Albany NY 12222
E-mail<EMAIL_ADDRESS>
Abstract

> One can drive safely with a GPS without memorizing a world map (not to
> mention the dark regions that humans have never explored). Such a locality-
> based attention mechanism has a profound implication on our understanding of
> how the brain works. This paper refines the existing embodied cognition
> framework by turning the locality from a constraint to an advantage.
> Analogous to GPS-based navigation, positioning represents a computationally
> more efficient solution to flexible behaviors than reconstruction. This
> simple intuition implies that positioning is all you need to understand
> cognitive functions generated by hippocampal-neocortical systems. That is,
> the neocortex generates thousands of local maps whose indexing is maintained
> by the hippocampus. Geometrically, we present a simple manifold positioning
> framework to explain the principle of localized embodied cognition. The
> positioning operation implemented by the attention mechanism can be
> interpreted as a nonlinear projection linking the discovery of local
> subspace structure by the neocortex (a sensorimotor machine interacting with
> the world locally) for the navigation task in mind $without$ discovering
> global manifold topology.

## Introduction

Sensory organs attached to the mammalian brain can access only a localized
region in space at any specific time. Such spatio-temporal locality of sensory
observation has a profound implication for the mental construction of the
internal world (a.k.a., internal representation (?)) including consciousness
by the mammalian brain. Nature has discovered a clever solution to learn a
global representation of the physical world with a local receptive field of
sensory cortex by flexible behavior, including spatial navigation. How do
mammalian brains encode the spatial information about the physical world into
the population of spike sequences in the temporal domain? Such a question is
at the holy grail of neuroscience whose importance to cognitive science is
equivalent to relativity theory in modern physics.

To formulate the above problem geometrically, we note that the observation
data acquired by localized sensory organs can be abstracted by the projection
of a manifold embedded in a high-dimensional to a low-dimensional subspace. It
is well recognized that nonlinear manifold, despite the sophisticated topology
globally, is locally isomorphic to a Euclidean space. Such fundamental
property has been the inspiration for the class of nonlinear dimensionality
reduction techniques (e.g., IsoMAP (?) and locally linear embedding (?)). An
important new insight brought by this work is that we do not need to
reconstruct the global manifold but only need to know the $position$ within
the global coordinates for goal-oriented behavior. This insight is in
alignment with intelligence without representation (?) and animate vision (?)
which attempt to put perception and action on an equal footing. Using GPS as
the metaphor, we only need access to a local map without the need to memorize
the global map anywhere and anytime. Such locality principle, when combined
with embodied cognition, is a fundamental advantage that has been exploited by
nature during the evolution of mammalian brains.

Through co-evolution of the hippocampus and the neocortex, nature solves the
manifold positioning problem by the simple attention mechanism (?), which can
be mathematically modeled by context (?) and computationally implemented by
the transformer architecture (?). At the level of individual neurons,
attention or selection of firing patterns is implemented by the nonlinear
effects of inhibitory networks (?). At the level of cortical columns,
canonical circuits implementing predictive processing (?) are responsible for
context-dependent modulation of prediction errors. At the level of cortical
areas, hippocampal indexing theory (?, ?) appears to be the most comprehensive
treatment of modeling hippocampal-neocortical interactions (?). By serving as
the neocortex’s librarian (pp. 285, (?), the hippocampus uses an indexing-
based positioning strategy to dynamically stabilize the sensorimotor
activities generated by the neocortex.

In this paper, we provide a geometric interpretation of the
positioning/indexing strategy underlying hippocampal-neocortical systems. It
is shown that the problem of navigating on an unknown manifold in an arbitrary
dimension can be solved by “think globally, fit locally” (?) - i.e., to
exploit the local subspace constraint of a nonlinear manifold, the only
solution that is universal and generalizable is to explicitly learn a
nonlinear mapping that connects the local geometry with the global topology.
Since we are not interested in manifold reconstruction (e.g., the dark areas
of the world map), an indexing or positioning strategy is sufficient for
“navigating” from one place to another. This intuition is consistent with
“using the world as its own model” (?) but we are taking one step forward by
noting that we only need a localized version of the world model due to the
locality constraint of our sensory organs. Using the map analogy from a
thousand brain theory (?), we need a librarian (i.e., the hippocampus) to
maintain efficient indexing and retrieval of 150,000 maps generated by the
neocortex. This librarian can easily manage more maps as the intelligence
evolves - note that the neocortex has a virtually infinite capacity thanks to
its small-world network organization (?); while the network organization of
the hippocampus does not change much from mammals to humans.

## Toy Example

To facilitate the illustration of our core idea, we introduce a toy example
based on marrying place cells as discovered in the hippocampus (?) with the
two-spiral dataset as widely used in manifold learning literature (?), as
shown in Fig. 1. Even though the maze experiments used in studying rodent
models have not used such a spiral setting (?), we think it is plausible to
assume that mammalian brains such as rodents can easily manage a maze with a
spiral shape. The deeper implication of this toy example is that if place
cells are interpreted as spatial context, we can generalize this hypothesized
experiment to higher dimensions. For example, if object recognition is
formulated as a maze problem in a higher-dimensional space, how can we
despositionign an intelligence machine for navigation?

a) b)

Figure 1: The place cells discovered in the hippocampus provide spatial
context for the task of navigation. Such contextual modeling can be exploited
to navigate the local Euclidean space of an arbitrary manifold by “memorizing”
its position in the global coordinates.

In conventional wisdom such as IsoMAP (?) and locally linear embedding (LLE)
(?), the objective is to exploit the local subspace constraint of the manifold
by nonlinear dimensionality reduction. The drawback of this class of solutions
is that they do not generalize well to arbitrary manifolds, especially in
higher dimensions. The fundamental reason behind this failure can be explained
by intelligence without representation (?) - under the framework of embodied
cognition (?), perception and action have an equal footing. From the
evolutionary perspective, goal-oriented behavior is the foundation of natural
intelligence. Therefore, $navigation$ on the manifold is easier than
$reconstruction$ of the manifold; and nature prefers simple but not simpler
solutions. A moment of thought can tell that a place cell-based navigation
strategy can be easily generalized to a manifold of arbitrary dimension or
topology. The job of indexing or map-keeping will become more involved but it
is still manageable by a librarian.

## Computational Modeling of Hippocampal-Neocortex Systems

We advocate a new conceptual framework called “localized embodied cognition”
with the locality principle in mind from problem formulation to solution
algorithms. We still use the world “as its own model”, but the way local
embodied cognition never attempts to solve a global problem locally. An
organism needs to couple sensory with motor systems to achieve various goals
but never stores knowledge about the global environment. Instead, the bodily
interactions with the environment (situatedness) are persistently maintained
by the neocortex (a single book)’s interaction with the hippocampus (a large
library). The cognitive map in the hippocampus offers a situational context to
interpret the sensorimotor interaction occurring in the neocortex. In this
section, we first present computational models for characterizing indexing
operation in the hippocampus and sensorimotor interaction in the neocortex and
then connect them with a universal attention mechanism.

a) b)

Figure 2: Computational modeling of hippocampal-neocortical coupling.

### Hippocampal Indexing Theory.

The key idea underlying hippocampal indexing theory (?, ?) is that the
hippocampus is “functional designed and anatomically situated” to handle the
diverse neocortical activities generated by sensorimotor interactions. By
projecting back to the neocortex, the hippocampus stores a collection of
indexes that can serve as the context for the storage and retrieval of
episodic memory (?). Recent findings indicate a more complex interaction
between hippocampal functioning and episodic memory, with an emphasis on the
cognitive map interpretation (?). This newer perspective considers the
hippocampus as creating spatial maps (cognitive maps (?)) for memory
formation. The theory also recognizes the hippocampus’s role in binding
different elements of an experience, including time and space, into a cohesive
memory (?). Furthermore, the relationship between place cells in the
hippocampus and memory formation suggests that these cells may function both
as part of a cognitive map and as index cells for memory (?).

### Neocortex as a Sensorimotor Machine.

One of the biggest mysteries in studying cortical columns is that they are
structures without a function (?). Instead, several canonical microcircuits
such as Douglas-Martin model (?) and Haeusler-Maass model (?) have been
studied for predictive coding in the literature. These models have revealed a
remarkable correspondence between the microcircuitry of the cortical column
and the connectivity required by predictive coding. In Douglas-Martin model
(?), information flow through the cortical column is characterized by a
stereotypical pattern of fast excitation followed by slower and longer-lasting
inhibition in the cat visual system. The three neuronal populations receive
thalamic drive and amplify transit thalamic inputs to generate sustained
activities while maintaining a balance between excitation and inhibition. This
model was further extended by Haeusler and Maass in (?), which closely
resembles the canonical circuits required by predictive coding. The
feedforward prediction errors from a lower cortical level arrive at granular
layers and are passed to excitatory and inhibitory interneurons in
supragranular layers, encoding expectations. Meanwhile, the connections
between excitatory and inhibitory neurons in supragranular layers enable deep
pyramidal cells and excitatory interneurons to generate context-dependent
feedback predictions, which descend to a lower hierarchical level.

### Hippcampal-Neocortical Interaction

. The coupling of the hippocampus with the neocortex is based on the principle
of localized embodied cognition (please refer to Fig. 2). Following the spirit
of intelligence without representation (?), we use the localized world as its
own model which nicely fits the functional design of the neocortex (i.e.,
detecting regularity in stimuli by sensorimotor interaction). To accommodate
the constantly changing properties of the global world model, an organism
needs a cognitive map for navigation. This map can be mathematically
abstracted by a nonlinear mapping between local subspace geometry as
discovered by the neocortex and global topology as maintained by the
hippocampus. Instead of reconstructing the entire manifold, we argue it is
sufficient to retrieve its local projection based on the context information.
This nonlinear mapping is essentially an attention mechanism that can be
implemented by a sparse and distributed memory (SDM) (?) (attention
approximates SDM (?)) or an associative memory (e.g., modern Hopfield network
(?) is intrinsically connected with the transformer architecture (?)). SDM-
based indexing/positioning supports continual learning (?). Real-world map-
based navigation serves as a perfect example of the plausibility of this
manifold positioning strategy. Why can we safely drive around using GPS
without memorizing the entire world? Because at any time and location, we only
need access to a localized version of the world map for navigation purposes.
Note that there is no need to reconstruct a map for an area if no one ever
goes there, which is consistent with goal-oriented behavior in natural
intelligence.

## Applications into Cognitive Science

In this section, we revisit four widely studied cognitive tasks from the new
perspective of locally navigating on a globally unknown manifold. The unifying
theme is that positioning is all you need to understand how the human brain
solves these problems by a universal cortical processing algorithm, as
envisaged by Vernon Mountcastle in 1957 (?).

### Spatial Navigation.

Cognitive maps organize knowledge to support flexible and adaptive behavior.
Among various behavioral tasks, spatial navigation is among the most well-
studied in the literature (?, ?). How do mammalian brains encode 2D physical
space information into 1D spike sequence? This question is at the core of
understanding the relationship between space and time in its simplest form.
The representation of spatial information (Euclidean or non-Euclidean) plays
an essential role in the building of cognitive maps. A graph-based
representation has been widely adopted for encoding the physical space into
action-related state space. A fundamental problem is the aliasing problem
because identical sensory observations can occur in different spatial
locations (?). The aliasing problem is solved by attention in the hippocampus
where the encoding of spatio-temporal context (index to the local map or
position on the global map) is handled by CA3-CA1 collateral system (pp. 326
(?)). The consolidation of semantic memory (analogy to global manifold
topology) from episodic memory (analogy to local Euclidean geometry)
eliminates the spatio-temporal context to support context-dependent inference
where context is defined by the attention window associated with different
spatial and temporal locations.

### Motor Control.

Motor control refers to the process of physically interacting with objects in
the world and manipulating them toward completing a specific task/goal (e.g.,
reach and grasp). In motor control and trajectory planning (?), internal
models consist of: 1) a forward component responsible for predicting sensory
consequences of the executed motor commands; and 2) an inverse dynamics model
that calculates forward motor commands from the desired trajectory
information. Both components require an attention-guided and action-oriented
body map (?) that is centered on the controlled object. The generalization
property of motor controls lies in the transformation from one coordinate to
another. For example, the task of grasping a water bottle differs from that of
grasping a coffee cup at the surface; but the principle of motor commands is
the same except for the change of coordinates (from egocentric for spatial
navigation to allocentric for motor control). Using the library as the
metaphor again, what the hippocampus needs to adapt is to $re-position$ the
neocortex - i.e., from one book of grasping without a handle to another with a
handle. Unlike navigation, object manipulation does not require path
integration for trajectory planning because the scale of motor control is the
same as that of the reference frame centered on the object.

### Object Detection.

In the spatial navigation of rodents, the environment or place learning is
achieved by ego-motion of animals, which dynamically changes the bearing and
sketch maps (?). Such behavior becomes more sophisticated as the evolution of
the visual cortex helps locate salient objects (a.k.a. landmarks) in the
environment (?). From a manifold positioning perspective, we argue that there
is an appealing analogy between the integrated map for spatial navigation and
the shape description for animate vision (?). That is, the problem of
object/landmark detection can also be solved by the inferior temporal (IT)
cortex (?) assuming that the spatial-temporal context is provided by the
hippocampus. Mathematically, a place cell can be abstracted by a Dirac
function $\delta(x,y)$ that only fires when the organism is in the
neighborhood of $(x,y)$. By analogy, a gnostic cell (a.k.a. concept cell (?,
?) can fire if and only if an object is detected by the neocortex in a latent
space (e.g., generalized Hough transform (?)). The spatial attention mechanism
implemented by the gaze becomes critically important here because it reduces
the computational burden by indexical reference (?). More importantly,
positioning of the gaze direction is naturally required by the visual cortex
to recognize an object based on the detection result (?).

### Language Comprehension.

Action-oriented representation facilitates understanding the constitution
between language and action in the theory of embodied language (?). By
restoring the central role played by the motor system in language processing,
we can gain deeper insight into the essence of language from a motor resonance
or action simulation perspective (?). The class of referential motor resonance
is the source of abstract thoughts or covert movements in our mental world,
which are hierarchically organized. The relationship between hierarchical
motor control (?) and the nested structure of natural languages (?) can be
better understood from a hierarchical extension of the attention perspective.
Both language and motor control demonstrate compositionality, whereby complex
structures are built from simpler elements according to combinatorial rules.
In natural language, sentences are composed of phrases composed of smaller
units such as words and morphemes. This syntactic structure manifests as the
ability to generate an infinite variety of sentences from a finite set of
words and grammatical rules. Similarly, in motor control, movements can be
decomposed into smaller units, such as motor primitives, which are organized
hierarchically to generate complex behaviors. Moreover, movements can comprise
sequences of simpler actions or motor primitives, combined to produce more
complex behaviors. The latest study on relating neural representation of the
hippocampal formation (e.g., Tolman-Eichenbaum Machine (?)) to transformer (?)
seems to support that attention is needed for both spatial navigation and
language comprehension. The curios position formula of tokens in the
transformer architecture using sine and cosine functions seems the de-aliasing
result that is conceptually similar to the omnidirectional place cells in
cognitive maps (pp. 330, (?)).

## References and Notes

  * 1. R. A. Brooks, Artificial intelligence 47, 139 (1991).
  * 2. J. B. Tenenbaum, V. d. Silva, J. C. Langford, science 290, 2319 (2000).
  * 3. S. T. Roweis, L. K. Saul, science 290, 2323 (2000).
  * 4. D. H. Ballard, Artificial Intelligence 48, 57 (1991).
  * 5. M. I. Posner, C. D. Gilbert, Proceedings of the National Academy of Sciences 96, 2585 (1999).
  * 6. N. Cesa-Bianchi, G. Lugosi, Prediction, learning, and games (Cambridge university press, 2006).
  * 7. A. Vaswani, et al., Advances in neural information processing systems 30 (2017).
  * 8. G. Buzsáki, Rhythms of the Brain (Oxford university press, 2006).
  * 9. G. B. Keller, T. D. Mrsic-Flogel, Neuron 100, 424 (2018).
  * 10. T. J. Teyler, P. DiScenna, Behavioral neuroscience 100, 147 (1986).
  * 11. T. J. Teyler, J. W. Rudy, Hippocampus 17, 1158 (2007).
  * 12. J. L. McClelland, B. L. McNaughton, R. C. O’Reilly, Psychological review 102, 419 (1995).
  * 13. L. K. Saul, S. T. Roweis (2003).
  * 14. J. Hawkins, A thousand brains: A new theory of intelligence (Hachette UK, 2021).
  * 15. J. O’keefe, L. Nadel, The hippocampus as a cognitive map (Oxford university press, 1978).
  * 16. E. C. Tolman, Psychological review 55, 189 (1948).
  * 17. M. Wilson, Psychonomic bulletin & review 9, 625 (2002).
  * 18. E. Tulving, Annual review of psychology 53, 1 (2002).
  * 19. J. C. Whittington, et al., Cell 183, 1249 (2020).
  * 20. J. C. Whittington, D. McCaffary, J. J. Bakermans, T. E. Behrens, Nature Neuroscience 25, 1257 (2022).
  * 21. B. P. Staresina, L. Davachi, Neuron 63, 267 (2009).
  * 22. M.-B. Moser, D. C. Rowland, E. I. Moser, Cold Spring Harbor perspectives in biology 7, a021808 (2015).
  * 23. J. C. Horton, D. L. Adams, Philosophical Transactions of the Royal Society B: Biological Sciences 360, 837 (2005).
  * 24. R. J. Douglas, K. Martin, The Journal of physiology 440, 735 (1991).
  * 25. S. Haeusler, W. Maass, Cerebral cortex 17, 149 (2007).
  * 26. P. Kanerva, Sparse distributed memory (MIT press, 1988).
  * 27. T. Bricken, C. Pehlevan, Advances in Neural Information Processing Systems 34, 15301 (2021).
  * 28. H. Ramsauer, et al., arXiv preprint arXiv:2008.02217 (2020).
  * 29. T. Bricken, X. Davies, D. Singh, D. Krotov, G. Kreiman, arXiv preprint arXiv:2303.11934 (2023).
  * 30. V. B. Mountcastle, Journal of neurophysiology 20, 408 (1957).
  * 31. M. Kawato, Current opinion in neurobiology 9, 718 (1999).
  * 32. E. M. Gordon, et al., Nature 617, 351 (2023).
  * 33. L. F. Jacobs, F. Schenk, Psychological review 110, 285 (2003).
  * 34. P. S. Churchland, V. S. Ramachandran, T. J. Sejnowski (1993).
  * 35. J. J. DiCarlo, D. Zoccolan, N. C. Rust, Neuron 73, 415 (2012).
  * 36. R. Q. Quiroga, L. Reddy, G. Kreiman, C. Koch, I. Fried, Nature 435, 1102 (2005).
  * 37. R. Q. Quiroga, G. Kreiman, C. Koch, I. Fried, Trends in cognitive sciences 12, 87 (2008).
  * 38. D. H. Ballard, Pattern recognition 13, 111 (1981).
  * 39. D. H. Ballard, IJCAI (Citeseer, 1989), vol. 89, pp. 1635–1641.
  * 40. M. H. Fischer, R. A. Zwaan, Quarterly journal of experimental psychology 61, 825 (2008).
  * 41. R. A. Zwaan, L. J. Taylor, Journal of Experimental Psychology: General 135, 1 (2006).
  * 42. J. Merel, M. Botvinick, G. Wayne, Nature communications 10, 5489 (2019).
  * 43. N. Chomsky (1957).
  * 44. J. C. Whittington, J. Warren, T. E. Behrens, International Conference on Learning Representations (2022).


# SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis

Jiaxin Pei ♣ Vítor Silva† Maarten Bos † Yozen Liu†
Leonardo Neves† David Jurgens♣ Francesco Barbieri†
♣School of Information, University of Michigan, Ann Arbor, MI, USA
†Snap Inc, Santa Monica, CA, USA
♣{pedropei<EMAIL_ADDRESS>
†{fbarbieri, vsilvasousa, maarten, yliu2<EMAIL_ADDRESS>

###### Abstract

We propose Mint, a new Multilingual intimacy analysis dataset covering 13,372
tweets in 10 languages including English, French, Spanish, Italian,
Portuguese, Korean, Dutch, Chinese, Hindi, and Arabic. We benchmarked a list
of popular multilingual pre-trained language models. The dataset is released
along with the SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis.

##  1 Introduction

Intimacy has long been viewed as a primary dimension of human relationships
and interpersonal interactions (Maslow, 1981; Sullivan, 2013; Prager, 1995).
Existing studies suggest that intimacy is an essential component of language
and can be modeled with computational methods (Pei and Jurgens, 2020). Textual
intimacy is an important social aspect of language, and automatically
analyzing it can help to reveal important social norms in various contexts
(Pei and Jurgens, 2020). Recognizing intimacy can also serve as an important
benchmark to test the ability of computational models to understand social
information (Hovy and Yang, 2021).

Despite the importance of intimacy in language, resources on textual intimacy
analysis remain rare. Pei and Jurgens (2020) annotated the first textual
intimacy dataset containing 2,397 English questions collected mostly from
social media posts and fictional dialogues. However, phrases following the
question structure are used for interrogative situations, and models trained
over it may not generalize well over texts in other forms of languages.

To further promote computational modeling of textual intimacy, we annotated a
new multilingual textual intimacy dataset named Mint. Mint covers tweets in 6
languages as the training data, including English, Spanish, French,
Portuguese, Italian, and Chinese, covering major languages used in The
Americas, Europe, and Asia. A total of 12,000 tweets are annotated for the 6
languages. To test the model generalizability under the zero-shot settings, we
also annotated small test sets for Dutch, Korean, Hindi, and Arabic (500
tweets for each).

We benchmarked a series of large multilingual pre-trained language models
including XLM-T (Barbieri et al., 2021), XLM-R (Conneau et al., 2019), BERT
(Devlin et al., 2018), DistillBERT (Sanh et al., 2019) and MiniLM (Wang et
al., 2020). We found that distilled models generally perform worse than other
normal models, while the XLM-R model trained over the twitter dataset (XLM-T)
performs the best on 7 languages. While the pre-trained language models are
able to achieve promising performance, zero-shot prediction of unseen
languages remains challenging especially for Korean and Hindi.

##  2 Data

We choose Twitter as the source of our dataset as Twitter is a public media
platform that naturally includes multilingual text data, and from our
analysis, a fair amount of intimate texts. In this section, we introduce the
data collection and annotation process for Mint.

###  2.1 Sampling

We use tweets sampled from 2018 to 2022. We use the lang_id key in the tweet
object to select English and Chinese tweets. For other languages, we use
fastText (Joulin et al., 2016b, a) for language
identification111https://fasttext.cc/docs/en/language-identification.html and
assign language labels when the model confidence is larger than 0.8. All the
mentions of unverified users are replaced with a special token “@user” during
pre-processing to remove noise from random and very infrequent usernames.

We fine-tune XLM-T, a multilingual RoBERTa model adapted to the Twitter domain
(Barbieri et al., 2021) over the annotated question intimacy dataset (Pei and
Jurgens, 2020). The fine-tuned model attained a Pearson’s $r$ of 0.80 for
English questions222Pei and Jurgens (2020) report a Pearson’s $r$ of 0.82
using RoBERTa-base. Given that XLM-T is pre-trained on tweets, a Pearson’s $r$
of 0.8 is reasonable. and we use it to estimate the intimacy of all the
collected tweets in 10 languages. Then, in the second step, we split the
tweets into 5 buckets based on the estimated intimacy and up-sampled
relatively more intimate tweets. We did bucket sampling for 1,000 English
tweets and randomly sampled another 1,000 English tweets as well as all the
tweets for the rest of the languages333We intended to do bucket sampling for
all the data, however, due to an issue in the pre-processing, we were only
able to do it for 1,000 tweets. We conducted further analyses for the
potential effect of this error. We found that the distribution of the final
annotated intimacy scores are not changed much while the fine-tuned XLM-T only
achieved a Pearson’s $r$ of 0.43 on the random sample, suggesting that the
model trained on Reddit questions may not be reliable enough to detect
intimacy in tweets. .

English | Intimacy
---|---
Ukrainian Railways Chief Says ‘Honest’ Belarusians Are Cutting Russian Supplies By Train http | 1.00
19:04h Temp: 28.9°F Dew Point: 19.40°F Wind:SSW 4.3mph Rain:0.00in. Baro:29.66 inHg via MeteoBridge 3.2 | 1.00
A team that shops together stays together…helping life go right @StateFarm http | 1.00
Leicester City fans - keep an eye on Ross Barkley. Could be moving to the Foxes on a permanent for £11m… #lcfc | 1.25
@user They aren’t open | 1.25
Kenya I vote for #Butter for #BestMusicVideo at the 2022 #iHeartAwards @BTS_twt | 1.25
That might have been the best episode of power ever | 1.40
@user Coming to USA if Trump loses in 2020. | 1.40
Change the formula to get a different result | 1.60
@user thank u | 2.50
@user Happy birthday! | 2.60
it’s the worst feeling when you feel like no matter how much u do for a person you’ll never get the same in return | 3.00
@user you’re not my mom | 3.00
@user @user Love you | 4.00
I am SO ecstatic I’m not married to a man who has cheated on me. | 4.33
My nails so mf ghetto. I’m embarrassed | 4.67
need a kiss | 4.75

Table 1: A sample of annotated tweets in English

###  2.2 Annotation

We recruited annotators from Prolific.co and paid them $15 USD per hour for
their annotations. We set a “first language” requirement during annotator pre-
screening. For example, an annotator must meet the requirement of “Spanish as
the first language” to annotate the intimacy of Spanish tweets. Intimacy is
annotated using a 5-point Likert scale where 1 indicates “Not intimate at all”
and 5 indicates “Very intimate”. In pilot annotations, we explored 7-point
likert scales as well as Best-Worst-Scaling (BWS) similar to Pei and Jurgens
(2020) and calculated the Krippendorff’s $\alpha$ to measure inter-annotator
agreement (IAA). We found that the 5-point-likert scale annotations ($\alpha$
= 0.38) achieve IAA similar to BWS ($\alpha$ = 0.36) and have higher IAA than
a 7-point likert scale ($\alpha$ = 0.25). For each language in the training
set, we collected annotations for 2,000 tweets. Because they annotated the
intimacy of tweets, the annotators could see sexual or potentially offensive
content during annotation. Therefore, we required annotators to be at least 18
years old to work on our task. Each annotator was explicitly notified about
the potential for sexual or offensive content and they signed a consent form
before starting the annotations. Each tweet was annotated by 7 annotators and
each annotator was shown 50 tweets. After the annotation, each annotator was
required to complete a post-study survey about their demographics including
gender, age, religion, country, education background, and occupation. For
tweets that were not in the target language or that did not make sense (e.g.
random characters), the annotators were instructed to annotate them as
_Invalid Tweet_. We use POTATO (Pei et al., 2022) to set up all the annotation
interfaces.

###  2.3 Quality control

Annotating textual intimacy is challenging because of the subjective nature of
intimacy perception and potential individual rating bias. We designed a series
of quality control procedures throughout the annotation process: (1) we
conducted 10 pilot studies on Prolific.com and revised our annotation
procedures according to attained IAA and participant feedback; (2) annotation
guidelines for each language were carefully translated by native
translators,444Chinese, Spanish, Dutch, French, Korean, Portuguese, Hindi, and
Italian guidelines are translated by native speakers in Snap Inc. and the
University of Michigan. The Arabic guideline was translated by one expert
translator and one expert proofreader recruited from Upwork.com, and both
translators were paid $13/h. which prompts the annotators to think about
intimacy in their own languages; (3) all instructions in the recruitment phase
were written in the annotator’s indicated first language in the recruitment
phase, which could potentially remove potential spam annotators in
crowdsourcing platforms; (4) we randomly inserted two attention test
questions555“This is a test question, please select N” where N was a random
number between 1-5 to identify potential spammers; (5) the annotators were
balanced by sex (based on Prolific’s built-in feature) and were also generally
diverse regarding other demographics (e.g. the annotators are from 73 unique
countries and regions), which allowed us to collect more population
representative ratings.

###  2.4 Post processing

We first removed annotations from users who failed the attention test since
they may have been spammers. No more than 2 annotators were removed in this
step, except for Hindi (26 removed), Korean (7 removed), and Arabic (4
removed). To remove potential noise in the crowdsourcing setting, similar to
trimmed mean (Millsap and Maydeu-Olivares, 2009), we removed one highest score
and one lowest score for tweets with at least five labels. After all the
processing, we kept the tweets with at least two valid scores. For external
test languages (i.e. Dutch, Hindi, Korean, and Arabic), we only kept tweets
with a relatively low label diversity (i.e. standard deviation lower than 1)
to ensure a good golden test set for the zero-shot setting66640%, 15%, 17%,
16% of tweets are removed for Hindi, Dutch, Korean, and Arabic respectively.
The final intimacy score is calculated as the mean score of all the remaining
labels for each tweet.

###  2.5 Annotation result

The final dataset includes 13,372 tweets annotated with the textual intimacy
score. Table 2 shows the final statistics for the annotated data. We attained
moderate inter-annotator-agreement, similar to previous work (Pei and Jurgens,
2020). Given the subjective nature of intimacy perception, we believe that
such an IAA score is promising. To further verify the quality of the
annotations, we conduct a split-half-reliability test (SHR; Johnson and Penny,
2022): randomly splitting labels into two groups and calculating the Pearson
correlation between the aggregated scores from the two groups. All the SHR
scores are above 0.63 with an average of 0.68, suggesting that the final
aggregated scores are reliable. Figure 1 shows the intimacy distribution of
the final dataset. The final dataset for English, Spanish, French, Italian,
Portuguese, and Chinese is split into training, validation, and test sets
following a ratio of 7:1:2, and all the tweets are held as the test set for
Arabic, Dutch, Korean, and Hindi.

Figure 1: The distribution of intimacy scores for each language

Language $\alpha$ SHR Amount English 0.48 0.69 1,983 Spanish 0.52 0.72 1,991
Portuguese 0.45 0.66 1,994 Italian 0.43 0.63 1,916 French 0.47 0.67 1,981
Chinese 0.44 0.64 1,996 Hindi 0.61 0.68 280 Korean 0.53 0.67 411 Dutch 0.48
0.68 413 Arabic 0.58 0.74 407

Table 2: Statistics for the annotated dataset

##  3 Baseline Models

We benchmark several baseline models on the tweet intimacy prediction task. We
compare the following multilingual pre-trained language models:

  1. 1.

BERT (Devlin et al., 2018): multilingual BERT model.

  2. 2.

XLM-R (Conneau et al., 2019): multilingual RoBERTa model.

  3. 3.

XLM-T (Barbieri et al., 2021): Multilingual RoBERTa model trained over 200M
tweets.

  4. 4.

DistillBERT (Sanh et al., 2019): Multilingual distilled BERT model.

  5. 5.

MiniLM (Wang et al., 2020): Multilingual MiniLM model.

All the models are trained with 10 epochs and the best performing model is
selected based on the dev set777we evaluate the model performance every 500
steps and choose the best model. We train each model with 5 different random
seeds and report the mean score. The learning rate is set as 0.001 and the
batch size is 64. We use AdamW as the optimizer.

model | XLM-T | BERT | XLM-R | DistillBERT | MiniLM
---|---|---|---|---|---
English | 0.70 | 0.59 | 0.65 | 0.55 | 0.61
Spanish | 0.73 | 0.62 | 0.64 | 0.61 | 0.67
Portuguese | 0.65 | 0.54 | 0.61 | 0.52 | 0.53
Italian | 0.70 | 0.57 | 0.67 | 0.58 | 0.62
French | 0.68 | 0.55 | 0.63 | 0.54 | 0.57
Chinese | 0.70 | 0.65 | 0.72 | 0.67 | 0.65
Hindi | 0.24 | 0.09 | 0.24 | 0.17 | 0.18
Dutch | 0.59 | 0.47 | 0.60 | 0.44 | 0.57
Korean | 0.35 | 0.32 | 0.33 | 0.26 | 0.41
Arabic | 0.64 | 0.35 | 0.48 | 0.32 | 0.38
overall | 0.58 | 0.48 | 0.53 | 0.52 | 0.53

Table 3: Performance of the baselines. The bottom four rows are tested under
the zero-shot setting.

Table 3 shows the performance of the baselines. We found that XLM-T achieved
the best performance over 7 languages, suggesting that domain-specific
language model training is beneficial for our tweet intimacy analysis task.

For zero-shot tasks, while XLM-T still performs the best on Hindi and Arabic,
XLM-R and MiniLM achieved the best result on Dutch and Korean, respectively.
Moreover, the zero-shot performance is generally lower compared with the tasks
with in-domain training, suggesting that the zero-shot task is challenging. We
encourage the Task participants to explore different strategies to improve the
zero-shot intimacy prediction performance.

##  4 Conclusion

In this paper, we present Mint, the dataset for SemEval 2023 Task 9:
Multilingual Tweet Intimacy Analysis and benchmarked a series of multilingual
pre-trained language models. While the models are generally able to predict
textual intimacy with promising performance, how to infer textual intimacy for
unseen language remains challenging especially for Korean and Hindi. We
encourage the SemEval participants to explore advanced methods for both
monolingual and multilingual tweet intimacy analysis tasks.

##  5 Acknowledgment

Our multilingual tweet intimacy annotation has been approved by the IRB office
of the University of Michigan (HUM00214259).

## References

  * Barbieri et al. (2021) Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. 2021.  Xlm-t: A multilingual language model toolkit for twitter.  _arXiv preprint arXiv:2104.12250_.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised cross-lingual representation learning at scale.  _arXiv preprint arXiv:1911.02116_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Hovy and Yang (2021) Dirk Hovy and Diyi Yang. 2021.  The importance of modeling social factors of language: Theory and practice.  In _The 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics.
  * Johnson and Penny (2022) Robert L. Johnson and Jim Penny. 2022.  Split-half reliability.  _The SAGE Encyclopedia of Research Design_.
  * Joulin et al. (2016a) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016a.  Fasttext.zip: Compressing text classification models.  _arXiv preprint arXiv:1612.03651_.
  * Joulin et al. (2016b) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016b.  Bag of tricks for efficient text classification.  _arXiv preprint arXiv:1607.01759_.
  * Maslow (1981) Abraham Harold Maslow. 1981.  _Motivation and personality_.  Prabhat Prakashan.
  * Millsap and Maydeu-Olivares (2009) Roger E Millsap and Alberto Maydeu-Olivares. 2009.  _The SAGE handbook of quantitative methods in psychology_.  Sage Publications.
  * Pei et al. (2022) Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Jackson Sargent, Apostolos Dedeloudis, and David Jurgens. 2022.  Potato: The portable text annotation tool.  _arXiv preprint arXiv:2212.08620_.
  * Pei and Jurgens (2020) Jiaxin Pei and David Jurgens. 2020.  Quantifying intimacy in language.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5307–5326.
  * Prager (1995) Karen Jean Prager. 1995.  _The psychology of intimacy_.  Guilford Press.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _ArXiv_ , abs/1910.01108.
  * Sullivan (2013) Harry Stack Sullivan. 2013.  _The interpersonal theory of psychiatry_.  Routledge.
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.