# Bridging Natural Language Processing and Psycholinguistics: computationally
grounded semantic similarity datasets for Basque and Spanish

Josu Goikoetxea, Miren Arantzeta, Itziar SanMartin222Correspondence authors:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We present a computationally-grounded word similarity dataset based on two
well-known Natural Language Processing resources; text corpora and knowledge
bases. This dataset aims to fulfil a gap in psycholinguistic research by
providing a variety of quantifications of semantic similarity in an extensive
set of noun pairs controlled by variables that play a significant role in
lexical processing. The dataset creation has consisted in three steps, 1)
computing four key psycholinguistic features for each noun; concreteness,
frequency, semantic and phonological neighbourhood density; 2) pairing nouns
across these four variables; 3) for each noun pair, assigning three types of
word similarity measurements, computed out of text, Wordnet and hybrid
embeddings. The present dataset includes noun pairs’ information in Basque and
European Spanish, but further work intends to extend it to more languages.

###### keywords:

psycholinguistics, natural language processing , word similarity , semantic
similarity , semantic distance , embeddings, corpora, WordNet, Basque ,
Spanish

††journal: XXXXX††journal: Journal of LaTeX Templates

##  1 Introduction

Semantic similarity is a measure of distance between items based on how
similar their meanings are. That is, it is a type of relationship based on
shared characteristics between concepts. For instance, ’cat’ has more
remarkable semantic similarity with ’tiger’ than ’rhinoceros’. In contrast,
semantic relatedness, frequently used interchangeably with semantic
similarity, denotes any relation between concepts, not necessarily
taxonomical. For instance, ’cat’ is related to ’tiger’, ’milk’, and
’veterinary’. In short, semantic similarity offers a metric of categorical
semantic relations, whereas semantic relatedness is closer to depicting
associative relations. By now, we will use these two concepts interchangeably
and show how the metrics used in the present dataset constitute independent
proxies of different types of semantic relations.

Semantic relations impact the underlying capability of many cognitive
processes, such as memory recall, mental imagery or language comprehension.
They have been the focus of extensive research in Natural Language Processing
(NLP) and psycholinguistics. There are computational tools to gauge the
strength of semantic connections between concepts. Two important semantic
resources in NLP are text corpora and knowledge bases (KB), both of them
useful in cognitive experiments involving semantic similarity. In the case of
KBs, they provide a quantitative and structured framework for representing
words’ meanings.

This work presents two automatically calculated semantic similarity datasets
restricted to nouns, to provide useful material to build up psycholinguistic
experiments; one in Basque and the other in Spanish. The similarity
measurements of noun pairs have been computed using vectors or embeddings
built up in text corpora and KB. In order to capture different nuances of
semantic similarity between noun pairs, in this work we employed three types
of word representations; text embeddings, KB-based embeddings and hybrid
embeddings (see 4.3). The selection of noun pairs has been conducted by
controlling for word frequency, concreteness, and the number of semantic and
phonologic neighbours due to the widely proven effect of these variables in
language processing. Controlling for these linguistic features allows
leveraging crucial lexical properties within pairs of nouns beyond semantic
similarity. It broadens the usability of the dataset and the interpretation of
the results.

In the following lines, we will describe the procedures that have been
followed in each stage of the construction of the dataset, as well as the
characteristics of the final dataset.

##  2 Related work

Psycholinguistics and NLP have been closely linked for a long time. In
semantic similarity computation, two methods have been mainly used; KB-based
and embedding-based. The former treats KBs as graphs and exploit their
complete structural information, so that similarity measurements are based on
the KB taxonomy. The latter encodes word meaning in a numeric vector or
embedding in a Euclidean space following the Distributional Hypothesis
(Harris, 1954). Thus, embeddings latently encode the semantic (and also
syntactic) features, and therefore, they offer comparable representations for
words.

Embedding measurements have revolutionized the field of NLP, increasing model
accuracy significantly in practically every task. Further, they have overcome
KB-based ones, proving to be more robust resources than the latter in word
similarity tasks (Lastra-Díaz et al., 2019).

In recent years, several works have given meaningful insights into the
correspondence between word-embedding features and human cognition. Despite
the misconceptions in understanding word embedding from a cognitive
perspective (Günther et al., 2019), they remain the most meaningful proxy for
human semantic representations. Thus, the performance of embeddings in the
semantic similarity tasks has gained attention lately in combination with a
variety of experimental methods, such as semantic priming (Ettinger and
Linzen, 2016; Auguste et al., 2017; Hollenstein et al., 2019), brain imaging
(Søgaard, 2016; Abnar et al., 2017; Ruan et al., 2016; Jain and Huth, 2018;
Rodrigues et al., 2018; Toneva and Wehbe, 2019) or eye-tracking (Luke and
Christianson, 2018; Cop et al., 2017; Hollenstein et al., 2019) (see also
(Bakarov, 2018)).

Mandera et al. (2017) proved that word vectors successfully explain semantic
priming data, stating that they equal or overperform human-rated association
datasets or feature norms. Salicchi et al. (2021) showed a strong correlation
between similarity measurements of contextual and non-contextual embeddings
and two English corpora annotated by eye-tracking measures. Hayes and
Henderson (2021) explored the relationship between the visual scene and
attention, grounding the semantic scene representation with word embeddings
and evidencing the strong relationship between the semantic similarity of a
scene region and the gaze-fixation pattern of the viewers.

Other researches on psycholinguistics have gauged the semantic distance of
words via KB-based methods, as the latter provide a means to quantify the
relationships between words and concepts within the semantic structure of a
language. Kenett et al. (2017) created a Hebrew KB, quantifying semantic
distance as the path length between word pairs by counting the number of steps
for traversing from one word to another in the KB. They stated that a distance
of 4 was the turning point for the performance in the semantic relatedness
judgement task and subsequent recall from memory. Similarly, Benedek et al.
(2017) proved that a KB based on semantic relatedness judgements was feasible
and valuable to test the associative and executive accounts of creativity.
Likewise, Jelodar et al. (2010) proposed a model for concrete nouns’ feature
vectors based on Wordnet (Miller, 1995) similarity measurements for predicting
brain activation patterns as measured with fMRI. It outperformed previous
models.

Psycholinguistic research on ageing is emerging, and in this context,
embeddings and KBs are also gaining momentum in the study of ageing lexico-
semantic processing (Wulff et al., 2019). For example, Broderick et al. (2021)
analysed the differences between the correspondence of two computational
measurements (semantic dissimilarity and lexical surprisal) and event-related
potentials in young and elderly subjects while performing a sentence
comprehension task. They revealed dissociable neural correlates between the
two populations, and showed that the two groups processed context-based
predictions differently. Cosgrove et al. (2021) also studied the impact of
ageing on the performance of a verbal fluency task based on network sciences,
and provided quantitative evidence of the decrease of flexibility in adults’
semantic memory networks.

##  3 Extra linguistic features included in the dataset

Although the core of this dataset is word similarity calculation, the dataset
has been controlled by four additional features: concreteness, frequency,
semantic neighbourhood density and phonological neighbourhood density. The
following sections summarize the scientific evidence highlighting these
variables’ significant role in psycholinguistics and NLP. A more profound
comprehension of the connections between various linguistic features and word
similarity from a cognitive viewpoint is needed. This dataset may lead to
gaining insight into ongoing studies.

###  3.1 Concreteness

Concreteness is a term to refer to the degree to which a word denotes a
tangible thing. This measurement was introduced by Paivio (1971), and it has
been proven to play a role in several aspects, such as working memory (Mate et
al., 2012), embodied cognition (Barsalou, 1999; Fischer and Zwaan, 2008; Hauk
et al., 2004), and neural representations on word processing (Wang et al.,
2010). The literature suggests that concrete words show thicker links to
associated semantic information and involve visual imagery processes.
Accordingly, they elicit faster response times and larger N400 and N700
electrophysiological signals (Schwanenflugel, 2010; Wang et al., 2010).
Remarkably, when contextual information and mental imageability are
controlled, response times become faster for abstract words. However, the
neural correlates do not change, suggesting that concrete words involve more
significant semantic processing during meaning activation (Barber et al.,
2013).

Due to the relevance of concreteness in psycholinguistic research, several
works on NLP have computationally predicted concreteness values using word
embeddings (Ljubešić et al., 2018; Charbonnier and Wartena, 2019; Incitti and
Snidaro, 2021), which have been proven to be useful in tasks like metaphor
detection (Tsvetkov et al., 2014; Alnafesah et al., 2020) and sentiment
analysis (Rothe et al., 2016; Long et al., 2019). In Long et al. (2019), for
example, the authors propose a cognition-grounded attention model in sentiment
analysis, considering concreteness for leveraging the model’s attention
mechanism. Further, concreteness seems to be gaining more attention in the NLP
field, as some concreteness norms rated by humans have been published in
various languages lately, being the English (Brysbaert et al., 2014b) and
Dutch (Brysbaert et al., 2014a) ones quite extensive, and the Croatian (Ćoso
et al., 2019), Russian (Solovyev et al., 2022), French (Bonin et al., 2018)
and Spanish (Guasch et al., 2016) ones more reduced.

###  3.2 Word frequency

Word frequency is also key in both psycholinguistics and NLP. In the former,
it is accepted that the rate at which something occurs in language affects how
individuals process and recall information. This effect has been widely
studied in lexical access (Balota and Chumbley, 1984; Balota et al., 2004;
Brysbaert and New, 2009) and recall tasks (MacLeod and Kampe, 1996; Gregg,
1976; Kinsbourne and George, 1974), which involve memory usage. The effect of
lexical frequency on human cognition has also been supported based on
electrophysiological correlates (Strijkers et al., 2010) and gaze-fixation
patterns in spoken-word recognition (Dahan et al., 2001) (for a review, see
Brysbaert et al. (2018).)

From a computational perspective, word frequency is also a fundamental feature
in Text Meaning, and Information Retrieval tasks, among others. Language
structure provides information about how important a word is in a text or
corpus by measuring its occurrences. In NLP, it can be used in a wide variety
of tasks, such as to determine the most frequent words in a language (Spink et
al., 2001), to identify rare words (Dave et al., 2003), to synthesise
information (Haghighi and Vanderwende, 2009) or to answer questions
automatically (Koehn and Knowles, 2017).

###  3.3 Semantic Neighborhood Density

The semantic structure of the lexicon has always played an essential role in
psycholinguistics and NLP, as the organization of knowledge is a pivotal
aspect of studying meaning. Semantic neighbourhood density***Following the
terminology in this field, we will refer to the size of the neighbourhood when
using the term ’density’, both for semantic and phonologic neighbourhoods.
(SND) of a word correlates with several cognitive processes and their
respective brain activations but remains a poorly explored field.

Different semantic neighbourhood size measures have been used in lexico-
semantic research (e.g., metrics based on feature semantics, co-occurrence,
and categorical relations), showing diverging effects of SND in lexical
processing. For instance, when SND is measured considering semantic
associations between words (or word co-occurrence is taken as a proxy of it),
words with large semantic neighbourhoods generate faster responses than words
with sparse semantic neighbourhoods in lexical decision tasks (Yates et al.,
2003; Buchanan et al., 2001; Locker et al., 2003). Opposite, when featural
semantic information is taken as a ground to calculate the density, words with
sparse neighbourhoods are related with faster recognition and naming (Rabovsky
et al., 2016; Reilly and Desai, 2017). In line with these findings, Abdel
Rahman (2007) shows that associative relations facilitate word processing,
whereas categorial connections between words create interference. Hence,
different metrics of SND tap different constructs that need to be understood
as complementary.

From an NLP perspective, KBs such as Wordnet (Miller, 1995), FrameNet (Baker
et al., 1998), BabelNet (Navigli and Ponzetto, 2010) or even Wikipedia provide
a structured and quantifiable framework of words that show a strong
correlation with our mental lexicon. Thus, they represent the primary
computationally-grounded source for the semantic analysis of the neighbourhood
of words via semantic relations.

###  3.4 Phonological Neighborhood Density

As with the SND, Phonological Neighborhood Density (PND) of words affects
lexical processing, which indicates a need to control for such variables when
designing a study. A word´s PND refers to the number of words in the lexicon
that can be formed by substituting a single phoneme of the target word.
Likewise, orthographic neighbourhood density (OND) is defined by the number of
words that can be formed by replacing a single letter of the target word
(Colheart et al.´s N metric) (Coltheart et al., 1977). These two measures
imply considerable differences in opaque languages, such as English, Arabic or
French, because phonemes do not present a one-to-one mapping into graphemes.
Some studies on opaque languages have argued that neighbourhood effects
reflect phonological rather than orthographic similarity (Mulatti et al.,
2006; Yates et al., 2004); when orthographic similarity is controlled,
phonological similarity still affects lexical decision times. However, PND and
OND measures may be used indistinctly in shallow languages such as Basque and
Spanish because there is a direct correspondence between phonemes and
graphemes, with a few exceptions. Thus, in the present work, orthographic
neighbours have been considered when computing the phonological neighbourhood
density of noun pairs.

The influence of phonologically related words has been explored in different
studies of phonological neighbourhood density, and it has shown relevant
effects in task-dependent language processing. In particular, PND seems to
exert competitive effects on word recognition tasks, but facilitatory effects
in production tasks (Dell and Gordon, 2011; Gahl et al., 2012). In spoken word
recognition, the acoustic stimulus activates potential candidates that are
phonologically similar. Thus, the larger the phonological neighbourhood of a
word, the harder it is to recognise the target word (Luce and Pisoni, 1998).
Paradoxically, in word production, dense phonological neighbours seem to ease
production. For example, (Vitevitch, 2002) found facilitatory effects of
neighbourhood density in picture naming in a study that controlled other key
factors such as frequency, neighbourhood frequency, familiarity and
phonotactic probability. Although most studies have attested facilitatory
effects in production, such effects are less consistent than in recognition
tasks.

##  4 Resources for the automatically created dataset

Two primary NLP sources of semantic information have been employed to create
the Basque and Spanish semantic similarity datasets based on embeddings; KBs
and text corpora. Regarding the former, the multilingual version of Wordnet
(Miller, 1995) has been used due to its reliability as a KB in NLP and the
abundance of associated libraries and tools. Table 1 summarises the corpora
and their derived embeddings which constitute the core of the resources in
this work:

Acronym Description Corpora txt Text corpora kb Wordnet-based pseudo-corpora
embeddings FTtxt Fastext over text corpora FTkb Fastext over wordnet-based
pseudo-corpora FThyb Combination of previous two embeddings

Table 1: Acronyms along with the description of the main resources used in
this work. The first group includes two types of corpora, KB and text corpora
(_kb_ and _txt_ , respectively). The second group describes the types of
embeddings, namely FasText over text corpora, over wordnet pseudo-corpora and
the meta-embeddings, which combine the first two types (_TF txt_, _TF kb_ and
_FT hyb_, respectively).

###  4.1 Wordnets and NLTK toolkit

Wordnet (Miller, 1995) is an English lexical database organized by concept and
meaning. Specifically, lexical forms of nouns, verbs, adjectives and adverbs
are grouped into sets of cognitive synonyms called synsets, each expressing a
language-independent concept. Further, each synset is linked to several
synsets via semantic relations such as hypernymy, hyponymy, meronymy and
antonymy, thus, creating a semantic network.

The semantic structure of Wordnet gives us a robust framework for the present
work. First, the nature of the semantic information it conveys around words
and concepts allows assigning two of the previously mentioned features to each
word of the dataset; SND and concreteness. Second, the semantic structure of
the WordNet grants coding in two of the three types of embeddings employed in
this work, namely wordnet and hybrid embeddings (see 4.3).

We have used the Open Multilingual Wordnet
(OMW)†††http://compling.hss.ntu.edu.sg/omw/ which is linked to the Princeton
WordNet 3.0 (PWN)‡‡‡https://wordnet.princeton.edu/. OMW extends the former
English Wordnet to 34 languages, including Basque and Spanish, and uses the
PWN synset structure. Thus, it maps the lexicalizations of all languages into
the same semantic network. This extension method based on English Wordnet’s
structure is called _expand-approach_ §§§In this paper, we will refer to the
original PWN and the one used in this work (Wordnet 3.0) as ’Wordnet’. In
contrast, we will use ’wordnet’ for the rest which derive from PWM such as the
Spanish and Basque ones.. We have used Python NLTK
toolkit¶¶¶https://www.nltk.org/howto/wordnet.html to extract concreteness and
SND features from Wordnet synsets (see section 5.2), for both Basque and
Spanish.

In parallel to the KB Wordnet, we have used language corpora as a
complementary semantic information resource, which is introduced in the next
section.

###  4.2 Corpora

In the following section, we describe the two types of corpora used during the
construction of the dataset; the text corpora and the wordnet-based KB pseudo-
corpora.

####  4.2.1 Text corpora

Text corpora are collections of vast amounts of texts, being usually annotated
(e.g., British National Corpus) ∥∥∥https://www.english-corpora.org/bnc/ or
even syntactically parsed (e.g., Penn TreeBank (Marcus et al., 1993)). Plain
text corpora with no structure have been used to create this dataset. The use
of plain text has allowed using greater text corpora in the computation of the
relationships of words and their contexts via word occurrence. Hence, we have
calculated directly from the corpora the similarity between words by using
word embeddings. In the case of Basque, it has also been used for extracting
word frequencies (see section 5.2.4).

In Spanish, we have employed the readily available 2018 Wikipedia dump text
corpora******https://linguatools.org/tools/corpora/wikipedia-monolingual-
corpora/, extracting the text from the dump using a script††††††xml2textx
available in the same site. The content of tables and maths have been
deleted.. Since the size of Wikipedia is insufficient for building a large
corpus in low-resourced languages such as Basque, web crawling has been used
to complement the corpora in this language, since it is an effective method
for collecting texts to compensate for this deficiency (Leturia, 2012). Thus
for the case of Basque, we have employed the publicly available Euscrawl
corpus‡‡‡‡‡‡http://ixa.ehu.eus/euscrawl/ (Artetxe et al., 2022).

Both Basque and Spanish corpora were pre-processed with the standard
procedure, that is, lowercase and tokenization. Token sparsity in Basque, for
being an agglutinative language, was avoided with an in-house stemmer. The
final Spanish and Basque corpus comprises 608 and 288 million tokens,
respectively.

####  4.2.2 Knowledge-based corpora

This kind of corpora is not as known as the text corpora, but it has recently
gained some attention in the NLP field. The knowledge-based corpora latently
contain the semantic structure of a KB (in our case Basque or Spanish
wordnets), and we have dubbed it as pseudo-corpora in the present work because
it is not human-understandable. This pseudo-corpora represents concepts and
lexicalizations of knowledge bases in a much more compact format than
traditional methods. As explained in the following sections (see 4.3), the
pseudo-corpora have been processed by a neural network model to encode the
semantic structure of Basque or Spanish wordnets in a continuous vector space.

To do this, we applied the monolingual method for English introduced by
Goikoetxea et al. (2015), but in the Basque and Spanish settings. This
technique uses a Monte Carlo method to compute the PageRank algorithm
Avrachenkov et al. (2007). The algorithm considers the KB as an undirected
graph comprised of concepts and links among concepts. It needs a dictionary
which associates words with concepts, as well as a damping factor $\alpha$
that determines the continuity of the random walk and the maximum number of
walks.

For creating every context ******Context refers to each of the lines in the
pseudo-corpora created by the algorithm. the algorithm starts in a random
concept and launches a random lexical form of the concept via the dictionary.
Afterwards, it decides whether to jump to another concept*†*†*†This will
depend on the parameter $\alpha$ and to launch a random lexical form of the
latter, or stops the walk and starts over a new walk. Finally, if the number
of walks reaches the maximum number of walks, it ends the process. Note that
the word is fed to a text file whenever the method launches a lexical form in
the walk.

Each line of the following example shows a different walk of the monolingual
algorithm in Wordnet 3.0, which is used by Goikoetxea et al. (2015). In this
case, every walk has a different length, and each jump from concept to concept
has launched a random lexicalization. It is worth mentioning that this pseudo-
corpus is not human-readable, but every walk gathers semantically related
words, so that implicitly it contains Wordnet’s structure. The following
example shows five random walks from an English Wordnet pseudo-corpus:

>
>             storyteller liar beguiler grifter dissimulation
>             revitalize strength delicate ethereal
>             paved patio terrace house living_room home
>             swimming dive
>             backlog fire re-afforest forest woods rainforest
>
>

As mentioned before, the monolingual version of this method is adapted to the
Basque and Spanish setup, using the dictionaries of both languages from OMW;
hence, aligned with Wordnet 3.0 semantic structure. This means that the former
English lexical forms have been translated to the target language, but the
semantic structure remains intact. The resultant wordnets’ size is not the
same for every language, since expand-approach wordnets do not have the same
number of lexicalizations as the former English Wordnet 3.0. As shown in Table
2, Basque and Spanish wordnets’ sizes are much smaller than the original one.

| Lexicalizations | Synsets
---|---|---
EN | 147306 | 136334
ES | 53039 | 55814
EU | 26701 | 30464
Table 2: Wordnet sizes for English (EN), Spanish (ES) and Basque (EU). Number
of lexicalizations and synsets in the middle and rightmost columns,
respectively.

The reduction of the size of the semantic structure and the number of lexical
forms in Basque and Spanish directly impacts the dimension of the wordnet-
based corpora in both languages. To prevent saturation and redundant
information as described by (Goikoetxea et al., 2015, 2018), the sizes of the
pseudo-corpora in both languages (see section 4.2) have been limited. Even
though the wordnet-based corpus is smaller in Basque, they are big enough to
encode their respective wordnets’ semantic structure (see section 4.3).

English Wordnet 3.0 with glosses consists of 147306 lexical forms, and
Goikoetxea et al. (2016) reached the peak performance in word similarity task
with 200 million random walks, which created an 1100 million token corpus. In
this work, we kept the same proportion between the number of lexical forms and
random walks as Goikoetxea et al. (2016) in order to ensure good performance.
Thus 72 million and 36.3 million random walks were performed for Spanish and
Basque, respectively, resulting in a corpus of 406 million tokens for the
former and 166 million for the latter.

###  4.3 Embeddings

Three types of static embeddings have been computed: text embeddings
($FT_{txt}$), wordnet-based embeddings ($FT_{kb}$), and hybrid embedding
($FT_{hyb}$). These word representations encompass the two sources of semantic
information (text and KB embeddings) related to word similarity (see section
2).

The text embeddings are low-dimensional word representations computed
following the Distributional Hypothesis Harris (1954). More precisely, a
neural network processes the whole text corpus by traversing the corpus word
by word, computing words’ representations based on the Distributional
Hypothesis. It calculates the representation of a given word based on the
representations of all the neighbours found within a predefined window while
traversing the corpus. Eventually, every token in the corpus ends up with a
vector representation (embedding) that latently encodes its semantic (and
syntactic) features.

In this work, the neural-based model fastText (Bojanowski et al., 2017) has
been used because of its more robust performance. It implements a variant of
the Distributional Hypothesis which, instead of exploiting the neighbouring
words’ information to compute a representation of a given word, exploits
subword information. Both text-based and wordnet-based embeddings have been
calculated using fastText.

In the case of Basque and Spanish text-based embeddings, we fed fastText with
their respective text corpora (see 4.2) separately. Regarding wordnet-based
ones, we encoded the semantic structure of Basque and Spanish wordnets in a
vector space following the method proposed by (Goikoetxea et al., 2015), which
comprises two steps. First, the creation of Basque and Spanish Wordnet pseudo-
corpus, as explained in the previous section. Second, processing of Basque and
Spanish pseudo-corpora with a neural-based model to obtain their respective
wordnet embeddings. In the original proposal, (Goikoetxea et al., 2015) used
word2vec (Mikolov et al., 2013), but as mentioned before, in the present work
fastText has been used with the same parameters as in Mikolov et al. (2018).

Recent works show that embeddings which combine semantic information from both
text and KB (i.e., hybrid embeddings) have an overall higher performance in
word similarity tasks. Lastra-Díaz et al. (2019) proved that hybrid embeddings
outperformed most ontology-based measures and the rest of the word embedding
models. Likewise, (Goikoetxea et al., 2016; García et al., 2020) proved that
hybrid embeddings are more explicative of human perception of semantic
distance than text or KB embeddings separately. In the present work, the
method proposed originally by Goikoetxea et al. (2016) and improved by
Goikoetxea et al. (2016) and García et al. (2020) will be implemented in the
computation of hybrid embeddings. In short, (García et al., 2020) proposal
consists in four steps*‡*‡*‡For further details, see García et al. (2020).:

  1. 1.

To compute separate text and wordnet embeddings.

  2. 2.

To map text embeddings space onto the wordnet one.

  3. 3.

To estimate word embeddings for both spaces.

  4. 4.

To combine equivalent text and wordnet embeddings.

In the creation of $FT_{hyb}$ embeddings, García et al. (2020) have been
strictly followed, employing vecmap (Artetxe et al., 2018) for the mapping of
text and wordnet embeddings.

####  4.3.1 Evaluation of the quality of the embeddings in similarity task

To verify the quality of the three types of embeddings, we have tested them in
a word similarity task. In NLP, word similarity is commonly calculated by the
cosine similarity of the angle between two-word embeddings, measuring the
similarity of the words they represent. The cosine similarity is determined by
computing the dot product of the two vectors and dividing it by the product of
the Euclidean norms of the vectors. The cosine similarity is a measure ranging
from 0 to 1, where 0 means the complete absence of similarity and 1 means
complete similarity (i.e., synonyms). The similarity measurement is
independent of the origins of the embeddings, in the way that that text,
wordnet and hybrid representations are processed in the same way.

In this paper, the term similarity has been used indistinctly, but the
difference between pure similarity and relatedness has been widely recognised
in cognitive sciences for a long time (Tversky, 1977). Pure similarity
measures the degree to which two concepts share semantic features, while
relatedness is the degree of association between two words. Regarding the
semantic relations involved, pure similarity includes synonymy and
hyponymy/hyperonymy. In contrast, relatedness encompasses the previous ones
and a wider variety of relations, such as meronymy, functional associations
and other unusual relations. For example, wolf and dog are taxonomically
linked by hypernymy relations in the same semantic structure and share many
features; thus, they have high similarity. In contrast, wolf and moon do not
share any semantic feature (low similarity) but are related by association;
hence, they have a high relatedness.

Cosine similarity measurement does not distinguish between these two aspects
but can be applied to different types of embeddings to measure distinctive
semantic relations. In this work, we have chosen three types of embeddings,
which perform differently in pure semantic similarity (Hill et al., 2015;
Agirre et al., 2009; Rubenstein and Goodenough, 1965) and relatedness
(Finkelstein et al., 2001; Bruni et al., 2014) measures. Broadly, wordnet
embeddings are more precise for measuring pure similarity relations, text
embeddings are more sensitive to compute relatedness, and hybrid embeddings
have been proven to be more robust for capturing semantic relations in general
(Goikoetxea et al., 2016, 2018; García et al., 2020).

Hence in this work, we chose the available similarity and relatedness datasets
for evaluating all embeddings in both languages. In Spanish, we operated with
the pure similarity datasets RG65 (RG) (Camacho Collados et al., 2015) and
SimLex999 (SL) (Etcheverry and Wonsever, 2016), and the relatedness dataset
Wordsim353 (WS) (Hassan and Mihalcea, 2009). In the case of Basque, we used
the pure similarity RG dataset and the relatedness dataset WS created by
Goikoetxea et al. (2018).

In both Spanish and Basque, $FT_{txt}$, $FT_{kb}$ and $FT_{hyb}$
representations have been compared with the baseline, using publicly available
text-based representations fastText *§*§*§https://fasttext.cc/docs/en/crawl-
vectors.html. All of our embeddings have been computed with the set of
parameters as in the baseline embeddings. The Spearman correlation results
between the embeddings and the gold standards are shown in table 3 and
discussed below:

|  | Dataset
---|---|---
|  | RG | WS | SL
EU | Baseline | 77.05 | 65.7 | —
$FT_{txt}$ | 77.86 | 73.31 | —
$FT_{kb}$ | 85.67 | 65.88 | —
$FT_{hyb}$ | 86.55 | 74.57 | —
ES | Baseline | 87.9 | 57.8 | 36.58
$FT_{txt}$ | 86.57 | 57.28 | 28.7
$FT_{kb}$ | 72.84 | 57.32 | 39.93
$FT_{hyb}$ | 87.25 | 63.45 | 40.57
Table 3: Word similarity task results in RG, Wordsim353 (WS) and SimLex999
(SL) datasets in Basque (EU) and Spanish (ES). Text ($FT_{txt}$), wordnet-
based ($FT_{kb}$) and hybrid embedding ($FT_{hyb}$) representations are
compared to their baselines. The best results for each dataset and language
are expressed in bold.

  * 1.

$FT_{txt}$ embeddings: $FT_{txt}$ results perform similarly to the baseline
ones in both languages, with two exceptions. One, the Basque $FT_{txt}$ result
in the WS dataset is higher than the baseline, likely due to the greater
corpus size Euscrawl. Second, the Spanish $FT_{txt}$ in the SL dataset is
lower than the baseline. The only plausible explanation for this result in the
Spanish SL may lie in the differences in the pre-processing of the corpus; the
baseline text corpus was tokenized with Europarl pre-processing
tools*¶*¶*¶https://www.statmt.org/europarl/, whereas the NLTK tokenizer has
been used in the present work.

  * 2.

$FT_{kb}$ embeddings: being this type of embeddings more suited for pure
similarity datasets (RG and SL) rather than for relatedness ones (WS),
$FT_{kb}$ results are higher than the baseline in the Basque RG and the
Spanish SL, but not in the Spanish RG. This underperformance in Spanish must
be interpreted with caution, as the RG dataset is small (64 pairs) and it has
low statistical power.

The most noticeable result, though, is in the relatedness WS dataset, in which
$FT_{kb}$ performs at baseline. Note that the baseline txt-based embeddings
are supposed to have a better performance in WS due to their capacity of
better capturing relatedness relations, so we expect them to obtain better
results than the $FT_{kb}$ measurement. However, the incorporation of gloss
relations (which are relatedness relations) when creating the wordnet-based
corpora may has enhanced the capability of $FT_{kb}$ embeddings to measure
relatedness.

  * 3.

$FT_{hyb}$ embeddings: comparing the results of $FT_{txt}$, $FT_{kb}$ and
$FT_{hyb}$ embeddings, we find that the combination of the first two into the
latter enhances their performance in similarity tasks for both languages
across all datasets. $FT_{hyb}$ also outperforms the baseline in all datasets,
with the only exception of RG in Spanish, likely due to its small size, as
mentioned above.

In sum, the three types of embeddings in this work have performed as expected,
and furthermore, $FT_{hyb}$ embeddings have shown overall best results.
Moreover, the quality of the embeddings in the Basque language has been proven
to be the best known to date. Note that the excellence of the embeddings is
critical in this work, as they are used to create the three semantic
similarity measurements (see section 5.1) that constitute the core features of
the dataset. Altogether, all embeddings created in this work have been
considered suitable to be included in the final dataset.

##  5 Building the dataset

This work has aimed to create a dataset of noun pairsthat compiles information
about the semantic similarity between them. As already introduced, the
semantic similarity has been calculated using three types of embeddings; text,
wordnet and hybrid embeddings. The linguistic features controlled in pairing
the nouns have been concreteness, word frequency, and semantic (SND) and
phonological (PND) neighbour density.

Each feature has been clustered via KNN classification (Cover and Hart, 1967)
to match pairs of words. That is, each pair of nouns has been set every time
there has been a coincidence in the four feature clusters. Despite maximizing
the number of potential noun pairs, a two-group clustering (low and high
values) has resulted balanced in concreteness, word frequency, SND and PND
(see Table 6).

Each line of the final dataset comprises the normalised values of features of
the noun pairs, their corresponding clusters and the three similarity
measurements. Each language has a separate dataset built by following the next
steps:

  * 1.

First, compute the four features for every noun in the corresponding Basque or
Spanish wordnet.

  * 2.

Second, find every possible noun pairs thatmatch the four linguistic features
via clustering.

  * 3.

Third, compute similarities for every noun pair.

  * 4.

Last, write every noun pair’s three similarity measurements along with their
features’ values (L2-normalised) and cluster number.

The following sections are dedicated to describing the details of the
measurements of every feature mentioned in the above paragraphs and the
creation of the final dataset.

###  5.1 Similarity measurements

Table 4 shows the percentages of semantic similarity values for all possible
noun pairs classified along five different ranks. Only nouns with the three
types of embeddings that match all four linguistic features are considered.
The percentages presented in table 4 show a bias towards lower rank similarity
values for all types of embeddings in both Basque and Spanish. This is even
more evident in the wordnet embeddings.

This phenomenon was already pointed out in a cross-lingual setting by Lample
and Conneau (2019). The author observed that fasText based embeddings’ cosine
similarity mean value was considerably lower than that achieved by XLM (Lample
and Conneau, 2019), a large cross-lingual language model. Authors suggested
that the nature of these two types of vector spaces may result in higher word
similarity values, because language models’ vector spaces are more compact
than the fasText due to the formers training in a sentence encoder. This
phenomenon only shows that the non-contextual embeddings like fasText or
word2vec are sparser than the language model ones. As seen in the results of
table 3, the different organization of the lexicon in the vector spaces does
not affect the performance of the embeddings in the word similarity task.

|  [HTML]C0C0C0EUS |  [HTML]C0C0C0ESP
---|---|---
|  [HTML]FFFFFF$FT_{hyb}$ |  [HTML]FFFFFF$FT_{kb}$ | $FT_{txt}$ | $FT_{hyb}$ | $FT_{kb}$ | $FT_{txt}$
0.0-0.2 | 89.1 | 98.44 | 84.9 | 93.52 | 98.68 | 89.13
0.2-0.4 | 10,3 | 1.48 | 13.97 | 6.09 | 1.24 | 10.12
0.4-0.6 | 0.54 | 0.066 | 1.045 | 0.36 | 0.06 | 0.71
0.6-0.8 | 0.025 | 0.009 | 0,048 | 0.015 | 0.007 | 0.03
0.8-1.0 | 0.00135 | 0.0015 | 0.0018 | 0.001 | 0.0013 | 0.0014
Table 4:  Percentages of word similarity values in the Basque and Spanish
dataset across five ranges and three types of embeddings; text ($FT_{txt}$ ),
wordnet ($FT_{kb}$ ), and hybrid ($FT_{hyb}$ ).

###  5.2 Features

This section describes the details of the calculation of each linguistic
feature before being L2-normalised and clustered in the final dataset.

####  5.2.1 Concreteness

Concreteness measurements were calculated automatically by exploiting
Wordnet’s taxonomy in Basque and Spanish. For doing so, Feng et al. (2011) was
followed, who proposed an algorithm for predicting word concreteness via
various lexical resources and features, namely, human ratings, lexical types,
latent semantic analysis dimensions, word frequency and length, and hypernymy
level. Among these several aspects, in this work, only the _Hypernymy Level_
was considered for computing concreteness. This is because even though human
ratings and lexical type are not included in this research, the rest are going
to be analysed and treated as separate features. *∥*∥*∥Although latent
semantic analysis is not used for word representations in the present word,
more advanced techniques of static word embeddings are included (see section
4.3). The _Hypernymy Level_ aspect scores the concreteness of a word following
its hypernymy relations in Wordnet’s hierarchical structure. For every noun in
Basque and Spanish, the mentioned method does the following in the
corresponding wordnet:

  * 1.

Check all of its synsets within a given noun.

  * 2.

For each synset, the method counts every hypernym from the source synset until
the topmost synset (i.e., Wordnet’s root node _entity_), thus scoring the
depth of the source synset in Wordnet’s tree structure.

  * 3.

Compute the concreteness of a given noun by averaging all the depths of its
synsets.

Note that the higher the depth, the more concrete the word is, and the lower
the depth more abstract it will be. For example, the word _car_ has five
synsets in Wordnet 3.0. If we choose the most common synset for that
noun*********The sense with the following definition in Wordnet: _a motor
vehicle with four wheels; usually propelled by an internal combustion engine._
, its hypernymy path is the following:

$\textbf{car}\rightarrow\emph{MotorVehicle}\rightarrow\emph{SelfPropelledVehicle}\rightarrow\emph{WheeledVehicle}\rightarrow\emph{Container}$
$\rightarrow\emph{Instrumentality}\rightarrow\emph{Artifact}\rightarrow\emph{Whole}\rightarrow\emph{Object}\rightarrow\emph{PhysicalEntity}\rightarrow\textbf{Entity}$

.

Thus, the depth of the path of the mentioned synset of _car_ is 10. The word
_Wheeled Vehicle_ has a depth of 7, meaning that it is less concrete than
_car_ and _ambulance_ a depth of 11, therefore being more concrete than _car_.
For the final score of the noun, the method computes all the depths of the
rest of the synsets and averages all of them.

####  5.2.2 Semantic neighbourhood density

This work’s framework for computing SND is also based on Wordnet’s semantic
structure. So, the semantic neighbours of every noun are computed by counting
all semantic relations of its synsets. A procedure similar to the one
described in section 5.2.1 has been used for every noun in Basque and Spanish:

  * 1.

Check all of its synsets within a given noun.

  * 2.

For each synset, count all its surrounding synsets linked by a Wordnet
semantic relation. This approach departs from the source synset and checks all
its semantic relations to find its neighbouring synsets.

  * 3.

Finally, compute the semantic neighbours of a noun averaging all the counts in
its synsets.

Only first-degree semantic neighbours have been considered. Regarding the
Wordnet semantic relations, we have taken every available synset-based
semantic relation in the NLTK toolkit and discarded the more surface-level
lexical relations like derivation and pertainymy.

Following the same example as in the previous section, the most common synset
for the noun _car_ has 31 hyponyms, 29 meronym parts and one hypernym. That
is, 61 first-degree semantic neighbours.

####  5.2.3 Phonological neighbourhood density

Phonological neighbours are pairs of words that differ in only one
phonological segment, such as cat and bat. For this work, phonological
neighbourhood size has been calculated with the Levenstein distance method
Levenshtein (1965). It has several advantages compared to other alternatives,
such as the Hamming Distance or the Jaro-Winkler distance; it accounts for
both substitution and insertion/deletion operations, so the distance is more
accurately computed, independent of the length of the strings.

This work assumes that two words are phonological neighbours if their
Levenstein distance is smaller or equal to one. For example, in the case of
the word _car_ , the number of phonological neighbours that meet the
conditions mentioned above is 31, with first-degree neighbouring words such as
_cat_ , _card_ , _scar_ , _ear_ , _jar_ and _tar_. Multiword expressions and
nouns with less than three characters in length have been excluded from the
dataset.

####  5.2.4 Word frequency

Word frequency measures have been calculated using Zipf frequencies; a base-10
logarithm of its occurrences per billion words. Zipf frequency allows for a
more comprehensive analysis than raw noun counts because it accounts for the
relative importance of a word in the corpus.

For Spanish, Python’s wordfreq library has been used to obtain Zipf
frequencies. Given the nature of the 10-base logarithm scale, Zipf values in
this library are within the range of 0-8. In Basque, Zipf frequencies have
been computed from the most extensive Basque public corpus, Euscrawl (Artetxe
et al., 2022). Euscrawl corpus has been pre-processed as described in 4.2, and
raw word counts have been calculated using fastext model from the gemsim
library in Python. Afterwards, those counts were converted into Zipf
frequencies, limiting their maximum and minimum values between 0 and 8.

####  5.2.5 Feature dictionaries

As mentioned in previous sections, each noun is accompanied by four features
in the dataset. An independent dictionary has been created for each feature
(i.e., concreteness, frequency, SND and PND) in Basque and Spanish, so that it
assigns the measurements of its feature along with its cluster number to each
noun. Once the feature dictionaries have been computed, the next step has been
building the final dataset (see section 5.3).

The lists of single-word nouns from Spanish and Basque wordnets have been
extracted via NLTK toolkit, constraining the candidates to 28647 and 22877 in
each language, respectively.

Although the Spanish wordnet nearly doubles the Basque one in size (see Table
2), the unbalance is not pronounced in the size of the feature dictionaries
because the number of available nouns in the NLTK toolkit is comparable across
these two languages. To improve the KNN classification of the nouns across
each feature, outliers have been removed using the interquartile range, and
L2-normalisation has been applied to every raw measurement. This adjustment
has diminished the size of the datasets but has reduced noisy samples and
balanced cluster sizes.

Table 5 shows the resulting feature dictionaries’ size and the mean value for
each feature.

|  |  [HTML]C0C0C0EU |  [HTML]C0C0C0ES
---|---|---|---
| size | 19660 | 14771
CNC | avg | 8.41 | 8.38
| size | 14380 | 12146
FRQ | avg | 6.39 | 2.95
| size | 18044 | 15534
SND | acg | 2.47 | 2.18
| size | 14671 | 14608
PND | avg | 1.75 | 1.38
Table 5: Concreteness (CNC), frequency (FRQ), phonological neighbourhood
density (PND) and semantic neighbourhood density (SND) dictionaries for Basque
(EU) and Spanish (ES). The upper line in each feature shows the size of a
specific dictionary, and the lower line shows the mean value of each
measurement.

Table 5 shows that the Basque feature dictionaries are slightly bigger than
the ones in Spanish, likely because there are more outliers in the Spanish
dataset. Regarding the average value of each feature, the only difference is
in frequency. On average, Basque nouns are more frequent than the Spanish
ones*††*††*††Note that the mean Zipf frequency value of the Basque words is 6,
meaning that a word appears once per a thousand words, whereas the mean Zipf
value in Spanish is 2, indicating that a word appears once per million words..
A reasonable explanation for this difference may be the size of the corpora
used to calculate word frequencies in each language. The Scrawl corpus is
smaller than the one used for computing Spanish Zipf frequencies in the NLTK
toolkit, and it has much fewer infrequent words. This may bias the Zipf
frequency towards a higher average value in Basque.

Table 6 shows the token distribution of all previous dictionaries in KNN-based
clusters. Features are clustered in two groups, differencing between high and
low values. The size of the dictionary of phonological neighbours features is
far below the rest. This is because nouns with 0 neighbours have been excluded
from the KNN classification (14671 in Basque and 12085 in Spanish) because
they tend to unbalance the cluster distribution, leaving the cluster with
high-PND value with almost no content. Therefore, nouns with 0 neighbours have
been extracted into a subgroup and treated as a separate cluster when creating
the final dataset.

|  [HTML]C0C0C0EUS |  [HTML]C0C0C0ESP
---|---|---
|  [HTML]FFFFFFL |  [HTML]FFFFFFH | L | H
CNC | 11373 | 8287 | 9692 | 6179
FRQ | 5646 | 8734 | 6103 | 6043
SND | 13915 | 4129 | 4772 | 9836
PND | 3791 | 974 | 1017 | 2432
Table 6: Concreteness (CNC), frequency (FRQ), phonological neighbourhood
density (PND) and semantic neighbourhood density (SND) dictionaries for Basque
(EU) and Spanish (ES). The upper line in each feature shows the size of a
specific dictionary, and the lower line shows the mean value of each
measurement

###  5.3 Construction of word pair matrix

The final step in this work has consisted of creating the noun pair dataset
out of the feature dictionaries described previously. There are two conditions
to fulfil to set a pair of nouns:

  * 1.

First, the two nouns composing the pair must be represented in the three types
of embeddings.

  * 2.

Second, the two nouns composing the pair mustshare cluster in all four
linguistic features.

The number of nouns that have fulfilled the first condition was more
considerable in Basque (n= 15110) than in Spanish (n= 8565). This phenomenon
is due to the high number of multiword expressions in the Spanish wordnet,
which are discharged from the dataset. The suppression of the multiword leads
to a lower overlap of text and wordnet embeddings and, therefore, a lower
amount of hybrid embeddings.

All the nouns that have fulfilled the first condition have been traversed to
find all possible pairs of nouns that match the four linguistic features, as
stated in the second condition. Every time a pair has been set, featural
values in concreteness, frequency, SND, and PND, as well as their cluster
identification*‡‡*‡‡*‡‡We used -1 number for 0 neighbour subgroup in
phonological neighbour matching, as that group was not part of a KNN cluster.
For the rest of the linguistic features, clustering was marked with 0 and 1 to
indicate low or high-value clustering, have been inserted in the dataset,
along with the three types of similarity measurements. Altogether, each nour
pair contains pieces of information spread over 15 columns.

##  6 Conclusions

Psycholinguistic evidence supports the idea that the overlap of semantic
features across concepts is crucial in the computation of semantic relations,
and by extension, in semantic processing. A comprehensive work on NLP has
shown that embeddings or vectors are a sensitive method to artificially
compute similarities across concepts by accounting for large number of
semantic properties, which may be applied to different types of linguistic
resources in order to uncover distinctive semantic relations. This work has
aimed to bridge computational linguistics and psycholinguistics to
automatically build a dataset of vectorised word similarity measures in Basque
and Spanish†*†*†*Datasets will be available upon acceptance.

We have presented a computationally grounded dataset that encompasses
different aspects of the semantic information present in both text corpora and
knowledge bases. On the one hand, each dataset includes three similarity
measurements based on their corresponding embedding computations, namely,
text-based, wordnet-based and hybrid embeddings. These measurements encode
different subtleties of meaning. Text-based embeddings are computed out of
word co-occurrence in large natural language corpora; being prone to better
measure relatedness relations. Wordnet-based embeddings encode the semantic
structure of Wordnet, so that they are considered a measurement of pure
similarity relations. Finally, hybrid embeddings combine both text-based and
wordnet-based embeddings, binding categorical and associative relations. In
addition, all the materials have been controlled for several linguistic
features (concreteness, frequency and semantic and phonemic neighbour size) to
adjust the dataset to various research interests and requirements.

##  7 Acknowledgements

This work was supported by the Leonardo grant of the BBVA Foundation for
Researchers and Cultural Creators 2021 (FP2157) (J.G., M.A., I.S.),
Consolidated Group funding from the Basque Government (IT1439/22-GIC21/132)
(M.A., I.S.), and the grant RYC2021-033222-I funded by the Ministry of Science
and Innovation/State Research Agency/ 10.13039/501100011033 and by the
European Union NextGenerationEU/Recovery, Transformation and Resilience Plan
of Spain (M.A.).

## References

  * Abnar et al. (2017) Abnar, S., Ahmed, R., Mijnheer, M., Zuidema, W., 2017. Experiential, distributional and dependency-based word embeddings have complementary roles in decoding brain activity. arXiv preprint arXiv:1711.09285.
  * Agirre et al. (2009) Agirre, E., Alfonseca, E., Hall, K., Kravalova, J., Paşca, M., Soroa, A., 2009\. A study on similarity and relatedness using distributional and WordNet-based approaches. In: Proceedings of HLT-NAACL. pp. 19–27.
  * Alnafesah et al. (2020) Alnafesah, G., Madabushi, H. T., Lee, M., 2020. Augmenting neural metaphor detection with concreteness. In: Proceedings of the Second Workshop on Figurative Language Processing. pp. 204–210.
  * Artetxe et al. (2022) Artetxe, M., Aldabe, I., Agerri, R., Perez-de Viñaspre, O., Soroa, A., 2022\. Does corpus quality really matter for low-resource languages? arXiv preprint arXiv:2203.08111.
  * Artetxe et al. (2018) Artetxe, M., Labaka, G., Agirre, E., 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 789–798.
  * Auguste et al. (2017) Auguste, J., Rey, A., Favre, B., 2017. Evaluation of word embeddings against cognitive processes: primed reaction times in lexical decision and naming tasks. In: Proceedings of the 2nd workshop on evaluating vector space representations for NLP. pp. 21–26.
  * Avrachenkov et al. (2007) Avrachenkov, K., Litvak, N., Nemirovsky, D., Osipova, N., 2007. Monte carlo methods in pagerank computation: When one iteration is sufficient. SIAM Journal on Numerical Analysis 45 (2), 890–904.
  * Bakarov (2018) Bakarov, A., 2018. A survey of word embeddings evaluation methods. arXiv preprint arXiv:1801.09536.
  * Baker et al. (1998) Baker, C. F., Fillmore, C. J., Lowe, J. B., 1998. The berkeley framenet project. In: COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics.
  * Balota and Chumbley (1984) Balota, D. A., Chumbley, J. I., 1984. Are lexical decisions a good measure of lexical access? the role of word frequency in the neglected decision stage. Journal of Experimental Psychology: Human perception and performance 10 (3), 340\.
  * Balota et al. (2004) Balota, D. A., Cortese, M. J., Sergent-Marshall, S. D., Spieler, D. H., Yap, M. J., 2004. Visual word recognition of single-syllable words. Journal of Experimental Psychology: General 133 (2), 283.
  * Barber et al. (2013) Barber, H. A., Otten, L. J., Kousta, S.-T., Vigliocco, G., 2013. Concreteness in word processing: Erp and behavioral effects in a lexical decision task. Brain and language 125 (1), 47–53.
  * Barsalou (1999) Barsalou, L. W., 1999. Perceptual symbol systems. Behavioral and Brain Sciences 22 (4), 577–660.
  * Benedek et al. (2017) Benedek, M., Kenett, Y. N., Umdasch, K., Anaki, D., Faust, M., Neubauer, A. C., 2017\. How semantic memory structure and intelligence contribute to creative thought: a network science approach. Thinking & Reasoning 23 (2), 158–183.
  * Bojanowski et al. (2017) Bojanowski, P., Grave, E., Joulin, A., Mikolov, T., 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics 5, 135–146.
  * Bonin et al. (2018) Bonin, P., Méot, A., Bugaiska, A., 2018. Concreteness norms for 1,659 french words: Relationships with other psycholinguistic variables and word recognition times. Behavior research methods 50 (6), 2366–2387.
  * Broderick et al. (2021) Broderick, M. P., Di Liberto, G. M., Anderson, A. J., Rofes, A., Lalor, E. C., 2021\. Dissociable electrophysiological measures of natural language processing reveal differences in speech comprehension strategy in healthy ageing. Scientific Reports 11 (1), 4963.
  * Bruni et al. (2014) Bruni, E., Tran, N.-K., Baroni, M., 2014. Multimodal distributional semantics. JAIR 49, 1–47.
  * Brysbaert et al. (2018) Brysbaert, M., Mandera, P., Keuleers, E., 2018. The word frequency effect in word processing: An updated review. Current Directions in Psychological Science 27 (1), 45–50.
  * Brysbaert and New (2009) Brysbaert, M., New, B., 2009. Moving beyond kučera and francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for american english. Behavior Research Methods 41 (4), 977–990.
  * Brysbaert et al. (2014a) Brysbaert, M., Stevens, M., De Deyne, S., Voorspoels, W., Storms, G., 2014a. Norms of age of acquisition and concreteness for 30,000 dutch words. Acta psychologica 150, 80–84.
  * Brysbaert et al. (2014b) Brysbaert, M., Warriner, A. B., Kuperman, V., 2014b. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior Research Methods 46 (3), 904–911.
  * Buchanan et al. (2001) Buchanan, L., Westbury, C., Burgess, C., 2001. Characterizing semantic space: Neighborhood effects in word recognition. Psychonomic Bulletin & Review 8 (3), 531–544.
  * Camacho Collados et al. (2015) Camacho Collados, J., Pilehvar, M. T., Navigli, R., 2015. A framework for the construction of monolingual and cross-lingual word similarity datasets. Association for Computational Linguistics.
  * Charbonnier and Wartena (2019) Charbonnier, J., Wartena, C., 2019. Predicting word concreteness and imagery. In: Proceedings of the 13th International Conference on Computational Semantics-Long Papers. Association for Computational Linguistics, pp. 176–187.
  * Coltheart et al. (1977) Coltheart, M., Davelaar, E., Jonasson, J., Besner, D., 1977. Access to the internal lexicon. Attention and Performance 6, 535–555.
  * Cop et al. (2017) Cop, U., Dirix, N., Drieghe, D., Duyck, W., 2017. Presenting geco: An eyetracking corpus of monolingual and bilingual sentence reading. Behavior research methods 49, 602–615.
  * Cosgrove et al. (2021) Cosgrove, A. L., Kenett, Y. N., Beaty, R. E., Diaz, M. T., 2021. Quantifying flexibility in thought: The resiliency of semantic networks differs across the lifespan. Cognition 211, 104631.
  * Ćoso et al. (2019) Ćoso, B., Guasch, M., Ferré, P., Hinojosa, J. A., 2019. Affective and concreteness norms for 3,022 croatian words. Quarterly Journal of Experimental Psychology 72 (9), 2302–2312.
  * Cover and Hart (1967) Cover, T., Hart, P., 1967. Nearest neighbor pattern classification. IEEE transactions on information theory 13 (1), 21–27.
  * Dahan et al. (2001) Dahan, D., Magnuson, J. S., Tanenhaus, M. K., 2001. Time course of frequency effects in spoken-word recognition: Evidence from eye movements. Cognitive Psychology 42 (4), 317–367.
  * Dave et al. (2003) Dave, K., Lawrence, S., Pennock, D. M., 2003. Mining the peanut gallery: Opinion extraction and semantic classification of product reviews. In: Proceedings of the 12th international conference on World Wide Web. pp. 519–528.
  * Dell and Gordon (2011) Dell, G., Gordon, J., 2011. Neighbors in the lexicon: Friends or foes? Phonetics and phonology in language comprehension and production, eds N. O. Schiller and A. S. Meyer, 9–38.
  * Etcheverry and Wonsever (2016) Etcheverry, M., Wonsever, D., 2016. Spanish word vectors from wikipedia. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16). pp. 3681–3685.
  * Ettinger and Linzen (2016) Ettinger, A., Linzen, T., 2016. Evaluating vector space models using human semantic priming results. In: Proceedings of the 1st workshop on evaluating vector-space representations for NLP. pp. 72–77.
  * Feng et al. (2011) Feng, S., Cai, Z., Crossley, S., McNamara, D. S., 2011. Simulating human ratings on word concreteness. In: Twenty-Fourth International FLAIRS Conference.
  * Finkelstein et al. (2001) Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G., Ruppin, E., 2001. Placing search in context: The concept revisited. In: Proceedings of the 10th international conference on World Wide Web. pp. 406–414.
  * Fischer and Zwaan (2008) Fischer, M. H., Zwaan, R. A., 2008. Embodied language: A review of the role of the motor system in language comprehension. Quarterly Journal of Experimental Psychology 61 (6), 82–850.
  * Gahl et al. (2012) Gahl, S., Yao, Y., Johnson, K., 2012. Why reduce? phonological neighborhood density and phonetic reduction in spontaneous speech. Journal of Memory and Language 66 (4), 789–806.
  * García et al. (2020) García, I., Agerri, R., Rigau, G., 2020. A common semantic space for monolingual and cross-lingual meta-embeddings. arXiv preprint arXiv:2001.06381.
  * Goikoetxea et al. (2016) Goikoetxea, J., Agirre, E., Soroa, A., 2016. Single or multiple? combining word representations independently learned from text and wordnet. In: Thirtieth AAAI Conference on Artificial Intelligence.
  * Goikoetxea et al. (2018) Goikoetxea, J., Soroa, A., Agirre, E., 2018. Bilingual embeddings with random walks over multilingual wordnets. Knowledge-Based Systems 150, 218–230.
  * Goikoetxea et al. (2015) Goikoetxea, J., Soroa, A., Agirre, E., Donostia, B. C., 2015. Random walks and neural network language models on knowledge bases. In: Proceedings of HLT-NAACL. pp. 1434–1439.
  * Gregg (1976) Gregg, V., 1976. Word frequency, recognition and recall.
  * Guasch et al. (2016) Guasch, M., Ferré, P., Fraga, I., 2016. Spanish norms for affective and lexico-semantic variables for 1,400 words. Behavior Research Methods 48 (4), 1358–1369.
  * Günther et al. (2019) Günther, F., Rinaldi, L., Marelli, M., 2019. Vector-space models of semantic representation from a cognitive perspective: A discussion of common misconceptions. Perspectives on Psychological Science 14 (6), 1006–1033.
  * Haghighi and Vanderwende (2009) Haghighi, A., Vanderwende, L., 2009. Exploring content models for multi-document summarization. In: Proceedings of human language technologies: The 2009 annual conference of the North American Chapter of the Association for Computational Linguistics. pp. 362–370.
  * Harris (1954) Harris, Z. S., 1954. Distributional structure. Word.
  * Hassan and Mihalcea (2009) Hassan, S., Mihalcea, R., 2009. Cross-lingual semantic relatedness using encyclopedic knowledge. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. pp. 1192–1201.
  * Hauk et al. (2004) Hauk, O., Johnsrude, I., Pulvermüller, F., 2004. Somatotopic representation of action words in human motor and premotor cortex. Neuron 41 (2), 301–307.
  * Hayes and Henderson (2021) Hayes, T. R., Henderson, J. M., 2021. Looking for semantic similarity: what a vector-space model of semantics can tell us about attention in real-world scenes. Psychological Science 32 (8), 1262–1270.
  * Hill et al. (2015) Hill, F., Reichart, R., Korhonen, A., 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics 41 (4), 665–695.
  * Hollenstein et al. (2019) Hollenstein, N., de la Torre, A., Langer, N., Zhang, C., 2019. Cognival: A framework for cognitive word embedding evaluation. arXiv preprint arXiv:1909.09001.
  * Incitti and Snidaro (2021) Incitti, F., Snidaro, L., 2021. Fusing contextual word embeddings for concreteness estimation. In: 2021 IEEE 24th International Conference on Information Fusion (FUSION). IEEE, pp. 1–8.
  * Jain and Huth (2018) Jain, S., Huth, A., 2018. Incorporating context into language encoding models for fmri. Advances in Neural Information Processing Systems 31.
  * Jelodar et al. (2010) Jelodar, A. B., Alizadeh, M., Khadivi, S., 2010. Wordnet based features for predicting brain activity associated with meanings of nouns. In: Proceedings of the NAACL HLT 2010 First Workshop on Computational Neurolinguistics. pp. 18–26.
  * Kenett et al. (2017) Kenett, Y. N., Levi, E., Anaki, D., Faust, M., 2017. The semantic distance task: Quantifying semantic distance with semantic network path length. Journal of Experimental Psychology: Learning, Memory, and Cognition 43 (9), 1470\.
  * Kinsbourne and George (1974) Kinsbourne, M., George, J., 1974. The mechanism of the word-frequency effect on recognition memory. Journal of Verbal Learning and Verbal Behavior 13 (1), 63–69.
  * Koehn and Knowles (2017) Koehn, P., Knowles, R., 2017. Six challenges for neural machine translation. arXiv preprint arXiv:1706.03872.
  * Lample and Conneau (2019) Lample, G., Conneau, A., 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291.
  * Lastra-Díaz et al. (2019) Lastra-Díaz, J. J., Goikoetxea, J., Taieb, M. A. H., García-Serrano, A., Aouicha, M. B., Agirre, E., 2019. A reproducible survey on word embeddings and ontology-based methods for word similarity: Linear combinations outperform the state of the art. Engineering Applications of Artificial Intelligence 85, 645–665.
  * Leturia (2012) Leturia, I., 2012. Evaluating different methods for automatically collecting large general corpora for basque from the web. In: Proceedings of COLING 2012\. pp. 1553–1570.
  * Levenshtein (1965) Levenshtein, V., 1965. Binary codes capable of correcting spurious insertions and deletion of ones. Problems of Information Transmission 1 (1), 8–17.
  * Ljubešić et al. (2018) Ljubešić, N., Fišer, D., Peti-Stantić, A., 2018. Predicting concreteness and imageability of words within and across languages via word embeddings. arXiv preprint arXiv:1807.02903.
  * Locker et al. (2003) Locker, L., Simpson, G., Yates, M., 2003. Semantic neighbourhood effects on the recognition of ambiguous words. Memory and Cognition (31), 505–515.
  * Long et al. (2019) Long, Y., Xiang, R., Lu, Q., Huang, C.-R., Li, M., 2019. Improving attention model based on cognition grounded data for sentiment analysis. IEEE Transactions on Affective Computing 12 (4), 900–912.
  * Luce and Pisoni (1998) Luce, P., Pisoni, D., 1998. Recognizing spoken words: The neighborhood activation model. Ear and Hearing 19 (1), 1–36.
  * Luke and Christianson (2018) Luke, S. G., Christianson, K., 2018. The provo corpus: A large eye-tracking corpus with predictability norms. Behavior research methods 50, 826–833.
  * MacLeod and Kampe (1996) MacLeod, C. M., Kampe, K. E., 1996. Word frequency effects on recall, recognition, and word fragment completion tests. Journal of experimental psychology: Learning, memory, and cognition 22 (1), 132.
  * Mandera et al. (2017) Mandera, P., Keuleers, E., Brysbaert, M., 2017. Explaining human performance in psycholinguistic tasks with models of semantic similarity based on prediction and counting: A review and empirical validation. Journal of Memory and Language 92, 57–78.
  * Marcus et al. (1993) Marcus, M., Santorini, B., Marcinkiewicz, M. A., 1993. Building a large annotated corpus of english: The penn treebank.
  * Mate et al. (2012) Mate, J., Allen, R. J., Baqués, J., 2012. What you say matters: Exploring visual–verbal interactions in visual working memory. The Quarterly Journal of Experimental Psychology 65 (3), 395–400.
  * Mikolov et al. (2018) Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., Joulin, A., 2018. Advances in pre-training distributed word representations. In: Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
  * Mikolov et al. (2013) Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., Dean, J., 2013. Distributed representations of words and phrases and their compositionality. In: Proceedings of Advances in neural information processing systems. pp. 3111–3119.
  * Miller (1995) Miller, G. A., 1995. Wordnet: a lexical database for english. Communications of the ACM 38 (11), 39–41.
  * Mulatti et al. (2006) Mulatti, C., Reynolds, M., Besner, D., 2006. Neighborhood effects in reading aloud: new findings and new challenges for computational models. Journal of Experimental Psychology: Human Perception and Performance 32 (4), 799–810.
  * Navigli and Ponzetto (2010) Navigli, R., Ponzetto, S. P., 2010. Babelnet: Building a very large multilingual semantic network. In: Proceedings of the 48th annual meeting of the association for computational linguistics. pp. 216–225.
  * Paivio (1971) Paivio, A., 1971. Imagery and verbal processes.
  * Rabovsky et al. (2016) Rabovsky, M., Schad, D., Abdel Rahman, R., 2016. Language production is facilitated by semantic richness but inhibited by semantic density: Evidence from picture naming. Cognition 146, 240–244.
  * Reilly and Desai (2017) Reilly, M., Desai, R., 2017. Effects of semantic neighborhood density in abstract and concrete words. Cognition 169, 46–53.
  * Rodrigues et al. (2018) Rodrigues, J., Branco, R., Silva, J., Saedi, C., Branco, A., 2018. Predicting brain activation with wordnet embeddings. In: Proceedings of the Eight Workshop on Cognitive Aspects of Computational Language Learning and Processing. pp. 1–5.
  * Rothe et al. (2016) Rothe, S., Ebert, S., Schütze, H., 2016. Ultradense word embeddings by orthogonal transformation. arXiv preprint arXiv:1602.07572.
  * Ruan et al. (2016) Ruan, Y.-P., Ling, Z.-H., Hu, Y., 2016. Exploring semantic representation in brain activity using word embeddings. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pp. 669–679.
  * Rubenstein and Goodenough (1965) Rubenstein, H., Goodenough, J. B., 1965. Contextual correlates of synonymy. Communications of the ACM 8 (10), 627–633.
  * Salicchi et al. (2021) Salicchi, L., Lenci, A., Chersoni, E., 2021. Looking for a role for word embeddings in eye-tracking features prediction: Does semantic similarity help? In: Proceedings of the 14th International Conference on Computational Semantics (IWCS). pp. 87–92.
  * Schwanenflugel (2010) Schwanenflugel, P., 2010. Why are abstract concepts hard to understand? the psychology of word meanings. Human Brain Mapping 31, 1459–1468.
  * Søgaard (2016) Søgaard, A., 2016. Evaluating word embeddings with fmri and eye-tracking. In: Proceedings of the 1st workshop on evaluating vector-space representations for NLP. pp. 116–121.
  * Solovyev et al. (2022) Solovyev, V., Yu, V., Andreeva, M., Zaikin, A., 2022. Russian dictionary with concreteness/abstractness indices. Russian Journal of Linguistics 26 (2), 515–549.
  * Spink et al. (2001) Spink, A., Wolfram, D., Jansen, M. B., Saracevic, T., 2001. Searching the web: The public and their queries. Journal of the American society for information science and technology 52 (3), 226–234.
  * Strijkers et al. (2010) Strijkers, K., Costa, A., Thierry, G., 2010. Tracking lexical access in speech production: electrophysiological correlates of word frequency and cognate effects. Cerebral Cortex 20 (4), 912–928.
  * Toneva and Wehbe (2019) Toneva, M., Wehbe, L., 2019. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). Advances in Neural Information Processing Systems 32.
  * Tsvetkov et al. (2014) Tsvetkov, Y., Boytsov, L., Gershman, A., Nyberg, E., Dyer, C., 2014. Metaphor detection with cross-lingual model transfer. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 248–258.
  * Tversky (1977) Tversky, A., 1977. Features of similarity. Psychological Review 84 (4), 327.
  * Vitevitch (2002) Vitevitch, M., 2002. The influence of phonological similarity neighbors on speech production. Journal of Experimental Psychology: Learning, Memory, and Cognition 28 (4), 735–747.
  * Wang et al. (2010) Wang, J., Conder, J., Blitzer, D., Shinkareva, S., 2010. Neural representation of abstract and concrete concepts: a meta-analysis of neuroimaging studies. Human Brain Mapping 31 (10), 1459–1468.
  * Wulff et al. (2019) Wulff, D. U., De Deyne, S., Jones, M. N., Mata, R., 2019. New perspectives on the aging lexicon. Trends in Cognitive Sciences 23 (8), 686–698.
  * Yates et al. (2004) Yates, M., Locker, L., Simpson, G., 2004. The influence of phonological neighborhood on visual word perception. Psychonomic Bulletin & Review 11, 452–457.
  * Yates et al. (2003) Yates, M., Locker, L., Simpson, G. B., 2003. Semantic and phonological influences on the processing of words and pseudohomophones. Memory & Cognition 31, 856–866.