# HistBERT: A Pre-trained Language Model for Diachronic Lexical Semantic
Analysis

Wenjun Qiu∗, Yang Xu†
∗Department of Electrical and Computer Engineering
† Department of Computer Science
University of Toronto
Toronto, Ontario, Canada
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Contextualized word embeddings have demonstrated state-of-the-art performance
in various natural language processing tasks including those that concern
historical semantic change. However, language models such as BERT was trained
primarily on contemporary corpus data. To investigate whether training on
historical corpus data improves diachronic semantic analysis, we present a
pre-trained BERT-based language model, HistBERT, trained on the balanced
Corpus of Historical American English. We examine the effectiveness of our
approach by comparing performance of the original BERT and that of HistBERT,
and we report promising results in word similarity and semantic shift
analysis. Our work suggests that the effectiveness of contextual embeddings in
diachronic semantic analysis is dependent on the temporal profile of the input
text and care should be taken in applying this methodology to study historical
semantic change.

##  1 Introduction

Semantic change is an extensively studied topic in historical linguistics
Traugott and Dasher (2001), but reliable quantification of semantic change
remains a challenging problem Tahmasebi et al. (2018). One of the most popular
approaches to quantifying semantic change relies on distributional semantic
models which have been applied to cases including but not restricted to the
automatic detection, modeling, and evaluation of laws of semantic change Sagi
et al. (2011); Cook and Stevenson (2010); Gulordava and Baroni (2011);
Kulkarni et al. (2015); Kim et al. (2014); Król (2017); Rosenfeld and Erk
(2018); Bamler and Mandt (2017); Schlechtweg et al. (2020a); Xu and Kemp
(2015); Hamilton et al. (2016); Dubossarsky et al. (2017).

Recent development in natural language processing (NLP) shows that
contextualized embeddings outperform word embeddings in quantifying word
meaning and semantic change Peters et al. (2018); Radford et al. (2018). For
instance, BERT has achieved state-of-the-art performance on a number of
natural language understanding tasks Devlin et al. (2019). Building on these
studies, Giulianelli et al. 2020 and Hu et al. 2019a construct contextualized
embeddings for words in a diachronic setting and investigate word meaning
change in vector space over historical times. Their work suggests that
contextualized embeddings offer an advantage in the quantification and
detection of historical semantic change.

While BERT-based models have shown promising results, we challenge one of the
basic assumptions in applying pre-trained BERT models for diachronic semantic
analysis: BERT was typically trained on BooksCorpus Zhu et al. (2015) and
English Wikipedia111https://www.english-corpora.org/wiki/ which often do not
appropriately reflect language use in historical times. Although the corpora
for pre-training BERT may contain some text from history, modern corpora
including Wikipedia contain mostly contemporary English text. This skewed pre-
training input leads naturally to several research questions that motivate our
study:

  1. 1.

Is it valid to use pre-trained BERT to model historical semantic change?

  2. 2.

Does this approach lead to semantic representations that are biased toward
modern language use?

  3. 3.

Can additional training on (balanced) historical data help to improve the
precision of BERT in quantifying historical semantic change?

These research questions are partly inspired by the study of Giulianelli et
al. 2020, where it was observed that issues arise when the original pre-
trained BERT model was applied directly to quantifying semantic change
(without controlling for the temporal profile of the input text). One
prominent example described in Giulianelli et al. 2020 was the insensitivity
of BERT in teasing apart historical and modern usages of the English word
“coach”. Specifically, the pre-trained BERT model considered that the
similarity between _stage coach_ and _go coach_ to be less than the similarity
between _stage coach_ and _Cinderella’s coach_. We believe that such
problematic cases exist precisely due to the lack of care in training BERT on
historical text, which forbade the model to disambiguate nuanced usage
differences across historical time periods.

In order to investigate the validity of this claim, we present a simple
alternative approach dubbed HistBERT, a BERT-based language model that
assimilates high-quality and more balanced historical data as opposed to
relying on training data skewed toward the contemporary period. We pre-train
HistBERT on longitudinal text data, particularly the Corpus of Historical
American English (COHA) Davies (2012), which is a well-known historical corpus
containing fictions and non-fictions, newspapers, and magazines from 1820s to
2010s. We demonstrate the effectiveness of HistBERT by comparing its precision
on historical semantic similarities against the original BERT model, and we
make the following contributions:

  1. 1.

We construct contextualized word embeddings by leveraging unsupervised pre-
training on BERT using the COHA dataset. The pre-trained language model,
HistBERT, is openly available here:
https://drive.google.com/drive/folders/1E4jZPzCatmS73-Mg7iacq4SXROSPsC6b?usp=sharing.

  2. 2.

We investigate the effectiveness of our approach by comparing performance of
the original BERT and that of HistBERT. We perform evaluation on various
semantic analysis tasks, and observe promising results in word similarity and
semantic shift analysis. The source code is available at:
https://github.com/wendyqiu/diachronicBert.

The remaining part of this paper is orgainzed as follows. We review related
work in Section 2. We describe the details of our model design and
implementation for HistBERT in Section 3. We present the results in Section 4,
along with with model interpretation and discussion. We close by addressing
limitations and future directions in Section 5.

##  2 Related Work

###  2.1 Diachronic Semantic Modeling

Pre-trained language model has been shown to be effective in lexical semantic
change analysis Tahmasebi et al. (2019). Existing models can be categorized
into two major types, namely the form-based and sense-based approach Kutuzov
et al. (2018). The difference lies in that form-based models compute an
embedding for each word, whereas sense-based models encode each word sense
into a vector. As an example, Hamilton et al. 2018 constructed HistWords, the
openly available diachronic embeddings generated from PPMI, SVD, and SGND
(word2vec). More recent methods presented probabilistic language model for
time-stamped text data to track the semantic evolution of individual words
Bamler and Mandt (2017); Rudolph and Blei (2018). Despite the fact that these
approaches have achieved reasonable performance, form-based approaches
aggregate all senses of a word into a single representation Hopper (1991)
which may not be desirable in the case of modeling historical semantic shifts.

To overcome such an issue, existing studies have explored the sense-based
approach. Frermann and Lapata 2016 proposed a Bayesian-based model named SCAN
to capture how a word’s senses evolve over time. In their model, senses are
expressed as a probability distribution over words, which can also change over
time. However, such models assume that time is discrete, divided into
consecutive intervals.

Recent studies have investigated the potential of contextualized word
representations to analyzing lexical semantic change. Hu et al. 2019b proposed
a sense representation and tracking framework based on BERT and achieved a
substantial improvement in word meaning change detection. Using similar
contextualized word embeddings, Giulianelli et al. 2020 proposed a model
without lexicographic supervision and demonstrated that the detected semantic
shifts are positively correlated with human judgements. They proposed
different metrics to quantify lexical semantic changes in addition to the
contextualized word representations, and demonstrated a positive correlation
between model predictions and human judgements. In this study, we focus on
extending this recent work and pre-train the contextualized language model
with more balanced historical text—an important aspect that has not been
explored previously.

###  2.2 Pre-trained Contextualized Language Models

Pre-trained contextualized lanaguage models such as BERT leverage the transfer
learning technique to achieve good performance in various downstream NLP tasks
Devlin et al. (2019). However, as it was trained on general domain corpora
such as Wikipedia, it lacks ability to perform specialized tasks in certain
niche areas. Existing work has suggested how to improve the basic BERT by
further training on domain-specific corpora. These embeddings were proven to
be more effective than the original general-purpose BERT embeddings. For
example, SciBERT Beltagy et al. (2019), which targets specific on scientific
articles, and BioBERT Lee et al. (2019), which is constructed for biomedical
text mining.

In this work, we identify the potential bias in BERT-based diachronic semantic
analysis due to relying on modern word usages as input and a lack of
historical input text, and we show how altering the pre-training input makes a
difference in the quantification of historical word semantics.

##  3 Methodology

Here we describe the technical details of how we constructed HistBERT.
Background information on the training corpus is covered in Section 3.1. The
design process of HistBERT is divided into three steps: pre-processing
(Section 3.2), pre-training (Section 3.3) and representation (or feature)
extraction (Section 3.4). We build upon the open-sourced
BERT222https://github.com/google-research/bert and follow the official
guidelines. Note that our implementation of the BERT-based models slightly
differs from that of Giulianelli et al. 2020, which is developed using Hugging
Face’s implementation. For computational efficiency, HistBERT is constructed
using _bert-base-uncased_ , which consists of 12 hidden layers with 768
dimensions.

###  3.1 Training Corpus

For the purpose of diachronic semantic analysis, we perform further pre-
training additionally to the original BERT using a large corpus with balanced
historical text. This way, the model would be exposed to language usages
throughout the historical period in question, and the updated model should in
principle capture contextualized meanings that are not presented in modern
English. In this study, we use the COHA corpus, which is one of the largest
structured corpora of historical English to date. COHA contains more than 475M
words, extracted from various genres, including fictions, non-fictions,
newspapers, magazines and so on. The historical text ranges from the 1820s to
the 2010s Davies (2012). To ensure the amount and quality of our training
data, we focus on the 10 decades between the 1910s and the 2000s, where there
are more than 20M texts for each decade.

###  3.2 Pre-processing

To pre-process the COHA dataset, we use the default BERT tokenizier and
perform tokenization on sentence level. All textual data are normalized by
lower-casing the input, stripping out accent markers, splitting punctuation,
and handling abbreviations embedded in the dataset. We perform an additional
step to eliminate the mismatch between the tokenization schemes of BERT and
COHA. For example, _$\langle do\rangle$ , $\langle n^{\prime}t\rangle$_ for
COHA, and _$\langle don^{\prime}t\rangle$_ for BERT. We remove such
contractions and converted them back to the raw-looking form. As suggested in
Gulordava and Baroni 2011, lemmatization has little effect on BERT models
trained on English texts. Therefore, we do not lemmatize our pre-training
texts. Finally, we organize our input files according to BERT’s pre-training
requirements:

  1. 1.

One sentence per line, which would be truncated during training if exceeds
_max_seq_length_.

  2. 2.

Documents are separated by empty newline.

  3. 3.

Truncate 2% of input segments randomly to improve the model robustness to non-
sentential input.

###  3.3 Pre-training

HistBERT models are constructed by continuing pre-training the original BERT.
For each model, the training starts from the last checkpoint of the original
BERT. Models are trained on the following two tasks: _masked LM_ and _next
sentence prediction_ , which align with the original BERT model. We use the
configuration shown in Table 1 to train our models on the openly available
BERT source code. We prepare multiple models trained on different settings to
examine the performance and robustness of HistBERT performs under different
situations.

Hyperparameters | HistBERT-proto | HistBERT-5 | HistBERT-10
---|---|---|---
max_seq_length | 128 | 128 | 128
max_predictions_per_seq | 20 | 20 | 20
masked_lm_prob | 0.15 | 0.15 | 0.15
train_batch_size | 32 | 32 | 32
num_warmup_steps | 10,000 | 500,000 | 500,000
num_train_steps | 10,000 | 200,000 | 200,000
learning_rate | 2e-5 | 2e-5 | 2e-5
Table 1: Pre-training hyperparameters for different HistBERT models

We train a prototype model, HistBERT-proto, to verify the feasibility of our
proposed idea, and to present results for a specific use case (Section 4.2).
The prototype model is trained on selected data, with reduced the breadth and
depth. As mentioned, we focus on the 20th century, where there are more than
20M texts per decade in COHA. For HistBERT-proto, we further reduce the depth
of training data by selecting documents that cover a specific keyword, in this
case, _“coach”_. We select this keyword as it is identified as a problematic
word in Giulianelli et al. 2020. Based on the original BERT model, the
similarity between _stage coach_ and _go coach_ is smaller than the similarity
between _stage coach_ and _Cinderella’s coach_ , which contradicts with the
common judgements. We suspect the reason to be the lack of training on
historical text data in the original BERT model. To test our hypothesis, we
build a prototype using textual data that contain the keyword “coach”. The
training hyperparameters for this prototype is much smaller than other models.
We limit the number of training steps to prevent overfitting, because the
training set consists of only 280M texts, much smaller than the full COHA.

A richer version of the model, HistBERT-5, is also trained on partial data
selected from COHA, covering only 5 decades of texts, ranging from 1910s to
1950s. In other words, it is to expose sufficient historical data to the
original BERT model, but focus only on the first half decade.

The most comprehensive model, HistBERT-10, is trained on data selected from
1910s to 2000s (i.e., 10 decades). This model is trained on a much larger
subset of COHA. We eliminate texts from the last decade (2010s), as they may
overlap with the original training data of BERT. The 10-decade textual data
also align with the time period of our evaluation set DUPS which was used in
Giulianelli et al. 2020. We investigate the impact of different training data
on the model performance by comparing these HistBERT models. Further analysis
is covered in 4.

Due to the large amount of data, we performed additional training using a
larger _max_seq_length_ , allowing the model to further learn its positional
embeddings. The reason why we did not use _max_seq_length_ $=512$ for all its
training iterations is that longer sequences are disproportionately expensive
due to the attention nature of BERT. As suggested by the original
authors333https://github.com/google-research/bert, a batch of 32 sequences of
length 512 is much more expensive than a batch of 128 sequences of length 128,
despite having the same amount of training data.

All 3 versions of HistBERT are trained from the last checkpoint of _bert-base-
uncased_. The pre-training process is done on a single Google Cloud v3-8 TPU
with 64 GB memory. The virtual instance was on 2 Intel Haswell vCPUs with 7.5
GB memory, and a 250 G of hard drive is associated with each instance. The
output models were stored in Google Cloud Storage Bucket. Each finalized model
has an average size of 1.3 GB. Each model takes up to 34 hours to complete.

###  3.4 Representation Extraction

Using different versions of HistBERT, we construct data representations from
the pre-trained language models. We extract word embeddings from HistBERT
using a similar approach as the prior work Giulianelli et al. (2020): for each
focus word $w$, we will find all sentences that contain $w$, with its position
denoted as $i$. Next, we feed these sentences into the pre-trained model and
obtain the according sentence embeddings. To find the exact embedding for each
focus word, we extract embeddings from position $i$ in each hidden layers and
sum them dimension-wise. For efficiency, we focus on the last 4 hidden layers,
which is the suggested number of layers to be used in BERT’s feature vector
extraction.

To measure word sense similarity, we compute the distance between the selected
vector representations. To quantify lexical semantic change, we compute a
cosine similarity score for each pairwise comparison between two word usages
from different times. We next describe the details for model evaluation.

##  4 Evaluation and Results

We evaluate HistBERT by comparing its performance against the original BERT
model. We observe an overall improvement in the positively correlation with
human annotations (Section 4.1). We further perform a case study on the
specific keyword “coach”, by providing a low-dimensional visualization in
Section 4.2, and analyzing the embedding shifts in Section 4.3. The
experiments were conducted on a local Ubuntu 20.04 system, with a single
Nvidia 1070 GPU and an Intel i7-7700 CPU @ 3.60GHz with 16.0 GB of memory.

###  4.1 Evaluation against DUPS Human Similarity Data

| BERT | HistBERT-proto | HistBERT-5 | HistBERT-10
---|---|---|---|---
| $\rho$ | p | $\rho$ | p | $\rho$ | p | $\rho$ | p
leaf | 0.2405 | 0.0010 | 0.3059 | 0.0010 | 0.2896 | 0.0010 | 0.2742 | 0.0010
sphere | 0.4961 | 0.0010 | 0.4339 | 0.0050 | 0.4359 | 0.0070 | 0.4124 | 0.0160
coach | 0.4338 | 0.0010 | 0.4443 | 0.0020 | 0.3625 | 0.0090 | 0.3931 | 0.0080
net | 0.2816 | 0.0010 | 0.2201 | 0.0010 | 0.2051 | 0.0020 | 0.2182 | 0.0020
federal | 0.1227 | 0.0010 | 0.1315 | 0.0010 | 0.1596 | 0.0010 | 0.1514 | 0.0010
compact | 0.2033 | 0.0080 | 0.2702 | 0.0010 | 0.2599 | 0.0010 | 0.2708 | 0.0010
signal | 0.2241 | 0.0090 | 0.2768 | 0.0010 | 0.1583 | 0.0830 | 0.1633 | 0.0850
mirror | 0.4985 | 0.0100 | 0.4945 | 0.0100 | 0.5307 | 0.0060 | 0.5629 | 0.0030
optical | 0.1902 | 0.0220 | 0.2542 | 0.0020 | 0.2739 | 0.0010 | 0.2554 | 0.0030
spine | 0.1819 | 0.0380 | 0.1593 | 0.0970 | 0.1534 | 0.1470 | 0.1502 | 0.1490
card | 0.3987 | 0.0535 | 0.4987 | 0.0160 | 0.5209 | 0.0090 | 0.5205 | 0.0100
brick | 0.4462 | 0.2340 | 0.4235 | 0.0400 | 0.5012 | 0.0140 | 0.5376 | 0.0030
virus | 0.1968 | 0.3870 | 0.5618 | 0.0040 | 0.4214 | 0.0480 | 0.4684 | 0.0470
disk | 0.1066 | 0.5010 | 0.1735 | 0.2710 | 0.1337 | 0.3860 | -0.0233 | 0.8790
energy | -0.0151 | 0.9260 | 0.1487 | 0.3820 | 0.2591 | 0.1260 | 0.2209 | 0.2360
virtual | 0.0088 | 0.9580 | -0.0415 | 0.7790 | 0.0547 | 0.6960 | 0.0878 | 0.5050
Table 2: Spearman’s rank correlations of different BERT models on DUPS. Words
that have a significant positive correlation are highlighted. Results that
outperform the BERT baseline are marked in bold: in the cases where the result
is significant, the coefficient $\rho$ will be marked; vice-versa, the p-value
will be marked. A line is shown to separates the 3 words that do not pass the
test of significance in any models.

As an overall evaluation, we determine whether the word embeddings output from
HistBERT align with human similarity judgements, and how well it performs in
comparison to the original BERT model. In this experiment, we test our models
using the Diachronic Usage Pair Similarity (DUPS)
dataset444https://doi.org/10.5281/zenodo.3773250 created by Giulianelli et al.
2020.

The DUPS dataset contains crowdsourced similarity judgements of English word
usage pairs from different time decades between the 20th century, each being
labelled by five human annotators. They sampled 16 words uniformly at random
from the GEMS dataset Gulordava and Baroni (2011), resulting in 3,285 usage
pairs. Each DUPS usage pair contains two text snippets that contains the same
focus word. Each snippet contains a label of time interval to which the usage
sentence belongs. A sample usage pair for the keyword “virus” is shown as
follows:

> Solomon’s software, recently acquired by data-fellows, offers a pair of
> virus scanners: Dr.Solomon’s Anti-virus Deluxe for Windows 95 and
> Dr.Solomon’s Virex for the macintosh OS. [1990–2000]

and

> Virus, of many forms, appears in every cancer patient and vitiates his
> blood, upsets the biochemistry homologous to the normal for the species.
> [1910–1920]

By averaging the five scores, we obtain a single human similarity judgment for
each usage pair. We create a dictionary of matrices to record the average
scores for all usage pairs, denoted as the _matrices of DUPS’s similarity
scores_.

We use the following steps to compute similarity scores using the original
BERT and our HistBERT models: given the sentence snippets provided in DUPS, we
obtain sentence embeddings from each BERT-based model, and extract the word
embedding for the focus word, resulting in two contextualized word vectors for
each usage pair. We compute the similarity scores between the two usages. For
each model, we create a dictionary of matrices that have the same dimension as
the _matrices of DUPS’s similarity scores_. We compute the Spearman’s rank
correlation between the human similarity scores and the similarity scores
obtained from each model. We use the Scikit-bio Python
library555http://scikit-bio.org/docs/0.1.3/index.html, which is developed
based on the Mantel test Mantel (1967). Table 2 presents the correlation
coefficients $\rho$ and p-values obtained for each focus word.

Recall that we use the tensorflow-based BERT library to extract features, and
create word embeddings based on the last 4 layers only. Because of the set up
difference between our work and the prior work by Giulianelli et al. 2020, the
correlation results of BERT model we present are slightly different from what
they reported. However, the words with significant positive correlations
remain the same, summing to 10 out of 16 words, as highlighted in the BERT
column in Table 2.

We observe an improvement in both the correlation coefficients and p-values.
Firstly, the number of words with significant positive correlation (p $\leq
0.05$) increases from 10 out of 16 to 12, 11, and 12 respectively for
HistBERT-proto, HistBERT-5 and HistBERT-10. Compared with BERT, our best
model, HistBERT-10, shows an increase in the average correlation of the
significant words, from $0.31$ to $0.35$. The range also increases from
$[0.12,0.5]$ to $[0.15,0.56]$. The performance difference can be explained by
the choices of training data: HistBERT-10 is trained with the largest amount
of additional textual data among all models. At the same time, it covers the
full 10 decades from the 1910s to 2000s, which align with the target time
period of DUPS.

For the 13 words that have any significant positive correlations in one or
more models, we observe 10 words that show an improvement under the
construction of HistBERT. For the remaining 3 words, as shown in the bottom of
the table below the line, HistBERT models result in lower p-values, indicating
a stronger deviation against the null hypothesis. The gain of significance is
likely due to the increased size of training samples for senses that occur
more frequently during the 20th century, yet are not very popular in modern
English. Among all 16 words, there is only 1 word showing a deviation from
significance. We believe that introducing more historical usages may decrease
the prediction accuracy on novel senses, as the language model shifts its
focus towards the historical usages since they are used in the most recent
training process. This may be related to catastrophic interference, which is
the tendency of a neural network to forget previously learned information upon
learning new information Ratcliff (1990); McCloskey and Cohen (1989).

Overall, we observe an improvement in 13 out of 16 words. The remaining 3
words are “sphere”, “net” and “spine”. We believe that the different senses of
these words do not specifically related to historical texts in the 20th
century. In such situations, there is no obvious advantage of the HistBERT
models over the original BERT, despite being trained on additional COHA data.

###  4.2 Case Study and Visualization of Semantic Shifts

Figure 1: PCA visualization for usages of “coach” before and after training on
historical data: vectors of BERT are labelled with numbers only, whereas
vectors produced by HistBERT-proto are labelled with [number]_new.

To gain insights into the HistBERT model, we perform a case study to further
illustrate how time-relevant data improves performance in diachronic semantic
analysis using BERT.

We focus on the keyword “coach” and visualize the embedding shifts from BERT
and HistBERT-proto. We prepare 12 sentences that cover the focus word “coach”.
All sentences are selected from either COHA or online resources such as
Wikipedia. Because the focus word occurs more than once in some sentences, the
total number of usages generated from these sentences is 17, as shown in the
legend of Figure 1. These sample sentences cover various usages of the word
“coach” grouped into 4 major categories:

  1. 1.

An enclosed vehicle with four wheels, pulled by horses, and in which people
used to travel.

  2. 2.

A large, comfortable bus that carries passengers on long journeys.

  3. 3.

Someone who trains a person or team of people in a particular sport.

  4. 4.

An American luxury design house specializing in handbags, luggage and ready-
to-wear.

As seen in the visualization, the original usages of “coach” generated from
BERT are labelled with numbers, and the same usages created by HistBERT-proto
are labelled as _[number]_new_. Note that sample 9 (circled in yellow) remains
at the same position before and after, whereas other types of usages shift
away from their original positions and where sample 9 locates. This is
expected because the sense of a sport coach is very different from the
remaining categories. We used sample 9 as a stable reference (or anchors) to
illustrate that the shifts of other usages are not random.

The right-most cluster is separated from the remaining usages (circled in
green). We observe that sample 4 and 12 move towards sample 2, forming a
closer cluster. The reduction of intra-cluster distance indicates that after
training on historical data, HistBERT-proto is able to better differentiate
_category 1_ from other usage types: all three samples in this cluster refer
to stagecoaches.

In the cluster circled in blue, sample 5, 6, and 11 all refer to _category 2_
, namely “express coach”, “coach travel”, and “go coach”. After the additional
training on COHA, our language model produces a tighter cluster of these
samples, which share similar meanings. Also notice that sample 1, which refers
to Cinderella’s pumpkin coach, moves away from the “express coach” cluster.

Observe that sample 1 not only shifts away from the cluster of _category 2_ ,
it also moves towards sample 8, and to the further bottom where sample 7, 3
and 10 locate. These samples refer to usages of “stage coach”, “stage-coach”,
and “coachman”. Despite sharing a similar sense, they are grouped into a
different cluster from sample 2, 4 and 12, due to the difference in their
lexical usages. The movement of sample 1 is also desirable: as an existing
issue of using BERT directly to generate usage representations, the usages of
“go coach” and “stage coach” are mixed into one cluster, whereas “Cinderella’s
coach” is left out in a separated cluster. In our newly constructed model, we
observe that the “go coach” cluster is moving away from the “stage coach”
cluster, and “Cinderella’s coach” is moving towards “stage coach”.

Also, observe that sample 5, which refers to “express coach”, shifts away from
the middle cluster that contains “Cinderella’s coach” and “stage-coach”, to
the very top where “express coach” and “go coach” samples are.

The remaining samples are located on the top-left corner. Sample 14, 15, 16
and 17 (circled in red) belong to _category 4_ : the American brand _Coach New
York_. Observe that sample 14, 16, and 17 are moving towards each others and
form a tighter cluster, reducing the intra-cluster distance; sample 15 shifts
along the x-axis of the cluster’s direction, but in opposite y-axis direction.

To summarize, we observe a decrease in the intra-cluster distance, whereas the
inter-cluster distance increases. We also observe some substantial shifts that
correct the misunderstood word sense representations from the original BERT
model.

###  4.3 Quantification of Semantic Shifts in Embedding Space

To go beyond the qualitative interpretations, we also quantify the shifts in
contextualized word embeddings presented in the PCA visualization. We measure
the pairwise similarity between each usage pair and compare the difference
between what BERT and HistBERT-proto produce. We define this quantity as the
inverse of embedding shift due to the additional training on historical data.

The average embedding shift of all 17 instances of “coach” is $-0.012$, which
is closed to $0$, indicating that the overall shift remains balanced. The
maximum increase of cosine similarity indicates that the usage pair undergoes
a change and become the most similar. We observe that the pair of
“Cinderella’s Coach” and “stagecoach” has the maximum increase in similarity
($0.066$). On the other hand, the maximum decrease occurs in usage pairs that
become the least similar. In our case, “(express) coach” and “(stage) coach”
has a negative change in similarity: $-0.13$.

Both of these results confirm our hypothesis that additional training on
historical data correct the misunderstood word sense representations from the
original model, and helps BERT to better represent words that undergo semantic
shifts. The effectiveness of contextual embeddings in diachronic semantic
analysis is dependent on the input quality and specifically temporal profile
of the input text.

##  5 Conclusion

We present HistBERT, a pre-trained BERT-based language model for historical
semantic analysis. We construct various versions of the model and evaluate
them against human similarity judgments. We also present a visualization to
illustrate semantic shifts due to the additional training on historical
textual data.

Our results indicate that training on historical data is beneficial for
diachronic semantic analysis using BERT. We suggest that the temporal profile
of the textual input should be carefully controlled for in order to optimize
the precision of the contextualized language model in a diachronic setting.

Future work could apply HistBERT to detect semantic shifts at a systematic
scale. For instance, experimenting on word usage clustering and changes
similar to Giulianelli et al. 2020, or detecting lexical semantic changes
using unsupervised methods, as described in SemEval-2020 Task 1 Schlechtweg et
al. (2020b), are promising venues for further exploration of the methodology
that we have presented here.

##  6 Acknowledgements

We thank David Lie for helpful feedback on this manuscript.

## References

  * Bamler and Mandt (2017) Robert Bamler and Stephan Mandt. 2017.  Dynamic word embeddings.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 380–389. PMLR.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A Pretrained Language Model for Scientific Text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3613–3618, Hong Kong, China. Association for Computational Linguistics.
  * Cook and Stevenson (2010) Paul Cook and Suzanne Stevenson. 2010.  Automatically identifying changes in the semantic orientation of words.  In _LREC_.
  * Davies (2012) Mark Davies. 2012.  Expanding horizons in historical linguistics with the 400-million word Corpus of Historical American English.  _Corpora_ , 7(2):121–157.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv:1810.04805 [cs]_.  ArXiv: 1810.04805.
  * Dubossarsky et al. (2017) Haim Dubossarsky, Daphna Weinshall, and Eitan Grossman. 2017.  Outta Control: Laws of Semantic Change and Inherent Biases in Word Representation Models.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1136–1145, Copenhagen, Denmark. Association for Computational Linguistics.
  * Frermann and Lapata (2016) Lea Frermann and Mirella Lapata. 2016.  A Bayesian Model of Diachronic Meaning Change.  _Transactions of the Association for Computational Linguistics_ , 4:31–45.
  * Giulianelli et al. (2020) Mario Giulianelli, Marco Del Tredici, and Raquel Fernández. 2020.  Analysing Lexical Semantic Change with Contextualised Word Representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3960–3973, Online. Association for Computational Linguistics.
  * Gulordava and Baroni (2011) Kristina Gulordava and Marco Baroni. 2011.  A distributional similarity approach to the detection of semantic change in the google books ngram corpus.  In _Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics_ , GEMS ’11, page 67–71, USA. Association for Computational Linguistics.
  * Hamilton et al. (2016) William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016.  Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1489–1501, Berlin, Germany. Association for Computational Linguistics.
  * Hamilton et al. (2018) William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2018.  Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change.  _arXiv:1605.09096 [cs]_.  ArXiv: 1605.09096.
  * Hopper (1991) Paul J. Hopper. 1991.  On some principles of grammaticization.  In Elizabeth Closs Traugott and Bernd Heine, editors, _Typological Studies in Language_ , volume 19:1, page 17. John Benjamins Publishing Company, Amsterdam.
  * Hu et al. (2019a) Renfen Hu, Shen Li, and Shichen Liang. 2019a.  Diachronic Sense Modeling with Deep Contextualized Word Embeddings: An Ecological View.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3899–3908, Florence, Italy. Association for Computational Linguistics.
  * Hu et al. (2019b) Renfen Hu, Shen Li, and Shichen Liang. 2019b.  Diachronic sense modeling with deep contextualized word embeddings: An ecological view.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3899–3908, Florence, Italy. Association for Computational Linguistics.
  * Kim et al. (2014) Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014.  Temporal Analysis of Language through Neural Language Models.  In _Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science_ , pages 61–65, Baltimore, MD, USA. Association for Computational Linguistics.
  * Król (2017) Marcin Król. 2017.  The past is not a foreign country: As a new Polish history museum opens in Gdansk, a jigsaw of censorship and counter stories are rewriting the country’s story of itself. Historian Marcin Król examines the past for insight into today’s battles.  _Index on Censorship_ , 46(1):74–76.
  * Kulkarni et al. (2015) Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015.  Statistically significant detection of linguistic change.  In _Proceedings of the 24th International Conference on World Wide Web_.
  * Kutuzov et al. (2018) Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, and Erik Velldal. 2018.  Diachronic word embeddings and semantic shifts: a survey.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1384–1397, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , page btz682.
  * Mantel (1967) Nathan Mantel. 1967.  The detection of disease clustering and a generalized regression approach.  _Cancer Research_ , 27(2 Part 1):209–220.
  * McCloskey and Cohen (1989) Michael McCloskey and Neal J. Cohen. 1989.  Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem.  In _Psychology of Learning and Motivation_ , volume 24, pages 109–165. Elsevier.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  _arXiv:1802.05365 [cs]_.  ArXiv: 1802.05365.
  * Radford et al. (2018) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018.  Language models are unsupervised multitask learners.
  * Ratcliff (1990) Roger Ratcliff. 1990.  Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. _Psychological Review_ , 97(2):285–308.
  * Rosenfeld and Erk (2018) Alex Rosenfeld and Katrin Erk. 2018.  Deep Neural Models of Semantic Shift.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 474–484, New Orleans, Louisiana. Association for Computational Linguistics.
  * Rudolph and Blei (2018) Maja Rudolph and David Blei. 2018.  Dynamic Embeddings for Language Evolution.  In _Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW ’18_ , pages 1003–1011, Lyon, France. ACM Press.
  * Sagi et al. (2011) Eyal Sagi, Stefan Kaufmann, and Brady Clark. 2011.  Tracing semantic change with latent semantic analysis.  _Current methods in historical semantics_ , 73:161–183.
  * Schlechtweg et al. (2020a) Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020a.  Semeval-2020 task 1: Unsupervised lexical semantic change detection.  _arXiv preprint arXiv:2007.11464_.
  * Schlechtweg et al. (2020b) Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020b.  SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection.  _arXiv:2007.11464 [cs]_.  ArXiv: 2007.11464.
  * Tahmasebi et al. (2018) Nina Tahmasebi, Lars Borin, and Adam Jatowt. 2018.  Survey of computational approaches to lexical semantic change.  _arXiv preprint arXiv:1811.06278_.
  * Tahmasebi et al. (2019) Nina Tahmasebi, Lars Borin, and Adam Jatowt. 2019.  Survey of Computational Approaches to Lexical Semantic Change.  _arXiv:1811.06278 [cs]_.  ArXiv: 1811.06278.
  * Traugott and Dasher (2001) Elizabeth Closs Traugott and Richard B Dasher. 2001.  _Regularity in semantic change_.  Cambridge University Press.
  * Xu and Kemp (2015) Yang Xu and Charles Kemp. 2015.  A computational evaluation of two laws of semantic change.  In _Proceedings of the 37th Annual Meeting of the Cognitive Science Society, CogSci 2015, Pasadena, California, USA, July 22-25, 2015_. cognitivesciencesociety.org.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books.  _arXiv:1506.06724 [cs]_.  ArXiv: 1506.06724.