# Towards Effective Time-Aware Language Representation: Exploring Enhanced
Temporal Understanding in Language Models

Jiexin Wang  South China University of TechnologyChina
<EMAIL_ADDRESS>,  Adam Jatowt  University of InnsbruckAustria
<EMAIL_ADDRESS>and  Yi Cai  South China University of TechnologyChina
<EMAIL_ADDRESS>
(2018)

###### Abstract.

In the evolving field of Natural Language Processing, understanding the
temporal context of text is increasingly crucial. This study investigates
methods to incorporate temporal information during pre-training, aiming to
achieve effective time-aware language representation for improved performance
on time-related tasks. In contrast to common pre-trained models like BERT,
which rely on synchronic document collections such as BookCorpus and
Wikipedia, our research introduces BiTimeBERT 2.0, a novel language model pre-
trained on a temporal news article collection. BiTimeBERT 2.0 utilizes this
temporal news collection, focusing on three innovative pre-training
objectives: Time-Aware Masked Language Modeling (TAMLM), Document Dating (DD),
and Time-Sensitive Entity Replacement (TSER). Each objective targets a unique
aspect of temporal information. TAMLM is designed to enhance the understanding
of temporal contexts and relations, DD integrates document timestamps as
chronological markers, and TSER focuses on the temporal dynamics of ”Person”
entities, recognizing their inherent temporal significance. The experimental
results consistently demonstrate that BiTimeBERT 2.0 outperforms models like
BERT and other existing pre-trained models, achieving substantial gains across
a variety of downstream NLP tasks and applications where time plays a pivotal
role.

language models, temporal information, news archives

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
emai; June 03–05, 2018; Woodstock, NY††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Information systems Content analysis and feature selection

##  1\. Introduction

In recent years, the digitization of news archives has significantly improved
our access to historical data. Digital archives offer a comprehensive
perspective of past events, societal shifts, and cultural evolution, assisting
researchers and historians in identifying pivotal moments and trends that have
played a major role in shaping today’s society (Korkeamäki and Kumpulainen,
2019).

Furthermore, incorporating the chronological data into Natural Language
Processing (NLP) and Information Retrieval (IR) systems has become
increasingly crucial. This has spurred the development of Temporal Information
Retrieval, a specialized branch of IR that concentrates on the time-related
aspects of queries and documents. Beyond information retrieval, temporal
information has found applications across a wide range of tasks, presenting
exciting avenues for research and innovation. Examples include web content
analysis (Calzarossa and Tessera, 2015; MacKay et al., 2017), social media
trends analysis (Chandrasekaran et al., 2020; Khan et al., 2020), temporal
reasoning (Jain et al., 2023; Zhou et al., 2020), timeline summarization
(Steen and Markert, 2019; Campos et al., 2021), event detection and ordering
(Strötgen and Gertz, 2012; Derczynski, 2017), event time prediction (Wang et
al., 2021a), question answering (Pasca, 2008; Wang et al., 2020), content
trustworthiness analysis (Yamamoto et al., 2008) and semantic change detection
(Rosin and Radinsky, 2022; Rosin et al., 2022). These applications underscore
the broad and varied impact of temporal information in numerous research
domains and practical applications.

Pre-trained language models like BERT (Devlin et al., 2018), T5 (Raffel et
al., 2020), and GPT-3 (Brown et al., 2020) have revolutionized NLP by enabling
deeper, context-aware linguistic representations. These models exhibit
remarkable performance in diverse NLP tasks, including text classification,
sentiment analysis, and machine translation. However, they often struggle to
capture essential domain-specific information, primarily due to their reliance
on training on general-purpose corpora like English Wikipedia and training
objectives not tailored to domain-specific characteristics. In scenarios where
understanding temporal dynamics is crucial, most existing language models face
challenges in effectively integrating temporal information, such as temporal
expressions and document timestamps. This capability is particularly vital in
processing news articles and in the aforementioned applications where time is
an important factor. Developing language models that are not only context-
aware but also adept at comprehending temporal aspects would promise a more
refined and accurate interpretation of temporal information, thereby
significantly enhancing their effectiveness in time-related tasks.

In this study, we present BiTimeBERT 2.0, a novel language model pre-trained
on a temporal news collection with three training objectives: _time-aware
masked language modeling_ (TAMLM), _document dating_ (DD), and _time-sensitive
entity replacement_ (TSER), as depicted in Figure 1(a). Each objective targets
a distinct aspect of temporal information. The first dimension, known as
_document timestamp_ , serves as a chronological marker, indicating the
creation date of each document and establishing a definite temporal reference.
The second dimension, _content time_ , relates to the temporal expressions
within the text, shedding light on the temporal context and the period of the
events discussed. These two explicit temporal dimensions have proven valuable
across various applications, such as temporal web search (Stack, 2006; Joho et
al., 2013), summarization (Allan et al., 2001; Barros et al., 2019), and event
ordering (Honovich et al., 2020), all benefiting from a nuanced understanding
of temporal aspects. Lastly, _time-sensitive entity_ , involves dynamic
entities like individuals, which often imply specific timeframes (Rijhwani and
Preotiuc-Pietro, 2020; Xu et al., 2022; González-Gallardo et al., 2023;
Agarwal et al., 2018). For example, mentioning Sam Altman likely indicates a
period around 2022 or later, coinciding with the emergence and significant
impact of ChatGPT. This is in contrast to temporally static entities like
”Location” entities or ”Organization” entities, which generally remain
unchanged over time, as illustrated by (Agarwal et al., 2018). Notably, our
study places a particular emphasis on ”Person” entities, due to their inherent
temporal significance. Although other time-sensitive entities like EVENT
(e.g., WWII) or PRODUCT (e.g., iPhone 15) also convey implicit temporal
references, they are comparatively rarer or pose greater challenges in precise
identification and thus are not the primary focus of this research.

In the experimental evaluation, we conduct extensive experiments to thoroughly
assess the effectiveness of the proposed pre-training tasks. The results from
these experiments unequivocally demonstrate that BiTimeBERT 2.0, harnessing
the three distinct types of temporal information through its pre-training
tasks, yields remarkably effective time-aware representations. Furthermore,
our findings underscore the substantial potential of BiTimeBERT 2.0 to
significantly enhance performance across a wide spectrum of time-related
downstream tasks, showcasing its versatility and applicability in various
applications.

This paper represents a substantial extension of our earlier work presented in
(Wang et al., 2023), introducing significant advancements to the BiTimeBERT
model. Firstly, the time-aware masked language modeling (TAMLM) task is
refined to include the masking of temporal signals that indicate temporal
relations, such as ”before”, ”after”, and ”during”. This refinement allows for
a deeper comprehension of temporal contexts compared to the previous approach,
which focused solely on masking temporal information. Secondly, we introduce
the time-sensitive entity replacement (TSER) objective, with a particular
focus on dynamic ”Person” entities. This objective is designed to further
enrich the model’s comprehension of temporal dynamics associated with
individuals. Additionally, BiTimeBERT 2.0 is pre-trained on a smaller, more
focused temporal news collection, which excludes sentences lacking temporal
information. This strategy effectively halves the training costs, reducing GPU
hours from 80 to 38, without compromising the quality of our results in most
cases. Furthermore, this paper explores various methodologies in employing
temporal information, experimenting with different training objectives to
enhance our understanding. These enhancements enable our study to provide a
more thorough exploration of time-aware language representations,
demonstrating a significant leap in both efficiency and effectiveness, thereby
contributing to advancement of temporal modeling within the realm of language
models.

To sum up, we make the following contributions in this work:

  1. (1)

We thoroughly explore the impact of integrating three distinct types of
temporal information into pre-trained language models. Our findings reveal
that infusing language models with temporal information notably improves their
performance across various downstream time-related tasks, highlighting the
importance of temporal context.

  2. (2)

We introduce BiTimeBERT 2.0, a novel time-aware language model specifically
trained through three new pre-training tasks. In addition, we implement
several strategic enhancements to achieve more effective time-aware
representations, including the refinement of the time-aware masked language
modeling, the introduction of time-sensitive entity replacement objectives,
efficient training with a more focused temporal news collection.

  3. (3)

We conduct extensive experiments on a variety of time-related tasks,
showcasing BiTimeBERT 2.0’s superiority over existing language models in
comprehending temporal information. This empirical validation not only
underscores our model’s enhanced ability in generating effective time-aware
representations, but also demonstrates its practical efficiencies in real-
world applications.

##  2\. Related Work

###  2.1. Temporal Dimensions of Text

According to Campos et al. (2014), there are two key temporal dimensions of a
document or a query: timestamp and content time (or focus time). The document
timestamp refers to the time when the document was created, whereas the
content time refers to the time mentioned or implicitly referred to in the
document’s content (e.g., a document published today111We use ”2024/01/29” as
the exact date of ”today” being the submission deadline. about WWII would have
its timestamp as January 29, 2024, and the content time as 1939-1945)
Similarly, the query timestamp indicates the time when the query was issued,
and its content time relates to the time period associated with the query
(e.g., a ”Winter Olympics 1978” query issued three month ago would have its
timestamp as October 2023, and would refer to the time period when the winter
sports took place in Edmonton, Alberta in 1987). Additionally, as a document
usually contains sentences related to different events that take place in
different time points,the document’s content time is typically represented by
a set of time intervals (Jatowt et al., 2013). For readers, timestamp
information aids in quickly locating the content on a timeline and helps
assessing the document’s up-to-dateness. On the other hand, content time can
help to strengthen our understanding of particular parts of document content.
For example, the development of events and their causal relations can be
understood by analyzing the relations between different content temporal
information scattered throughout text content. Recently, exploiting these two
kinds of temporal information in documents and queries has been gaining
increased importance in IR and NLP. Their interplay can be utilized to develop
time-specific search and exploration applications (Alonso et al., 2011; Campos
et al., 2014; Kanhabua and Anand, 2016), such as temporal web search (Stack,
2006; Joho et al., 2013), temporal question answering (Wang et al., 2020,
2021b), search results diversification (Berberich and Bedathur, 2013; Styskin
et al., 2011) and clustering (Alonso et al., 2009; Svore et al., 2012),
summarization (Allan et al., 2001; Barros et al., 2019), historical event
ordering (Honovich et al., 2020) and so on.

In addition to the two temporal dimensions previously discussed, many named
entities inherently carry implicit temporal information. Research has
demonstrated that incorporating temporal information into named entity
recognition can significantly improve accuracy (Rijhwani and Preotiuc-Pietro,
2020; Xu et al., 2022; González-Gallardo et al., 2023), thereby confirming
that named entities are closely tied to specific time periods. Furthermore,
Agarwal et al. (2018) show that integrating temporal information into named
entity disambiguation yields impressive results, particularly for ”Person”
type entities. While ”Location” and ”Organization” entities experience
moderate performance improvements, ”Person” entities demonstrate a substantial
increase in accuracy, from 9.63% to 39.91%. This significant improvement can
be attributed to the relatively shorter life spans (or activity time spans) of
person entities, making them more time-sensitive compared to other entity
types. Motivated by these findings, our study places special emphasis on
”Person” entities, recognizing their inherent temporal significance. Although
other time-sensitive entities such as products (e.g., iPhone 15) do carry
implicit temporal references, they are comparatively rarer and present more
challenges in terms of precise identification.

###  2.2. General Pre-trained Language Models

The field of Natural Language Processing (NLP) has witnessed a transformative
change with the advent of pre-trained language models. Early models such as
Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) lay the
groundwork by capturing word-level representations, but it is the introduction
of contextual models such as ELMo (Peters et al., 2018) that first enable a
deeper understanding of word usage within specific contexts. The emergence of
Transformer-based models, notably BERT (Devlin et al., 2018), T5 (Raffel et
al., 2020), and GPT (Radford et al., 2018), mark a pivotal shift. These
models, leveraging the Transformer architecture (Vaswani et al., 2017), are
typically trained on large general-purpose corpora of text and then fine-tuned
for specific NLP tasks, which offer advanced context-aware capabilities. BERT,
in particular, utilizes masked language modeling (MLM) and next sentence
prediction (NSP) to enhance understanding of text sequences. RoBERTa (Liu et
al., 2019) further refines BERT’s approach by optimizing key hyperparameters
and removing the NSP task, while ALBERT (Lan et al., 2019) introduces sentence
order prediction (SOP), achieving higher performance with fewer parameters.
Further innovations include XLM (Lample and Conneau, 2019), which incorporates
translation language modeling (TLM) for cross-lingual training, XLNet (Joshi
et al., 2020) with its permuted language modeling (PLM), and SpanBERT (Joshi
et al., 2020), which extends BERT by masking contiguous spans and employing a
span boundary objective. Despite these advancements, a notable limitation of
these language models is their reliance on general-purpose text corpora, and
training objectives not tailored for specific domains, which restricts their
effectiveness in domain-specific applications.

###  2.3. Language Models for Specific Domains

To address the limitations of general-purpose language models, there has been
a trend towards developing models tailored to specific domains, leading to
enhanced performance in domain-specific tasks. Some studies adapt pre-trained
models to specific domains by directly applying BERT’s pre-training tasks on
domain-specific corpora. For instance, SciBERT (Beltagy et al., 2019) is
trained on scientific texts, BioBERT (Lee et al., 2020) on biomedical
documents, ClinicalBERT (Huang et al., 2019) on clinical records, LegalBERT
(Chalkidis et al., 2020) on legal documents. These domain-adapted models have
shown remarkable effectiveness in their respective fields, such as BioBERT’s
improved accuracy in biomedical entity recognition and biomedical relation
extraction. Furthermore, there is a growing focus on continually pre-training
existing models for targeted applications using specific training objectives.
SentiLARE, proposed by Ke et al. (2019), continually pre-trains the RoBERTa
model with a label-aware masked language model for sentiment analysis. WKLM
(Weakly Supervised Knowledge-Pretrained Language Model) (Xiong et al., 2019),
another adaptation, enhances BERT for entity-related tasks through an entity
replacement objective, improving the model’s understanding of real-world
entities. Similarly, Althammer et al. (2021) introduce linguistically informed
masking (LIM), a domain adaptive objective for the patent domain and perform
continual pre-training on BERT and SciBERT. These models are increasingly
refined for specific domains and tasks, facilitating more precise and
effective language representation. The trend towards domain-specific
adaptations underscores the importance of tailoring language models to the
intricacies of specialized fields, thereby enhancing their performance and
applicability.

###  2.4. Incorporating Temporal Information into Language Models

In recent years, the integration of temporal information into language models
has garnered considerable research interest, leading to various innovative
approaches aimed at enhancing these models’ temporal understanding
(Giulianelli et al., 2020; Dhingra et al., 2021; Rosin and Radinsky, 2022;
Rosin et al., 2022; Cole et al., 2023). One notable example is the work of
Dhingra et al. (2021), who modified the pre-training process by incorporating
timestamp information into the masked language modeling (MLM) objective. This
approach facilitates the acquisition of temporally-scoped knowledge, showing
improved performance in tasks like question answering. Similarly, Cole et al.
(2023) propose the Temporal Span Masking task (TSM), an adaptation of Salient
Span Masking (SSM) (Guu et al., 2020). TSM specifically focuses on masking
temporal expressions in sentences and trains the model to reconstruct these
expressions. In contrast to these models, which predominantly use Transformer
encoder-decoder architectures, other studies have explored the utility of
Transformer encoder-only models. These models have proven effective in
semantic change detection, a task that requires the model to identify and
understand the extent of semantic shifts in words over time. In this domain,
Giulianelli et al. (2020) propose the first unsupervised approach, utilizing
contextualized embeddings from BERT for semantic change detection. Further
advancements include the work of (Rosin and Radinsky, 2022) and (Rosin et al.,
2022). Rosin and Radinsky (2022) enhances self-attention mechanisms (Vaswani
et al., 2017) by integrating timestamp data to compute attention scores. On
the other hand, (Rosin et al., 2022) introduces TempoBERT, a time-aware
variant BERT model. TempoBERT pre-processes input texts by appending timestamp
information and masking these tokens during training, achieving state-of-the-
art results in semantic change detection.

The integration of temporal information into language models, while leading to
significant advancements, still faces several challenges that limit their
applicability and effectiveness. A primary weakness is that existing time-
aware models focus on either content temporal information or document
timestamps, overlooking the necessity of integrating both dimensions for a
comprehensive temporal understanding. Additionally, the pre-training of these
models often relies on coarse temporal granularity, such as years or decades,
lacking the necessary precision for finer granularities like months. This
shortcoming is particularly evident in tasks requiring detailed time-sensitive
analysis, for instance, in web content analysis (Calzarossa and Tessera, 2015;
MacKay et al., 2017). Moreover, existing models, while innovative in their
approach, often fall short in delivering effective time-aware language
representations. For instance, in experiments involving sentence time
prediction tasks, e.g., Rosin et al. (2022) show that TempoBERT exhibits a
performance decline when compared to a fine-tuned BERT model.

Compared with the aforementioned approaches, our model effectively exploits
both timestamp and content time during its pre-training phase on a temporal
news collection. This extended work takes a step further by utilizing temporal
signals, specifically the words that indicate temporal relations like
”before”, ”after”, and ”during”. Moreover, we place a special emphasis on
time-sensitive entities, with a particular focus on the ”Person” entity type
during pre-training. In our experimental evaluations, we demonstrate that
BiTimeBERT 2.0 not only delivers more effective time-aware language
representations but also outperforms existing language models in tasks
requiring a comprehensive understanding of temporal contexts.

##  3\. Method

(a) An illustration of BiTimeBERT 2.0 training, which includes the TAMLM, DD
and TSER tasks.

(b) An illustrative input example for the BiTimeBERT 2.0 model during the pre-
training phase.

Figure 1. A comprehensive overview of the BiTimeBERT 2.0 model training
process, incorporating the TAMLM, DD, and TSER tasks, along with an
illustrative input example during the pre-training phase.

In this section, we introduce BiTimeBERT 2.0, the proposed time-aware language
model built on the Transformer encoder (Vaswani et al., 2017), with its
training process illustrated in Figure 1(a). BiTimeBERT 2.0 differentiates
itself from existing language models such as BERT (Devlin et al., 2018) and
TempoBERT (Rosin et al., 2022) in four critical ways, each contributing to its
enhanced time-aware representation capabilities. Firstly, unlike general pre-
trained language models that rely on synchronic datasets such as Wikipedia or
web crawls, BiTimeBERT 2.0 is trained on a news article collection that spans
over two decades.222The temporal news collection utilized in both this study
and the previous one is the New York Times Annotated Corpus (NYT corpus),
encompassing 1.8 million news articles spanning from January 1, 1987, to June
19, 2007. While models like RoBERTa (Liu et al., 2019) do include news
datasets in their training (e.g., CC-NEWS (Nagel, 2016)), they often fail to
adequately prioritize or leverage the temporal dimensions of their documents.
Note that we employ a more focused temporal news collection in this work,
deliberately excluding sentences lacking temporal content, significantly
reducing training costs while maintaining most high-quality results333Details
regarding the pre-training corpus processing are provided in Section 4.1..
Secondly, BiTimeBERT 2.0 employs a novel _time-aware masked language modeling_
(TAMLM) strategy. Unlike traditional random token masking, TAMLM prioritizes
the masking of temporal information within the text. This technique not only
incorporates the inherent domain knowledge but also enriches the model’s
comprehension of temporal information embedded in the document content. The
TAMLM has been further refined to specifically mask temporal signals that
indicate temporal relations, such as ”before”, ”after”, and ”during”,
enhancing the model’s understanding of temporal contexts. Third, BiTimeBERT
2.0 replaces the conventional Next Sentence Prediction (NSP) objective with a
_document dating_ (DD) task, effectively integrating timestamp information
during training. As document dating involves time prediction, this objective
enriches the model with both time-related and task-oriented knowledge,
potentially aiding in improving the performance in other time-related tasks.
Lastly, this extended work introduces the _time-sensitive entity replacement_
(TSER) objective, with a specific emphasis on dynamic ”Person” entities. This
addition significantly enhances the model’s capacity to analyze and interpret
temporal dynamics, thereby increasing its overall temporal sensitivity. Figure
1(a) illustrates the three proposed objectives: TAMLM, DD, and TSER, whereas
Figure 1(b) provides an illustrative example of the input for the model during
the pre-training phase.444The selected news article, titled ”Add a Drunken-
Driving Charge to Busta Rhymes’s Troubles,” is published in The New York Times
on 2007/05/04. BiTimeBERT 2.0 is jointly trained on these tasks, incorporating
three distinct additional layers based on the output of its Transformer
network. All objectives employ cross entropy as the loss function.

It is important to note that in our exploration of BiTimeBERT 2.0, we also
experiment with additional pre-training tasks that harness temporal
information in different ways, namely Time-Sensitive Entity Masked Language
Modeling (TSEMLM) and Temporal Relation Word Replacement (TRWR). TSEMLM is
designed to mask ”Person” entities in text following the masking of content
time. TRWR, on the other hand, involves replacing temporal signals within the
text: in 50% of the cases, the temporal signals are substituted with
alternatives of a different temporal nature, such as changing ”before” to
”after” or ”during”. Both TSEMLM and TRWR, detailed in Section 4.4, were
conceived with the goal of examining whether incorporating ”Person” entities
and temporal signals via different designed objectives could enhance model
performance. However, as our experimental results will later reveal, while
they have shown promising results in certain downstream tasks, they are found
to be less effective compared to the combined impact of TAMLM, DD, and TSER in
BiTimeBERT 2.0. Subsequent sections will provide detailed elaborations on
these three objectives of BiTimeBERT 2.0.

###  3.1. Time-aware Masked Language Modeling

The first pre-training objective, Time-Aware Masked Language Modeling (TAMLM),
integrates both content time and temporal signals during pre-training. Content
time refers to explicit temporal information embedded within the document’s
content, while temporal signals are specific words that indicate temporal
relations. Following (Jia et al., 2018), we classify temporal signals into
three types, each representing a different temporal relation: BEFORE, AFTER,
and OVERLAP. For instance, BEFORE type includes words like ”before” and ”prior
to”, AFTER type includes words such as ”after” and ”following”, and OVERLAP
type includes words like ”during” and ”in”. Both content time and temporal
signals are instrumental in understanding event developments and discerning
relationships between events mentioned in texts, as evidenced in applications
like timeline summarization (Yu et al., 2021) and temporal question answering
(Wang et al., 2020, 2021b).

Consider a token sequence $X=(x_{1},x_{2},...,x_{n})$, where each $x_{i}$
$(1\leq i\leq n)$ represents a token in the vocabulary. First, we identify
temporal expressions within the document content using spaCy,555Note that in
this step, we also identify ”Person” entities, which will be utilized in the
TSER objective. as highlighted in deep pink in Figure 1(a). These recognized
temporal expressions form the set $T=(t_{1},t_{2},...,t_{m})$, where each
$t_{i}$ $(1\leq i\leq m)$ represents a specific temporal expression found in
the text. Second, based on the temporal signals list taken from (Jia et al.,
2018) and the extracted temporal expressions, we locate temporal signals
within the text, as indicated by the light pink font at the bottom of Figure
1(a). These signals form the temporal signal set $S$. Third, unlike BERT’s
masking scheme where 15% of the sequence tokens are randomly sampled directly,
we prioritize the extracted temporal expressions and temporal signals.
Specifically, we randomly sample 30% of the entire temporal expressions in $T$
and 30% of the entire temporal signals in $S$, as exemplified by ”the 1990s”
and ”Before” in Figure 1(a). This choice of 30% was determined through
experimentation, as it yielded the best performance in most cases.
Subsequently, we continue to randomly sample other tokens that are not part of
$T$ or $S$, until a total of 15% of the tokens are sampled. For instance, in
Figure 1(a), ”born” and ”figure” are sampled and masked, while the temporal
signal ”during” and temporal expression ”2006” are not allowed to be sampled.
Finally, similar to the BERT methodology, 80% of the sampled tokens are
replaced with ”[MASK]”, 10% with random tokens, and 10% with the original
tokens. For instance, in Figure 1(b), which provides a more detailed input
example during pre-training, ”clear” is sampled and remains unchanged, ”music”
is sampled and replaced with the random token ”next”, and other sampled tokens
are replaced with ”[MASK]”.

Through the TAMLM masking scheme, BiTimeBERT 2.0 is explicitly prompted to
incorporate both the inherent domain knowledge embedded in the news article
collection and the explicit temporal information, encompassing content time
and temporal signals. This strategy fosters the model’s ability to integrate
knowledge about interrelated events and understand the connections among
various temporal expressions. Specifically, this approach forces the model to
consider the contextual relationships of surrounding, unmasked temporal
details when predicting tokens for masked temporal expressions or signals. For
instance, consider the example illustrated in Figure 1(b), where the model is
presented with a text involving masked temporal expressions pertaining to the
career of Mr. Smith, a notable figure in the rap music. The text indicates
that his influential career spanned during a masked period, and before a
specific year (masked as ”[MASK] 2006”). To accurately reconstruct these
masked tokens, BiTimeBERT 2.0 must infer from the available context that Mr.
Smith’s major career phase likely occurred in the 1990s and that a notable
shift in his stance on violence and criminality emerged around 2006. This
example clearly demonstrates the effectiveness of the TAMLM scheme in
compelling the model to utilize both context and surrounding temporal cues.
This fosters precise predictions of masked temporal elements, thereby
significantly enriching the model’s grasp of temporal dynamics and enhancing
its overall temporal understanding.

###  3.2. Document Dating

The second pre-training objective, Document Dating (DD), is specifically
designed to integrate the timestamp information during its pre-training phase.
In news archives, each article is typically annotated with a timestamp
denoting its publication date. As discussed earlier, the timestamp plays an
important role in numerous applications, particularly in temporal information
retrieval (Li and Croft, 2003; Kanhabua and Nørvåg, 2010; Wang et al., 2021b).

In the DD objective, the [CLS] token is inserted at the beginning of the input
sequence, and its representation, $h_{[CLS]}$, serves as the contextual
representation for the entire sequence. However, different from the
conventional approach of binary classification for next sentence prediction
(e.g., BERT), the [CLS] token is employed to predict the document timestamp,
as indicated in green font in Figure 1(a). An important hyper-parameter in
this task is the granularity of the timestamp, which can vary from decade,
year, month, to day temporal granularity. For instance, the timestamp of the
news article in Figure 1(a) can be represented as ”2007/05” under month
granularity and simply ”2007” under year granularity. It is noteworthy that
existing models like TempoBERT (Rosin et al., 2022) typically adopt broader
temporal granularities, such as years or decades, in their pre-training phase.
While this approach facilitates easier and more accurate timestamp predictions
during pre-training, it often falls short in challenging time-related
downstream tasks or in time-sensitive applications, such as event occurrence
time prediction (Wang et al., 2021a) and web content analysis (Calzarossa and
Tessera, 2015; MacKay et al., 2017). Our prior research has demonstrated that
utilizing month granularity yields the most effective results across a variety
of downstream tasks. Therefore, we have continued to employ month granularity
in this work. However, an interesting observation from our studies is that the
use of day granularity often leads to relatively poorer performance. We
attribute this to the increased complexity of predicting timestamps at such a
fine granularity. To illustrate, in the New York Times Annotated Corpus used
in our studies, which covers the period from 1987/01/01 to 2007/06/19, the
class number for document dating under day granularity is a staggering 7,475,
compared to just 246 under month granularity. This exponential increase in
class number necessitates a more extensive pre-training dataset to achieve
effective training outcomes.

The DD objective, focused on predicting temporal aspects, significantly
augments the model by infusing it with time-related and task-specific
knowledge. This empowers the model on downstream time-related tasks,
especially those constrained by small dataset sizes that are typically
insufficient for effective training using task-agnostic language models.
Similar ideas have been extensively employed in other studies where language
models are adapted to task-specific knowledge through task-oriented pre-
training objectives, leading to improved model performance. For instance, Xu
et al. (2019) continually train BERT through a reading comprehension
objective, achieving improved results in review reading comprehension tasks.
Likewise, Ke et al. (2019) propose the Sentilare model, which is trained to
classify sentence sentiment during pre-training, and demonstrates strong
performance on sentiment analysis tasks. Moreover, Han et al. (2021) employ
MLM during pre-training alongside the proposed utterance relevance
classification objective, demonstrating good results in response selection
tasks.

###  3.3. Time-sensitive Entity Replacement

The third and final pre-training objective of BiTimeBERT 2.0, Time-sensitive
Entity Replacement (TSER), which concentrates on dynamic entities, marks a
noteworthy improvement from our previous work (Wang et al., 2023). This
objective is inspired by the recognition that named entities, especially
”Person” entities, inherently carry temporal significance. Previous research
has demonstrated that integrating temporal information into named entity
recognition can significantly enhance accuracy (Rijhwani and Preotiuc-Pietro,
2020; Xu et al., 2022; González-Gallardo et al., 2023), affirming the close
association between named entities and specific time periods. Agarwal et al.
(2018) also illustrate that the integration of temporal information into named
entity disambiguation yields impressive results, particularly for entities of
the ”Person” type. Building upon this insight, the TSER objective in
BiTimeBERT 2.0 specifically focuses on ”Person” entities, given their
pronounced temporal sensitivity.

In the TSER objective, we first construct a set of ”Person” entities for each
month within the pre-training corpus, comprising individuals mentioned in
documents published in that specific month666Given that the New York Times
Annotated Corpus spans 246 months from January 1987 to June 2007, this yields
a collection of 246 distinct monthly ”Person” entity sets.. The identification
of these entities is performed using spaCy, concurrently with the recognition
of temporal expressions for the TAMLM task. Subsequently, for a given token
sequence processed through the TAMLM’s masking procedure, we focus on ”Person”
entities that have not been sampled. For example, in Figure 1(b), entities
like ”Smith” and ”Tupac Shakur”are considered, whereas ”Notorious B.I.G.” is
excluded as its tokens are partially masked in TAMLM. Next, in 50% of cases,
unsampled ”Person” entities are replaced with others randomly selected from
the same month’s ”Person” entity set, while no replacement is done for the
other 50%. An example of this can be observed in Figure 1(b), where ”Tupac
Shakur” is replaced by ”Antonin Scalia” from the same month’s set, while
”Smith” is left unchanged. Finally, the model uses the concatenated boundary
word representations of each ”Person” entity to make a binary prediction
(”replaced” vs. ”not replaced”), akin to the methodology in WKLM (Xiong et
al., 2019).

By incorporating the TSER objective into BiTimeBERT 2.0, we aim to enhance the
model’s ability to analyze and interpret temporal nuances associated with
dynamic entities, particularly ”Person” entities. This capability is crucial
in understanding the evolving context of these entities over time. Accurate
identification and contextualization of ”Person” entities within their
specific temporal frames also enable BiTimeBERT 2.0 to offer in-depth insights
into how related historical events and cultural evolution are framed in news
texts. For example, a mention of Sam Altman is likely to point to a period
around 2022 or later, coinciding with the notable development and influence of
ChatGPT. Thus, by capturing the temporal information inherent in ”Person”
entities, BiTimeBERT 2.0 could provide a more dynamic and time-aware analysis
of news content.

##  4\. Experimental Settings

###  4.1. Pre-training Corpus Preprocessing and BiTimeBERT 2.0 Implementation

In this study, we utilize the New York Times Annotated Corpus (NYT corpus)
(Sandhaus, 2008) for pre-training BiTimeBERT 2.0. This corpus, encompassing
over 1.8 million news articles from January 1987 to June 2007, stands as a
well-established temporal news collection and has been widely used in Temporal
Information Retrieval research (Campos et al., 2014; Kanhabua et al., 2015).
Building upon our previous findings (Wang et al., 2023), which demonstrated
the corpus’s efficacy in generating time-aware language representations, we
further refine this dataset to optimize our model’s training efficiency and
effectiveness. This refined dataset, termed the Refined NYT (RNYT) corpus,
specifically excludes sentences that lack content time. Consequently, articles
devoid of any content time are omitted, resulting in a more focused corpus of
approximately 1.4 million articles. This strategic refinement substantially
reduces the training costs, cutting GPU hours from 80 to 38, without notably
compromising the quality of the outcomes in most cases. we note that before
the corpus pre-processing, 50,000 original articles were randomly sampled and
removed from the NYT corpus. This subset is exclusively designated for
experiments on the document dating downstream task, as detailed in Section
4.2. Therefore, these articles are excluded from the Refined NYT (RNYT) corpus
to ensure the validity of our experiments.

As our method is applicable to all Transformer encoder-based language models,
we adopt BERT (Devlin et al., 2018) as the base framework. To mitigate the
high costs of training from scratch, we initiate BiTimeBERT 2.0 with the
parameters of the pre-trained $BERT_{BASE}$ (cased) model, which serves as a
robust starting point for further training. BiTimeBERT 2.0 is continually pre-
trained on the RNYT corpus for 10 epochs, incorporating the TAMLM, DD, and
TSER tasks777This process takes approximately 38 hours on a single NVIDIA A100
GPU, significantly reducing the time compared to our prior research using the
entire NYT corpus (80 hours).. In the pre-training phase, we configure the
model with a maximum sequence length of 512 and a batch size of 8. The
optimization is performed using the AdamW optimizer (Kingma and Ba, 2014),
with a learning rate set to 3e-5 and gradient accumulation fixed at 8.

###  4.2. Downstream Tasks

We rigorously evaluate BiTimeBERT 2.0 across six datasets, encompassing three
distinct time-related downstream tasks: _event occurrence time estimation_ ,
_document dating_ , and _semantic change detection_. These tasks has
significant implications in various domains of NLP and IR, such as enhancing
the accuracy of historical event timelines (Steen and Markert, 2019; Campos et
al., 2021; Yu et al., 2021), improving the precision of content retrieval
based on specific time frames (Wang et al., 2020), and tracking the evolution
of language and meanings in digital archives (Wevers and Koolen, 2020;
Tahmasebi et al., 2010). Each task is designed to test a specific aspect of
the model’s temporal understanding capabilities. For the task of event
occurrence time estimation, we utilize the EventTime dataset (Wang et al.,
2021a) and the WOTD dataset (Honovich et al., 2020). This task focuses on
accurately predicting the occurred time of specific events. In the document
dating task, aimed at determining the publication timestamps of documents, we
employ the NYT-Timestamp dataset and the TDA-Timestamp dataset. Finally, the
task of semantic change detection aims to determine whether and to what extent
the meanings of a set of target words have changed over time. For this task,
we leverage the LiverpoolFC dataset (Del Tredici et al., 2018) and the
SemEval-English dataset (Schlechtweg et al., 2020), both of which are time-
annotated corpora specifically curated for assessing semantic evolution. To
determine how well a model can detect changes in the meaning of words over
time, we measure its performance by comparing the model’s assessment of
semantic shift for each target word to the semantic index (i.e., the ground
truth).

Table 1. Statistics of six datasets across three distinct downstream tasks.

Dataset | Size | Time Span | Source | Granularity | Task
---|---|---|---|---|---
EventTime | 22,398 | 1987-2007 |  | Wikipedia &
---
”On This Day” Website
Year, Month, Day |  | Event Occurrence
---
Time Estimation
WOTD | 6,809 | 1302-2018 | Wikipedia Website | Year |  | Event Occurrence
---
Time Estimation
NYT-Timestamp | 50,000 | 1987-2007 | News Archive | Year, Month, Day | Document Dating
TDA-Timestamp | 50,000 | 1785-2009 | News Archive | Year, Month, Day | Document Dating
(c) Statistics of event occurrence time estimation datasets and document
dating datasets.

Dataset | Target Words | C1 Source | C1 Time Period | C2 Source | C2 Time Period
---|---|---|---|---|---
LiverpoolFC | 97 | Reddit | 2011–2013 | Reddit | 2017
SemEval-English | 37 | CCOHA | 1810–1860 | CCOHA | 1960–2010
(d) Statistics of semantic change detection datasets.

Table 2. Sample data from six datasets across three distinct downstream tasks.

Dataset | Text (Event Description or Document Content) | Time
---|---|---
EventTime |  | Mary Kay LeTourneau, 36, former teacher, who violated probation
---
by seeing 14 year old father of her baby, sentenced to 7 years.
2006-09
WOTD |  | Louis XVI of France gives his public assent to Civil Constitution of
---
the Clergy during the French Revolution.
1790
NYT-Timestamp |  | NASA managers said Wednesday that they would probably ”never
---
know for sure” what caused unexpected and mysterious debris to
appear around the space shuttle Atlantis in the past two days …
2006-09
TDA-Timestamp |  | Sir,-When a guest some 50 rears ago at an indigo factory in Bihar, I
---
remember two Moslems paving mv host a visit accompanied by a \”
wolf-child ’ that was on a lead and progressed on her arms from …
1927-04
(e) Sample data from event occurrence time estimation datasets and document
dating datasets.

Dataset | Target Words with their Corresponding Shift Indices
---|---
LiverpoolFC | toxic (0.857); f5 (0.556); arrangement (0.413); vomit (0) …
SemEval-English | plane (0.882); tip (0.678); attack (0.143); chairman (0) …
(f) Sample data from semantic change detection datasets.

The details of the six datasets are discussed below:

  1. (1)

EventTime (Wang et al., 2021a): This dataset contains descriptions and
occurrence times of 22,398 events from January 1987 to June 2007, sourced from
Wikipedia year pages888https://en.wikipedia.org/wiki/List_of_years and ”On
This Day” website.999https://www.onthisday.com/dates-by-year.php Since the
compared SOTA method (Wang et al., 2021a) performs search on the entire NYT
corpus, we introduce an additional dataset named EventTime-WithTop1Doc. The
aim is to simulate a comparable input setting to (Wang et al., 2021a). In the
EventTime-WithTop1Doc dataset, we extract the top-1 relevant document for each
event in the NYT corpus using the same retrieval method (BM25). Following
this, the new model input is constructed by integrating the target event
description alongside the appended timestamp and text content of the top-1
document.

  2. (2)

WOTD (Honovich et al., 2020): This dataset was scraped from Wikipedia’s ”On
this day”
webpages,101010https://en.wikipedia.org/wiki/Wikipedia:On_this_day/Today,
accessed 01/2024. comprising 6,809 short event descriptions along with their
occurrence year information. WOTD encompasses 635 classes, each corresponding
to a different occurrence year. The earliest year is 1302, and the latest is
2018, with a median year of 1855.0 and a mean of 1818.7. Additionally, the
authors provide contextual information (CI) about each event, consisting of
relevant sentences extracted from Wikipedia through a series of carefully
designed filtering steps, including key entity extraction and sentence
filtering.111111The contextual information of the WOTD example in Table 2
contains sentence like ”French forces were rebuilt, and feeling bitter about
having lost many of France’s overseas colonies to the British Empire during
the Seven Years’ War, Louis XVI was eager to give the American rebels
financial and military support.” For testing purposes, we evaluate two
versions of this dataset: one with contextual information (CI) and the other
without (No_CI). It is important to note that only year information is
provided as gold labels, limiting the models to predicting time at the year
granularity. Furthermore, the time span of the WOTD dataset (1302-2018)
significantly exceeds that of the NYT corpus (1987-2007) used for pre-
training. This allows us to assess the models’ robustness when confronted with
a dataset spanning a much longer and older time range.

  3. (3)

NYT-Timestamp: NYT-Timestamp is employed for evaluating the models in the
document dating task, utilizing 50,000 separate news articles from the NYT
corpus (Sandhaus, 2008), as discussed in Section 4.1.

  4. (4)

TDA-Timestamp121212https://www.gale.com/binaries/content/assets/gale-us-
en/primary-sources/intl-gps/ghn-factsheets-
fy18/ghn_factsheet_fy18_website_tda.pdf: We also evaluate document dating on
the Times Digital Archive (TDA), which contains over 12 million news articles
published across more than 200 years (1785-2012),131313Note that despite TDA
containing more articles and spanning a longer time period, the high number of
OCR errors in TDA led us to use it only for testing, not for pre-training. In
comparison with the NYT, errors are relatively common in TDA, especially in
the early years (Niklas, 2010). The average error rate from 1785 to 1932 was
found to be above 30%, with the highest rate reaching about 60%. Similar to
NYT-Timestamp, we randomly sample 50,000 articles from the TDA corpus, with
the time spans of the sampled articles ranging from ”1785/01/10” to
”2009-12-31”. Similar to the WOTD dataset, this extended time span aids in
comparing the robustness of different models. Such a long time span, akin to
the WOTD dataset, aids in comparing the robustness of different models.

  5. (5)

LiverpoolFC (Del Tredici et al., 2018): The LiverpoolFC corpus, composed of
Reddit posts from the Liverpool Football Club subreddit, is tailored for
short-term meaning shift analysis in online communities. The data spans two
distinct periods: 2011-2013 and 2017. In our study, we perform minimal text
pre-processing, mainly removing URLs, following the methodology of Rosin et
al. (2022). The semantic change evaluation utilizes a set of 97 words from
this corpus, each annotated with semantic shift labels by the members of the
LiverpoolFC subreddit (Del Tredici et al., 2018). These labels, ranging from 0
to 1, represent the degree of semantic change, with the average judgement
serving as a gold standard semantic shift index.

  6. (6)

SemEval-English (Schlechtweg et al., 2020): In contrast to the LiverpoolFC
dataset, the SemEval-English dataset from SemEval-2020 Task 1 focuses on long-
term semantic change detection, covering two centuries. SemEval-English
dataset spans two distinct periods: 1810-1860 and 1960-2010. This task
comprises two subtasks: binary classification of whether a word sense has been
gained or lost (subtask 1) and ranking a list of words according to their
degree of semantic change (subtask 2). We utilize the dataset for subtask 2,
consisting of 37 target words accompanied by graded labels indicating semantic
shift. These target words are carefully balanced in terms of part of speech
(POS) and frequency. Same as prior work (Rosin et al., 2022), we omit POS tags
from both the corpus and the evaluation set.

Table 1 provides a summary of the basic statistics for the six datasets,
containing two subtables for ease of reference. Table 1(c) details the four
datasets utilized for event occurrence time estimation and document dating
tasks, while Table 1(d) focuses on the two datasets employed for semantic
change detection. Notably, the WOTD and TDA-Timestamp datasets encompass
significantly longer time spans compared to the pre-training NYT corpus.
Similarly, the SemEval-English dataset spans across two centuries, extending
far beyond the NYT corpus timeframe. The LiverpoolFC corpus, meanwhile,
contains text from a timeframe not covered in the NYT corpus.

Table 2 includes sample data from each of the six datasets. As shown in Table
1(e), the datasets for EventTime, NYT-Timestamp, and TDA-Timestamp include
detailed occurrence time or timestamp information. This allows the models
tested on these datasets to be fine-tuned for estimating time with various
temporal granularities (e.g., year or month). Conversely, models fine-tuned on
WOTD are restricted to predicting time at a year granularity. The complexity
of the task increases when estimating time at finer granularities, such as
months, due to the substantial increase in the number of possible labels. For
instance, the TDA dataset, when considered at month granularity, comprises
2,688 distinct labels corresponding to the total number of months covered.
Furthermore, Table 1(f) offers examples of target words along with their
semantic shift indices, illustrating the extent of semantic change over time.
For example, the word ”plane” exhibits a notable semantic shift from the
1810-1860 period to 1960-2010. In the earlier period, it predominantly
referred to a flat surface, with the first successful powered airplane only
being invented in 1903. However, by 1960-2010, its meaning expanded to include
the now-common definition of a flying vehicle.

###  4.3. Evaluation Metrics

In our study, the first two downstream tasks involve time prediction. For
these, we adopt accuracy (ACC) and mean absolute error (MAE) as our evaluation
metrics, same as the approach in (Wang et al., 2021a, 2023). For the semantic
shift detection, we measure performance based on the correlation between the
ground truth semantic shift index and the model’s assessment for each word in
the evaluation set. Thus, we utilize both Pearson and Spearman correlation
coefficients for this task, aligning with the methodology in (Rosin et al.,
2022). These metrics are explained as follows:

  1. 1)

Accuracy (ACC): This metric calculates the percentage of events for which the
model correctly predicts the occurrence time.

  2. 2)

Mean absolute error (MAE): MAE represents the average of the absolute
differences between the model’s predicted time and the actual occurrence time
of events, tailored to the specified temporal granularity141414For example, an
MAE of 1 at year granularity implies an average temporal distance error of 1
year, whereas at month granularity, it implies an average error of 1 month..

  3. 3)

Pearson: Pearson correlation assesses the linear correlation between the
predicted semantic shifts and the ground truth. It evaluates how well the
model’s predictions mirror the actual changes in word meanings, considering
the exact predicted values.

  4. 4)

Spearman: Unlike Pearson, Spearman coefficient considers the rank order of the
words. It evaluates how well the relationship between the predicted semantic
shifts and the ground truth can be described using a monotonic function.

It is important to note that, with the exception of the WOTD dataset which
only contains year information, all models in our study could potentially be
evaluated under three different temporal granularities (day, month, and year),
for both the event occurrence time estimation and document dating tasks.
However, we observed that all the pre-trained language models exhibited
relatively poor performance when evaluated at the day granularity level. Given
this consistent underperformance across models, we have chosen not to include
the results evaluated at day granularity in our analysis.

###  4.4. Compared Models

BiTimeBERT 2.0 is evaluated against a range of models to establish its
effectiveness in various time-related tasks. The compared models include:

  1. (1)

RG: The results are estimated through random guess. We compute the average
performance across 1,000 random selections to establish a lower-bound
benchmark for our datasets.

  2. (2)

SOTA-EOTE: The state-of-the-art results for event occurrence time estimation
on the EventTime and WOTD datasets, as reported in (Wang et al., 2021a) and
(Honovich et al., 2020), respectively. Note that the two methods are not based
on language models, and both consist of complex rules or steps of searching
and filtering results to obtain the features for estimating the correct date.
Thus, due to their task-specific nature, these methods may not be easily or
quickly applicable in similar tasks.

  3. (3)

BERT Variants:

     * •

BERT-Orig: The original $BERT_{BASE}$ (cased) model (Devlin et al., 2018),
trained on the BooksCorpus (Zhu et al., 2015) and English Wikipedia.

     * •

BERT-NYT: This variant is the BERT model continually pre-trained on the full
NYT corpus for 10 epochs, employing Masked Language Modelling (MLM) and Next
Sentence Prediction (NSP) tasks.

     * •

BERT-RNYT: Similar to BERT-NYT, but pre-trained on the Refined NYT corpus.

  4. (4)

BERT+TIR: This model, as explored in our previous work (Wang et al., 2023),
represents an alternative approach to incorporating content temporal
information. BERT+TIR involves the continuous pre-training of the BERT model
on the NYT corpus for 10 epochs, utilizing both MLM and TIR (temporal
information replacement) tasks. TIR specifically entails the replacement of
temporal expressions within the input sequence. In 50% of the instances, these
temporal expressions are substituted with others that maintain the same
temporal granularity. This method aims to explore a different facet of using
content time information.

  5. (5)

TempoBERT: TempoBERT (Rosin et al., 2022) is a time-aware BERT model that
incorporates timestamp information by prepending texts with timestamps and
masking them during training, as discussed in Section 2.4. We compare our
model to TempoBERT in the semantic change detection task, where TempoBERT is
primarily focused. Rosin et al. (2022) have also demonstrated that TempoBERT
experiences a performance decline when compared to a fine-tuned BERT model in
time prediction tasks.

  6. (6)

BiTimeBERT Variants:

     * •

BiTimeBERT: Our previous version of BiTimeBERT, continually pre-trained on the
NYT corpus for 10 epochs using TAMLM and DD tasks (Wang et al., 2023).

     * •

BiTimeBERT-RNYT: This variant of BiTimeBERT undergoes pre-training on the
Refined NYT corpus, incorporating MLM and NSP tasks.

     * •

BiTimeBERT+TSEMLM: This variant incorporates the TSEMLM (time-sensitive entity
masked language modeling) task alongside TAMLM and DD, and is pre-trained on
the Refined NYT corpus. The TSEMLM task specifically targets the masking and
subsequent prediction of ”Person” entities in the text after content temporal
information is masked. By masking 30% of ”Person” entities, TSEMLM aims to
enhance the model’s sensitivity to the temporal dynamics associated with
individuals. This additional focus on ”Person” entities is implemented to
explore whether an alternative approach to incorporating time-sensitive entity
information can yield improved performance.

     * •

BiTimeBERT+TRWR: This variant extends the original BiTimeBERT by including the
(temporal relation word replacement) task, alongside the TAMLM and DD tasks,
and is pre-trained on the Refined NYT corpus. TRWR specifically focuses on
replacing temporal signals within the text: in 50% of instances, it replaces
these signals with alternatives of a different temporal nature (e.g.,
substituting ”before” with ”after” or ”during”). This task challenges the
model to accurately identify and adapt to these altered temporal contexts. The
primary goal of incorporating TRWR is to assess whether this approach of
temporal signal replacement can further improve the model’s performance.

     * •

BiTimeBERT 2.0: Our latest version of BiTimeBERT, continually pre-trained on
the Refined NYT corpus. This version incorporates the improvements from the
TAMLM, DD, and TSER tasks, with an enhanced emphasis on the interpretation of
temporal signals during the TAMLM process.

###  4.5. Fine-tuning Setting

For the tasks of event occurrence time prediction and document dating, we
fine-tune the aforementioned language models using their respective datasets.
In each language model, the final hidden state of the first token, denoted as
$h_{[CLS]}$, is utilized as the representative vector for the entire sequence.
A softmax classifier, parameterized by a matrix $X\in\mathbb{R}^{KxH}$ where
$K$ is the number of categories in the dataset, is then appended. For all
settings, a dropout rate of 0.1 is applied. The cross-entropy loss is
optimized using the Adam optimizer, with a learning rate of 2e-5 and a batch
size of 16. Considering the nature of the input data, the maximum sequence
length for fine-tuning on the EventTime and WOTD datasets is set to 128, as
these typically involve shorter event descriptions. In contrast, for
EventTime-WithTop1Doc, NYT-Timestamp, and TDA-Timestamp, where input sequences
can be significantly longer, we set the maximum length to 512.

Regarding semantic change detection, we first adapt the language models to
their respective domains using either the LiverpoolFC corpus or the SemEval-
English corpus, depending on the dataset in focus. To ensure a fair comparison
with TempoBERT (Rosin et al., 2022), we adopt the same training
hyperparameters. Specifically, for LiverpoolFC, we set the learning rate to
1e-7 and train for one epoch, whereas for SemEval-English, we use a learning
rate of 1e-6 and train for two epochs. To ensure uniformity in domain
adaptation, all models are continually trained using Masked Language Modeling
(MLM). It is important to note that, due to the sentence-level nature of these
corpora which typically lack content time information, we do not test Time-
Aware Masked Language Modeling (TAMLM) in this phase. After the language
models are trained and adapted to the specific domains, we employ TempoBERT’s
methodology to generate representations of target words for each time period.
These representations are then used to assess the degree of semantic change of
the words, by using cosine distance (cos_dist). Specifically, Pearson’s
correlation coefficient and Spearman’s rank correlation coefficient are
calculated to evaluate the model’s effectiveness. Additionally, we report
results using the ”time-diff” distance method, which is tailored to TempoBERT,
as introduced by Rosin et al. (2022).

##  5\. Experimental Results

###  5.1. Main Results

Table 3. Performance comparison of different models on the event occurrence time estimation task, evaluated across two distinct datasets under different settings. Bold font indicates the best performing model while underline is used to mark the second best one. Model | EventTime | EventTime-WithTop1Doc | WOTD | WOTD-WithCI
---|---|---|---|---
Year | Month | Year | Month | ACC | MAE | ACC | MAE
ACC | MAE | ACC | MAE | ACC | MAE | ACC | MAE
RG | 4.77 | 6.92 | 0.41 | 81.60 | 4.77 | 6.92 | 0.40 | 81.70 | 0.16 | 217.72 | 0.15 | 217.57
SOTA-EOTE | - | - | - | - | 40.93 | 3.01 | 30.89 | 36.19 | 11.40 | - | 13.10 | -
BERT-Orig | 21.65 | 3.47 | 5.09 | 43.81 | 35.98 | 3.89 | 5.98 | 37.95 | 7.20 | 52.58 | 9.69 | 41.16
Models Pre-trained using the NYT corpus
BERT-NYT | 21.25 | 3.56 | 5.18 | 43.50 | 34.46 | 4.45 | 8.21 | 34.14 | 8.08 | 53.75 | 19.97 | 36.47
BERT+TIR | 25.40 | 3.23 | 6.83 | 40.45 | 36.47 | 3.54 | 17.01 | 31.72 | 10.13 | 54.92 | 18.36 | 35.99
BiTimeBERT | 31.91 | 3.12 | 12.99 | 34.79 |  41.96 | 2.40 |  25.76 | 28.86 |  11.60 |  48.51 |  23.05 | 33.70
Models Pre-trained using the Refined NYT corpus
BERT-RNYT | 23.44 | 3.37 | 5.91 | 43.30 | 34.69 | 2.72 | 12.46 | 34.92 | 8.08 | 49.46 | 19.24 | 33.40
BiTimeBERT-RNYT | 28.48 | 3.24 | 8.84 | 38.43 | 39.22 | 2.67 | 19.55 | 30.34 | 10.28 | 52.71 | 21.73 | 35.11
BiTimeBERT+TSEMLM | 28.21 | 3.22 | 9.33 | 39.95 | 38.66 | 2.58 | 18.48 | 31.80 | 10.28 | 57.00 | 21.15 | 36.19
BiTimeBERT+TRWR | 27.68 |  3.17 | 9.11 | 38.45 | 40.31 |  2.31 | 19.87 |  29.74 | 10.72 | 53.84 | 22.17 |  31.84
BiTimeBERT 2.0 |  29.29 |  3.17 |  10.71 |  36.85 | 42.01 | 2.25 | 20.27 | 30.15 | 12.33 | 46.63 | 24.82 | 30.30
Table 4. Performance comparison of different models on the document dating task, evaluated using the NYT-Timestamp and TDA-Timestamp datasets. Model | NYT-Timestamp | TDA-Timestamp
---|---|---
Year | Month | Year | Month
ACC | MAE | ACC | MAE | ACC | MAE | ACC | MAE
RG | 4.77 | 7.06 | 0.41 | 81.79 | 0.45 | 75.39 | 0.04 | 873.88
BERT-Orig | 35.00 | 1.64 | 2.56 | 22.74 | 15.84 | 44.87 | 0.80 | 632.66
Models Pre-trained using the NYT corpus
BERT-NYT | 38.74 | 1.41 | 8.24 | 18.35 | 15.04 | 45.16 | 0.66 | 669.02
BERT+TIR | 48.06 | 1.09 | 20.30 | 13.54 | 17.72 | 43.53 | 1.26 | 589.69
BiTimeBERT | 58.72 | 0.80 | 31.10 | 9.54 |  19.00 | 40.11 |  2.38 |  580.25
Models Pre-trained using the Refined NYT corpus
BERT-RNYT | 39.00 | 1.49 | 12.76 | 19.37 | 15.92 | 45.89 | 0.78 | 624.63
BiTimeBERT-RNYT | 50.06 | 1.08 | 23.62 | 13.26 | 16.44 | 43.59 | 1.04 | 583.49
BiTimeBERT+TSEMLM | 49.64 | 1.13 | 23.22 | 13.40 | 17.64 | 42.86 | 1.08 | 613.07
BiTimeBERT+TRWR | 52.18 | 1.15 | 25.38 | 12.88 | 17.40 | 42.97 | 1.36 | 597.26
BiTimeBERT 2.0 |  54.48 |  1.05 |  28.92 |  10.36 | 19.86 |  42.26 | 2.46 | 569.09

####  5.1.1. Event Occurrence Time Estimation

Table 3 presents the results of various models in estimating event occurrence
time, using the EventTime and WOTD datasets. Notably, BiTimeBERT 2.0
consistently outperforms other models pre-trained on the Refined NYT corpus in
terms of ACC and MAE across different settings, including year/month
granularities, with/without top-1 document and with/without contextual
information. When compared to state-of-the-art event occurrence time
estimation (SOTA-EOTE) methods, BiTimeBERT 2.0 generally achieves superior
results, with the exception of the ACC metric for EventTime-WithTop1Doc under
month granularity. Interestingly, despite being trained on a significantly
smaller corpus, BiTimeBERT 2.0 exhibits comparable performance to the original
BiTimeBERT (Wang et al., 2023), which was pre-trained using the full corpus.
Moreover, BiTimeBERT 2.0 achieves state-of-the-art performance on EventTime-
WithTop1Doc under year granularity and excels in the WOTD dataset cross
different settings. This highlights the effectiveness of our refined training
approach, which also significantly reduces pre-training time. Additionally, we
observe that BERT-Orig yields poor results compared to the other two BERT
variants, and BERT-RNYT, despite being trained on a substantially smaller
corpus, outperforms BERT-NYT in most cases. This suggests that, for temporal
news collections, a more focused and refined dataset like the Refined NYT
corpus may yield more effective results when applying BERT objectives.
Furthermore, BiTimeBERT variants incorporating TSEMLM and TRWR objectives also
demonstrate commendable performance, surpassing BERT+TIR which is trained on
the complete NYT corpus using the temporal information replacement task. These
findings also suggest the potential benefits of integrating time-sensitive
entities and temporal signals in training.

When examining the performance on the original EventTime dataset, BiTimeBERT
2.0 shows impressive gains over BERT-RNYT, ranging from 24.95% to 81.21% in
ACC and from 5.93% to 14.89% in MAE across year and month granularities. The
original BiTimeBERT, trained with TAMLM and DD objectives, maintains a strong
performance, validating the effectiveness of these objectives in capturing
detailed temporal information within the news collection. BiTimeBERT 2.0,
while not quite matching BiTimeBERT, still achieves commendable second-best
results, illustrating the successful integration of the enhancements.

In evaluating the EventTime-WithTop1Doc dataset, wherein the top-1 relevant
document is considered, significant improvements are observed with BiTimeBERT
2.0 compared to BERT-RNYT models trained using the entire NYT corpus.
Specifically, at the month granularity, BiTimeBERT 2.0 exhibits a substantial
enhancement of 146.89% and 11.68% in ACC and MAE, respectively. In addition,
its superiority over BERT-Orig is even more pronounced, achieving an
impressive improvement of 246.15% and 20.55% in ACC and MAE at the month
granularity, respectively. Notably, the SOTA method involves a time-consuming
preparation of input for time estimation, relying on multivariate time series
constructed from the top-50 retrieved documents through complex steps like
sentence similarity computation. In contrast, our approach utilizes the
simpler top-1 document ranked by BM25. We believe that the performance of
BiTimeBERT 2.0 could be further enhanced by incorporating more useful
information through advanced information retrieval techniques. Additionally,
in comparison with BiTimeBERT, BiTimeBERT 2.0 excels particularly under year
granularity, while being less effective in the month granularity.

Nonetheless, when considering the WOTD dataset, BiTimeBERT 2.0 demonstrates
superior performance across different settings. Specifically, the improvements
in ACC and MAE for WOTD, compared to BiTimeBERT, are 6.29% and 3.38%,
respectively. This performance further enhances to 7.67% and 10.08% when
incorporating contextual information.151515As elucidated in Section 4.2,
contextual information comprises relevant sentences extracted from Wikipedia,
serving as external knowledge. The noteworthy achievement of BiTimeBERT 2.0 on
WOTD, a challenging dataset with a timespan significantly longer than that of
the pre-training corpus, underscores its robust generalization ability.

####  5.1.2. Document Dating

Table 4 presents the results of the document dating task. Notably, all
language models exhibit poor performance under month granularity for TDA-
Timestamp, likely due to the challenge posed by the 2,627 month labels and the
broad timestamp range spanning from 1785 to 2009 across 50,000 articles in
TDA-Timestamp. Consequently, our primary comparison focuses on NYT-Timestamp
at both year and month granularities, along with TDA-Timestamp at year
granularity. Similarly, BiTimeBERT consistently outperforms other language
models pre-trained with the Refined NYT corpus. For example, when evaluating
using NYT-Timestamp at year and month granularities, the improvement over
BERT-RNYT ranges from 39.69% to 122.64% in terms of ACC. For TDA-Timestamp at
year granularity, the ACC and MAE improvements are 24.74% and 7.91%,
respectively. Additionally, BiTimeBERT+TSEMLM and BiTimeBERT+TRWR also yield
relatively favorable results on document dating, surpassing BERT+TIR but
falling short of BiTimeBERT 2.0. However, when comparing with the original
BiTimeBERT, BiTimeBERT outperforms the second-best BiTimeBERT 2.0 on the NYT-
Timestamp dataset, while being less effective on the TDA-Timestamp dataset in
most cases. This could be attributed to BiTimeBERT being trained on the
significantly larger NYT articles corpus, encompassing articles published in
similar time periods as those in the NYT-Timestamp dataset, potentially
causing information leakage during the pre-training phase.

Table 5. Performance comparison of different models on the semantic change detection task, evaluated using the LiverpoolFC and SemEval-Eng datasets. Model | LiverpoolFC | SemEval-Eng
---|---|---
Pearson | Spearman | Pearson | Spearman
BERT-Orig | 0.414 | 0.454 | 0.483 | 0.416
TempoBERT (cos_dist) | 0.371 | 0.451 | 0.538 | 0.467
TempoBERT (time-diff) | 0.637 | 0.620 | 0.208 | 0.381
Models Pre-trained using the NYT corpus
BERT-NYT | 0.431 | 0.463 | 0.510 | 0.422
BiTimeBERT | 0.468 | 0.476 |  0.616 |  0.476
Models Pre-trained using the Refined NYT corpus
BERT-RNYT | 0.437 | 0.453 | 0.555 | 0.415
BiTimeBERT-RNYT | 0.435 | 0.457 | 0.585 | 0.403
BiTimeBERT+TSETAMLM | 0.431 | 0.425 | 0.586 | 0.468
BiTimeBERT+TRWR | 0.472 | 0.429 | 0.606 | 0.427
BiTimeBERT 2.0 |  0.493 |  0.498 | 0.635 | 0.506

####  5.1.3. Semantic Change Detection

In the semantic change detection task, as presented in Table 5, we assess the
extent to which the meanings of target words have evolved over time using the
relevant corpora. BiTimeBERT 2.0 is compared with TempoBERT (Rosin et al.,
2022), a time-aware BERT variant that introduces timestamps into texts and
masks these during training. Contrasting with TempoBERT that only integrates
timestamp information, BiTimeBERT 2.0 additionally leverages content temporal
information and time-sensitive entity data. While TempoBERT’s evaluation
involves its tailored time-diff and cosine distance (cos_dist) methods, other
models, including BiTimeBERT 2.0, primarily use cos_dist to calculate Pearson
and Spearman correlation scores. Focusing on cos_dist, BiTimeBERT 2.0
demonstrates superior performance, particularly on the long-term SemEval-
English corpus, with statistically significant correlations (p ¡ 0.005). For
instance, in comparison to TempoBERT on SemEval-English, BiTimeBERT 2.0 shows
an enhancement of 18.29% and 8.35% in Pearson and Spearman scores,
respectively. Additionally, it surpasses the original BiTimeBERT’s performance
on the same corpus by 3.08% (Pearson) and 6.30% (Spearman), and on LiverpoolFC
by 5.34% (Pearson) and 4.62% (Spearman). Interestingly, while TempoBERT, using
its tailored time-diff method, attains the best results on the short-term
LiverpoolFC corpus, it undergoes a significant performance drop on the long-
term SemEval-English corpus when applying the same method. In contrast,
BiTimeBERT maintains relatively stable and robust results across various
corpora types using cos_dist. This observation underscores the necessity of
selecting the appropriate measurement method (cos_dist or time-diff) based on
the temporal span of the corpus (long-term or short-term) for effective
semantic change detection with TempoBERT.

###  5.2. Additional Analysis

####  5.2.1. Ablation Study

To assess the impact of the three objectives in BiTimeBERT 2.0, we perform an
ablation analysis and present the results in Table 7 and Table 7. This
analysis involves comparing four distinct model configurations, each combining
different pre-training tasks, and testing them across four datasets. Notably,
we exclude the TDA-Timestamp dataset at month granularity from our tests due
to its poor performance in this setting. MLM, TAMLM, DD and TSER indicate the
corresponding models trained using only MLM, TAMLM, DD or TSER tasks,
respectively. TAMLM+DD means the model utilizing both the TAMLM and the DD
objectives, aligning with the original BiTimeBERT model’s approach. The
TAMLM+TSER model integrates the TAMLM with the TSER objectives, and the
DD+TSER model combines the DD with the TSER objectives. Lastly, the
TAMLM+DD+TSER configuration encompasses all three proposed objectives,
representing the complete BiTimeBERT 2.0 model. To ensure a fair and effective
comparison, all model variants are continually pre-trained on the Refined NYT
corpus for 3 epochs, using $BERT_{BASE}$ model as their starting point.

As shown in Table 7 and Table 7, TAMLM+DD+TSER, which uses the three pre-
training tasks of BiTimeBERT 2.0, consistently achieves superior results
across the four datasets. This underscores the collective contribution of the
TAMLM, DD, and TSER objectives to the model’s performance, facilitating the
generation of effective time-aware language representations. When examining
models that incorporate only two of the three objectives, we observe a trend
where these models generally outperform those utilizing a single objective.
Particularly, the TAMLM+DD model, embodying two of the original BiTimeBERT’s
pre-training tasks, consistently ranks second-best in most scenarios.
Interestingly, while the TAMLM+DD and DD+TSER models show relatively higher
performance than TAMLM+TSER on the NYT-Timestamp dataset, their effectiveness
is more evenly matched on the TDA-Timestamp dataset. This could be attributed
to potential information leakage during pre-training on the NYT corpus, which
may have influenced the performance on the NYT-Timestamp dataset and may not
apply to the TDA dataset. Focusing on models utilizing a single objective, we
find that the TAMLM model consistently surpasses the MLM model. Moreover, the
DD model generally outperforms other single-objective models, particularly
excelling in the document dating task. This finding confirms the benefit of
aligning pre-training and downstream tasks. Although TSER alone is less
effective compared to TAMLM and DD, its combination with these tasks, such as
in TSER+TAMLM, consistently enhances the model’s performance, indicating the
value of integrating time-sensitive entity information for a more
comprehensive temporal understanding.

Table 6. Ablation test on event occurrence time estimation task. All models
are pre-trained on the Refined NYT corpus for 3 epochs.

Model | EventTime | EventTime-WithTop1Doc | WOTD | WOTD-WithCI
---|---|---|---|---
Year | Month | Year | Month | ACC | MAE | ACC | MAE
ACC | MAE | ACC | MAE | ACC | MAE | ACC | MAE
MLM | 20.79 | 3.62 | 5.22 | 44.52 | 37.42 | 2.94 | 11.47 | 32.10 | 7.66 | 58.08 | 16.74 | 33.49
TAMLM | 22.28 | 3.42 | 6.73 | 43.42 | 38.46 | 2.74 | 12.59 | 30.50 | 8.64 | 54.22 | 18.21 | 30.38
DD | 23.54 | 3.66 | 8.01 | 41.74 | 39.73 | 2.73 | 15.39 | 32.11 | 9.69 | 62.90 | 17.47 | 43.53
TSER | 20.71 | 3.69 | 5.15 | 48.03 | 36.96 | 2.83 | 8.40 | 34.88 | 6.02 | 54.06 | 14.78 | 42.91
TAMLM+DD |  25.98 |  3.28 |  8.44 | 38.22 | 39.38 | 2.69 |  19.69 | 29.77 | 10.28 | 55.61 | 22.03 | 32.70
TAMLM+TSER | 23.54 | 3.46 | 7.40 | 42.42 |  40.27 |  2.66 | 16.21 |  29.91 | 9.84 |  53.29 | 22.03 |  33.00
DD+TSER | 25.62 | 3.43 | 8.17 | 40.07 | 40.09 | 2.57 | 17.86 | 31.35 |  11.16 | 56.91 |  22.61 | 38.43
TAMLM+DD+TSER | 27.19 | 3.19 | 10.05 |  38.48 | 41.96 | 2.71 | 19.80 | 29.99 | 11.89 | 50.57 | 23.49 | 33.51

Table 7. Ablation test on document dating task. All models are pre-trained on
the Refined NYT corpus for 3 epochs.

Model | NYT-Timestamp | TDA-Timestamp
---|---|---
Year | Month | Year
ACC | MAE | ACC | MAE | ACC | MAE
MLM | 32.24 | 1.47 | 6.15 | 18.36 | 13.11 | 44.29
TAMLM | 34.98 | 1.32 | 10.01 | 17.01 | 14.46 | 42.90
DD | 44.48 | 1.51 | 19.28 | 15.77 | 15.63 | 45.13
TSER | 32.12 | 1.46 | 8.38 | 19.16 | 15.10 | 45.29
TAMLM+DD |  48.22 | 1.17 | 21.76 |  13.64 | 16.64 | 44.83
TAMLM+TSER | 39.92 | 1.33 | 19.28 | 14.16 | 17.00 |  41.90
DD+TSER | 47.94 | 1.15 |  22.22 | 13.93 |  17.50 | 41.14
TAMLM+DD+TSER | 50.06 |  1.16 | 24.22 | 13.52 | 18.02 | 42.41

Figure 2. Comparative analysis of sampled event types VS. event types that are
accurately predicted by BiTimeBERT 2.0 in event occurrence time estimation
task.

####  5.2.2. Analysis of Event Type Distribution and Prediction Accuracy in
Time Estimation Tasks

In our subsequent analysis, we aim to investigate the types of events that
BiTimeBERT 2.0 accurately predicts, particularly focusing on the EventTime-
WithTop1Doc and WOTD datasets at year granularity. We randomly select 200
instances from each dataset and manually categorize these events into 10
distinct types: ”Political and Governmental Events”, ”Sports and Athletic
Events”, ”Arts, Entertainment, and Media”, ”Scientific and Technological
Advances”, ”Natural Disasters and Environmental Events”, ”Cultural, Social,
and Religious Events”, ”Economic and Business Events”, ”Legal and Judicial
Events”, ”Military and Conflict Events”, and ”Miscellaneous Events”. We then
analyze the events correctly predicted by the BiTimeBERT 2.0. Of the 200
instances in the EventTime-WithTop1Doc dataset, BiTimeBERT 2.0 accurately
predicts 67 instances, while in the WOTD dataset, it correctly predicts 50
instances. In Figure 2, the subfigures (a) and (b) illustrate the overall
distributions of the 200 sampled events for EventTime-WithTop1Doc and WOTD,
respectively, while (c) and (d) display the distributions of the event types
among those correctly predicted by BiTimeBERT 2.0. To further quantify this
analysis, we calculated the recall for each event type, essentially the ratio
of accurately predicted events to the total number of instances for that
specific category. For example, among the 200 sampled instances, there were 37
’Political and Governmental Events’, of which 14 were correctly predicted by
BiTimeBERT 2.0. Thus, the recall ratio for this category is 0.3783 (14 out of
37). This metric, depicted in subfigures (e) and (f) of Figure 2, provides a
nuanced understanding of the model’s predictive accuracy across different
event types.

When comparing subfigures (a) and (b) in Figure 2, we can see that the event
types within the EventTime-WithTop1Doc dataset are more evenly distributed,
with four types each exceeding 10%. However, in both datasets, ”Political and
Governmental Events” and ”Military and Conflict Events” are the two most
prevalent categories. However, a closer analysis of the correctly predicted
events, as shown in subfigures (c) and (d) of Figure 2, reveals that
”Political and Governmental Events” continue to dominate, maintaining a
significant proportion similar to their original distribution. Interestingly,
while most event types follow a similar distribution pattern as seen in
subfigures (a) and (b), a notable reduction in the representation of ”Military
and Conflict Events” is observed in the EventTime-WithTop1Doc dataset. This
shift indicates a varying degree of predictive accuracy for different event
types, with ”Political and Governmental Events” being more consistently and
accurately predicted by BiTimeBERT 2.0. The proficiency of BiTimeBERT 2.0 in
accurately predicting the occurrence times across various event types becomes
evident when examining Figure 2 (c) and (d), which depict the ratio of
correctly predicted events to the total number of sampled events for each
type. Notably, in the case of the EventTime-WithTop1Doc dataset, where the
event times fall within the span of the pre-training corpus, BiTimeBERT 2.0
demonstrates a relatively uniform prediction accuracy across most event types.
This consistency suggests the model’s robust understanding and handling of
temporal information within its training scope. In contrast, the prediction
accuracy on the WOTD dataset, which features a broader and more varied
historical scope, shows more diversity in its results. This variance in the
WOTD dataset underscores the challenges and nuances associated with predicting
times for events that extend beyond the temporal range encountered during the
model’s pre-training.

Figure 3. Accuracy Trends of BiTimeBERT and BERT Over Time

####  5.2.3. Model Accuracy Trends Over Time

In this section, we delve into the accuracy trends of BiTimeBERT 2.0 and BERT-
RNYT in estimating event occurrence times, based on the main results obtained
in Section 5.1.1. Figure 3 plots the accuracy trends of BiTimeBERT 2.0 and
BERT-RNYT on EventTime-WithTop1Doc and WOTD-WithCI under year granularity. It
is important to note the distinct time span characteristics of these datasets:
EventTime-WithTop1Doc aligns with the pre-training NYT corpus spanning
1987-2007, whereas WOTD-WithCI covers a broader range, extending from 1302 to
2018. For EventTime-WithTop1Doc, our analysis focuses on the models’ yearly
accuracy from 1987 to 2007. Conversely, for WOTD-WithCI, we assess accuracy in
50-year intervals due to its extensive time span. Our observations reveal that
BiTimeBERT 2.0 generally outperforms BERT-RNYT in both datasets over time,
underscoring the effectiveness of BiTimeBERT 2.0’s three integrated
objectives. Specifically, in the EventTime-WithTop1Doc dataset, BiTimeBERT 2.0
demonstrates notably higher accuracy in the earlier years, particularly in
1989 and 1997, with the smaller scale of improvement in later years. For WOTD-
WithCI, BiTimeBERT 2.0 is either consistently better or performs same as BERT-
NYT at every time point. The largest improvement is observed from 1700 to 1850
and then from 1900 onwards, with many of these periods falling outside the
pre-training corpus’s temporal scope. These results underscores BiTimeBERT
2.0’s robustness and adaptability, highlighting its ability to effectively
handle events from a wide range of time periods, even those well beyond its
training data. This trend not only reflects the model’s strong generalization
capabilities but also suggests that BiTimeBERT 2.0’s performance could be
further augmented by training on larger and more temporally diverse news
collections, making it particularly suited for diverse and historically
extensive datasets.

##  6\. Conclusions

In this paper, we have presented a novel and effective language representation
model called BiTimeBERT 2.0, which is specifically designed to enhance
temporal understanding of text and has demonstrates exceptional performance
across various time-related tasks. BiTimeBERT 2.0 is trained over a temporal
news collection through three innovative pre-training objectives: Time-Aware
Masked Language Modeling (TAMLM), Document Dating (DD), and Time-Sensitive
Entity Replacement (TSER). Each objective targets a unique aspect of temporal
information, encompassing timestamp, content time and time-sensitive entities.
In addition to these advancements, our refined approach to corpus processing,
which in most cases significantly reduces the training cost without
compromising the quality, further enhances the model’s applicability and
efficiency. Through comprehensive experiments, we have validated the
effectiveness of these pre-training tasks. The findings from our study
underscore that BiTimeBERT 2.0 does not only excel in generating time-aware
representations but also significantly boosts performance across a spectrum of
time-related downstream tasks. This demonstrates its potential as a powerful
tool for temporal analysis in various NLP applications. In the future, we aim
to further expand the capabilities of BiTimeBERT 2.0 by exploring the
integration of additional time-sensitive entities, such as Products and
Events, as these entities inherently carry temporal information.

## References

  * (1)
  * Agarwal et al. (2018) Prabal Agarwal, Jannik Strötgen, et al. 2018\.  Dianed: Time-aware named entity disambiguation for diachronic corpora. In _ACL_. 686–693.
  * Allan et al. (2001) James Allan, Rahul Gupta, and Vikas Khandelwal. 2001.  Temporal summaries of new topics. In _Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval_. 10–18.
  * Alonso et al. (2009) Omar Alonso, Michael Gertz, and Ricardo Baeza-Yates. 2009.  Clustering and exploring search results using timeline constructions. In _Proceedings of the 18th ACM conference on Information and knowledge management_. 97–106.
  * Alonso et al. (2011) Omar Alonso, Jannik Strötgen, Ricardo Baeza-Yates, and Michael Gertz. 2011.  Temporal Information Retrieval: Challenges and Opportunities.  _Twaw_ 11 (2011), 1–8.
  * Althammer et al. (2021) Sophia Althammer, Mark Buckley, Sebastian Hofstätter, and Allan Hanbury. 2021.  Linguistically Informed Masking for Representation Learning in the Patent Domain.  _arXiv preprint arXiv:2106.05768_ (2021).
  * Barros et al. (2019) Cristina Barros, Elena Lloret, Estela Saquete, and Borja Navarro-Colorado. 2019.  NATSUM: Narrative abstractive summarization through cross-document timeline generation.  _Information Processing & Management_ 56, 5 (2019), 1775–1793.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  Scibert: A pretrained language model for scientific text.  _arXiv preprint arXiv:1903.10676_ (2019).
  * Berberich and Bedathur (2013) Klaus Berberich and Srikanta Bedathur. 2013.  Temporal diversification of search results. In _Proceedings of SIGIR 2013 workshop on time-aware information access_.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ (2020).
  * Calzarossa and Tessera (2015) Maria Carla Calzarossa and Daniele Tessera. 2015.  Modeling and predicting temporal patterns of web content changes.  _Journal of Network and Computer Applications_ 56 (2015), 115–123.
  * Campos et al. (2014) Ricardo Campos, Gaël Dias, Alípio M Jorge, and Adam Jatowt. 2014.  Survey of temporal information retrieval and related applications.  _ACM Computing Surveys (CSUR)_ 47, 2 (2014), 1–41.
  * Campos et al. (2021) Ricardo Campos, Arian Pasquali, Adam Jatowt, Vítor Mangaravite, and Alípio Mário Jorge. 2021.  Automatic Generation of Timelines for Past-Web Events.  In _The Past Web_. Springer, 225–242.
  * Chalkidis et al. (2020) Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020.  LEGAL-BERT: The muppets straight out of law school.  _arXiv preprint arXiv:2010.02559_ (2020).
  * Chandrasekaran et al. (2020) Ranganathan Chandrasekaran, Vikalp Mehta, Tejali Valkunde, and Evangelos Moustakas. 2020.  Topics, trends, and sentiments of tweets about the COVID-19 pandemic: Temporal infoveillance study.  _Journal of medical Internet research_ 22, 10 (2020), e22624.
  * Cole et al. (2023) Jeremy R Cole, Aditi Chaudhary, Bhuwan Dhingra, and Partha Talukdar. 2023.  Salient Span Masking for Temporal Understanding.  _arXiv preprint arXiv:2303.12860_ (2023).
  * Del Tredici et al. (2018) Marco Del Tredici, Raquel Fernández, and Gemma Boleda. 2018.  Short-term meaning shift: A distributional exploration.  _arXiv preprint arXiv:1809.03169_ (2018).
  * Derczynski (2017) Leon Derczynski. 2017.  _Automatically Ordering Events and Times in Text_. Vol. 677.  https://doi.org/10.1007/978-3-319-47241-6
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Dhingra et al. (2021) Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2021.  Time-aware language models as temporal knowledge bases.  _arXiv preprint arXiv:2106.15110_ (2021).
  * Giulianelli et al. (2020) Mario Giulianelli, Marco Del Tredici, and Raquel Fernández. 2020.  Analysing lexical semantic change with contextualised word representations.  _arXiv preprint arXiv:2004.14118_ (2020).
  * González-Gallardo et al. (2023) Carlos-Emiliano González-Gallardo, Emanuela Boros, Edward Giamphy, Ahmed Hamdi, José G Moreno, and Antoine Doucet. 2023.  Injecting Temporal-Aware Knowledge in Historical Named Entity Recognition. In _European Conference on Information Retrieval_. Springer, 377–393.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.  Retrieval augmented language model pre-training. In _International conference on machine learning_. PMLR, 3929–3938.
  * Han et al. (2021) Janghoon Han, Taesuk Hong, Byoungjae Kim, Youngjoong Ko, and Jungyun Seo. 2021.  Fine-grained Post-training for Improving Retrieval-based Dialogue Systems. In _Proceedings of the NAACL 2021: Human Language Technologies_. 1549–1558.
  * Honovich et al. (2020) Or Honovich, Lucas Torroba Hennigen, Omri Abend, and Shay B Cohen. 2020.  Machine reading of historical events. In _Proceedings of ACL 2020_. 7486–7497.
  * Huang et al. (2019) Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019.  Clinicalbert: Modeling clinical notes and predicting hospital readmission.  _arXiv preprint arXiv:1904.05342_ (2019).
  * Jain et al. (2023) Raghav Jain, Daivik Sojitra, Arkadeep Acharya, Sriparna Saha, Adam Jatowt, and Sandipan Dandapat. 2023.  Do Language Models Have a Common Sense regarding Time? Revisiting Temporal Commonsense Reasoning in the Era of Large Language Models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_. 6750–6774.
  * Jatowt et al. (2013) Adam Jatowt, Ching-Man Au Yeung, and Katsumi Tanaka. 2013.  Estimating Document Focus Time. In _Proceedings of CIKM 2013_. 2273–2278.
  * Jia et al. (2018) Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018.  Tempquestions: A benchmark for temporal question answering. In _Companion Proceedings of the The Web Conference 2018_. 1057–1062.
  * Joho et al. (2013) Hideo Joho, Adam Jatowt, and Blanco Roi. 2013.  A survey of temporal web search experience. In _Proceedings of the 22nd International Conference on World Wide Web_. 1101–1108.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020.  Spanbert: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ 8 (2020), 64–77.
  * Kanhabua and Anand (2016) Nattiya Kanhabua and Avishek Anand. 2016.  Temporal information retrieval. In _Proceedings of SIGIR 2016_. 1235–1238.
  * Kanhabua et al. (2015) Nattiya Kanhabua, Roi Blanco, and Kjetil Nørvåg. 2015.  Temporal Information Retrieval.  _Foundations and Trends in Information Retrieval_ 9, 2 (2015), 91–208.  https://doi.org/10.1561/1500000043
  * Kanhabua and Nørvåg (2010) Nattiya Kanhabua and Kjetil Nørvåg. 2010.  Determining time of queries for re-ranking search results. In _International conference on theory and practice of digital libraries_. Springer, 261–272.
  * Ke et al. (2019) Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, and Minlie Huang. 2019.  SentiLARE: Sentiment-aware language representation learning with linguistic knowledge.  _arXiv preprint arXiv:1911.02493_ (2019).
  * Khan et al. (2020) Asif Khan, Jian Ping Li, Naeem Ahmad, Shuchi Sethi, Amin Ul Haq, Sarosh H Patel, and Sabit Rahim. 2020.  Predicting emerging trends on social media by modeling it as temporal bipartite networks.  _IEEE Access_ 8 (2020), 39635–39646.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Korkeamäki and Kumpulainen (2019) Laura Korkeamäki and Sanna Kumpulainen. 2019.  Interacting with Digital Documents: A Real Life Study of Historians’ Task Processes, Actions and Goals. In _Proceedings of the 2019 Conference on Human Information Interaction and Retrieval_ (Glasgow, Scotland UK) _(CHIIR ’19)_. ACM, New York, NY, USA, 35–43.  https://doi.org/10.1145/3295750.3298931
  * Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019.  Cross-lingual language model pretraining.  _arXiv preprint arXiv:1901.07291_ (2019).
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_ (2019).
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ 36, 4 (2020), 1234–1240.
  * Li and Croft (2003) Xiaoyan Li and W Bruce Croft. 2003.  Time-based language models. In _Proceedings of CIKM 2003_. ACM, 469–475.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * MacKay et al. (2017) Kelly MacKay, Danielle Barbe, Christine M Van Winkle, and Elizabeth Halpenny. 2017.  Social media activity in a festival context: Temporal and content analysis.  _International Journal of Contemporary Hospitality Management_ 29, 2 (2017), 669–689.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ (2013).
  * Nagel (2016) Sebastian Nagel. 2016.  Cc-news.  _URL: http://web. archive. org/save/http://commoncrawl. org/2016/10/newsdatasetavailable_ (2016).
  * Niklas (2010) Kai Niklas. 2010.  Unsupervised post-correction of OCR errors.  _Master’s thesis. Leibniz Universität Hannover_ (2010).
  * Pasca (2008) Marius Pasca. 2008.  Towards Temporal Web Search. In _SAC_. 1117–1121.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_. 1532–1543.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  arXiv:1802.05365 [cs.CL]
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.  (2018).
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ 21, 1 (2020), 5485–5551.
  * Rijhwani and Preotiuc-Pietro (2020) Shruti Rijhwani and Daniel Preotiuc-Pietro. 2020.  Temporally-informed analysis of named entity recognition. In _ACL_. 1–13.
  * Rosin et al. (2022) Guy D Rosin, Ido Guy, and Kira Radinsky. 2022.  Time masking for temporal language models. In _Proceedings of WSDM 2022_. 833–841.
  * Rosin and Radinsky (2022) Guy D Rosin and Kira Radinsky. 2022.  Temporal Attention for Language Models.  _arXiv preprint arXiv:2202.02093_ (2022).
  * Sandhaus (2008) Evan Sandhaus. 2008.  The new york times annotated corpus. LDC2008T19.  _Linguistic Data Consortium, Philadelphia_ 6, 12 (2008), e26752.
  * Schlechtweg et al. (2020) Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020.  SemEval-2020 task 1: Unsupervised lexical semantic change detection.  _arXiv preprint arXiv:2007.11464_ (2020).
  * Stack (2006) Michael Stack. 2006.  Full text search of web archive collections.  _Proc. of IWAW_ (2006).
  * Steen and Markert (2019) Julius Steen and Katja Markert. 2019.  Abstractive Timeline Summarization. In _the 2nd Workshop on New Frontiers in Summarization_. 21–31.
  * Strötgen and Gertz (2012) Jannik Strötgen and Michael Gertz. 2012.  Event-centric search and exploration in document collections. In _JCDL_. 223–232.
  * Styskin et al. (2011) Andrey Styskin, Fedor Romanenko, Fedor Vorobyev, and Pavel Serdyukov. 2011.  Recency ranking by diversification of result set. In _Proceedings of CIKM 2011_. 1949–1952.
  * Svore et al. (2012) Krysta M Svore, Jaime Teevan, Susan T Dumais, and Anagha Kulkarni. 2012.  Creating temporally dynamic web search snippets. In _In SIGIR 2012_. 1045–1046.
  * Tahmasebi et al. (2010) Nina Tahmasebi, Gideon Zenz, Tereza Iofciu, and Thomas Risse. 2010.  Terminology Evolution Module for Web Archives in the LiWA Context. In _10th International Web Archiving Workshop_. 55–62.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Advances in neural information processing systems_. 5998–6008.
  * Wang et al. (2020) Jiexin Wang, Adam Jatowt, Michael Färber, and Masatoshi Yoshikawa. 2020.  Answering Event-Related Questions over Long-Term News Article Archives. In _European Conference on Information Retrieval_. Springer, 774–789.
  * Wang et al. (2021b) Jiexin Wang, Adam Jatowt, Michael Färber, and Masatoshi Yoshikawa. 2021b.  Improving question answering for event-focused questions in temporal collections of news articles.  _Information Retrieval Journal_ (2021), 1–26.
  * Wang et al. (2021a) Jiexin Wang, Adam Jatowt, and Masatoshi Yoshikawa. 2021a.  Event Occurrence Date Estimation based on Multivariate Time Series Analysis over Temporal Document Collections. In _Proceedings of SIGIR 2021_. 398–407.
  * Wang et al. (2023) Jiexin Wang, Adam Jatowt, Masatoshi Yoshikawa, and Yi Cai. 2023.  Bitimebert: Extending pre-trained language representations with bi-temporal information. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 812–821.
  * Wevers and Koolen (2020) Melvin Wevers and Marijn Koolen. 2020.  Digital begriffsgeschichte: Tracing semantic change using word embeddings.  _Historical Methods: A Journal of Quantitative and Interdisciplinary History_ 53, 4 (2020), 226–243.
  * Xiong et al. (2019) Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2019.  Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model.  _arXiv preprint arXiv:1912.09637_ (2019).
  * Xu et al. (2022) Chengjin Xu, Fenglong Su, Bo Xiong, and Jens Lehmann. 2022.  Time-aware entity alignment using temporal relational attention. In _Proceedings of the ACM Web Conference 2022_. 788–797.
  * Xu et al. (2019) Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. 2019.  BERT post-training for review reading comprehension and aspect-based sentiment analysis.  _arXiv preprint arXiv:1904.02232_ (2019).
  * Yamamoto et al. (2008) Yusuke Yamamoto, Taro Tezuka, Adam Jatowt, and Katsumi Tanaka. 2008.  Supporting Judgment of Fact Trustworthiness Considering Temporal and Sentimental Aspects. In _Web Information Systems Engineering - WISE 2008, 9th International Conference, Auckland, New Zealand, September 1-3, 2008. Proceedings_ _(Lecture Notes in Computer Science, Vol. 5175)_ , James Bailey, David Maier, Klaus-Dieter Schewe, Bernhard Thalheim, and Xiaoyang Sean Wang (Eds.). Springer, 206–220.  https://doi.org/10.1007/978-3-540-85481-4_17
  * Yu et al. (2021) Yi Yu, Adam Jatowt, Antoine Doucet, Kazunari Sugiyama, and Masatoshi Yoshikawa. 2021.  Multi-timeline summarization (mtls): Improving timeline summarization by generating multiple summaries. In _In ACL 2021_. 377–387.
  * Zhou et al. (2020) Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2020.  Temporal reasoning on implicit events from distant supervision.  _arXiv preprint arXiv:2010.12753_ (2020).
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In _Proceedings of the IEEE international conference on computer vision_. 19–27.