# Neighborhood Contrastive Learning for Scientific Document Representations
with Citation Embeddings

Malte Ostendorff
DFKI GmbH
<EMAIL_ADDRESS>
&Nils Rethmeier
DFKI GmbH
<EMAIL_ADDRESS>
&Isabelle Augenstein
University of Copenhagen
<EMAIL_ADDRESS>
Bela Gipp
University of Wuppertal
<EMAIL_ADDRESS>
&Georg Rehm
DFKI GmbH
<EMAIL_ADDRESS>
###### Abstract

Learning scientific document representations can be substantially improved
through contrastive learning objectives, where the challenge lies in creating
positive and negative training samples that encode the desired similarity
semantics. Prior work relies on discrete citation relations to generate
contrast samples. However, discrete citations enforce a hard cut-off to
similarity. This is counter-intuitive to similarity-based learning, and
ignores that scientific papers can be very similar despite lacking a direct
citation – a core problem of finding related research. Instead, we use
controlled nearest neighbor sampling over citation graph embeddings for
contrastive learning. This control allows us to learn continuous similarity,
to sample hard-to-learn negatives _and positives_ , and also to avoid
collisions between negative and positive samples by controlling the sampling
margin between them. The resulting method SciNCL outperforms the state-of-the-
art on the SciDocs benchmark. Furthermore, we demonstrate that it can train
(or tune) models sample-efficiently, and that it can be combined with recent
training-efficient methods. Perhaps surprisingly, even training a general-
domain language model this way outperforms baselines pretrained in-domain.

##  1 Introduction

Figure 1:  Starting from a query paper

in a citation graph embedding space. Hard positives

are citation graph embeddings that are sampled from a similar (close) context
of

, but are not so close that their gradients collapse easily. Hard (to
classify) negatives

(red band) are close to positives (green band) up to a _sampling induced
margin_. Easy negatives

are very dissimilar (distant) from the query paper

.

Large pretrained language models (LLMs) achieve state-of-the-art results
through fine-tuning on many NLP tasks Rogers et al. (2020). However, the
sentence or document embeddings derived from LLMs are of lesser quality
compared to simple baselines like GloVe Reimers and Gurevych (2019), as their
embedding space suffers from being anisotropic, i.e. poorly defined in some
areas Li et al. (2020).

One approach that has recently gained attention is the combination of LLMs
with contrastive fine-tuning to improve the semantic textual similarity
between document representations Wu et al. (2020); Gao et al. (2021). These
contrastive methods learn to distinguish between pairs of similar and
dissimilar texts. As part of metric learning, they traditionally focused on
defining new loss functions, while Musgrave et al. (2020) showed that newer
metric losses lead to insignificant performance gains when compared fairly.
Instead, recent works on self and supervised contrastive learning has started
to focus on developing techniques that generate better positive and negative
data augmentations for efficient contrastive learning Tian et al. (2020);
Rethmeier and Augenstein (2021a); Rethmeier and Augenstein (2021b); Shorten et
al. (2021).

In this paper, we focus on learning scientific document representations
(SDRs). The core distinguishing feature of this domain is the presence of
citation information. SDR methods like SciBERT Beltagy et al. (2019) pretrain
a Transformer on domain-specific text but neglect citations. The current
state-of-the-art by Cohan et al. (2020) uses discrete citation information to
generate positive and negative samples for contrastive fine-tuning of SciBERT
via a triplet loss Schroff et al. (2015). Cited papers are used to generate
positive samples, while non-cited papers are negative samples.

This discrete cut-off to similarity is counter-intuitive to (continuous)
similarity-based learning. It encourages overfitting to human similarity
annotations, i.e. citations, which may reflect politeness and policy rather
than semantic similarity Pasternack (1969). Such sample generation may also
cause positive and negative samples to collide, e.g. Saunshi et al. (2019)
have shown that generating more negatives harms contrastive optimization
performance if collision are not avoided. Instead, the generation of _non-
colliding_ contrastive samples should be based on a continuous similarity
function that allow us to find semantically similar papers, despite a lack of
direct citation.

#### Contributions:

  * •

We propose neighborhood contrastive learning for scientific document
representations with citation graph embeddings (SciNCL).

  * •

We sample similar and dissimilar papers from neighboring citation graph
embeddings, such that both are hard to learn to avoid long training times and
gradient collapse.

  * •

As in recent contrastive learning works, we address sample generation
semantics based on contrastive learning theory insights rather than designing
new loss functions.

  * •

We compare against the state-of-the-art approach SPECTER Cohan et al. (2020)
and other strong methods on the SciDocs benchmark and find that SciNCL
outperforms SPECTER on average and on 9 of 12 metrics.

  * •

Finally, we demonstrate that with SciNCL, using only 1% of the training data,
starting with a general-domain language model, or training only the bias terms
of the model is sufficient to outperform the baselines.

  * •

Our code and models are publicly available.111
https://github.com/malteos/scincl

##  2 Related Work

#### Contrastive Learning

pulls representations of similar data points (positives) closer together,
while representations of dissimilar documents (negatives) are pushed apart. A
common contrastive objective is the triplet loss Schroff et al. (2015) that
Cohan et al. (2020) used for scientific document representation learning, as
we describe below. However, as Musgrave et al. (2020); Rethmeier and
Augenstein (2021a) point out, contrastive objectives work best when specific
requirements are respected. (Req. 1) Views of the same data should introduce
new information, i.e. the mutual information between views should be minimized
Tian et al. (2020). We use citation graph embeddings to generate contrast
label information that supplements text-based similarity. (Req. 2) For
training time and sample efficiency, negative samples should be hard to
classify, but should also not collide with positives Saunshi et al. (2019).
(Req. 3) Recent works like Musgrave et al. (2020); Khosla et al. (2020) use
multiple positives. However, positives need to be consistently close to each
other Wang and Isola (2020), since positives and negatives may otherwise
collide, e.g., Cohan et al. (2020) consider only ‘citations by the query’ as
similarity signal and not ‘citations to the query’. Such unidirectional
similarity does not guarantee that a negative paper (not cited by the query)
may cite the query paper and thus could cause collisions, the more we sample.
Our method treats both citing and being cited as positives (Req. 2), while it
also generates hard negatives and hard positives (Req. 2+3). Hard negatives
are close to but do not overlap positives (red band in Fig. 1). Hard positives
are close, but not trivially close to the query document (green band in Fig.
1).

#### Scientific Document Representations

based on Transformers Vaswani et al. (2017) and pretrained on domain-specific
text dominate today’s scientific document processing. There are SciBERT
Beltagy et al. (2019), BioBERT Lee et al. (2019) and SciGPT2 Luu et al.
(2021), to name a few. Recent works modify these domain LLMs to support cite-
worthiness detection Wright and Augenstein (2021), similarity Ostendorff et
al. (2020) or fact checking Wadden et al. (2020).

Aside from text, citations are a valuable signal for the similarity of
research papers. Paper (node) representations can be learned using the
citation graph Wu et al. (2019); Perozzi et al. (2014); Grover and Leskovec
(2016). Especially for recommendations of papers or citations, hybrid
combinations of text and citation features are often employed Han et al.
(2018); Jeong et al. (2020); Molloy et al. (2020); Färber and Sampath (2020);
Holm et al. (2022).

Closest to SciNCL are Citeomatic Bhagavatula et al. (2018) and SPECTER Cohan
et al. (2020). While Citeomatic relies on bag-of-words for its textual
features, SPECTER is based on SciBERT. Both leverage citations to learn a
triplet-based document embedding model, whereby positive samples are papers
cited in the query. Easy negatives are random papers not cited by the query.
Hard negatives are citations of citations – papers referenced in positive
citations of the query, but are not cited directly by it. Citeomatic also uses
a second type of hard negatives, which are the nearest neighbors of a query
that are not cited by it.

Unlike our approach, Citeomatic does not use the neighborhood of citation
embeddings, but instead relies on the actual document embeddings from the
previous epoch. Despite being related to SciNCL, the sampling approaches
employed in Citeomatic and SPECTER do not account for the pitfalls of using
discrete citations as signal for paper similarity. Our work addresses this
issue.

#### Triple Mining

remains a challenge in NLP due to the discrete nature of language which makes
data augmentation less trivial as compared to computer vision Gao et al.
(2021). Examples for augmentation strategies are: translation Fang et al.
(2020), or word deletion and reordering Wu et al. (2020). Positives and
negatives can be sampled based on the sentence position within a document
Giorgi et al. (2021). Gao et al. (2021) utilize supervised entailment datasets
for the triple generation.

Language- and text-independent approaches are also applied: Kim et al. (2021)
use intermediate BERT hidden state for positive sampling and Wu et al. (2021)
add noise to representations to obtain negative samples. Xiong et al. (2020)
present an approach similar to SciNCL where they sample hard negatives from
the k nearest neighbors in the embedding space derived from the previous model
checkpoint. While Xiong et al. rely only on textual data, SciNCL integrates
also citation information which are especially valuable in the scientific
context as Cohan et al. (2020) have shown.

##  3 Methodology

Our goal is to learn task-independent representations for scientific
documents. To do so we sample three document representation vectors and learn
their similarity. For a given query paper vector ${\bm{d}}^{Q}$, we sample a
positive (similar) paper vector ${\bm{d}}^{+}$ and a negative (dissimilar)
paper vector ${\bm{d}}^{-}$. This produces a ‘query, positive, negative’
triple $({\bm{d}}^{Q},{\bm{d}}^{+},{\bm{d}}^{-})$ – represented by (  ,  ,  )
in Fig. 1. To learn paper similarity, we need to define three components: (§
3.1) how to calculate document vectors ${\bm{d}}$ for the loss over triplets
$\mathcal{L}$; (§ 3.2) how citations provide similarity between papers; and (§
3.3) how negative and positive papers $({\bm{d}}^{-},{\bm{d}}^{+})$ are
sampled as (dis-)similar documents from the neighborhood of a query paper
${\bm{d}}^{Q}$.

###  3.1 Contrastive Learning Objective

Given the textual content of a document $d$ (paper), the goal is to derive a
dense vector representation ${\bm{d}}$ that best encodes the document
information and can be used in downstream tasks. A Transformer language model
$f$ (SciBERT; Beltagy et al. (2019)) encodes documents $d$ into vector
representations $f(d)={\bm{d}}$. The input to the language model is the title
and abstract separated by the [SEP] token.222Cohan et al. (2019) evaluated
other inputs (venue or author) but found the title and abstract to perform
best. The final layer hidden state of the [CLS] token is then used as a
document representation $f(d)={\bm{d}}$.

Training with a masked language modeling objectives alone has been shown to
produce sub-optimal document representations Li et al. (2020); Gao et al.
(2021). Thus, similar to the SDR state-of-the-art method SPECTER Cohan et al.
(2020), we continue training the SciBERT model Beltagy et al. (2019) using a
self-supervised triplet margin loss Schroff et al. (2015):

$\mathcal{L}=\max\Big{\\{}\lVert{\bm{d}}^{Q}{-}{\bm{d}}^{+}\lVert_{2}{-}\lVert{\bm{d}}^{Q}{-}{\bm{d}}^{-}\lVert_{2}+\xi,0\Big{\\}}$

Here, $\xi$ is a slack term ($\xi=1$ as in SPECTER) and
$\lVert\Delta{\bm{d}}\lVert_{2}$ is the $L^{2}$ norm, used as a distance
function. However, the SPECTER sampling method has significant drawbacks. We
will describe these issues and our contrastive learning theory guided
improvements in detail below in § 3.2.

###  3.2 Citation Neighborhood Sampling

Compared to the textual content of a paper, citations provide an outside view
on a paper and its relation to the scientific literature Elkiss et al. (2008),
which is why citations are traditionally used as a similarity measure in
library science Kessler (1963); Small (1973). However, using citations as a
discrete similarity signal, as done in Cohan et al. (2020), has its pitfalls.
Their method defines papers cited by the query as positives, while paper
citing the query could be treated as negatives. This means that _positive and
negative learning information collides_ between citation directions, which
Saunshi et al. (2019) have shown to deteriorate performance. Furthermore, a
cited paper can have a low similarity with the citing paper given the many
motivations a citation can have Teufel et al. (2006). Likewise, a similar
paper might not be cited.

To overcome these limitations, we learn citation embeddings first and then use
the citation neighborhood around a given query paper $d^{Q}$ to construct
similar (positive) and dissimilar (negative) samples for contrast by using
neighborhood information from either kNN (I) or a distance metric Sim (II-IV)
as detailed in § 3.3. This builds on the intuition that nodes connected by
edges should be close to each other in the embedding space Perozzi et al.
(2014); Grover and Leskovec (2016). Using citation embeddings allows us to:
(1) sample paper similarity on a continuous scale, which makes it possible to:
(2) define hard to learn positives, as well as (3) hard or easy to learn
negatives. Points (2-3) are important in making contrastive learning efficient
as will describe below in § 3.3.

###  3.3 Positives and Negatives Sampling

#### Positive samples:

$d^{+}$ should be semantically similar to the query paper $d^{Q}$, i.e.
sampled close to the query embedding ${\bm{d}}^{Q}$. Additionally, as Wang and
Isola (2020) find, positives should be sampled from comparable locations
(distances from the query) in embedding space and be dissimilar enough from
the query embedding, to avoid gradient collapse (zero gradients). Therefore,
we sample $c^{+}$ positive (similar) papers from a close neighborhood around
query embedding ${\bm{d}}^{Q}$ $(k^{+}-c^{+},k^{+}]$, i.e. the green band in
Fig. 1. When sampling from kNN neighbors, we use a small $k^{+}$ to find
positives and later analyze the impact of $k^{+}$ in Fig. 2.

#### Negative samples:

can be divided into easy  and hard  negative samples (light and dark red in
Fig. 1). Sampling more hard negatives is known to improve contrastive learning
Bucher et al. (2016); Wu et al. (2017). However, we make sure to sample hard
negatives (red band in Fig. 1) such that they are close to potential positives
but do not collide with positives (green band), by using a tunable ‘sampling
induced margin’. We do so, since Saunshi et al. (2019) showed that sampling a
larger number of hard negatives only improves performance _if the negatives do
not collide with positive samples_ , since collisions make the learning signal
noisy. That is, in the margin between hard negatives and positives we expect
positives and negatives to collide, thus we avoid sampling from this region.
To generate a diverse self-supervised citation similarity signal for
contrastive SDR learning, we also sample easy negatives that are farther from
the query than hard negatives. For negatives, the $k^{-}$ should be large when
sampling via kNN to ensure samples are dissimilar from the query paper.

###  3.4 Sampling Strategies

As described in § 3.2 and § 3.3, our approach improves upon the method by
Cohan et al. (2020). Therefore, we reuse their sampling parameters (5 triples
per query paper) and then further optimize our methods’ hyperparameters. For
example, to train the triplet loss, we generate the same amount of
$({\bm{d}}^{Q},{\bm{d}}^{+},{\bm{d}}^{-})$ triples per query paper as SPECTER
Cohan et al. (2020). To be precise, this means we generate $c^{+}{=}5$
positives (as explained in § 3.3). We also generate 5 negatives, three easy
negatives $c^{-}_{\text{easy}}{=}3$ and two hard negatives
$c^{-}_{\text{hard}}{=}2$, as described in § 3.3.

Below, we describe four strategies (I-IV) for sampling triplets. These either
sample neighboring papers from citation embeddings (I-II), by random sampling
(III), or using both strategies (IV). For each strategy, let $c^{\prime}$ be
the number of samples for either positives $c^{+}$, easy negatives
$c^{-}_{\text{easy}}$, or hard negatives $c^{-}_{\text{hard}}$.

#### Citation Graph Embeddings:

We train a graph embedding model $f_{c}$ on citations extracted from the
Semantic Scholar Open Research Corpus (S2ORC; Lo et al., 2020) to get citation
embeddings $C$. We utilize PyTorch BigGraph Lerer et al. (2019), which allows
for training on large graphs with modest hardware requirements. The resulting
graph embeddings perform well using the default training settings from Lerer
et al. (2019), but given more computational resources, careful tuning may
produce even better-performing embeddings. Nonetheless, we conducted a narrow
parameter search based on link prediction – see Appendix A.4.

#### (I) K-nearest neighbors (kNN):

Assuming a given citation embedding model $f_{c}$ and a search index (e.g.,
FAISS § 4.3), we run $KNN(f_{c}(d^{Q}),C)$ and take $c^{\prime}$ samples from
a range of the $(k-c^{\prime},k]$ nearest neighbors around the query paper
$d^{Q}$ with its neighbors $N{=}\\{n_{1},n_{2},n_{3},\dots\\}$, whereby
neighbor $n_{i}$ is the $i$-th nearest neighbor citation. For instance, for
$c^{\prime}{=}3$ and $k{=}10$ the corresponding samples would be the three
neighbors descending from the tenth neighbor: $n_{8}$, $n_{9}$, and $n_{10}$.
To reduce computing effort, we sample the neighbors $N$ only once via
$[0;\max(k^{+},k^{-}_{\text{hard}})]$, and then generate triples by range-
selection in $N$; i.e. positives $=(k^{+}-c^{+};k^{+}]$, and hard negatives
$=(k^{-}_{\text{hard}}-c^{-}_{\text{hard}};k^{-}_{\text{hard}}]$.

#### (II) Similarity threshold (Sim):

Take $c^{\prime}$ papers that are within the similarity threshold $t$ of a
query paper $d^{Q}$ such that $s(f_{c}(d^{Q}),f_{c}(d_{i}))<t$, where $s$ is
the cosine similarity function. For example, given the similarity scores
$S{=}\\{0.9,0.8,0.7,0.1\\}$ (ascending order, the higher the similarity is the
closer the candidate embedding to the query embedding is) with
$c^{\prime}{=}2$ and $t{=}0.5$, the two candidates with the largest similarity
scores and larger than the threshold would be $0.8$ and $0.7$. The
corresponding papers would be selected as samples. While the positive
threshold $t^{+}$ should close to 1, the negative threshold $t^{-}$ should be
small to ensure samples are dissimilar from $d^{Q}$.

#### (III) Random sampling

Sample any $c^{\prime}$ papers without replacement from the corpus.

#### (IV) Filtered random

Like (III) but excluding the papers that are retrieved by kNN or Sim, i.e.,
all neighbors within the largest $k$ or $n_{i}$ with $i{<=}k$ are excluded.

The kNN and Sim sampling strategies introduce hyperparameters ($k$ or $t$)
that allow for the _controlled sampling of positives or negatives_ with
different difficulty (from easy to hard depending on the hyperparameter).
Specifically, in Fig. 1 these hyperparameters define the tunable _sample
induced margin_ between positives and negatives, as well as the width and
position of the positive sample band (green) and negative sample band (red)
around the query sample. Besides the strategies above, we experiment with
k-means clustering and sorted random sampling, neither of which performs well
(see negatives results in Appendix A.5).

##  4 Experiments

We next introduce our experimental setting including the data used for
training and evaluation, as well as implementation details.

###  4.1 Evaluation Dataset

We evaluate the document representations on the SciDocs benchmark Cohan et al.
(2020). A key difference to other benchmarks is that embeddings are the input
to the individual tasks without explicit fine-tuning. The SciDocs benchmark
consists of the following four tasks:

Document classification (CLS) with labels from Medical Subject Headings (MeSH)
Lipscomb (2000) and Microsoft Academic Graph (MAG) Sinha et al. (2015)
evaluated with the F1 metric. Co-views and co-reads (USR) prediction based on
the L2 distance between embeddings. Co-views are papers viewed in a single
browsing session. Co-read refers to a user accessing the PDF of a paper. Both
user activities are evaluated using Mean Average Precision (map) and
Normalized Discounted Cumulative Gain (ndcg). Direct and co-citation (CITE)
prediction based on the L2 distance between the embeddings. map and ndcg are
the evaluation metrics. Recommendations (REC) generation based on embeddings
and paper metadata to rank a set of “similar papers” for a given paper. An
offline evaluation with historical clickthrough data determines the
performance using Precision@1 (P@1) and ndcg.

###  4.2 Training Data

With our experiments, we mainly compare against SPECTER Cohan et al. (2020) on
the SciDocs benchmark. However, we found 40.5% of SciDocs’s papers leaking
into SPECTER’s training data (§ A.2). To be transparent about this leakage, we
conduct experiments with two training datasets: (1) a replication of SPECTER’s
training data and (2) a random subset of S2ORC.

With (1), we replicate the training data from SPECTER as closely as possible
including its leakage. Unfortunately SPECTER’s data does not provide a mapping
to S2ORC which is the basis for our citation embedding model. Citation
embeddings based on SPECTER’s data are not feasible since citation data is not
provided. We successfully map 96.2% of SPECTER’s query papers and 83.3% of the
corpus from which positives and negatives are sampled. To account for the
missing papers, we randomly sample papers from S2ORC such that the absolute
number of papers is identical with SPECTER. When we sample from S2ORC we
exclude the papers from SciDocs to not add any additional leakage.

With (2), we select a random subset from S2ORC that does not contain any of
the mapped SciDocs papers. This avoids SPECTER’s leakage but also makes the
scores reported in Cohan et al. (2020) less comparable. We successfully map
98.6% of the SciDocs papers to S2ORC. Thus, only the remaining 1.4% of the
SciDocs papers could leak into this training set.

The details of the dataset creation are described in Appendix A.1 and A.3.
Both training sets yield 684K triples (same count as SPECTER). Also, the ratio
of training triples per query remains the same (§ 3.4). Our citation embedding
model is trained on the S2ORC citation graph. In (1), we include all SPECTER
papers even if they are part of SciDocs, the remaining SciDocs papers are
excluded (52.5 nodes and 463M edges). In (2), all mapped SciDocs papers are
excluded (52.4M nodes and 447M edges) such that we avoid leakage also for the
citation embedding model.

###  4.3 Training and Implementation

We replicate the training setup from SPECTER as closely as possible. We
implement SciNCL using Huggingface Transformers Wolf et al. (2020), initialize
the model with SciBERT’s weights Beltagy et al. (2019), and train via the
triplet loss (Equation 3.1). The optimizer is Adam with weight decay Kingma
and Ba (2015); Loshchilov and Hutter (2019) and learning rate
$\lambda{=}2^{-5}$. To explore the effect of computing efficient fine-tuning
we also train a BitFit model Zaken et al. (2021) with $\lambda{=}1^{-4}$ (§
7.2). We train SciNCL on two NVIDIA GeForce RTX 6000 (24G) for 2 epochs
(approx. 24 hours of training time) with batch size 8 and gradient
accumulation for an effective batch size of 32 (same as SPECTER). The graph
embedding training is performed on a Intel Xeon Gold 6230 CPU with 60 cores
and takes approx. 6 hours. The kNN and Sim strategies are implemented with
FAISS Johnson et al. (2021) using a flat index (exhaustive search) and take
less than 30min.

###  4.4 Baseline Methods

Task $\rightarrow$ | Classification | User activity prediction | Citation prediction |  Recomm. |  Avg.
---|---|---|---|---|---
Subtask $\rightarrow$ | MAG | MeSH | Co-View | Co-Read | Cite | Co-Cite
1-13 Model $\downarrow$ / Metric $\rightarrow$ | F1 | F1 | map | ndcg | map | ndcg | map | ndcg | map | ndcg | ndcg | P@1
Oracle SciDocs $\dagger$ | 87.1 | 94.8 | 87.2 | 93.5 | 88.7 | 94.6 | 92.3 | 96.8 | 91.4 | 96.4 | 53.8 | 19.4 | 83.0
Doc2Vec* (2014) | 66.2 | 69.2 | 67.8 | 82.9 | 64.9 | 81.6 | 65.3 | 82.2 | 67.1 | 83.4 | 51.7 | 16.9 | 66.6
fastText-sum* (2017) | 78.1 | 84.1 | 76.5 | 87.9 | 75.3 | 87.4 | 74.6 | 88.1 | 77.8 | 89.6 | 52.5 | 18.0 | 74.1
ELMo* (2018) | 77.0 | 75.7 | 70.3 | 84.3 | 67.4 | 82.6 | 65.8 | 82.6 | 68.5 | 83.8 | 52.5 | 18.2 | 69.0
Citeomatic* (2018) | 67.1 | 75.7 | 81.1 | 90.2 | 80.5 | 90.2 | 86.3 | 94.1 | 84.4 | 92.8 | 52.5 | 17.3 | 76.0
SGC* (2019) | 76.8 | 82.7 | 77.2 | 88.0 | 75.7 | 87.5 | 91.6 | 96.2 | 84.1 | 92.5 | 52.7 | 18.2 | 76.9
BERT (2019) | 79.9 | 74.3 | 59.9 | 78.3 | 57.1 | 76.4 | 54.3 | 75.1 | 57.9 | 77.3 | 52.1 | 18.1 | 63.4
SciBERT* (2019) | 79.7 | 80.7 | 50.7 | 73.1 | 47.7 | 71.1 | 48.3 | 71.7 | 49.7 | 72.6 | 52.1 | 17.9 | 59.6
BioBERT (2019) | 77.2 | 73.0 | 53.3 | 74.0 | 50.6 | 72.2 | 45.5 | 69.0 | 49.4 | 71.8 | 52.0 | 17.9 | 58.8
CiteBERT (2021) | 78.8 | 74.8 | 53.2 | 73.6 | 49.9 | 71.3 | 45.0 | 67.9 | 50.3 | 72.1 | 51.6 | 17.0 | 58.8
SPECTER* (2020) | 82.0 | 86.4 | 83.6 | 91.5 | 84.5 | 92.4 | 88.3 | 94.9 | 88.1 | 94.8 | 53.9 | 20.0 | 80.0
Replicated SPECTER training data (w/ leakage):
SciNCL (ours) | 81.4 | 88.7 | 85.3 | 92.3 | 87.5 | 93.9 | 93.6 | 97.3 | 91.6 | 96.4 | 53.9 | 19.3 | 81.8
$\pm\ \sigma$ w/ ten seeds | .449 | .422 | .128 | .08 | .162 | .118 | .104 | .054 | .099 | .066 | .203 | .356 | .064
Random S2ORC training data (w/o leakage):
SPECTER | 81.3 | 88.4 | 83.1 | 91.3 | 84.0 | 92.1 | 86.2 | 93.9 | 87.8 | 94.7 | 52.2 | 17.5 | 79.4
SciNCL (ours) | 81.3 | 89.4 | 84.3 | 91.8 | 85.6 | 92.8 | 91.4 | 96.3 | 90.1 | 95.7 | 54.3 | 19.9 | 81.1

Table 1: Results on the SciDocs benchmark. With replicated SPECTER training
data, our SciNCL approach surpasses the previous best avg. score by 1.8 points
and also outperforms the baselines in 9 of 12 task metrics. Our scores are
reported as mean and standard deviation $\sigma$ over ten random seeds. With
training data randomly sampled from S2ORC, SciNCL outperforms SPECTER in terms
of avg. score with 1.7 points. The baseline scores with * are taken from Cohan
et al. (2020). Oracle SciDocs $\dagger$ represents the upper bound of the
performance with triples generated based on SciDocs test and validation data.

We compare to 10 prior approaches: Doc2Vec Le and Mikolov (2014), weighted sum
of in-domain fastText word embeddings Bojanowski et al. (2017), averaged
contextualized token-level representations from ELMO Peters et al. (2018),
BERT Devlin et al. (2019) a state-of-the-art LLM pretrained on general-domain
text, BioBERT-Base-Cased-v1.2 Lee et al. (2019) a BERT variation for
biomedical text, SciBERT Beltagy et al. (2019) a BERT variation for scientific
text, CiteBERT Wright and Augenstein (2021) a SciBERT variation fine-tuned on
cite-worthiness detection, the graph-convolution approach SGC Wu et al.
(2019), Citeomatic Bhagavatula et al. (2018), and SPECTER Cohan et al. (2020).
If not otherwise mentioned, all BERT variations are used in their base-uncased
versions.

Also, we include the results of Oracle SciDocs which is identical to
SPECTER/SciNCL except that its triples are generated from the labels of
validation and test set of SciDocs. For example, papers with the same MAG
labels are positives and papers with different labels are negatives.
Similarly, the ground truth for the other tasks is used, i.e., clicked
recommendations are considered as positives etc.333We under-sample triples
from the classification tasks to ensure a balanced triple distribution over
the tasks. In total, this procedure creates 106K training triples for Oracle
SciDocs. Accordingly, Oracle SciDocs represents an estimate for the upper
bound for the results that can be achieved with the current setting (triplet
margin loss and SciBERT encoder).

##  5 Overall Results

§ 4.4 shows our main results, comparing SciNCL with the best validation
performance against prior approaches. With replicated SPECTER training data
(w/ leakage), SciNCL achieves an average performance of 81.8 across all
metrics, which is a 1.8 point absolute improvement over SPECTER - the next-
best baseline. When trained on the random subset of S2ORC without any SciDocs
papers (w/o leakage), the improvement of SciNCL over SPECTER is relative
consistent with 1.7 points but generally lower (79.4 avg. score). In the
following, we refer to the results obtained through training on the replicated
SPECTER data (w/ leakage) if not otherwise mentioned.

We find the best validation performance based on SPECTER’s data when positives
and hard negative are sampled with kNN, whereby positives are $k^{+}{=}25$,
and hard negatives are $k^{-}_{\text{hard}}{=}4000$ (§ 6). Easy negatives are
generated through filtered random sampling. As random sampling accounts for a
large fraction of the triples (in the form of easy negatives), we report the
mean scores and standard deviation based on ten random seeds
($\text{seed}\in[0,9]$).

For MAG classification, SPECTER achieves the best result with 82.0 F1 followed
by SciNCL with 81.4 F1 (-0.6 points). For MeSH classification, SciNCL yields
the highest score with 88.7 F1 (+2.3 compared to SPECTER). Both classification
tasks have in common that the chosen training settings lead to over-fitting.
Changing the training by using only 1% training data, SciNCL yields 82.2
F1@MAG (Tab. 2). In all user activity and citation tasks, SciNCL yields higher
scores than all baselines. It is notable that SciNCL also outperforms SGC on
direct citation prediction, where SGC outperforms SPECTER in terms of ndcg. On
the recommender task, SPECTER yields the best P@1 with 20.0, whereas SciNCL is
slightly worst with 19.3 P@1 (in terms of ndcg SciNCL and SPECTER are on par).

When training SPECTER and SciNCL without leakage, SciNCL outperforms SPECTER
even in 11 of 12 metrics and is on par in the other metric. This suggests that
SciNCL’s hyperparameters have a low corpus dependency since they were only
optimized on the corpus with leakage.

Regarding the LLM baselines, we observe that the general-domain BERT, with a
score of 63.4, outperforms the domain-specific BERT variants, namely SciBERT
(59.6), BioBERT (58.8), and CiteBERT (58.8). Still, all LLMs without
contrastive objectives yield substantially worse results (even compared to
Doc2Vec or fastText). This emphasizes the anisotropy problem of embeddings
directly extracted from current LLMs.

In summary, we show that SciNCL’s triple selection leads on average to a
performance improvement on SciDocs, with most gains being observed for user
activity and citation tasks. The gain from 80.0 to 81.8 is particularly
notable given that even Oracle SciDocs yields with 83.0 an only marginally
higher avg. score despite using SciDocs directly for the triple selection.
Appendix A.7 shows examples of triples generated with SciNCL.

##  6 Impact of Sample Difficulty

The benefit of SciNCL is that the hyperparameters of the sampling strategies
can be tuned (§ 3.3) to learn without sample collisions. In this section, we
present the results of this tuning procedure. We optimize the sampling
strategies for positives and hard or easy negatives with partial grid search
on a random sample of 10% of the replicated SPECTER training data (sampling
based on queries). Our experiments show that optimizations on this subset
correlate with the entire dataset. The scores in Fig. 2 and 3 are reported as
the mean over three random seeds including standard deviations.

###  6.1 Positive Samples

Figure 2: Results on the validation set w.r.t. positive sampling with kNN when
using 10% training data.

Fig. 2 shows the average scores on the SciDocs validation set depending on the
selection of positives with the kNN strategy. We only change $k^{+}$, while
negative sampling remains fixed to its best setting (§ 6.2). The Sim strategy
is omitted for positive sampling since it yields a poor performance throughout
all tasks (Appendix A.5)

The performance is relatively stable for $k^{+}{<}100$ with peak at
$k^{+}{=}25$, for $k^{+}{>}100$ the performance declines as $k^{+}$ increases.
Wang and Isola (2020) state that positive samples should be semantically
similar to each other, but not too similar to the query. For example, at
$k^{+}{=}5$, positives may be a bit “too easy” to learn, such that they
produce less informative gradients than the optimal setting $k^{+}{=}25$.
Similarly, making $k^{+}$ too large leads to the _sampling induced margin_
being too small, such that _positives collide with negative samples_ , which
creates contrastive label noise that degrades performance (Saunshi et al.,
2019).

Another observation is the standard deviation $\sigma$: One would expect
$\sigma$ to be independent of $k^{+}$ since random seeds affect only the
negatives. However, positives and negatives interact with each other through
the triplet margin loss. Therefore, $\sigma$ is also affected by $k^{+}$.

###  6.2 Hard Negative Samples

Figure 3: Results on the validation set w.r.t. hard negative sampling with kNN
using 10% training data.

Fig. 3 presents the validation results with kNN strategy and
$k^{-}_{\text{hard}}$ and the best setting for positives ($k^{+}{=}25$). The
performance increases with increasing $k^{-}_{\text{hard}}$, until the
performance plateaus for $2000{<}k^{-}_{\text{hard}}{<}4000$ with a peak at
$k^{-}_{\text{hard}}{=}4000$. This plateau can also be observed in the test
performance, where $k^{-}_{\text{hard}}{=}2000$ and
$k^{-}_{\text{hard}}{=}3000$ yield a marginally lower score of 81.6 and 81.7
(Tab. 2). For $k^{-}_{\text{hard}}{>}4000$, the performance starts to decline
again. This suggests that for large $k^{-}_{\text{hard}}$ the samples are not
“hard enough”. The need for hard negatives confirms the findings of Cohan et
al. (2020).

Intuitively, the kNN strategy should suffer from a centrality or hubness
problem. How many neighbors are semantically similar strongly depends on the
query paper itself. A popular and frequently cited paper has many more similar
neighbors than a niche paper. To test this assumption, we also evaluate the
Sim strategy that should account for the hubness problem. However, Sim
underperforms with a score of 81.7 (Tab. 2) independent from different
similarity thresholds (Appendix A.5).

###  6.3 Easy Negative Samples

Filtered random sampling of easy negatives yields the best validation
performance compared pure random sampling (Tab. 2). However, the performance
difference is marginal. When rounded to one decimal, their average test scores
are identical. The marginal difference is caused by the large corpus size and
the resulting small probability of randomly sampling one paper from the kNN
results. But without filtering, the effect of random seeds increases, since we
find a higher standard deviation compared to the one with filtering.

As a potential way to decrease randomness, we experiment with other approaches
like k-means clustering but find that they decrease the performance (Appendix
A.5).

##  7 Ablation Analysis

In addition to sample difficulty, we evaluate the performance impact of data
quantity, trainable parameters, and language model initialization.

| CLS | USR | CITE | REC | Avg. | $\Delta$
---|---|---|---|---|---|---
SciNCL | 85.0 | 88.8 | 94.7 | 36.6 | 81.8 | –
SPECTER | 84.2 | 88.4 | 91.5 | 36.9 | 80.0 | -1.8
$k^{-}_{\text{hard}}{=}2000$ | 84.9 | 88.8 | 94.7 | 36.1 | 81.6 | -0.2
$k^{-}_{\text{hard}}{=}3000$ | 84.5 | 88.7 | 94.6 | 36.9 | 81.7 | -0.1
hard neg. w/ Sim | 84.4 | 88.9 | 94.7 | 36.9 | 81.7 | -0.1
easy neg. w/ random | 85.1 | 88.8 | 94.7 | 36.6 | 81.8 | 0.0
Init. w/ BERT-Base | 83.4 | 88.4 | 93.8 | 37.5 | 81.2 | -0.6
Init. w/ BERT-Large | 84.6 | 88.7 | 94.1 | 36.4 | 81.4 | -0.4
Init. w/ BioBERT | 83.7 | 88.6 | 93.8 | 37.7 | 81.4 | -0.4
1% training data | 85.2 | 88.3 | 92.7 | 36.1 | 80.8 | -1.0
10% training data | 85.1 | 88.7 | 93.5 | 36.2 | 81.1 | -0.6
BitFit training | 85.8 | 88.6 | 93.7 | 35.3 | 81.2 | -0.5
Table 2: Ablations. Numbers are averages over tasks of the SciDocs test set,
average score over all metrics, and rounded absolute difference to SciNCL.

###  7.1 Initial Language Models

Tab. 2 shows the effect of initializing the model weights not with SciBERT but
with general-domain LLMs (BERT-Base and BERT-Large) or with BioBERT. The
initialization with other LLMs decreases the performance. However, the decline
is marginal (BERT-Base -0.6, BERT-Large -0.4, BioBERT -0.4) and all LLMs
outperform the SPECTER baseline. For the recommendation task, in which SPECTER
is superior over SciNCL, BioBERT outperform SPECTER. This indicates that the
improved triple mining of SciNCL has a greater domain adaption effect than
pretraining on domain-specific literature. Given that pretraining of LLMs
requires a magnitude more resources than the fine-tuning with SciNCL, our
approach can be a solution for resource-limited use cases.

###  7.2 Data and Computing Efficiency

The last three rows of Tab. 2 show the results regarding data and computing
efficiency. When keeping the citation graph unchanged but training with only
10% of the original triples, SciNCL still yields a score of 81.1 (-0.6). Even
with only 1% (6840 triples), SciNCL achieves a score of 80.8 that is 1.0
points less than with 100% but still 0.8 points more than the SPECTER
baseline. With this data efficiency, one could manually create triples or use
existing supervised datasets as Gao et al. (2021).

Lastly, we evaluate BitFit training Zaken et al. (2021), which only trains the
bias terms of the model while freezing all other parameters. This corresponds
to training only 0.1% of the original parameters. With BitFit, SciNCL yields a
considerable score of 81.2 (-0.5 points). As a result, SciNCL could be trained
on the same hardware with even larger (general-domain) language models (§
7.1).

##  8 Conclusion

We present a novel approach for contrastive learning of scientific document
embeddings that addresses the challenge of selecting informative positive and
negative samples. By leveraging citation graph embeddings for sample
generation, SciNCL achieves a score of 81.8 on the SciDocs benchmark, a 1.8
point improvement over the previous best method SPECTER. This is purely
achieved by introducing tunable sample difficulty and avoiding collisions
between positive and negative samples, while existing LLM and data setups can
be reused. This improvement over SPECTER can be also observed when excluding
the SciDocs papers from the training corpus. Furthermore, SciNCL’s improvement
from 80.0 to 81.8 is particularly notable given that even triples, which are
generated with SciDocs’s test and validation data, yield with 83.0 only a
marginally higher score.

Our work highlights the importance of sample generation in a contrastive
learning setting. We show that 1% of training triples are already sufficient
to outperform SPECTER, whereas the remaining 99% provide only 1.0 additional
points (80.8 to 81.8). This sample efficiency is achieved by adding reasonable
effort for sample generation (graph embedding training and nearest neighbor
search). We also demonstrate that in-domain language model pretraining (like
SciBERT) is beneficial, while general-domain LLMs can achieve a comparable
performance and even outperform SPECTER. This indicates that controlling
sample difficulty and avoiding collisions is more effective than in-domain
pretraining, especially in scenarios where training a LLM from scratch is
infeasible.

## Acknowledgements

The work presented in this article has received funding from the German
Federal Ministry of Education and Research (BMBF) through the project QURATOR
(no. 03WKDA1A).

## References

  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A Pretrained Language Model for Scientific Text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3613–3618, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Bhagavatula et al. (2018) Chandra Bhagavatula, Sergey Feldman, Russell Power, and Waleed Ammar. 2018.  Content-based citation recommendation.  _NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \- Proceedings of the Conference_ , 1:238–251.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching Word Vectors with Subword Information.  _Transactions of the Association for Computational Linguistics_ , 5:135–146.
  * Bucher et al. (2016) Maxime Bucher, Stéphane Herbin, and Frédéric Jurie. 2016.  Hard negative mining for metric learning based zero-shot classification.  In _Computer Vision – ECCV 2016 Workshops_ , pages 524–531, Cham. Springer International Publishing.
  * Cohan et al. (2019) Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019.  Structural Scaffolds for Citation Intent Classification in Scientific Publications.  In _Proceedings of the 2019 Conference of the North_ , volume 1, pages 3586–3596, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020.  SPECTER: Document-level Representation Learning using Citation-informed Transformers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2270–2282, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Elkiss et al. (2008) Aaron Elkiss, Siwei Shen, Anthony Fader, Güneş Erkan, David States, and Dragomir Radev. 2008.  Blind men and elephants: What do citation summaries tell us about a research article? _Journal of the American Society for Information Science and Technology_ , 59(1):51–62.
  * Fang et al. (2020) Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. 2020.  CERT: Contrastive Self-supervised Learning for Language Understanding.  pages 1–16.
  * Färber and Sampath (2020) Michael Färber and Ashwath Sampath. 2020.  HybridCite: A Hybrid Model for Context-Aware Citation Recommendation.  In _Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020_ , pages 117–126, New York, NY, USA. ACM.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple Contrastive Learning of Sentence Embeddings.  _arXiv:2104.08821_.
  * Giorgi et al. (2021) John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021.  DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 879–895, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Grover and Leskovec (2016) Aditya Grover and Jure Leskovec. 2016.  node2vec: Scalable Feature Learning for Networks.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ’16_ , pages 855–864, New York, New York, USA. ACM Press.
  * Han et al. (2018) Jialong Han, Yan Song, Wayne Xin Zhao, Shuming Shi, and Haisong Zhang. 2018.  hyperdoc2vec: Distributed Representations of Hypertext Documents.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , volume 1, pages 2384–2394, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Holm et al. (2022) Andreas Nugaard Holm, Barbara Plank, Dustin Wright, and Isabelle Augenstein. 2022\.  Longitudinal Citation Prediction using Temporal Graph Neural Networks.
  * Jeong et al. (2020) Chanwoo Jeong, Sion Jang, Hyuna Shin, Eunjeong Lucy Park, and Sungchul Choi. 2020\.  A context-aware citation recommendation model with bert and graph convolutional networks.  _Scientometrics_ , pages 1–16.
  * Johnson et al. (2021) Jeff Johnson, Matthijs Douze, and Herve Jegou. 2021.  Billion-Scale Similarity Search with GPUs.  _IEEE Transactions on Big Data_ , 7(3):535–547.
  * Kessler (1963) M. M. Kessler. 1963.  Bibliographic coupling between scientific papers.  _American Documentation_ , 14(1):10–25.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised contrastive learning.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 18661–18673. Curran Associates, Inc.
  * Kim et al. (2021) Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021.  Self-Guided Contrastive Learning for BERT Sentence Representations.  pages 2528–2540.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Lei Ba. 2015.  Adam: A method for stochastic optimization.  _3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings_ , pages 1–15.
  * Le and Mikolov (2014) Quoc V. Le and Tomas Mikolov. 2014.  Distributed Representations of Sentences and Documents.  _Proceedings of the 31st International Conference on Machine Learning_ , 32:1188–1196.
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , pages 1–8.
  * Lerer et al. (2019) Adam Lerer, Ledell Wu, Jiajun Shen, Timothee Lacroix, Luca Wehrstedt, Abhijit Bose, and Alex Peysakhovich. 2019.  PyTorch-BigGraph: A Large-scale Graph Embedding System.  In _Proceedings of The Conference on Systems and Machine Learning_.
  * Li et al. (2020) Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020.  On the Sentence Embeddings from Pre-trained Language Models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9119–9130, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Lipscomb (2000) Carolyn E. Lipscomb. 2000.  Medical subject headings (mesh).  _Bulletin of the Medical Library Association_ , 88 3:265–6.
  * Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020.  S2ORC: The Semantic Scholar Open Research Corpus.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4969–4983, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  _7th International Conference on Learning Representations, ICLR 2019_.
  * Luu et al. (2021) Kelvin Luu, Xinyi Wu, Rik Koncel-Kedziorski, Kyle Lo, Isabel Cachola, and Noah A. Smith. 2021.  Explaining Relationships Between Scientific Documents.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2130–2144, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Molloy et al. (2020) Paul Molloy, Joeran Beel, and Akiko Aizawa. 2020.  Virtual Citation Proximity (VCP): Learning a Hypothetical In-Text Citation-Proximity Metric For Uncited Documents.  In _Proceedings of the 8th International Workshop on Mining Scientific Publications_ , pages 1–8.
  * Musgrave et al. (2020) Kevin Musgrave, Serge J. Belongie, and Ser-Nam Lim. 2020.  A metric learning reality check.  In _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV_ , pages 681–699.
  * Ostendorff et al. (2020) Malte Ostendorff, Terry Ruas, Till Blume, Bela Gipp, and Georg Rehm. 2020.  Aspect-based Document Similarity for Research Papers.  In _Proceedings of the 28th International Conference on Computational Linguistics (COLING 2020)_.
  * Pasternack (1969) Simon Pasternack. 1969.  The scientific enterprise: Public knowledge. an essay concerning the social dimension of science.  _Science_ , 164(3880):669–670.
  * Perozzi et al. (2014) Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014.  DeepWalk: online learning of social representations.  In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD ’14_ , pages 701–710, New York, New York, USA. ACM Press.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep Contextualized Word Representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3980–3990, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Rethmeier and Augenstein (2021a) Nils Rethmeier and Isabelle Augenstein. 2021a.  A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectives.  _arXiv:2102.12982_.
  * Rethmeier and Augenstein (2021b) Nils Rethmeier and Isabelle Augenstein. 2021b.  Data-Efficient Pretraining via Contrastive Self-Supervision.
  * Rogers et al. (2020) Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020.  A primer in bertology: What we know about how BERT works.  _Trans. Assoc. Comput. Linguistics_ , 8:842–866.
  * Saunshi et al. (2019) Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. 2019.  A theoretical analysis of contrastive unsupervised representation learning.  In _ICML_ , volume 97 of _PMLR_. PMLR.
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015.  Facenet: A unified embedding for face recognition and clustering.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015_ , pages 815–823.
  * Shorten et al. (2021) Connor Shorten, Taghi M. Khoshgoftaar, and Borko Furht. 2021.  Text data augmentation for deep learning.  _J. Big Data_ , 8(1):101.
  * Sinha et al. (2015) Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Paul Hsu, and Kuansan Wang. 2015.  An overview of microsoft academic service (mas) and applications.  _Proceedings of the 24th International Conference on World Wide Web_.
  * Small (1973) Henry Small. 1973.  Co-citation in the scientific literature: A new measure of the relationship between two documents.  _Journal of the American Society for Information Science_ , 24(4):265–269.
  * Teufel et al. (2006) Simone Teufel, Advaith Siddharthan, and Dan Tidhar. 2006.  Automatic classification of citation function.  In _Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing - EMNLP ’06_ , page 103, Morristown, NJ, USA. Association for Computational Linguistics.
  * Tian et al. (2020) Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. 2020.  What makes for good views for contrastive learning? In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention Is All You Need.  _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , pages 6000–6010.
  * Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020.  Fact or Fiction: Verifying Scientific Claims.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7534–7550, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Wang and Isola (2020) Tongzhou Wang and Phillip Isola. 2020.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 9929–9939. PMLR.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2020.  Transformers: State-of-the-Art Natural Language Processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45.
  * Wright and Augenstein (2021) Dustin Wright and Isabelle Augenstein. 2021.  CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understanding.  In _Findings of ACL-IJCNLP_. Association for Computational Linguistics.
  * Wu et al. (2017) Chao-yuan Wu, R. Manmatha, Alexander J Smola, and Philipp Krahenbuhl. 2017.  Sampling Matters in Deep Embedding Learning.  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 2859–2867. IEEE.
  * Wu et al. (2019) Felix Wu, Tianyi Zhang, Amauri Holanda de Souza, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. 2019.  Simplifying Graph Convolutional Networks.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 6861-6871, pages 815–826. PMLR.
  * Wu et al. (2021) Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2021.  Smoothed Contrastive Learning for Unsupervised Sentence Embedding.
  * Wu et al. (2020) Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020.  CLEAR: Contrastive Learning for Sentence Representation.  _arXiv:2012.15466_.
  * Xiong et al. (2020) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020.  Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval.  pages 1–16.
  * Zaken et al. (2021) Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021.  BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models.  _arXiv:2106.10199_.

##  Appendix A Appendix

###  A.1 Mapping to S2ORC

Table 3: Mapping to S2ORC citation graph S2ORC mapping | Success rate
---|---
SciDocs papers |
\- with S2ORC IDs | 220,815 / 223,932 (98.6%)
\- in S2ORC graph | 197,811 / 223,932 (88.3%)
SPECTER papers |
\- with S2ORC IDs | 311,094 / 311,860 (99.7%)
\- in S2ORC graph | 260,014 / 311,860 (83.3%)

Neither the SPECTER training data nor the SciDocs test data comes with a
mapping to the S2ORC dataset, which we use for the training of the citation
embedding model. However, to replicate SPECTER’s training data and to avoid
leakage of SciDocs test data such a mapping is needed. Therefore, we try to
map the papers to S2ORC based on PDF hashes and exact title matches. The
remaining paper metadata is collected through the Semantic Scholar API. Tab. 3
summarizes the outcome of mapping procedure. Failed mappings can be attributed
to papers being unavailable through the Semantic Scholar API (e.g., retracted
papers) or papers not being part of S2ORC citation graph.

###  A.2 SPECTER-SciDocs Leakage

When replicating SPECTER Cohan et al. (2020), we found a substantial overlap
between the papers444https://github.com/allenai/specter/issues/2 used during
the model training and the papers from their SciDocs
benchmark555https://github.com/allenai/scidocs. In both datasets, papers are
associated with Semantic Scholar IDs. Thus, no custom ID mapping as in § A.1
is required to identify papers that leak from training to test data. From the
311,860 unique papers used in SPECTER’s training data, we find 79,201 papers
(25.4%) in the test set of SciDocs and 79,609 papers (25.5%) in its validation
set. When combining test and validation set, there is a total overlap of
126,176 papers (40.5%). However, this overlap affect only the ‘unsupervised’
paper metadata (title, abstract, citations, etc.) and not the gold labels used
in SciDocs (e.g., MAG labels or clicked recommendations).

###  A.3 Dataset Creation

As describe in § 4.2, we conduct our experiments on two datasets. Both
datasets rely on the citation graph of S2ORC Lo et al. (2020). More
specifically, S2ORC with the version identifier 20200705v1 is used. The full
citation graph consists of 52.6M nodes (papers) and 467M edges (citations).
Tab. 4 presents statistics on the datasets and their overlap with SPECTER and
SciDocs. The steps to reproduce both datasets are:

#### Replicated SPECTER (w/ leakage)

In order to replicate SPECTER’s training data and do not increase the leakage,
we exclude all SciDocs papers which are not used by SPECTER from the S2ORC
citation graph. This means that apart from the 110,538 SPECTER papers not a
single other SciDocs paper is included. The resulting citation graph has 52.5M
nodes and 463M edges and is used for training the citation graph embeddings.

For the SciNCL triple selection, we also replicate SPECTER’s query papers and
its corpus from which positive and negatives are sampled. Our mapping and the
underlying citation graph allows us to use 227,869 of 248,007 SPECTER’s papers
for training. Regarding query papers, we use 131,644 of 136,820 SPECTER’s
query papers. To align the number training triples with the one from SPECTER,
additional papers are randomly sampled from the filtered citation graph.

#### Random S2ORC subset (w/o leakage)

To avoid leakage, we exclude all successfully mapped SciDocs papers from the
S2ORC citation graph. After filtering the graph has 52.3 nodes and 447M edges.
The citation graph embedding model is trained on this graph.

Next, we reproduce triplet selection from SPECTER. Any random 136,820 query
papers are selected from the filtered graph. For each query, we generate five
positives (cited by the query), two hard negatives (citation of citation), and
three random nodes from the filtered S2ORC citation graphs. This sampling
produces 684,100 training triples with 680,967 unique papers IDs (more
compared to the replicated SPECTER dataset). Based on these triples the
SPECTER model for this dataset is trained with the same model settings and
hyperparameters as SciNCL (second last row in § 4.4).

Lastly, the SciNCL triples are generated based on the citation graph
embeddings of the same 680,967 unique papers IDs, i.e, the FAISS index
contains only these papers and not the remaining S2ORC papers. Also, the same
136,820 query papers are used.

Table 4: Statistics for our two datasets and their overlap with SPECTER and SciDocs respectively. |  | Replicated
---
SPECTER
(w/ leakage)
| Random
---
S2ORC subset
(w/o leakage)
Training triples | 684,100 | 684,100
Unique paper IDs | 248,007 | 680,967
\- in SPECTER | 227,869 | 9,182
\- in SciDocs | 110,538 | 0
| \- in SciDocs
---
and in SPECTER
110,538 | 0
Query paper IDs | 136,820 | 136,820
\- in SciDocs | 69,306 | 0
\- in SPECTER queries | 131,644 | 463
Citation graph |  |
\- Nodes | 52,526,134 | 52,373,977
\- Edges | 463,697,639 | 447,697,727

###  A.4 Graph Embedding Evaluation

To evaluate the underlying citation graph embeddings, we experiment with a few
of BigGraph’s hyperparameters. We trained embeddings with different dimensions
$d{=}\\{128,512,768\\}$ and different distance measures (cosine similarity and
dot product) on 99% of the data and test the remaining 1% on the link
prediction task. An evaluation of the graph embeddings with SciDocs is not
possible since we could not map the papers used in SciDocs to the S2ORC
corpus. All variations are trained for 20 epochs, margin $m{=}0.15$, and
learning rate $\lambda{=}0.1$ (based on the recommended settings by Lerer et
al. (2019)).

Table 5: Link prediction performance of BigGraph embeddings trained on S2ORC citation graph with different dimensions and distance measures. Dim. | Dist. | MRR | Hits@1 | Hits@10 | AUC
---|---|---|---|---|---
128 | Cos. | 54.09 | 43.39 | 75.21 | 85.75
128 | Dot | 89.75 | 85.84 | 96.13 | 97.70
512 | Dot | 94.60 | 92.47 | 97.64 | 98.64
768 | Dot | 95.12 | 93.22 | 97.77 | 98.74

Tab. 5 shows the link prediction performance measured in MRR, Hits@1, Hits@10,
and AUC. Dot product is substantially better than cosine similarity as
distance measure. Also, there is a positive correlation between the
performance and the size of the embeddings. The larger the embedding size the
better link prediction performance. Graph embeddings with $d{=}768$ were the
largest possible size given our compute resources (available disk space was
the limiting factor).

###  A.5 Negative Results

We tried additional sampling strategies and model modification of which none
led to an performance improvement.

#### Undirected Citations

Our graph embedding model considers citations as directed edges by default. We
also train a SciNCL model with undirected citations by first converting a
single edge $(a,b)$ into the two edges $(a,b)$ and $(b,a)$. This approach
yields a slightly worse performance (81.7 avg. score; -0.1 points) and,
therefore, was discarded for the final experiments.

####  kNN with interval large than $c$

Our best results are achieved with kNN where the size of the neighbor interval
$(k-c^{\prime};k]$ is equal to the number of samples $c^{\prime}$ that the
strategy should generate. In addition to this, we also experimented with large
intervals, e.g., $(1000;2000]$, from which $c^{\prime}$ papers are randomly
sampled. This approach yields comparable results but suffers from a larger
effect of randomness and is therefore more difficult to optimize.

#### K-Means Cluster for Easy Negatives

Easy negatives are supposed to be far away from the query. Random sampling
from a large corpus ensures this as our results show. As an alternative
approach, we tried k-means clustering whereby we selected easy negatives from
the centroid that has a given distance to the query’s centroid. However, this
decreased the performance.

#### Hard Negatives With Similarity Threshold

Figure 4: Results on the validation set w.r.t. hard negative sampling with Sim
using 10% training data.

As shown in Tab. 2, hard negative sampling with k nearest neighbors
outperforms absolute similarity sampling. Fig. 4 show the validation results
for different similarity thresholds. A similar pattern as in Fig. 3 can be
seen. When the negatives are closer to the query paper (larger similarity
threshold $t$), the validation score decreases.

#### Positives with Similarity Threshold

Positive sampling with Sim performs poorly since even for small $t^{+}<0.5$
many query papers do not have any neighbors within this similarity threshold
(more than 40%). Solving this issue would require changing the set of query
papers which we omit for comparability to SPECTER.

#### Sorted Random

Simple random sampling does not ensure if a sample is far or close to the
query. To integrate a distance measure in the random sampling, we first sample
$n$ candidates, then order the candidates according to their distance to the
query, and lastly select the $c^{\prime}$ candidates that are the closest or
furthest to the query as samples.

#### Mask Language Modeling

Giorgi et al. (2021) show that combining a contrastive loss with a mask
language modeling loss can improve text representation learning. However, in
our experiments a combined function decreases the performance on SciDocs,
probably due to the effects found by Li et al. (2020).

#### Graph Embedding Prediction Loss

We combine the triplet loss (Equation 3.1) with a MSE loss of the predicted
embedding and the graph embeddings. This approach yields a comparable
performance but adds additional computational complexity and was therefore
discarded for the final experiments.

###  A.6 Task-specific Results

(a) Classification

(b) User activities

(c) Citation

(d) Recommendation

Figure 5: Task-level validation performance w.r.t. $k^{+}$ with kNN strategy
using 10% training data.

(a) Classification

(b) User activities

(c) Citation

(d) Recommendation

Figure 6: Task-level validation performance w.r.t. $k^{-}_{\text{hard}}$ with
kNN strategy using 10% training data.

Fig. 5 and 6 present the validation performance like in § 6 but on a task-
level and not as an average over all tasks. The plots show that the optimal
$k^{+}$ and $k^{-}_{\text{hard}}$ values are partially task dependent.

###  A.7 Examples

Tab. 6 lists three examples of query papers with their corresponding positive
and negative samples. The complete set of triples that we use during training
are available in our code repository1.

Table 6: Example query papers with their positive and negative samples. Query: | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
---|---
Positives: |  |  • A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
---
• Looking for ELMo’s Friends: Sentence-Level Pretraining Beyond Language
Modeling
• GLUE : A MultiTask Benchmark and Analysis Platform for Natural Language
Understanding
• Dissecting Contextual Word Embeddings: Architecture and Representation
• Universal Transformers
Negatives: |  |  • Planning for decentralized control of multiple robots under uncertainty
---
• Graph-Based Relational Data Visualization
• Linked Stream Data Processing
• Topic Modeling Using Distributed Word Embeddings
• Adversarially-Trained Normalized Noisy-Feature Auto-Encoder for Text
Generation
|
Query: | BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Positives: |  |  • Exploring Word Embedding for Drug Name Recognition
---
• A neural joint model for entity and relation extraction from biomedical text
• Event Detection with Hybrid Neural Architecture
• Improving chemical disease relation extraction with rich features and weakly
labeled data
• GLUE : A MultiTask Benchmark and Analysis Platform for Natural Language
Understanding
Negatives: |  |  • Weakly Supervised Facial Attribute Manipulation via Deep Adversarial Network
---
• Applying the Clique Percolation Method to analyzing cross-market branch
banking …
• Perpetual environmentally powered sensor networks
• Labelling strategies for hierarchical multi-label classification techniques
• Domain Aware Neural Dialog System
|
Query: | A Context-Aware Citation Recommendation Model with BERT and Graph Convolutional Networks
Positives: |  |  • Content-based citation analysis: The next generation of citation analysis
---
• ScisummNet: A Large Annotated Dataset and Content-Impact Models for
Scientific Paper …
• Citation Block Determination Using Textual Coherence
• Discourse Segmentation Of Multi-Party Conversation
• Argumentative Zoning for Improved Citation Indexing
Negatives: |  |  • Adaptive Quantization for Hashing: An Information-Based Approach to Learning …
---
• Trap Design for Vibratory Bowl Feeders
• Software system for the Mars 2020 mission sampling and caching testbeds
• Applications of Rhetorical Structure Theory
• Text summarization for Malayalam documents — An experience