#  CausalCite: A Causal Formulation of Paper Citations

Ishan Kumar1, Zhijing Jin1,2††footnotemark: , Ehsan Mokhtarian3, Siyuan
Guo1,4, Yuen Chen5,
Negar Kiyavash3, Mrinmaya Sachan2, Bernhard Schölkopf1
1Max Planck Institute for Intelligent Systems, Tübingen, Germany
2ETH Zürich, Switzerland
3EPFL, Switzerland
4University of Cambridge, UK
5University of Illinois Urbana-Champaign, US

Equal contribution. Emails<EMAIL_ADDRESS>and<EMAIL_ADDRESS>
###### Abstract

Evaluating the significance of a paper is pivotal yet challenging for the
scientific community. While the citation count is the most commonly used proxy
for this purpose, they are widely criticized for failing to accurately reflect
a paper’s true impact. In this work, we propose a causal inference method,
TextMatch, which adapts the traditional matching framework to high-dimensional
text embeddings. Specifically, we encode each paper using the text embeddings
by large language models (LLMs), extract similar samples by cosine similarity,
and synthesize a counterfactual sample by the weighted average of similar
papers according to their similarity values. We apply the resulting metric,
called CausalCite, as a causal formulation of paper citations. We show its
effectiveness on various criteria, such as high correlation with paper impact
as reported by scientific experts on a previous dataset of 1K papers, (test-
of-time) awards for past papers, and its stability across various sub-fields
of AI. We also provide a set of findings that can serve as suggested ways for
future researchers to use our metric for a better understanding of a paper’s
quality.111Our code and data are at https://github.com/causalNLP/causal-cite.

##  1 Introduction

Recent years have seen explosive growth in the number of scientific
publications, making it increasingly challenging for scientists to navigate
the vast landscape of scientific literature. Therefore, identifying a good
paper has become a crucial challenge for the scientific community, not only
for technical research purposes, but also for decision-making, such as funding
allocation (Carlsson, 2009), research evaluation (Moed, 2006), recruitment
criteria (Gary Holden & Barker, 2005), and university ranking and evaluation
(Piro & Sivertsen, 2016).

A traditional method to recognize paper quality is peer review, a mechanism
that requires large efforts, and yet has inherent randomness and flaws (Cortes
& Lawrence, 2021; Rogers et al., 2023; Shah, 2022; Prechelt et al., 2018;
Resnik et al., 2008). Moreover, the number of papers after peer review is
still overwhelmingly massive for researchers to read, leaving the challenge of
identifying truly impactful research under-addressed. Another commonly used
metric is citations. However, this metric faces criticism for biases, such as
a preference for survey, toolkit, and dataset papers (Zhu et al., 2015;
Valenzuela-Escarcega et al., 2015). Citations, together with altmetrics
(Wilsdon et al., 2015), which incorporates the social media attention to a
paper, often have biases towards papers with extensive publicity and promotion
or those authored by established figures in the field.

To provide a more equitable assessment of paper quality, we employ the causal
inference framework (Hernán & Robins, 2010) to quantify a paper’s impact by
how much of the academic success in the follow-up papers should be causally
attributed to this paper. We introduce CausalCite, an enhanced version of
citations that poses the following _counterfactual question_ : “had this paper
never been published, what would happen to its follow-up studies?” To compute
the causal attribution for each follow-up paper, we synthetically construct a
counterfactual paper that addresses a similar topic, but is not built on the
paper of interest. Our approach allows CausalCite to quantify scientific
impact through a causal lens, offering an alternative understanding of a
paper’s impact within the academic community.

To tackle this problem, we need to address its main challenge – performing
causal inference on high-dimensional text data. Leveraging the current
advancement of large language models (LLMs), we adapt causal inference methods
to the high-dimensional space created by LLMs, marrying the causal inference
and natural language processing (NLP) domains. Specifically, we propose
TextMatch, a new method which uses LLMs to encode an academic paper as a high-
dimensional text embedding and then operationalizes the matching method by a
combination of techniques in information retrieval (Manning et al., 2008) and
semantic text similarity matching (Majumder et al., 2016; Chandrasekaran &
Mago, 2022), such as BM25 (Robertson & Zaragoza, 2009) and cosine similarity
matching (Majumder et al., 2016). After finding close matches for each paper,
we leverage all the close matches to synthesize a counterfactual sample by a
weighted average according to their similarity scores. This enables a stable
causal effect estimation. Finally, we quantify the counterfactual causal
effect as the difference between the synthesized control sample and the
original sample.

We conduct extensive experiments utilizing the Semantic Scholar corpus (Lo et
al., 2020; Kinney et al., 2023), which consists of $206$M papers and $2.4$B
citation links. We empirically validate CausalCite by showing higher
predictive accuracy of paper impact (as judged by scientific experts on a past
dataset of 1K papers (Zhu et al., 2015)) compared to citations and other
previous impact assessment metrics. We further show a stronger correlation of
the metric with the test-of-time (ToT) paper awards. Furthermore, we show
that, unlike citation counts, our metric exhibits a greater balance across
various research domains in AI, e.g., general AI, NLP and computer vision
(CV). While citation numbers for papers in those domains vary significantly –
for example, while an average CV paper has many more citations on average than
an NLP paper, our CausalCite scores papers across AI sub-fields more
similarly. After demonstrating the desirable property of our metric, we
present several case studies of its applications. Our evaluations reveal that
the quality of conference best papers is noisier on average than that of ToT
papers (Section 7.1). We find that our CausalCite follows a power law
distribution (Section 7.2), and conduct a case study showcasing CausalCite for
some renowned papers (Section 7.3). Finally, we utilize our metric to identify
high-quality papers that are less recognized by citation counts (Section 7.4).

Contributions. (1) We introduce CausalCite, a counterfactual causal effect-
based formulation for paper citations. (2) We introduce TextMatch, a novel
method that leverages LLMs and causal inference to estimate the counterfactual
causal effect of a paper. (3) We conduct comprehensive analyses, encompassing
three performance evaluations and four findings using our new metric.

##  2 Problem Formulation

###  2.1 Graph Notations

Our problem formulation involves a citation graph and a causal graph. We use
lowercase letters for specific papers and uppercase for an arbitrary paper
treated as a random variable.

####  2.1.0.0.1 Citation Graph

In the citation graph $\mathbb{G}\coloneqq(\mathbb{P},\mathbb{L})$,
$\mathbb{P}$ is a set of papers, and each edge $\ell_{i,j}\in\mathbb{L}$
indicates that an earlier paper ${p}_{i}$ influences (i.e., is cited by) a
follow-up paper ${p}_{j}$. To obtain the citation graph, we use the Semantic
Scholar Academic Graph dataset (Kinney et al., 2023) with 206M papers and 2.4B
citation edges.

####  2.1.0.0.2 Causal Graph

The causal graph, shown in Figure 1, highlights the contribution of a paper
$a$ to a follow-up paper $b$. It uses a binary variable $T$ to indicate if $a$
influences $b$ and an effect variable $Y$ to represent the success of $b$. We
use $\log_{10}$ of citation to quantify $Y$, although other scoring metrics
are applicable. In the causal graph, the source of randomness originates from
paper $b$, and the random variables pertain to properties of paper $b$. We
introduce two sets of variables in this causal graph: (i) The set of
confounders, which are the common causes of $T$ and $Y$. For instance, the
research area of $b$ impacts both the likelihood of a paper citing $a$ and its
own citation count. (ii) Descendants of the treatment, comprising mediators
(e.g., paper $a$ influencing the quality of paper $b$ and subsequently
influencing its citations) and colliders (e.g., both the influence from $a$
and the citations of $b$ itself influencing later awards received by $b$).

###  2.2 CausalCite Indices

In this section, we introduce various indices that measure the causal impact
of a paper.

####  2.2.0.0.1 Two-Paper Interaction: Pairwise Causal Impact (PCI).

To examine the causal impact of a paper $a$ on a follow-up paper $b$, we
define the pairwise causal impact $\mathrm{PCI}(a,b)$ by unit-level causal
effect:

$\displaystyle\mathrm{PCI}(a,b)\coloneqq y^{t=1}-y^{t=0}~{},$ (1)

where we compare the outcomes $Y$ of the paper $b$ had it been influenced by
paper $a$ or not, denoted as the actual $y^{t=1}$ and the counterfactual
$y^{t=0}$, respectively. Note that the counterfactual $y^{t=0}$ can never be
observed, but only estimated by statistical methods, as we will discuss in
Sections 3 and 4.

####  2.2.0.0.2 Single-Paper Quality Metrics: Total Causal Impact (TCI) and
Average Causal Impact (ACI).

Let $\bm{S}$ denote the set of all follow-up studies of paper $a$. We define
total causal impact $\mathrm{TCI}(a)$ as the sum of the pairwise causal impact
index $\mathrm{PCI}(a,b)$ across all possible $b$’s. That is,

$\mathrm{TCI}(a)\coloneqq\sum_{b\in\bm{S}}\mathrm{PCI}(a,b)~{}.$ (2)

This definition provides an aggregated measure of a paper’s influence across
all its follow-up papers.

As the causal inference literature is usually interested in the average
treatment effect, we further define the average causal impact (ACI) index as
the average per paper PCI:

$\mathrm{ACI}(a)\coloneqq\frac{\mathrm{TCI}(a)}{|\bm{S}|}=\frac{1}{|\bm{S}|}\sum_{b\in\bm{S}}\left(y^{t=1}-y^{t=0}\right).$
(3)

To establish its causal linkage more explicitly, we note that
$\mathrm{ACI}(a)$ is equal to the average treatment effect on the treated
(ATT) of paper $a$ (Rosenbaum & Rubin, 1983).

##  3 Classical Causal Inference Methods

In this section, we provide a brief overview on existing methods in causal
effect estimation and describe their limitations on our research question.

Figure 1: The causal graph of our study. We are interested in the treatment
effect of the influence from $a$ to the resulting citations of $B$ (outlined
in blue). To obtain this causal effect, we consider three sets of variables:
the confounder set which we need to control for (colored in the blue area),
the mediator and collider set which we need to explicitly exclude (colored in
gray with the red text “Not to control for”), and the non-shared ancestor set
which does not need any action (colored in gray with the gray text “No action
needed”).

####  3.0.0.0.1 Blocking the Backdoor Paths.

It is important to control for the right set of variables for causal effect
estimation. For example, in CausalCite, we control the domain of the paper to
estimate causal impact. Specifically, Pearl (1995) shows one need to block all
backdoor paths from the treatment to the effect variable, allowing only the
association along the $T\rightarrow Y$ path. Hence, we want to control for
certain factors, such as the research topic across the treatment and control
groups.

Using principles in do-calculus (Pearl, 1995), such variables to control for
must include all the confounders, as marked in the blue box in Figure 1, but
exclude the descendant variables of the treatment, as marked in the gray boxes
with the red note “Not to control for” in Figure 1. As a note, if we
unintentionally control for descendants of the treatment, such as the paper’s
reported model performance, then we might get an undesired control group, such
as papers that do not build on $a$ but still achieve as good performance as
those that build on $a$ (Imagine a paper that did not build LLMs but still
achieves the same performance as LLM-based models, then this paper must be
extra strong, deviating from the normal paper in the control group).
Technically, by avoiding the descendants of the treatment, we prevent
undesired effects, such as blocking mediators or accounting for colliders,
which will result in erroneous inference.

####  3.0.0.0.2 Randomized Control Trials (RCTs).

The ideal way to obtain causal effects is through randomized control trials
(RCTs). For example, when testing a drug, we randomly split all patients into
two groups, the control group and the treatment group, where the random
splitting ensures the same distribution of the confounders across the two
groups such as gender and age. However, RCTs are usually not easily
achievable, in some cases too expensive (e.g., tracking hundreds of people’s
daily lives for 50 years), and in other cases unethical (e.g., forcing a
random person to smoke), or infeasible (e.g., getting a time machine to change
a past event in history).

For our research question on a paper’s impact, utilizing RCTs is impractical.
Because it is infeasible to randomly divide researchers into two groups,
instructing one group to base their research on a specific paper $a$ while the
other group does not, and then observe the citation count of their papers
years later.

####  3.0.0.0.3 Ratio Matching

In the absence of RCTs, matching serves as an alternate method for determining
causal effects from observational data. In this case, we can let the treatment
assignment happen naturally, such as taking the naturally existing set of
papers and running causal inference by adjusting for the variables that block
all paths. Given a set of naturally observed papers, one of the most commonly
used causal inference methods is ratio matching (Rosenbaum & Rubin, 1983),
whose basic idea is to iterate over all possible values $\bm{x}$ of the
adjustment variables $\bm{X}$ and obtain the difference between the treatment
group $\mathcal{T}$ and control group $\mathcal{C}$.

While ratio matching is practical when there is a small set of values for the
adjustment variables to sum over, its applicability dwindles with high-
dimensional variables like text embeddings in our context. This scenario may
generate numerous intervals to sum over, presenting numerical challenges and
potential breaches of the positivity assumption.

####  3.0.0.0.4 One-to-One Matching

To handle the adjustment variables in the high-dimensional space, one possible
way is to avoid pre-defining all their possible intervals. but instead,
iterate over each unit in the treatment group to match for its closest control
unit. Consider a given follow-up paper $b$, and a set of candidate control
papers $\bm{C}$, where each paper $c_{i}$ has a citation count $y_{i}$, and
vector representation $\bm{t}_{i}$ of the confounders (e.g., topic, storyline,
etc). The one-to-one matching method estimates PCI as

$\displaystyle\widehat{\mathrm{PCI}}(a,b)=y_{b}-y_{\operatorname*{argmax}_{c_{i}\in\bm{C}}m_{i}}=y_{b}-y_{\operatorname*{argmax}_{c_{i}\in\bm{C}}\mathrm{sim}(\bm{t}_{y},\bm{t}_{i})}~{},$
(4)

where we approximate the counterfactual sample by the paper $c_{i}\in\bm{C}$
which is the most similar to paper $b$ by the matchedness score $m_{i}$, which
is obtained by the cosine similarity $\mathrm{sim}$ of the confounder vectors.
A limitation of this one-to-one matching method is that it might induce large
instability in the result, as only taking one paper with similar contents may
have a large variance in citations when the matched paper slightly differs.

##  4 TextMatch: Stabilizing Text Matching by Synthesis

###  4.1 The TextMatch Method

To fill in the gap of the existing matching methods, we propose TextMatch,
which mitigates the instability issue of the one-to-one matching by replacing
it with a convex combination of a set of matched samples to form a synthetic
counterfactual sample. Specifically, we identify a set of papers
$c_{i}\in\bm{C}$ with high matchedness scores $m_{i}$ to the paper $b$, and
synthesize the counterfactual sample by an interpolation of them:

$\displaystyle\widehat{\mathrm{PCI}}(a,b)=y_{b}-\sum_{c_{i}\in\bm{C}}w_{i}y_{i}=y_{b}-\sum_{c_{i}\in\bm{C}}\frac{m_{i}}{\sum_{i}m_{i}}y_{i}~{},$
(5)

where the weight $w_{i}$ of each paper $c_{i}$ is proportional to the
matchedness score $m_{i}$ and normalized.

The contributions of our method are as follows: (1) we adapt the traditional
matching methods from low-dimensional covariates to any high-dimensional
variables such as text embeddings; (2) different from the ratio matching, we
do not stratify the covariates, but synthesize a counterfactual sample for
each observed treated units; (3) due to this iteration over each treated unit
instead of taking the population-level statistics, we closely control for
exogenous variables for the ATT estimation, which circumvents that need for
the structural causal models; (4) we further stabilize the estimand by a
convex combination of a set of similar papers. Note that the contribution of
Eq. 5 might seem to bear similarity with synthetic control, but they are
fundamentally different, in that synthetic control runs on time series, and
fit for the weights $w_{i}$ by linear regression between the time series of
the treated unit and a set of time series from the control units, using each
time step’s values in the regression loss function. Therefore, our TextMatch
proposal is novel and constitutes a crucial step to bridge the traditional
matching methods to high-dimensional text embeddings.

###  4.2 Overall Algorithm

Using TextMatch as the backbone theoretical method, we develop the overall
algorithm to calculate the causal impact indices as follows.

####  4.2.0.0.1 Obtaining the Control Variables

Given the two papers $a$ and $b$, we first collect the set of confounders
using the causal graph in Figure 1: paper $b$’s publication year, research
area, research question, and storyline. When collecting the confounders, we
have direct access to the year variable, but for the remaining ones, we
unleash the power of NLP to represent them by text embeddings in a rich, high-
dimensional space, instead of manually coding them as categorical variables.
We describe the process below. Starting from any given paper $b$ with its
title and abstract, we not only need to model these contents, but also need to
pay attention not to accidentally control for any descendant of the treatment,
such as the quality/performance (through expressions such as “we achieved 90%
accuracy”). Therefore, we first exclude expressions about the quality or
performance of the paper, such as the notion “state-of-the-art” and the exact
performance numbers in arabic numbers. Then, we concatenate the title and
abstract to encode them into a text variable $\bm{t}_{b}$ later for the
matching.

####  4.2.0.0.2 Selecting the Text Encoder

When projecting the text into the vector space, we need a text encoder with a
strong representation power for scientific publications, and is sensitive
towards two-paper similarity comparisons regarding their research topics,
questions, and storylines. For the representation power for scientific
publications, instead of general-domain models such as BERT (Devlin et al.,
2019) and RoBERTa (Liu et al., 2019), we consider LLM variants pretrained on
large-scale scientific text, such as SciBERT (Beltagy et al., 2019), SPECTER
(Cohan et al., 2020), and MPNet (Song et al., 2020).

To check the capability for two-paper similarity comparisons, We conduct a
small-scale empirical study comparing human-ranked paper similarity and model-
identified semantic similarity. We find that MPNet correlates the best with
human judgments, and give more distinct scores to papers with different levels
of similarity. This capability advantage may be attributed to its Siamese
network objectives in the training process (Song et al., 2020). See more
details in Section A.3.

####  4.2.0.0.3 Enabling the Matching Method at Scale

Since we use one of the largest available paper database, the Semantic Scholar
dataset (Kinney et al., 2023) of 206M papers with 2.4B citation links, we need
to optimize our algorithm for large-scale paper matching. The vast paper size
is a critical problem, as for example, even after we filter by the publication
year, the number of papers in the same year could be up to 8.8M, and on
average 7.5M per year since 2010.

In order to conduct text matching across millions of papers, we use the
combination of two NLP tasks, first information retrieval (IR) (Manning et
al., 2008) and semantic textual similarity (STS) (Majumder et al., 2016;
Chandrasekaran & Mago, 2022). IR is used for large-scale, efficient document-
level text retrieval, an example of which is Google search, and we use the
most commonly used IR method, BM25 (Robertson & Zaragoza, 2009), as the
preparation step for the text matching. Briefly, it is a bag-of-words
retrieval function that uses term frequencies and document lengths to estimate
relevancy between two text documents. Deploying this method, we can find a set
of candidate papers for, for example, two million papers, at a speed 250x
faster than the text embedding cosine similarity matching.

We develop a first-coarse-then-fine-grained approach, which first obtains a
broad set of 100 candidate papers using BM25, and then runs the fine-grained
matching by cosine similarity. In the cosine similarity matching process, we
use the MPNet model to encode the text of each paper $c_{i}$ into an embedding
$\bm{t}_{i}$, with which we get the matchedness score $m_{i}$ according to Eq.
4, and the normalized weight $w_{i}$ by Eq. 5.

####  4.2.0.0.4 Deriving the Causal Impact Indices by Our Novel Numerical
Estimation Method

After using the above method to obtain each $\mathrm{PCI}(a,b)$, we aggregate
them to obtain the ACI and TCI in Algorithm 1.

Algorithm 1 Algorithm to derive the causal impact indices $\mathrm{ACI}$ and
$\mathrm{TCI}$

1:Input: Paper $a$, all its follow-up papers $\bm{B}$, a random subset
$\bm{B}^{\prime}$, and non-follow-up papers $\bm{C}$.

2:procedure GetACI($a$)

3: $\mathrm{ACI}\leftarrow 0$

4: for each $b_{i}$ in $\bm{B}^{\prime}$ do

5: $I_{i}\leftarrow\text{getPCI}(a,b_{i},\bm{C})$

6: $\mathrm{ACI}\leftarrow\mathrm{ACI}+\frac{1}{|\bm{B}^{\prime}|}\cdot I_{i}$

7: end for

8: $\mathrm{TCI}\leftarrow\mathrm{ACI}\cdot|\bm{B}|$

9: return $\mathrm{ACI}$ and $\mathrm{TCI}$

10:end procedure

Given the vast paper numbers mentioned above, there is also a critical
challenge when aggregating the total causal impact because the number of
follow-up papers for a study can be up to tens of thousands, such as the
57,200 citations for the ImageNet paper (Deng et al., 2009). To address this
problem, we innovate a numerical estimation method using a carefully designed
random subset of the follow-up papers.

A naive method to achieve this is through Monte Carlo (MC) sampling. However,
the basis assumption behind MC sampling is the law of large numbers (LLN), but
unfortunately, it requires a very large sample size when it comes to long-
tailed distributions, which is the usual case of citations. Since citations
are very likely to be concentrated in the head part of the distribution, we
will not be able to afford the computers of budget for a huge sample size that
covers the narrow head area. Instead, we propose a novel numerical estimation
method for the follow-up paper sampling, inspired by importance sampling
(Singh, 2014; Kloek & van Dijk, 1976).

Our numerical estimation method works as follows. First, we propose the
formulation that the relation between ACI and the TCI is an integral over all
possible paper $b$’s. Then, we formulated the above sampling problem as
integral estimation or area-under-the-curve estimation. We draw inspiration
from Simpson’s method, which estimates integrals by binning the input variable
into small intervals. Analogously, we bin the large set of follow-up papers
into $n$ equally-sized intervals and perform random sampling over each bin,
which we then sum over. In this way, we make sure that our samples come from
all parts of the long-tailed distribution and are a more accurate numerical
estimate for the actual TCI.

##  5 Experimental Setup

####  5.0.0.0.1 Dataset

We use the Semantic Scholar dataset (Lo et al., 2020; Kinney et al.,
2023)222https://api.semanticscholar.org/api-docs/datasets which includes a
paper corpus of 206M scientific papers, and a citation graph of 2.4B+ citation
edges. For each paper, we obtain the title and abstract for the matching
process. We list some more details of the dataset in Appendix B, such as the
number of papers reaching 8M per year after 2012.

####  5.0.0.0.2 Implementation Details

For the text embedding model, we compared SciBERT (Beltagy et al., 2019),
SPECTER (Cohan et al., 2020), and MPNet (Song et al., 2020), among which we
find that MPNet performs the best, outperforming the next one by 1.12 points
on our annotated dataset described in Section A.3. We deploy the all-mpnet-
base-v2 checkpoint of the MPNet using the transformers Python package (Wolf et
al., 2020), and set the batch size to be 32. For the set of matched papers, we
consider papers with cosine similarity scores higher than 0.81, and take the
top ten papers above the threshold. To enable efficient operations on the
large-scale citation graph, we use the Dask framework,333https://dask.org/
which optimizes for data processing and distributed computing. We optimize our
program to take around 100GB RAM, and on average 20-30 minutes for each
$\mathrm{PCI}(a,b)$ after matching against up to millions of candidates. More
implementation details are in Section A.1. For the estimation of TCI, we
empirically select the sample size to be 40, which is a balance between the
computational time and performance, as found in Section A.2.

##  6 Performance Evaluation

The contribution of a paper is inherently multi-dimensional, making it
infeasible to encapsulate its richness fully through a scalar. Yet the demand
for a single, comprehensible metric for research impact persists, fueling the
continued use of traditional citations despite their known limitations. In
this section, we show how our new metrics significantly improve upon
traditional citations by providing quantitative evaluations comparing the
effectiveness of citations, Semantic Scholar’s highly influential (SSHI)
citations (Valenzuela-Escarcega et al., 2015), and our CausalCite metric.

###  6.1 Author-Identified Paper Impact

In this experiment, we follow the evaluation setup in Valenzuela-Escarcega et
al. (2015) to use an annotated dataset (Zhu et al., 2015) comprised of 1,037
papers, annotated according to whether they serve as significant prior work
for a given follow-up study. Although paper quality evaluation can be
dauntingly tricky, this dataset was cleverly annotated by first collecting a
set of follow-up studies and letting one of the authors of each paper go
through the references they cite and select the ones that significantly impact
their work. In other words, for a given paper $b$, each reference $a$ is
annotated as whether $a$ has significantly impacts $b$ or not.

Metric | Accuracy
---|---
Citations | 71.33
SSHI Citations | 75.25
CausalCite | 80.29
Table 1: Accuracy of all three citation metrics.

Table 1 reports the accuracy of our CausalCite metric, together with two
existing citation metrics: citations, and SSHI citations (Valenzuela-Escarcega
et al., 2015). See the detailed derivation of the accuracy scores in Section
C.2. From this table, we can see that our CausalCite metric achieves the
highest accuracy, 80.29%, which is 5 points higher than SSHI, and 9 points
higher than the traditional citations.

###  6.2 Test-of-Time Paper Analysis

The test-of-time (ToT) paper award is a prestigious honor bestowed upon papers
that have made substantial and enduring impacts in their field. In this
section, we collect a dataset of $792$ papers, including $72$ ToT papers, and
a control group of $10$ randomly selected non-ToT papers from the same
conference and year as each ToT paper. To collect this ToT paper dataset, we
look into ten leading AI conferences spanning general AI (NeurIPS, ICLR, ICML,
and AAAI), NLP (ACL, EMNLP, and NAACL), and CV (CVPR, ECCV, and ICCV), for
which we go through each of their websites to identify all available ToT
papers.444We get this list by selecting the top conferences on Google Scholar
using the h5-Index ranking in each of the above domains: general AI (link), CV
(link), and NLP (link).

Metric | Corr. Coef.
---|---
Citations | 0.491
SSHI Citations | 0.317
TCI | 0.640
Table 2: Correlation coefficients of each metric and ToT paper award by Point
Biserial Correlation (Tate, 1954).

In Table 2, we show the correlations of various metrics with the ToT awards.
In this table, CausalCite achieves the highest correlation of $0.639$, which
is $+30.14\%$ better than that of citations. Furthermore, we visualize the
correspondence of our metric and ToT in Figure 2. The CausalCite distributions
of ToT vs. non-ToT papers show a substantial difference in Figure 2(a). We
also show three examples of ToT papers in Figure 2(b), where the ToT papers
differ from the non-ToT papers by one or two orders of magnitude.

(a) Distributions of ToT and non-ToT papers. The CausalCite values of non-ToT
papers is on average 142, and ToT papers on average 1,623.

(b) The CausalCite values of three example ToT papers from general AI, NLP,
and CV.

Figure 2: Correspondence of our CausalCite metric and ToT paper awards.

###  6.3 Topic Invariance of CausalCite

Research Area | ACI | Citations | SSHI
---|---|---|---
General AI (n=16) | 0.748 | 2,024 | 267
CV (n=36) | 0.734 | 7,238 | 1,088
NLP (n=20) | 0.763 | 1,785 | 461
Table 3: The average of each metric by research area on our collected set of
72 ToT papers.

A well-known issue with citations is their inconsistency across different
fields. What might be considered a large number of citations in one field
might be seen as average in another. In contrast, we show that our ACI index
does not suffer from this issue. We show this using our ToT dataset, where we
control for the quality of the papers to be ToT but vary the domain by the
three fields: general AI, CV, and NLP. We observe in Table 3 that even though
some domains have significantly more citations (for instance, CV ToT papers
have, on average, $4.05$ times more citations than NLP), the ACI remains
consistent across various fields.

##  7 Findings

Having demonstrated the effectiveness of our metrics, we now explore some
open-ended questions: (1) Do best papers have high causal impact? (Section
7.1) (2) How does the CausalCite value distribute across papers? (Section 7.2)
(3) What is the impact of some famous papers evaluated by CausalCite? (Section
7.3) (4) Can we use this metric to correct for citations? (Section 7.4).

###  7.1 Do Best Papers Have High Causal Impact?

Selecting best paper awards is an arguably much harder task than ToT papers,
as it is difficult to predict of the impact of a paper when it is just newly
published. Therefore, we are interested in the actual causal impact of best
papers. Similar to our study on ToT papers, we collect a dataset of $444$
papers including $74$ best papers and a control set of random $5$ non-best
papers from the same conference in the same year, using the same set of the
top ten leading AI conferences. We find that the correlation of the CausalCite
metric with best papers is $0.348$, which is very low compared to the $0.639$
correlation with the ToT papers. This shows that the best papers do not
necessarily have a high causal impact. One interpretation can be that the best
paper evaluation is a forecasting task, which is much more challenging than
the retrospective task of ToT paper selection.

Figure 3: The distribution of TCI values by percentile on a random set of 100
papers, which shows a long tail indicating that high impact is concentrated in
a relatively small portion of papers.

###  7.2 What Is the Curve Shape of the CausalCite Distributions?

We explore how the CausalCite scores are distributed across papers in general.
We plot Figure 3 using a random set of 100 papers from the Semantic Scholar
dataset, which is a reasonably large size given the computation budget
mentioned in 5.0.0.0.2. From this plot, we can see a power law distribution
with a long tail, echoing with the common belief that the paper impact follows
the power law, with high impact concentrated in a relatively small portion of
papers.

###  7.3 Selected Paper Case Study

Paper Name | TCI | Citations | ACI
---|---|---|---
Transformers | 52,507 | 68,064 | 0.771
BERT | 40,675 | 59,486 | 0.683
RoBERTa | 6,932 | 14,434 | 0.480
Table 4: Case study of some famous papers.

In addition to the overall distribution of our CausalCite values, we also look
at its correspondence to some selected papers. We show in Table 4 the causal
impact index values for a set of famous papers. For example, we know that the
Transformer paper (Vaswani et al., 2017) is a more foundational work than its
follow-up work BERT (Devlin et al., 2019), and BERT is more foundational than
its later variant, RoBERTa (Liu et al., 2019). And we can see this monotonic
trend in the TCI and ACI values of the three papers too. Again, this is a
preliminary case study, and we welcome future work to cover more papers.

###  7.4 Discovering Good Papers Less Recognized by Citations

Another important contribution of our metric is that it can help discover
papers that are traditionally overlooked by citations. To achieve the
discovery, we formulate the problem as outlier detection, where we first use a
linear projection to handle the trivial alignment of citations and CausalCite,
and then analyze the outliers using the interquartile range (IQR) method
(Smiti, 2020). See the exact calculation in Section C.1. We show the three
subsets of papers in Table 5, where the two outlier categories, the overcited
and undercited papers, correspond to the false positive and false negative
oversight by citations, respectively. An additional note is that, when we look
into some characteristics of the three categories, we find that the citation
frequency in result section, i.e., the percentage of times they are cited in
results section compared to all the citations, correlates with these
categories. Specifically, we find that the undercited papers tend to have more
of its citations concentrated in the results section, which usually indicates
that this paper constitutes an important baseline for a the follow-up study,
while the overcited papers tend to be cited out of the results section, which
tends to imply a less significant citation.

Paper Category | Citations in Result Section (%) | Residual
---|---|---
Overcited Papers (7.04% of the Papers) | 1.26 | -1.792
Aligned Papers (91.20%) | 1.51 | 0.118
Undercited Papers (1.76%) | 1.90 | 1.047
Table 5: We use our CausalCite metric to discover outlier papers that are
overlooked by citations. We find that the undercited papers (with high TCIs,
but fewer citations than what they should have been deserved) are usually more
cited in the results sections, which indicates a more significant role in the
follow-up papers.

##  8 Conclusion

In this study, we propose CausalCite, a novel causal formulation for paper
citations. Our method combines traditional causal inference methods with the
recent advancement of NLP in LLMs to provide a new causal outlook on paper
impact by answering the causal question: ”Had this paper never been published,
what would be the impact on this paper’s current follow-up studies?”. With
extensive experiments and analyses using expert ratings and test-of-time
papers as criteria for impact, our new CausalCite metric demonstrates clear
improvements over the traditional citation metrics. Finally, we use this
metric to investigate several open-ended questions like “Do best papers have
high causal impact?”, conduct a case study of famous papers, and suggest
future usage of our metric for discovering good papers less recognized by
citations for the scientific community.

## Author Contributions

This project originates as part of the AI Scholar series of projects that
Zhijing Jin started since 2021, as she identified that causal inference over
papers is a valuable research setting with sufficient data and rich causal
phenomenon. Bernhard Schölkopf came up with the formulation that the action of
citation itself has a causal nature, and can thus be formulated as a causal
inference question. Zhijing, Bernhard, and Siyuan Guo settled down the overall
project design.

After the initial idea formulation, Ishan Kumar and Zhijing Jin
operationalized the entire project, with vast efforts in identifying the data
source; improving the theoretical formulation (together with Ehsan Mokhtarian,
Bernhard, and Negar Kiyavash); speeding up the code efficiency; designing the
evaluation and analysis protocols (with the insightful supervision from
Mrinmaya Sachan and Bernhard, and suggestions from Siyuan); and implementing
all the evaluations (with the help of Yuen Chen). In the writing stage,
Mrinmaya gave substantial guidance to structure the storyline of the paper,
and Zhijing, Ehsan, Ishan, and Mrinmaya contributed significantly to the
writing, with various help and suggestions from all the other authors.

## Acknowledgment

During the idea formulation stage, we are grateful for the research
discussions with Kun Zhang on the vision of the AI Scholar series of projects.
During the implementation of our paper, we thank Zhiheng Lyu for his
suggestions on efficient computer algorithm over massive graphs and large
data. We also thank labmates from Max Planck Institute for constructive
feedback and help on data annotation. We thank Vincent Berenz, Felix Leeb, and
Luigi Gresele for their generous support with computation resources.

This material is based in part upon works supported by the German Federal
Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ:
01IS18039B; by the Machine Learning Cluster of Excellence, EXC number 2064/1 –
Project number 390727645; by the John Templeton Foundation (grant #61156); by
a Responsible AI grant by the Haslerstiftung; and an ETH Grant (ETH-19 21-1).
Zhijing Jin is supported by PhD fellowships from the Future of Life Institute
and Open Philanthropy, as well as the travel support from ELISE (GA no 951847)
for the ELLIS program.

## References

  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan.  SciBERT: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 3615–3620, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1371.  URL https://aclanthology.org/D19-1371.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  _ArXiv_ , abs/2005.14165, 2020.  URL https://api.semanticscholar.org/CorpusID:218971783.
  * Carlsson (2009) Håkan Carlsson.  Allocation of research funds using bibliometric indicators–asset and challenge to swedish higher education sector.  2009\.
  * Chandrasekaran & Mago (2022) Dhivya Chandrasekaran and Vijay Mago.  Evolution of semantic similarity - A survey.  _ACM Comput. Surv._ , 54(2):41:1–41:37, 2022\.  doi: 10.1145/3440755.  URL https://doi.org/10.1145/3440755.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  Palm: Scaling language modeling with pathways.  _J. Mach. Learn. Res._ , 24:240:1–240:113, 2022.  URL https://api.semanticscholar.org/CorpusID:247951931.
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld.  SPECTER: Document-level Representation Learning using Citation-informed Transformers.  In _ACL_ , 2020.
  * Cortes & Lawrence (2021) Corinna Cortes and Neil D. Lawrence.  Inconsistency in conference peer review: Revisiting the 2014 neurips experiment.  _CoRR_ , abs/2109.09774, 2021.  URL https://arxiv.org/abs/2109.09774.
  * Courant et al. (1952) Ernest D. Courant, Milton Stanley Livingston, and Hartland S. Snyder.  The strong-focusing synchrotron-a new high energy accelerator.  _Physical Review_ , 88:1190–1196, 1952.  URL https://api.semanticscholar.org/CorpusID:121090542.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In _Computer Vision and Pattern Recognition (CVPR)_ , pp. 248–255, 2009.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1423.  URL https://aclanthology.org/N19-1423.
  * Fang & Zhan (2015) Xing Fang and Justin Zhijun Zhan.  Sentiment analysis using product review data.  _Journal of Big Data_ , 2:1–14, 2015.  URL https://api.semanticscholar.org/CorpusID:16177937.
  * Gary Holden & Barker (2005) Gary Rosenberg Gary Holden and Kathleen Barker.  Bibliometrics.  _Social Work in Health Care_ , 41(3-4):67–92, 2005.  doi: 10.1300/J010v41n03\\_03.  URL https://doi.org/10.1300/J010v41n03_03.
  * Hernán & Robins (2010) Miguel A Hernán and James M Robins.  Causal inference, 2010.
  * Kinney et al. (2023) Rodney Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, Miles Crawford, Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob Evans, Sergey Feldman, Joseph Gorney, David Graham, Fangzhou Hu, Regan Huff, Daniel King, Sebastian Kohlmeier, Bailey Kuehl, Michael Langan, Daniel Lin, Haokun Liu, Kyle Lo, Jaron Lochner, Kelsey MacMillan, Tyler Murray, Chris Newell, Smita Rao, Shaurya Rohatgi, Paul Sayre, Zejiang Shen, Amanpreet Singh, Luca Soldaini, Shivashankar Subramanian, Amber Tanaka, Alex D. Wade, Linda Wagner, Lucy Lu Wang, Chris Wilhelm, Caroline Wu, Jiangjiang Yang, Angele Zamarron, Madeleine van Zuylen, and Daniel S. Weld.  The semantic scholar open data platform.  _CoRR_ , abs/2301.10140, 2023.  doi: 10.48550/arXiv.2301.10140.  URL https://doi.org/10.48550/arXiv.2301.10140.
  * Kloek & van Dijk (1976) Teun Kloek and Herman K. van Dijk.  Bayesian estimates of equation system parameters, an application of integration by monte carlo.  _Econometrica_ , 46:1–19, 1976.  URL https://api.semanticscholar.org/CorpusID:53634601.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld.  S2ORC: The semantic scholar open research corpus.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 4969–4983, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.447.  URL https://aclanthology.org/2020.acl-main.447.
  * Majumder et al. (2016) Goutam Majumder, Partha Pakray, Alexander Gelbukh, and David Pinto.  Semantic textual similarity methods, tools, and applications: A survey.  _Computación y Sistemas_ , 20, 2016.  URL https://api.semanticscholar.org/CorpusID:7828379.
  * Manning et al. (2008) Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze.  Introduction to information retrieval.  In _J. Assoc. Inf. Sci. Technol._ , 2008.  URL https://api.semanticscholar.org/CorpusID:262867651.
  * Moed (2006) Henk F Moed.  _Citation analysis in research evaluation_ , volume 9.  Springer Science & Business Media, 2006.
  * Pearl (1995) Judea Pearl.  Causal diagrams for empirical research.  _Biometrika_ , 82(4):669–688, 1995.
  * Piro & Sivertsen (2016) Fredrik Niclas Piro and Gunnar Sivertsen.  How can differences in international university rankings be explained?  _Scientometrics_ , 109(3):2263–2278, December 2016.  doi: 10.1007/s11192-016-2056-5.  URL https://ideas.repec.org/a/spr/scient/v109y2016i3d10.1007_s11192-016-2056-5.html.
  * Prechelt et al. (2018) Lutz Prechelt, Daniel Graziotin, and Daniel Méndez Fernández.  A community’s perspective on the status and future of peer review in software engineering.  _Information and Software Technology_ , 95:75–85, 2018\.
  * Radford & Narasimhan (2018) Alec Radford and Karthik Narasimhan.  Improving language understanding by generative pre-training.  2018\.  URL https://api.semanticscholar.org/CorpusID:49313245.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.  URL https://api.semanticscholar.org/CorpusID:160025533.
  * Reimers & Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Conference on Empirical Methods in Natural Language Processing_ , 2019.
  * Resnik et al. (2008) David B Resnik, Christina Gutierrez-Ford, and Shyamal Peddada.  Perceptions of ethical problems with scientific journal peer review: an exploratory study.  _Science and engineering ethics_ , 14(3):305–310, 2008.
  * Robertson & Zaragoza (2009) Stephen E. Robertson and Hugo Zaragoza.  The probabilistic relevance framework: Bm25 and beyond.  _Found. Trends Inf. Retr._ , 3:333–389, 2009.  URL https://api.semanticscholar.org/CorpusID:207178704.
  * Rogers et al. (2023) Anna Rogers, Marzena Karpinska, Jordan Boyd-Graber, and Naoaki Okazaki.  Program chairs’ report on peer review at acl 2023.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. xl–lxxv, Toronto, Canada, July 2023. Association for Computational Linguistics.  URL https://aclanthology.org/2023.acl-long.report.
  * Rombach et al. (2021) Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 10674–10685, 2021.  URL https://api.semanticscholar.org/CorpusID:245335280.
  * Rosenbaum & Rubin (1983) Paul R Rosenbaum and Donald B Rubin.  The central role of the propensity score in observational studies for causal effects.  _Biometrika_ , 70(1):41–55, 1983.
  * Shah (2022) Nihar B Shah.  An overview of challenges, experiments, and computational solutions in peer review.  _Communications of the ACM_ , 65(6):76–87, 2022\.  URL http://bit.ly/PeerReviewOverview.
  * Singh (2014) Surya Nath Singh.  Sampling techniques & determination of sample size in applied statistics research : an overview.  2014\.  URL https://api.semanticscholar.org/CorpusID:29130207.
  * Smiti (2020) Abir Smiti.  A critical overview of outlier detection methods.  _Computer Science Review_ , 38:100306, 2020.  ISSN 1574-0137.  doi: https://doi.org/10.1016/j.cosrev.2020.100306.  URL https://www.sciencedirect.com/science/article/pii/S1574013720304068.
  * Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu.  Mpnet: Masked and permuted pre-training for language understanding.  _arXiv preprint arXiv:2004.09297_ , 2020.
  * Tate (1954) Robert F. Tate.  Correlation between a discrete and a continuous variable. point-biserial correlation.  _Annals of Mathematical Statistics_ , 25:603–607, 1954\.  URL https://api.semanticscholar.org/CorpusID:121004643.
  * Valenzuela et al. (2015) Marco Valenzuela, Vu Ha, and Oren Etzioni.  Identifying meaningful citations.  In Cornelia Caragea, C. Lee Giles, Narayan L. Bhamidipati, Doina Caragea, Sujatha Das Gollapalli, Saurabh Kataria, Huan Liu, and Feng Xia (eds.), _Scholarly Big Data: AI Perspectives, Challenges, and Ideas, Papers from the 2015 AAAI Workshop, Austin, Texas, USA, January, 2015_ , volume WS-15-13 of _AAAI Technical Report_. AAAI Press, 2015.  URL http://aaai.org/ocs/index.php/WS/AAAIW15/paper/view/10185.
  * Valenzuela-Escarcega et al. (2015) Marco Antonio Valenzuela-Escarcega, Vu A. Ha, and Oren Etzioni.  Identifying meaningful citations.  In _AAAI Workshop: Scholarly Big Data_ , 2015.  URL https://api.semanticscholar.org/CorpusID:2538517.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pp. 5998–6008, 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
  * Wilsdon et al. (2015) James Wilsdon, Liz Allen, Eleonora Belfiore, Philip Campbell, Stephen Curry, Steven A. Hill, Richard Jones, Roger J. P. Kain, Simon Kerridge, Mike A Thelwall, Jane Tinkler, Ian Viney, Paul Wouters, Jude Hill, and Brandon Johnson.  The metric tide: report of the independent review of the role of metrics in research assessment and management.  2015\.  URL https://api.semanticscholar.org/CorpusID:109985240.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pp. 38–45, Online, October 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-demos.6.  URL https://aclanthology.org/2020.emnlp-demos.6.
  * Zainuddin & Selamat (2014) Nurulhuda Zainuddin and Ali Selamat.  Sentiment analysis using support vector machine.  _2014 International Conference on Computer, Communications, and Control Technology (I4CT)_ , pp. 333–337, 2014.  URL https://api.semanticscholar.org/CorpusID:43161567.
  * Zhu et al. (2015) Xiao-Dan Zhu, Peter D. Turney, Daniel Lemire, and André Vellino.  Measuring academic influence: Not all citations are equal.  _Journal of the Association for Information Science and Technology_ , 66, 2015.  URL https://api.semanticscholar.org/CorpusID:3023580.

Appendix

##  Appendix A Additional implementation details

###  A.1 Time and Space Complexity Details

For the time cost of running the causal impact indices, each
$\mathrm{PCI}(a,b)$ takes around 1,500 seconds, or 25 minutes. Multiplying
this by 40 samples per paper $a$, we spend 16.67 hours to calculate each ACI
or TCI for the paper’s overall impact. For a fine-grained division into the
time cost, the majority of the time is spend on the BM25 indexing (800s) and
the sentence embedding cosine similarities calculation (400s). The rest of the
time-consuming steps are the BFS search (150-200s every time) to identify
descendants and non-descendants of a paper.

For the space complexity, we loaded the 2.4B edges of the citation graph into
a parquet gzip format for faster loading, and use Dask’s lazy load operation
to load it part by part to RAM for better parallelization. The program can fit
into different sizes of RAMs by modifying the number of partitions and
reducing the number of workers in Dask, at the cost of an increased
computation time. On the hard disk, citation graph takes up 19G space, and
paper data takes 11G.

###  A.2 Numerical Estimation Method: Finding the Sample Size

For our numerical estimation method, we first calculate the ACI on a subset of
carefully sampled papers and then aggregate it to TCI. One design choice
question is how to decide the size of this random subset. In our case, we need
to balance both the computation time (25 minutes per pairwise paper impact)
and the estimation accuracy. To identify the best sample size, we conduct a
small-scale study, first obtaining the TCI using our upper-bound budget of
$n=100$ samples and then gradually decreasing the number of samples to see if
there is a stable point in the middle which also leads to a result close to
that obtained with 100 samples. In Figure 4, we show the trade-off of the two
curves, the error curve and time cost, where we can see $n=40$ seems to be a
good point balancing the two. It is at the elbow of the arrow curve, making it
relatively close to the estimation result of $n=100$, and also in the meantime
vastly saving our computational budget, enabling us to run efficient
experiments for more analyses.

$0$$20$$40$$60$$80$$100$$0$$50$$100$$150$Sample Size $n$Error PercentageTCI
error$0$$1{,}000$$2{,}000$$3{,}000$MinutesComputational time Figure 4:  We
show the trade-off of two curves: the error curve (orange), and the time cost
curve (blue). For the error curve, we see an elbow point at around $n=40$,
when the error starts to be small. The curve for the computational time is
linear, taking 25 minutes for each paper. Balancing the trade-offs, we decided
to choose the sample size $n=40$.

###  A.3 Experiment to Select the Best Embedding Method

When selecting the text encoder for our TextMatch method, we compare among the
three LLMs pre-trained on scientific papers, SciBERT, MPNet, and SPECTER.
Specifically, we conduct a small-scale experiment to see how much the
similarities scores based on the embedding of each model align with human
annotations. As for the annotation process, we first collect a set of random
papers as pivot papers, and for each pivot paper, we identify ten papers, from
the most similar to the least, with monotonically decreasing similarity. We
collect a total of 100 papers consisting of ten such collections, for which we
show an example in Table 6. Then we see how the resulting similarity scores
conform to this order by deducting the percentage of papers that are out of
place in the ranking. We find that MPNet performs the best, achieving an
accuracy of 82%, which is 10 points better the second best one, SPECTER, which
gets 72%, and 18 points better than SciBERT with a score of 64%. See our data
annotation and method evaluation in the GitHub.

Paper Index | Title | SciBERT | SPECTER | MPNet
---|---|---|---|---
Pivot Paper: GPT-3 (Brown et al., 2020) |  |  |
1 (Most similar) | PaLM (Chowdhery et al., 2022) | 0.9787 | 0.8689 | 0.7679
2 | GPT-2 (Radford et al., 2019) | 0.9346 | 0.9064 | 0.8196
3 | GPT (Radford & Narasimhan, 2018) | 0.9488 | 0.8778 | 0.7790
4 | BERT (Devlin et al., 2019) | 0.9430 | 0.8321 | 0.6784
5 | Transformers (Vaswani et al., 2017) | 0.9202 | 0.8644 | 0.6385
6 | SciBERT (Beltagy et al., 2019) | 0.8396 | 0.8112 | 0.5667
7 | Latent Diffusion Models (Rombach et al., 2021) | 0.9586 | 0.7755 | 0.4567
8 | Sentiment Analysis Using DL (Fang & Zhan, 2015) | 0.7775 | 0.7298 | 0.2911
9 | Sentiment Analysis Using ML (Zainuddin & Selamat, 2014) | 0.6462 | 0.6403 | 0.2563
10 (Least similar) | New High Energy Accelerator (Courant et al., 1952) | 0.8033 | 0.5617 | 0.0359
Table 6:  An example collection of papers with monotonically decreasing
similarity to the pivot paper. As can be seen from the similarities scores
produced by the three text embedding methods, MPNet corresponds to the ground
truth the most, and also shows clear score distinctions between less similar
and more similar papers.

##  Appendix B Dataset overview

For the Semantic Scholar dataset (Kinney et al., 2023; Lo et al., 2020), we
obtain the set of 206M papers using the “Papers” endpoint to get the Paper Id,
Title, Abstract, Year, Citation Count, Influential Citation Count (Valenzuela
et al., 2015),and the Reference Count for each paper. The papers come from a
variety of fields such as law, computer science, and linguistics, chemistry,
material science, physics, geology etc. For the citation network with 2.4B
edges, we use the Semantic Scholar Citations API to get each edge of the
citation graph in a triplet format of (fromPaper, toPaper,
isInfluentialCitations).

From Figure 5, we can see that, in general, the number of publications shows
an explosive increase in recent years. The left plot shows the number of
papers publish the per year, which reaches on average 7.5M per year since
2010. The right plot shows the number of references each paper cites, which
also increases from less than five before 1970s, to around 25 in recent years.

Figure 5: The left figure illustrates the number of papers published per year,
with paper data spanning from 1684 to 2023. We can see that in recent years
since 2010, there are more than 7 million papers each year. The right figure
shows the year-wise average of the number of references per paper, also with a
sharply increasing trend. Both statistics support the need of our paper, which
helps distinguish the quality of scientific studies given such massive growths
of papers.

##  Appendix C Additional Analyses

###  C.1 Citation Outlier Analysis

For the outlier detection, we first visualize the scatter plot between our
CausalCite and citations. Then, we fit a log-linear regression to learn the
line $\log(\mathrm{TCI})=1.026\log(\mathrm{Cit})-0.541$, as shown in Figure 6,
with a root mean squared error (RMSE) of 0.6807. After fitting the function,
we use the interquartile range (IQR) method (Smiti, 2020), which identify as
outliers any samples that are either lower than the first quartile by over 1.5
IQR, or higher than the third quartile by more than 1.5 IQR, where IQR is the
difference between the first and third quartile.

We denote as overcited papers the ones that are identified as outliers by the
IQR method due to too many citations than what it should have deserved given
the CausalCite value. Symmetrically, we denote as undercited papers the ones
that are identified as outliers by the IQR method due to too few citations
than what it should have deserved given the CausalCite value. And we denote
the non-outlier papers as the aligned ones.

Figure 6: The scatter plot between our CausalCite and citations, with the
fitted function as $\log(\mathrm{TCI})=1.026*\log(\mathrm{Cit})-0.541$, and a
non-outlier band width of 0.8809.

###  C.2 Additional Information for the Author-Identified Paper Impact
Experiment

As mentioned in the main paper, the dataset is annotated by pivoting on each
paper $b$, and going through each of its references $a$ to label whether $a$
has a significant influence on $b$ or not. We show an example of paper $b$ and
all its 31 references in Table 7. We calculate the accuracy of each metric
with the spirit that each non-significant paper’s impact value should be lower
than a significant paper’s. Specifically, we go through the score of each non-
significant paper, and count its accuracy as 100% if it is lower than all the
significant papers’, or the more general form
$n_{\mathrm{lower}}/|\mathrm{Sig}|$ of conformity, where $n_{\mathrm{lower}}$
is the number of significant papers which it is lower than, and
$|\mathrm{Sig}|$ is the total number of significant papers. Then we report the
overall accuracy for each score by averaging the accuracy numbers on each non-
significant paper. To illustrate the idea better, we show the calculated
accuracy numbers for all three metrics on our example batch in Table 7.

References of the Paper “Sorting improves word-aligned bitmap indexes” | Label | PCI | Citations | SSHI
---|---|---|---|---
\- A Quantitative Analysis and Performance Study for Similarity-Search Methods in High-Dimensional Spaces | 0 | 3.519 | 1777 | 156
\- Optimizing bitmap indices with efficient compression | 0 | 3.519 | 375 | 40
\- Data Warehouses And Olap: Concepts, Architectures And Solutions | 0 | 3.526 | 187 | 11
\- Histogram-aware sorting for enhanced word-aligned compression in bitmap indexes | 0 | 3.543 | 17 | 1
\- CubiST++: Evaluating Ad-Hoc CUBE Queries Using Statistics Trees | 0 | 3.543 | 5 | 1
\- Improving Performance of Sparse Matrix-Vector Multiplication | 0 | 3.543 | 114 | 11
\- Binary Gray Codes with Long Bit Runs | 0 | 3.543 | 53 | 4
\- Analysis of Basic Data Reordering Techniques | 0 | 3.543 | 16 | 1
\- Tree Based Indexes Versus Bitmap Indexes: A Performance Study | 0 | 3.543 | 24 | 0
\- Secondary indexing in one dimension: beyond b-trees and bitmap indexes | 0 | 3.543 | 10 | 1
\- A comparison of five probabilistic view-size estimation techniques in OLAP | 0 | 3.543 | 24 | 1
\- Compression techniques for fast external sorting | 0 | 3.543 | 16 | 0
\- A Note on Graph Coloring Extensions and List-Colorings | 0 | 3.543 | 33 | 1
\- Using Multiset Discrimination to Solve Language Processing Problems Without Hashing | 0 | 3.543 | 52 | 2
\- Monotone Gray Codes and the Middle Levels Problem | 0 | 3.543 | 80 | 5
\- The Art in Computer Programming | 0 | 3.543 | 9242 | 678
\- An Efficient Multi-Component Indexing Embedded Bitmap Compression for Data Reorganization | 0 | 3.543 | 8 | 2
\- The LitOLAP Project: Data Warehousing with Literature | 0 | 3.543 | 8 | 0
\- Multi-resolution bitmap indexes for scientific data | 0 | 3.583 | 96 | 3
\- Notes on design and implementation of compressed bit vectors | 0 | 3.583 | 81 | 12
\- Compressing Large Boolean Matrices using Reordering Techniques | 0 | 3.595 | 88 | 7
\- Compressing bitmap indices by data reorganization | 1 | 3.595 | 53 | 4
\- Model 204 Architecture and Performance | 0 | 3.635 | 238 | 10
\- On the performance of bitmap indices for high cardinality attributes | 1 | 3.654 | 196 | 10
\- A performance comparison of bitmap indexes | 0 | 3.655 | 86 | 9
\- Minimizing I/O Costs of Multi-Dimensional Queries with Bitmap Indices | 0 | 3.692 | 16 | 0
\- Evaluation Strategies for Bitmap Indices with Binning | 0 | 3.692 | 69 | 3
\- C-Store: A Column-oriented DBMS | 0 | 3.710 | 1241 | 111
\- Byte-aligned bitmap compression | 0 | 3.793 | 209 | 48
\- Bit Transposed Files | 0 | 3.837 | 84 | 10
\- Space efficient bitmap indexing | 0 | 4.011 | 96 | 16
Table 7: All the reference papers for a given study “Sorting improves word-
aligned bitmap indexes.” Among all its 31 references, we boldface the
reference papers that are annotated to be significant influencers. For the
three metrics, PCI, citations, and SSHI, we report their impact scores for
each reference paper on the given study, where we mark a score in green when
it conforms to the rule that a non-significant paper’s value should be lower
than that of a significant paper, and mark a score in dark green if it
conforms to the rule to have a lower score than one of the significant paper,
but violates the rule, i.e., having a higher score than the other significant
paper. In this example, our PCI metric has an accuracy score of 79.3%, which
is higher than both citations (68.1%), and SSHI (65.0%).

###  C.3 Step Curve for PCI Values Given a Fixed Paper $b$

Apart from the long-tailed curve shape of TCI in Section 7.2, we also look
into the pairwise paper impacts by PCI. If we fix the paper $b$, we can see
that $\mathrm{PCI}(\cdot,b)$ often has a step curve shape in Figure 7. The
reason behind it lies in the nature of PCI, which is calculated based on the
top K papers that are similar in content with paper $b$, but do not cite paper
$a$. When we go through different references, e.g., from $a_{1}$ to $a_{2}$ of
the same paper $b$, the semantically matched top K papers could still be
largely the same pool, and only change when some papers in the pool need to be
swapped when releasing the constraint to be that they can cite $a_{1}$, and
adding the constraint that they cannot cite $a_{2}$.

Figure 7:  We take an example paper $b$, Sentence BERT (Reimers & Gurevych,
2019), and plot its PCI values with all its reference paper $a$’s. We can see
clearly that there is a plateau in the curve, showing a step function-like
nature.