# How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired
Analysis

Shaobo Li1, Xiaoguang Li${}^{2\hskip 0.3pt*\hskip 0.3pt}\thanks{\hskip 0.5pt
Corresponding authors<EMAIL_ADDRESS><EMAIL_ADDRESS>Lifeng Shang2, Zhenhua Dong2,
Chengjie Sun1†, Bingquan Liu1, Zhenzhou Ji1, Xin Jiang2 and Qun Liu2
1Harbin Institute of Technology
2Huawei Noah’s Ark Lab
<EMAIL_ADDRESS>{sunchengjie, liubq<EMAIL_ADDRESS>
{lixiaoguang11, shang.lifeng, dongzhenhua, Jiang.Xin<EMAIL_ADDRESS>
Authors contribute equally. Corresponding authors<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recently, there has been a trend to investigate the factual knowledge captured
by Pre-trained Language Models (PLMs). Many works show the PLMs’ ability to
fill in the missing factual words in cloze-style prompts such as “Dante was
born in [mask].” However, it is still a mystery how PLMs generate the results
correctly: relying on effective clues or shortcut patterns? We try to answer
this question by a causal-inspired analysis that quantitatively measures and
evaluates the word-level patterns that PLMs depend on to generate the missing
words. We check the words that have three typical associations with the
missing words: knowledge-dependent, positionally close, and highly co-
occurred. Our analysis shows: (1) PLMs generate the missing factual words more
by the positionally close and highly co-occurred words than the knowledge-
dependent words; (2) the dependence on the knowledge-dependent words is more
effective than the positionally close and highly co-occurred words.
Accordingly, we conclude that the PLMs capture the factual knowledge
ineffectively because of depending on the inadequate associations.

##  1 Introduction

Do Pre-trained Language Models (PLMs) capture factual knowledge? LAMA
benchmark Petroni et al. (2019) answers this question by quantitatively
measuring the factual knowledge captured in PLMs: query PLMs with cloze-style
prompts such as “Dante was born in [mask]?” Filling in the mask with the
correct word “Florence” is considered a successful capture of the
corresponding factual knowledge. The percentage of correct fillings over all
the prompts can be used to estimate the amount of factual knowledge captured.
PLMs show a surprisingly strong ability to capture factual knowledge in such
probings Jiang et al. (2020); Shin et al. (2020); Zhong et al. (2021), which
elicits further research on a more in-depth question Cao et al. (2021); Elazar
et al. (2021a): How do PLMs capture the factual knowledge? In this paper, we
try to answer this question with a two-fold analysis:

###### Research Question 1

Which association do PLMs depend on to capture factual knowledge?

###### Research Question 2

Is the association on which PLMs depend effective in capturing factual
knowledge?

Knowledge-Dependent:
---
Columbus born between 25 August and 31 October 1451, died 20 May 1506 was an
Italian explorer.
Positionally Close:
Columbus born between 25 August and 31 October 1451, died 20 May 1506 was an
Italian explorer.
Highly Co-occurred:
Columbus born between 25 August and 31 October 1451, died 20 May 1506 was an
Italian explorer.
Figure 1: The associations we investigated. The underlined words are the
missing words that need to be generated. The bold words, which hold specific
associations with the missing words, are considered as the word-level patterns
that PLMs may use to generate the missing words.

(a) Dependence measure.

(b) Effectiveness measure.

Figure 2: The overview of the proposed analysis framework. The dependence
measure quantifies how much the PLMs depend on each association to capture
factual knowledge when per-training. The effectiveness measure evaluates
whether the dependence on an association is good for the factual knowledge
performance in probing.

We use association to refer to the explicit association between the missing
words and the remaining words in the context. We define three typical
associations between words. Figure 1 illustrates these associations in a mask-
filling sample.

###### Definition 1

Knowledge-Dependent (KD): According to a Knowledge Base (KB), the missing
words can be deterministically predicted when providing the remaining words.

###### Definition 2

Positionally Close (PC): The remaining words are positionally close to the
missing words.

###### Definition 3

Highly Co-occurred (HC): The remaining words have a higher co-occurrence
frequency with the missing words.

Question 1 investigates how much PLMs depend on a specific group of remaining
words to predict the missing words in pre-training samples. We select the
remaining words to be investigated according to their association with the
missing words. We propose a causal-inspired method to quantify the word-level
dependence in each sample. The average dependence on the remaining words that
hold the same association with the missing words over all the samples
indicates how PLMs rely on this association to predict the missing words. We
refer to this average dependence as dependence on the association. The above
analysis is named dependence measure.

In Question 2, we reveal the effectiveness of dependence by the correlation
between the quantified dependence on associations and the factual knowledge
capturing performance. The performance is probed with additionally crafted
cloze-style promptsElazar et al. (2021a). The more the dependence on an
association positive correlates with the probing performance, the more
effective this association is. We refer to the second analysis as
effectiveness measure. According the experiment results, we have the following
observations:

###### Observation 1

The PLMs depend more on the positional close and highly co-occurred
associations than the knowledge-dependent association to capture factual
knowledge.

###### Observation 2

Depending on the knowledge-dependent association is more effective for factual
knowledge capture than positional close and highly co-occurred associations.

By connecting the two observations, we can answer the question of “how PLMs
capture factual knowledge” as: The PLMs are capturing factual knowledge
ineffectively since the PLMs depend more on the PC and HC association than the
KD association.

The contribution of this paper can be summarized as follows: (1) We quantify
the word-level dependence for mask filling with a causal-inspired method,
revealing the word-level patterns that PLMs use to predict the missing words
quantitatively. (2) We compare the effectiveness of the dependence on
different associations, which provides direct insights for improving PLMs for
factual knowledge capture. (3) This paper introduces causal theories into PLMs
by formulating the effect measurement process in mask language modeling. It
paves the path to measure the causal effects between entities or events
described in natural language.

Association | Knowledge-Dependent | Positionally Close | Highly Co-occurred
---|---|---|---
Input |  $W_{t}$ born between 25 August and 31 October 1451, $W_{t}$ $W_{o}$ was an Italian explorer. |  Columbus born between 25 August and 31 October 1451, $W_{t}$ $W_{o}$ $W_{t}$ an Italian explorer. |  $W_{t}$ born between 25 August and 31 October 1451, died $W_{o}$ was an Italian $W_{t}$.
SCM |  |  |
Table 1: To analyze the dependence of associations, we do interventions on
treatment words to reveal their causal effects on the outcome words.

##  2 Method

###  2.1 Overview

We take a quick overview of our two-fold analysis with a running example in
Figure 2. Figure 2(a) illustrates how to measure the dependence on the
remaining words “Columbus” and “died” when predicting the missing words “20
May 1506.” We let the PLM generate the missing words based on the original
input first, then mask the remaining words in the input and let PLMs generate
again. The difference between these two predictions is quantified and used to
measure the dependence. The remaining and missing words hold the knowledge-
dependent association in this sample. We repeat this measure on all the
samples whose remaining and missing words have the KD association. Then the
dependence on the KD association can be estimated by the average of the
quantified difference.

Figure 2(b) measures the effectiveness of the dependence on each association
by calculating the correlation coefficient between the dependence and the
probing performance. Following Petroni et al. (2019); Elazar et al. (2021a),
the probing performance is indicated by the prediction accuracy and
consistency when querying on the same fact with different prompts. Since the
dependence on associations are quantified in the dependence measure, we can
calculate the correlation coefficient between the dependence and performance
over all the samples. Their correlation measures whether the dependence on an
association is harmful or beneficial to the performance, showing the
effectiveness of the dependence on an association quantitatively.

Section Outline We organize the rest of this section as follows. In Section
2.2.1, we formalize how we quantify the dependence with the causal effect
estimation. Section 2.2.2 gives detail about how we build the probing samples
for different associations. Section 2.3.1 introduces the metrics we used to
indicate the performance of factual knowledge capture. Section 2.3.2 describes
the details about the effectiveness measure of associations.

###  2.2 Quantify the Dependence on Associations

####  2.2.1 Causal Effect Estimation for PLMs

To study the causal effect of the different input words, we build a Structured
Causal Model (SCM) for the missing words generation process and apply
interventions on some input words to estimate their effect quantitatively. We
consider the missing words as outcome words and the remaining words that hold
a certain association (e.g. positionally close) with the outcome words as
treatment words. Then, we can formally represent the word generation process
with SCM, as the following structural equations:

$\begin{split}&w_{c}=f(\mathbb{I}),w_{t}={\rm PLM}(w_{c})\\\ &w_{o}={\rm
PLM}(w_{c},w_{t}).\end{split}$ (1)

Separate the words in a sentence into three groups: treatment words $W_{t}$,
outcome words $W_{o}$, and context words $W_{c}$ (specified by $w_{t}$,
$w_{o}$, and $w_{c}$ respectively). Equation 1 formulates the following data
generation process: (1) Sample a sentence from the natural text space
$\mathbb{I}$ and get the context words $w_{c}$ using function $f$. (2)
Generate the treatment words $w_{t}$ by the PLM based on $w_{c}$ only. (3)
Generate outcome words $w_{o}$ based on both the $w_{c}$ and $w_{t}$.

To obtain the quantitative causal effect of treatment words $W_{t}$ on outcome
words $W_{o}$, we apply the do-calculus $do()$ Pearl (2009) on treatment words
$W_{t}$ to introduce interventions for estimating the causal effect. $do()$
denotes the operation of forcibly setting the value of $W_{t}$. Then the
causal effect of $W_{t}$ on $W_{o}$ can be estimated by the Average Treatment
Effect (ATE) Rubin (1974):

$\begin{split}&\mathbb{E}\left[P(W_{o}|do(W_{t}=\hat{w}_{t}))\right]\\\
-&\mathbb{E}\left[P(W_{o}|do(W_{t}=w_{m}))\right].\end{split}$ (2)

Accordingly, we define ATE for PLMs as:

$\begin{split}\tau&=\sum_{\mathbb{I}}{\rm
PLM}\left(do(W_{t}=\hat{w}_{t}),w_{c}\right)P(s)\\\ &-\sum_{\mathbb{I}}{\rm
PLM}\left(do(W_{t}=w_{m}),w_{c}\right)P(s),\end{split}$ (3)

where $\hat{w}_{t}$ is the ground truth of the treatment words $W_{t}$ (the
original value of $W_{t}$ without intervention). $w_{m}$ is the intervention
value (several [mask]s) for $W_{t}$, and we use it to simulate removing the
ground-truth value $\hat{w}_{t}$ from the input. $P(s)$ denotes the
probability of selecting the sample $s$ that consists of $w_{t}$, $w_{o}$, and
$w_{c}$ from $\mathbb{I}$. $\rm PLM(\cdot,\cdot)$ denotes the output of PLMs
with certain input. Table 1 illustrates the interventions on the SCM for
different associations.

The raw output of PLMs is a probability distribution over fixed vocabulary. We
transform the output into reciprocal rank to quantify the differences:

${\rm
PLM}_{k}{(w_{t},w_{c})}=\begin{cases}\frac{1}{rank_{\hat{w}_{o}}},&\text{if
}rank_{\hat{w}_{o}}\leq k\\\ 0,&\text{otherwise}\end{cases}.$ (4)

$\hat{w}_{o}$ is the ground-truth outcome words. $rank_{\hat{w}_{o}}$ is the
rank position of $\hat{w}_{o}$ according to the generation probability of
$\hat{w}_{o}$ output by ${\rm PLM}(w_{c},w_{t})$. We set $k$ to 100 and use
${\rm PLM_{100}}$ to replace ${\rm PLM}$ in Equation 3 to calculate ATE. The
ATE reflects the effect of $W_{t}$ on the prediction of $W_{o}$, it can be
regarded as a quantitative estimation of how much PLMs depend on $W_{t}$ when
generating $W_{o}$.

####  2.2.2 Mark words by Associations

Wikipedia is a rich source of knowledge Thom et al. (2007); Hassanzadeh
(2021), and most of the PLMs nowadays have been pre-trained on Wikipedia
Devlin et al. (2019); Liu et al. (2019); Lan et al. (2019), so we take
Wikipedia sentences as pre-training samples to construct the probing samples
for dependence measure. We probe the mask-filling on these sentences to
analyze what PLMs based on when capturing factual knowledge in pre-training.

The outcome we want to observe is the predictions of factual words in the
sentences. In order to locate the factual words, we align each sentence with a
triplet (subject, predicate, object) in the KB. The words that correspond to
the object serve as outcome words $W_{o}$ for observation, and the remaining
words that hold an explicit association with $W_{o}$ are marked as treatment
words $W_{t}$ for intervention. For different associations, the $W_{t}$ is
identified as:

  1. 1.

Knowledge-Dependent: all the remaining words correspond to the predicate and
object in the same triplet with the $W_{t}$.

  2. 2.

Positionally Close: the remaining words closest to $W_{o}$.

  3. 3.

Highly Co-occurred: the remaining words that have higher Pointwise Mutual
Information (PMI) Church and Hanks (1990) with $W_{o}$. The PMI is calculated
over all the Wikipedia sentences using the following equation:

${\rm PMI}(w_{i};\hat{w}_{o})=\frac{P(w_{i}|\hat{w}_{o})}{P(w_{i})},$ (5)

where $\hat{w}_{o}$ is a group of words (a span) and $w_{i}$ is a single word.

  4. 4.

We further define a Random (R) association, where the $W_{t}$ are randomly
selected remaining words. It provides some empirical support for how much the
modifications in the context affect the mask-filling output.

Accordingly, one sentence yields four probing samples for the four
associations, respectively. The four probing samples share the same $W_{o}$
but use different words as $W_{t}$ to show the dependence on different
associations when predicting the same $W_{o}$. We preserve that the number of
words in $W_{t}$ is the same among different associations. For example, if
there are two words used as $W_{t}$ by the KD association, we select the top
two closest words with $W_{o}$ as $W_{t}$ for PC, and the words have the top
two PMI with $W_{o}$ for HC. We can obtain a set of probing samples for each
association. The sample sets for different associations source from the same
set of sentences and have the same size.

###  2.3 Measure the Effectiveness of Associations

This section investigates which association can lead to better performance on
factual knowledge capture. We first define the metrics to evaluate the
performance, then we measure the effectiveness of an association by relating
the dependence on this association with probing performance.

####  2.3.1 Metrics for Factual Knowledge Probing

Section 2.2 uses the original Wikipedia sentence as pre-training samples to
quantify the dependence PLMs used to capture the corresponding fact in pre-
training. The performance of capturing the corresponding fact is probed by
having PLMs fill masks on crafted quires. We construct these queries by
instantiating templates on triplets Petroni et al. (2019). $T_{i}(s)$ denotes
the $i$-th query for the fact corresponds to $s$. The accuracy ${\rm mrr}$ of
capturing this fact is obtained by averaging over all the predictions obtained
with different queries:

${\rm mrr}\left(s\right)=\frac{1}{n}\sum_{i}^{n}{\rm PLM}_{k}({T_{i}(s)}),$
(6)

${\rm PLM}_{k}({T_{i}(s)})$ denotes the reciprocal rank of the ground truth in
the PLM’s output for query $T_{i}(s)$, it is defined in Equation 4.

The consistency of the capture is indicated by the percentage of the pairs of
queries that have the same result Elazar et al. (2021a):

${\rm con}\left(s\right)=\frac{\sum_{i\neq j}\mathds{1}_{{\rm
PLM}({T_{i}(s)})={\rm PLM}(T_{j}(s))}}{n(n-1)}.$ (7)

There are $n$ different queries on every fact, and we can get
$\binom{n}{2}=n(n-1)$ pairs of predictions in total. ${\rm PLM}({T_{i}(s)})$
denotes the top-1 output for the query $T_{i}(s)$. The value of
$\mathds{1}_{{\rm PLM}({T_{i}(s)})={\rm PLM}(T_{j}(s))}$ is an indicator
function that takes the value 1 if the PLMs returns identical at top-1 for
${T_{i}(s)}$ and ${T_{j}(s)}$ and 0 otherwise. The PLMs are better on the
consistency metric if they keep the predictions consistent when queries only
vary on the surface forms. _E.g_., the two queries “Dante was born in [mask]”
and “The birthday of Dante is [mask]” should return the same results.

Finally, we evaluate the factual knowledge capture performance by jointly
examining the accuracy and consistency Elazar et al. (2021a):

${\rm test}(s)={\rm mrr}\left(s\right)\cdot{\rm con}\left(s\right)$ (8)

${\rm test}(s)$ measures the probing performance on template-based queries. We
also define a metric to measure how well the PLMs memorize the missing words
in pre-training samples (Wikipedia sentences):

${\rm train}(s)={\rm PLM}_{k}(s).$ (9)

####  2.3.2 Correlate Performance with Dependence

We have quantified the dependence on each association and defined the metrics
for probing performance in the above sections. We then calculate the Pearson
correlation coefficient Kirch (2008) between dependence and probing
performance to reveal the effectiveness of different associations. An
association is considered more effective if the probing performance positively
correlates with its dependence more.

Because only part of the facts has available templates, the samples in the
dependence measure without templates are ignored in the calculation. The
factual knowledge captured by different PLMs may vary significantly due to the
differences in model scale, pre-training data, or other settings. To make the
correlation coefficient comparable between different PLMs, we calculate the
correlation only on the factual knowledge gathered correctly by the PLM.
_I.e_., only the pre-training samples with ${\rm train}(s)={\rm PLM}_{k}(s)=1$
are involved.

Sample in Dependence Measure
---
# Wikipedia sentences | 4,779,753
# Different triplets | 3,795,229
# Different predicates | 565
# Sentences with synthetic templates | 1,119,875
Queries in Effectiveness Measure
# Template-based queries | 7,645,635
# Different triplets | 654,112
# Different predicates | 38
# Different templates | 328
Table 2: Statistics of the probing data.

##  3 Experiments and Discussions

Model | Accuracy |  Dependences on Associations ($k=100$)
---|---|---
KD | PC | HC | R
BERT-base-cased | 0.3623 | 0.1585 | 0.4085 | 0.1779 | 0.1081
BERT-large-cased | 0.3692 | 0.1603 | 0.4113 | 0.1791 | 0.0996
BERT-large-cased-wwm | 0.5030 | 0.1384 | 0.4477 | 0.2305 | 0.1072
SpanBERT-large | 0.5223 | 0.1351 | 0.3679 | 0.2383 | 0.1157
RoBERTa-base | 0.3511 | 0.1352 | 0.3926 | 0.2093 | 0.1053
RoBERTa-large | 0.4276 | 0.1360 | 0.3962 | 0.2162 | 0.0985
BERT-large-uncased-wwm | 0.5035 | 0.1410 | 0.4350 | 0.2290 | 0.1089
ALBERT-xxlarge-v2 | 0.4758 | 0.2852 | 0.4338 | 0.3801 | 0.2704
Table 3: The quantified dependence on associations. Accuracy denotes the
performance of filling in the masks in pre-training samples. The PLMs use
cased and uncased vocabularies are separated.

(a) BERT-large-cased-wwm

(b) RoBERTa-large

(c) SpanBERT-large

(d) ALBERT-xxlarge-v2

Figure 3: The correlations between the dependence on associations and the probing performance on factual knowledge capture. |  Pre-training samples for the dependence measure (Wikipedia Sentence) | Dep.
---|---|---
|  KD: Kimwenza is a community in the Democratic Republic of the Congo in the Mont Ngafula commune in the south of the capital, Kinshasa . | 0.8564
Case 1 |  PC: Kimwenza is a community in the Democratic Republic of the Congo in the Mont Ngafula commune in the south of the capital, Kinshasa . | 0.0000
HC: Kimwenza is a community in the Democratic Republic of the Congo in the Mont Ngafula commune in the south of the capital, Kinshasa . | 0.0000
Template-based Queries for the effectiveness measure | MRR
The capital of Congo is Kinshasa . | 1.0
Kinshasa is the capital of Congo . | 1.0
Congo’s capital is Kinshasa . | 1.0
Pre-training samples for the dependence measure (Wikipedia Sentence) | ATE
|  KD: Drayton is a hamlet in England, in the county of Northamptonshire, $\ldots$, hundred of Fawsley, ¾ of a mile on the low-lying north western side of the town of Daventry . | 0.0000
Case 2 |  PC: Drayton is a hamlet in England, $\ldots$, in the parish and union of Daventry, hundred of Fawsley, ¾ of a mile on the low-lying north western side of the town of Daventry . | 0.9496
HC: Drayton is a hamlet in England, $\ldots$, in the parish and union of Daventry, hundred of Fawsley, ¾ of a mile on the low-lying north western side of the town of Daventry . | 0.8452
Template-based Queries for the effectiveness measure | MRR
Drayton is located in Daventry . | 0.0
Drayton is in Daventry . | 0.0
Drayton can be found in Daventry . | 0.0
Table 4: Two cases from SpanBERT-large. The quantified dependence on
associations (denoted by Dep.) and the performance of factual knowledge
capture (denoted by MRR).

###  3.1 Probing Data and PLMs

We use the TREX dataset Elsahar et al. (2018), which aligns KB triplets with
Wikipedia sentences, to construct the samples for the dependence measure
following the definition in Section 2.2.2. We employ the templates from Elazar
et al. (2021a) to construct the queries to probe the factual knowledge for the
effectiveness measure. Table 2 shows the statistics for the data in the
dependence measure and the effectiveness measure. The PLMs we analyzed include
BERT Devlin et al. (2019), RoBERTa Liu et al. (2019), SpanBERT Joshi et al.
(2020), and ALBERT Lan et al. (2019).

###  3.2 Dependence on Associations

The dependence on an association is the average ATE over the probing samples
whose treatment words hold that association with the outcome words. Table 3
shows the quantified dependence of different associations. The accuracy in
Table 3 represents the accuracy of recovering the masked factual words in pre-
training samples, revealing how well does PLMs memorize the pre-training
samples. It is calculated by Equation 9 with $k=1$.

We find a general trend over all the picked PLMs: the Positionally Close (PC)
association takes the dominant effect on the prediction results, the Highly
Co-occurred (HC) association comes second, and the least for the Knowledge-
Dependent (KD) association. The trend does not change much as increasing the
model scale (large _vs_. base), using additional training data (RoBERTa _vs_.
BERT), or improving the masking strategy (SpanBERT _vs_. BERT). Consequently,
the accuracy drops the most when perturbing the positionally close words but
least on knowledge-dependent words111Table 6 in the Appendix shows the
accuracy decrease when perturbing the different associations..

The results provide quantitative evidence for Question 1 of “Which association
do PLMs depend on to capture factual knowledge?:” PLMs prefer the associations
founded with positionally close or the highly co-occurred words to the
knowledge-based clues. It is different from how a conventional KB works, e.g.,
an object can be retrieved by the corresponding subject and predicate.

###  3.3 Correlations between Dependence and Performance

We show the correlation between association’s dependence and the probing
performance in Figure 3. Each point in the figure represents a piece of
factual knowledge $s$. We refer to it as a fact for convenience. The
horizontal axis indicates ${\rm test}(s)$ for the fact, showing the probing
performance of the fact with effectiveness measure. The vertical axis shows
the dependence of associations when capturing this fact, which is quantified
by the causal effect estimation defined in Section 2.2.1. The straight lines
are the regression lines and different associations are shown in different
line styles222We standardize the quantified value of dependence (denoted as
Std. Dependence) and plot a bucket of facts as a single point to show the
trends clearly. The correlations without standardization for more PLMs are in
Table 8.

As we can see from the results, the dependence on the KD association
positively correlates with the probing performance. The dependence on the HC
association has a slightly positive correlation or almost has no correlations
sometimes (such as ALBERT in Figure 3(d)). The PC association holds a negative
correlation with the performance.

These results can give an empirical answer to “Is the association on which
PLMs depend effective in capturing factual knowledge?:” the more PLMs depend
on the Knowledge-Dependent (KD) association, the better PLMs can capture the
corresponding factual knowledge. Meanwhile, relying much on the positionally
close association is harmful to the probing performance.

The dependence measure results reveal that the PLMs depend most on the
positionally close but least on the knowledge-dependent association. However,
in effectiveness measure, we find that the positionally close association is
the most ineffective for factual knowledge capture while the knowledge-
dependent association is the most effective. By connecting the two results, we
can conclude the answer to the question in the title: The PLMs do not capture
factual knowledge ideally, since they depend more on the ineffective
associations than the effective one.

###  3.4 Case Study

To illustrate the analysis result intuitively, we show two cases with
SpanBERT-large in Table 4. The MRR shows the probing performance on the
template-based query (calculated by Equation 6). In Case 1, the knowledge-
dependent association gains the biggest effect, and the predictions are robust
in all the template-based probing. However, the positionally close association
takes the main effect in Case 2, while the PLM fails to recall the word
“England” with the template-based queries.

(a) BERT-large-cased-wwm

(b) RoBERTa-large

(c) SpanBERT-large

(d) ALBERT-xxlarge-v2

Figure 4: The correlations between the dependence on associations and the
mask-filling accuracy on the pre-training samples (Wikipedia sentences).

###  3.5 Discussions

Generality of the Proposed Probing Method Generally, the dependence measure
offers a way to measure how much the word-level patterns cause the prediction
of missing words in Mask Language Model (MLM). Because words are readable,
directly visible, and can be manipulated from the input side directly, the
word-level patterns can provide more intuitive interpretations than numeric
representation vectors Elazar et al. (2021b) or neurons Vig et al. . We use
the proposed method to estimate the causal effect of three typical
associations in this paper, while this method can be easily adapted to
quantify the dependence on any word-level patterns.

Reconsidering “PLM as KB” If we want to use a PLM like a KB, whether the PLM
has the same inner workflow as KBs deserves to be considered. The prevalent
KBs index knowledge as subject-predicate-object triplets and can infer with
triplets Speer et al. (2017); Bollacker et al. (2008); Vrandečić and Krötzsch
(2014). However, we find out that the knowledge-dependent association, which
represents the process of inferring a missing object based on the given
subject and predicate, has the lowest dependence in the PLMs. It provides
evidence that the PLMs work quite differently with KBs and can not serve
stably as KBs for now.

Overfiting and Generalization Figure 4 shows the correlations between the
dependence on associations and the mask-filling accuracy on pre-training
samples (referred to as memorizing accuracy). The memorizing accuracy
increases most as the dependence of the PC association increases,
demonstrating that the more PLMs depend on the positionally close words, the
better PLMs can recover the pre-training samples. However, there is an
opposite trend in probing performance as shown in Figure 3. The additionally
crafted queries used to evaluate the probing performance are mostly unseen in
per-training. If we consider these queries as the test set and the pre-
training samples as the train set, we can conclude that the dependence on the
PC associations makes the PLMs tend to overfit the training data and degrade
the generalization on the test set.

Factual Knowledge Capture in Pre-training We want to focus on the pre-training
samples that help PLMs to capture factual knowledge. So we reconstruct the
pre-training samples that predict some missing factual words (object) based on
the factual clues (subject, predicate). We conduct the dependence measure on
these samples to investigate how the factual knowledge is captured in pre-
training. The mask-filling accuracy on these pre-training samples denotes how
well PLMs memorize them in pre-training. We name it as “train” in Equation 9
and “Memorizing Accuracy” in Figure 4.

Overlap between Associations The clues for different associations overlap
sometimes, e.g., some remaining words may hold the KD and PC associations with
the missing words at the same time. The overlaps do not impair the estimations
because we use a set of samples to estimate the effect of each association.
The samples that hold the same association stay in the same set, and the
average causal effect in all these samples is the quantified dependence of
this association. The sample sets are quite different for different
associations. Table 5 shows the corresponding statistics of the overlaps.

##  4 Related Works

Probing Factual Knowledge in PLMs Factual Knowledge Probing in PLMs has
attracted much attention recently. LAMA Petroni et al. (2019) propose a
benchmark that probes the factual knowledge in the PLMs with cloze-style
prompts and shows PLMs’ ability to capture factual knowledge. This ability can
be further explored by tuning the prompts for probing Jiang et al. (2020);
Shin et al. (2020); Zhong et al. (2021).

Motivated by the probing results, some recent works analyze the captured
factual knowledge from more perspectives. Cao et al. (2021) analyze the
distribution of predictions and the answer leakage in probing. Poerner et al.
(2020) propose that the PLMs could predict based on some correlation between
surface forms rather than infer according to facts. Elazar et al. (2021a)
reveal that the PLMs’ outputs are inconsistent as querying the same fact with
different prompts.

This paper proposes a more fine-grained inspection of word-level patterns in
the input. In addition to constructing more challenging probing data as input
or analyzing the outputs more detailedly, we try to reveal the inner mechanism
of PLMs by conducting intervention on the input and then observing the change
in the output.

Causal-inspired Interpretations in NLP A causal-inspired approach to
explanation is to generate counterfactual examples and then compare the
predictions Feder et al. (2021a). Feder et al. (2021b) propose a framework for
producing explanation for NLP models using counterfactual representation. Vig
et al.  analyze the effect of neurons (or attention heads) on the gender bias
using causal mediation analysis. In this paper, we revisit the word-level
post-hot interpretation Sun et al. (2021); Li et al.  from a causal-effect
perspective: intervene on some specified words in the input and measure the
difference in the output to estimate the causal effect of these words.
Furthermore, we evaluate the effectiveness of different causes by calculating
correlations between their effects and performance. As far as we know, our
work is the first study to probe and evaluate word-level patterns in the
factual knowledge capture task.

##  5 Conclusion

In this paper, we try to answer the question of How Pre-trained Language
Models Capture Factual Knowledge by measuring and evaluating different
associations that PLMs use to capture factual knowledge. We present three
word-level associations, knowledge-dependent, positionally close, and highly
co-occurred in the analysis. The analysis results show that the PLMs rely more
on the ineffective positionally close and highly co-occurred associations when
capturing factual knowledge, and somewhat ignore the effective knowledge-
dependent clues. These findings indicate that we should pay more attention to
the knowledge-dependent association to let PLMs capture factual knowledge
better.

## References

  * Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008\.  Freebase: a collaboratively created graph database for structuring human knowledge.  In _Proceedings of the 2008 ACM SIGMOD international conference on Management of data_ , pages 1247–1250.
  * Cao et al. (2021) Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021.  Knowledgeable or educated guess? revisiting language models as knowledge bases.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1860–1874, Online. Association for Computational Linguistics.
  * Church and Hanks (1990) Kenneth Ward Church and Patrick Hanks. 1990.  Word association norms, mutual information, and lexicography.  _Computational Linguistics_ , 16(1):22–29.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Elazar et al. (2021a) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021a.  Measuring and improving consistency in pretrained language models.  _Transactions of the Association for Computational Linguistics_ , 9:1012–1031.
  * Elazar et al. (2021b) Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021b.  Amnesic probing: Behavioral explanation with amnesic counterfactuals.  _Transactions of the Association for Computational Linguistics_ , 9:160–175.
  * Elsahar et al. (2018) Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018.  T-rex: A large scale alignment of natural language with knowledge base triples.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Feder et al. (2021a) Amir Feder, Katherine A Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E Roberts, et al. 2021a.  Causal inference in natural language processing: Estimation, prediction, interpretation and beyond.  _arXiv preprint arXiv:2109.00725_.
  * Feder et al. (2021b) Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. 2021b.  Causalm: Causal model explanation through counterfactual language models.  _Computational Linguistics_ , 47(2):333–386.
  * Hassanzadeh (2021) Oktie Hassanzadeh. 2021.  Predicting the future with wikidata and wikipedia.  In _Proceedings of the ISWC_.
  * Jiang et al. (2020) Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020.  How can we know what language models know? _Transactions of the Association for Computational Linguistics_ , 8:423–438.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020.  SpanBERT: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77.
  * Kassner et al. (2021) Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021.  Multilingual LAMA: Investigating knowledge in multilingual pretrained language models.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 3250–3258, Online. Association for Computational Linguistics.
  * Kirch (2008) Wilhelm Kirch, editor. 2008.  _Pearson’s Correlation Coefficient_, pages 1090–1091. Springer Netherlands, Dordrecht.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_.
  * (16) Jiwei Li, Will Monroe, and Dan Jurafsky.  Understanding neural networks through representation erasure.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Pearl (2009) Judea Pearl. 2009.  _Causality_.  Cambridge university press.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Poerner et al. (2020) Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020.  E-BERT: Efficient-yet-effective entity embeddings for BERT.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 803–818, Online. Association for Computational Linguistics.
  * Rubin (1974) Donald B Rubin. 1974.  Estimating causal effects of treatments in randomized and nonrandomized studies.  _Journal of educational Psychology_ , 66(5):688.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020.  AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4222–4235, Online. Association for Computational Linguistics.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _Thirty-first AAAI conference on artificial intelligence_.
  * Sun et al. (2021) Xiaofei Sun, Diyi Yang, Xiaoya Li, Tianwei Zhang, Yuxian Meng, Qiu Han, Guoyin Wang, Eduard Hovy, and Jiwei Li. 2021.  Interpreting deep learning models in natural language processing: A review.  _arXiv preprint arXiv:2110.10470_.
  * Thom et al. (2007) James A Thom, Jovan Pehcevski, and Anne-Marie Vercoustre. 2007.  Use of wikipedia categories in entity ranking.  _arXiv preprint arXiv:0711.2917_.
  * (26) Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber.  Investigating gender bias in language models using causal mediation analysis.
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: a free collaborative knowledgebase.  _Communications of the ACM_ , 57(10):78–85.
  * Zhong et al. (2021) Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021.  Factual probing is [MASK]: Learning vs. learning to recall.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5017–5033, Online. Association for Computational Linguistics.

## Appendix

##  Appendix A Probing Sample Construction for the Dependence Measure

We detail how we construct the probing samples for the dependence measure as
follows. We take a subject-predicate-object triplet in
Wikidata333https://www.wikidata.org/ as a piece of factual knowledge Petroni
et al. (2019); Cao et al. (2021); Kassner et al. (2021); Elazar et al.
(2021a). A subject-predicate-object triplet is aligned with a Wikipedia
sentence by matching the subject, predicate, and object with their
corresponding spans, respectively. A subject-predicate-object triplet is
aligned with a Wikipedia sentence by matching the subject, predicate, and
object with their corresponding spans, respectively. The words that correspond
to the object are the factual words that are masked and need to be predicted,
and we investigate how the different remaining words contribute to the
prediction.

The remaining words that have three typical associations with the masked words
are considered in the analysis. The rules to identify the remaining words that
have the Knowledge-Dependent (KD) association are:

  1. 1.

The $W_{o}$ and the $W_{t}$ describe the same subject-predicate-object triplet
in KB.

  2. 2.

The $W_{t}$ are the natural language description of the subject and predicate,
the $W_{o}$ are that for the object.

  3. 3.

If the subject and predicate corresponding to $W_{t}$ are given correctly
(_i.e_., $W_{t}=\hat{w}_{t}$), the ground-truth value of the object is unique
in the KB.

The first rule makes the $W_{t}$ and $W_{o}$ grounded in the same piece of
factual knowledge. The second rule makes predicting the outcome words similar
to inferring the object using a KB when giving the subject and predicate. The
third rule means if the treatment words are given correctly, there should be
one and only one ground-truth value for the object. The third rule is similar
to the N-1 relationship Cao et al. (2021) in KB and lets the ground-truth
treatment words can be regarded as a sufficient condition to predict the
unique outcome words deterministically.

We use the T-REx444https://hadyelsahar.github.io/t-rex/ dataset to provide the
initial alignment between KB triplets and the Wikipedia sentences. We use the
aliases in KB as keys for fuzzy string match (Levenshtein distance is less
than 2, stemming before matching, etc.) to align more subjects, predicates,
and objects with spans in the sentence. The sentences that have no aligned
triplet are filtered out.

Sometimes, the outcome words in a single sentence relate to multiple triplets
that satisfy the rules for KB. E.g., there are two groups of remaining words
that can infer the outcome words deterministically based on KB. We select them
all as the $W_{t}$ when probing the KB association and keep the number of the
masked words be the same in interventions for the other associations. $D_{\rm
KD}$, $D_{\rm PC}$, and $D_{\rm HC}$ denote the sample sets for the Knowledge-
Dependent (KD), Positionally Close (PC), and Highly Co-occurred (HC)
associations, respectively.

For the Highly Co-occurred association (HC), the remaining words that are
top-$k$ in PMI with the ground-truth outcome words $\hat{w}_{o}$ are selected
as $W_{t}$. The $k$ is the number of words with the KD associations for the
same sentence. The PMI between words is calculated by all the Wikipedia
sentences. If the $\hat{w}_{o}$ consists of multiple words, occurring with all
the words in $\hat{w}_{o}$ altogether are taken as co-occurring. The order of
the words in $\hat{w}_{o}$ are ignored for efficiency. Table 5 shows more
details about the probing samples.

Probing Samples in Dependence Measure |
---|---
# Average treatment words | 4.0023
# Average outcome words | 1.8588
# Average words | 23.1031
Word-level Overlap Between Associations |
$D_{\rm KD}\cap D_{\rm HC}$ | 44.25% (8,455,641)
$D_{\rm KD}\cap D_{\rm PC}$ | 20.83% (3,981,305)
$D_{\rm PC}\cap D_{\rm HC}$ | 18.75% (3,582,576)
$D_{\rm KD}\cap D_{\rm PC}\cap D_{\rm HC}$ | 8.3349% (1,592,560)
Sample-level Overlap Between Associations |
$D_{\rm KD}\cap D_{\rm HC}$ | 3.12% (149,479)
$D_{\rm KD}\cap D_{\rm PC}$ | 0.06% (2,995)
$D_{\rm PC}\cap D_{\rm HC}$ | 0.12% (5,777)
$D_{\rm KD}\cap D_{\rm PC}\cap D_{\rm HC}$ | 0.0001% (547)
Template-based Queries in Effectiveness Measure |
# Average treatment words | 1.9484
# Average outcome words | 1.5844
# Average word per sample | 6.9617
Table 5: Statistic of the probing data in the dependence measure.

##  Appendix B More Probing Results

The Pearson correlation coefficients between the dependence on associations
(raw value without standardization) and the performance are shown in Table 8.
The three metrics, accuracy (defined in Equation 6), consistency (defined in
Equation 7), and the overall performance metric (defined in Equation 8), are
reported respectively. The correlation coefficients between the dependence and
the performance are consistent with the slopes of the regression lines in
Figure 3. Table 6 shows the accuracy decreasing results after masking the
treatments words when generating the missing words in Wikipedia sentences.

Model |  Input Context (Accuracy, %)
---|---
Complete | w/o KD | w/o PC | w/o HC | w/o R
BERT-base-cased | 36.23 | 22.05 (-14.17) |  2.67 (-33.56) | 19.68 (-16.54) | 26.71 ( -9.52)
BERT-large-cased | 36.92 | 22.11 (-14.81) |  2.70 (-34.22) | 19.04 (-17.88) | 27.11 ( -9.81)
BERT-large-cased-wwm | 50.30 | 35.22 (-15.08) | 11.50 (-38.80) | 26.09 (-24.21) | 39.04 (-11.25)
SpanBERT-large | 52.23 | 36.87 (-15.35) | 16.66 (-35.57) | 26.78 (-25.44) | 39.87 (-12.35)
RoBERTa-base | 35.11 | 23.07 (-12.03) |  4.81 (-30.30) | 16.38 (-18.72) | 25.70 ( -9.41)
RoBERTa-large | 42.76 | 28.26 (-14.50) |  8.98 (-33.78) | 21.21 (-21.54) | 32.67 (-10.09)
BERT-base-uncased | 36.90 | 23.53 (-13.37) |  3.33 (-33.58) | 20.78 (-16.12) | 28.53 ( -8.37)
BERT-large-uncased | 38.62 | 24.58 (-14.04) |  2.88 (-35.74) | 21.61 (-17.00) | 29.83 ( -8.79)
BERT-large-uncased-wwm | 50.35 | 35.03 (-15.31) | 11.92 (-38.43) | 26.16 (-24.18) | 39.03 (-11.32)
ALBERT-xxlarge-v2 | 47.58 | 23.02 (-24.56) | 11.58 (-36.00) | 15.25 (-32.34) | 24.42 (-23.16)
ALBERT-xlarge-v2 | 40.87 | 16.13 (-24.74) |  8.15 (-32.72) |  9.57 (-31.30) | 15.11 (-25.76)
ALBERT-large-v2 | 39.02 |  8.03 (-30.99) |  3.82 (-35.20) |  4.52 (-34.50) |  9.00 (-30.02)
ALBERT-base-v2 | 32.76 |  3.63 (-29.13) |  1.74 (-31.03) |  1.76 (-31.00) |  3.63 (-29.13)
ALBERT-xxlarge-v1 | 47.50 | 23.68 (-23.83) | 12.20 (-35.30) | 15.79 (-31.71) | 25.27 (-22.24)
ALBERT-xlarge-v1 | 48.19 | 21.15 (-27.04) | 11.81 (-36.38) | 12.82 (-35.38) | 22.57 (-25.62)
ALBERT-large-v1 | 43.64 | 14.14 (-29.50) |  7.52 (-36.12) |  7.93 (-35.71) | 14.80 (-28.84)
ALBERT-base-v1 | 40.95 | 27.39 (-13.56) | 13.13 (-27.83) | 19.30 (-21.65) | 30.05 (-10.90)
Table 6: The accuracy of the predictions when different treatment words are missing. Model |  ATE of Association ($k=100$/$k=1$)
---|---
KD | PC | HC | R
BERT-base-cased | 0.1585/0.1416 | 0.4085/0.3354 | 0.1779/0.1654 | 0.1081/0.0950
BERT-large-cased | 0.1603/0.1480 | 0.4113/0.3420 | 0.1791/0.1788 | 0.0996/0.0979
BERT-large-cased-wwm | 0.1384/0.1506 | 0.4477/0.3879 | 0.2305/0.2421 | 0.1072/0.1124
SpanBERT-large | 0.1351/0.1533 | 0.3679/0.3555 | 0.2383/0.2544 | 0.1157/0.1233
RoBERTa-large | 0.1360/0.1438 | 0.3962/0.3366 | 0.2162/0.2154 | 0.0985/0.0997
RoBERTa-base | 0.1352/0.1192 | 0.3926/0.3018 | 0.2093/0.1872 | 0.1053/0.0929
BERT-base-uncased | 0.1439/0.1337 | 0.4112/0.3357 | 0.1643/0.1612 | 0.0901/0.0837
BERT-large-uncased | 0.1522/0.1403 | 0.4401/0.3573 | 0.1713/0.1700 | 0.0946/0.0879
BERT-large-uncased-wwm | 0.1410/0.1531 | 0.4350/0.3842 | 0.2290/0.2418 | 0.1089/0.1131
ALBERT-base-v2 | 0.3987/0.2911 | 0.4269/0.3100 | 0.4269/0.3100 | 0.4021/0.2911
ALBERT-large-v2 | 0.4075/0.3098 | 0.4716/0.3519 | 0.4566/0.3450 | 0.3958/0.3001
ALBERT-xlarge-v2 | 0.3279/0.2474 | 0.4336/0.3272 | 0.4170/0.3130 | 0.3468/0.2576
ALBERT-xxlarge-v2 | 0.2852/0.2457 | 0.4338/0.3601 | 0.3801/0.3234 | 0.2704/0.2317
ALBERT-base-v1 | 0.1429/0.1355 | 0.3235/0.2782 | 0.2287/0.2165 | 0.1167/0.1089
ALBERT-large-v1 | 0.3638/0.2951 | 0.4569/0.3612 | 0.4488/0.3571 | 0.3621/0.2884
ALBERT-xlarge-v1 | 0.3223/0.2705 | 0.4425/0.3639 | 0.4266/0.3538 | 0.3116/0.2563
ALBERT-xxlarge-v1 | 0.2761/0.2384 | 0.4230/0.3531 | 0.3708/0.3171 | 0.2590/0.2224
Table 7: The ATE of associations in more PLMs. Model |  Associations (joint/accuracy/consistency)
---|---
KD | PC | HC | R
BERT-base-cased | 0.1523/0.2222/0.0768 | -0.2156/-0.1839/-0.1597 |  0.0011/ 0.0180/-0.0137 | -0.0823/-0.0774/-0.0621
BERT-large-cased | 0.1398/0.1879/0.0788 | -0.1638/-0.1120/-0.1295 | -0.0017/ 0.0095/-0.0141 | -0.0810/-0.0741/-0.0638
BERT-large-cased-wwm | 0.2492/0.2795/0.1627 | -0.1904/-0.1783/-0.1290 |  0.0800/ 0.0851/ 0.0422 | -0.0487/-0.0455/-0.0417
SpanBERT-large | 0.2463/0.2784/0.1382 | -0.1068/-0.0781/-0.1187 |  0.1017/ 0.1175/ 0.0254 | -0.0384/-0.0307/-0.0391
RoBERTa-base | 0.2432/0.3062/0.1223 | -0.0414/-0.0440/-0.0366 |  0.0966/ 0.1252/ 0.0297 | -0.0201/-0.0054/-0.0423
RoBERTa-large | 0.2212/0.2666/0.1141 | -0.1131/-0.1455/-0.0642 |  0.0749/ 0.0911/ 0.0156 | -0.0311/-0.0441/-0.0236
BERT-base-uncased | 0.1635/0.2233/0.0954 | -0.1454/-0.1269/-0.1182 |  0.0130/ 0.0410/-0.0114 | -0.0659/-0.0630/-0.0596
BERT-large-uncased | 0.1507/0.2022/0.0749 | -0.2056/-0.1900/-0.1084 |  0.0247/ 0.0355/ 0.0085 | -0.0671/-0.0667/-0.0454
BERT-large-uncased-wwm | 0.2526/0.2776/0.1593 | -0.1772/-0.1589/-0.1346 |  0.0866/ 0.0886/ 0.0344 | -0.0462/-0.0419/-0.0401
ALBERT-base-v2 | 0.0453/0.0530/0.0347 | -0.1054/-0.1333/-0.0417 |  0.0071/-0.0005/ 0.0371 | -0.0886/-0.1186/-0.0117
ALBERT-large-v2 | 0.0809/0.0988/0.0370 | -0.1130/-0.1457/-0.0826 |  0.0201/ 0.0158/ 0.0093 | -0.0822/-0.1061/-0.0707
ALBERT-xlarge-v2 | 0.1515/0.2161/0.1064 | -0.1184/-0.1152/-0.0759 |  0.0278/ 0.0524/ 0.0154 | -0.0997/-0.0978/-0.0627
ALBERT-xxlarge-v2 | 0.1685/0.1954/0.1347 | -0.1549/-0.1552/-0.1039 |  0.0445/ 0.0492/ 0.0496 | -0.0783/-0.0759/-0.0559
ALBERT-base-v1 | 0.3034/0.3563/0.1764 | -0.0650/-0.0504/-0.1010 |  0.1564/ 0.1724/ 0.0497 |  0.0111/ 0.0175/-0.0202
ALBERT-large-v1 | 0.1466/0.1741/0.1145 | -0.0384/-0.0639/-0.0037 |  0.0598/ 0.0530/ 0.0606 | -0.0186/-0.0408/ 0.0026
ALBERT-xlarge-v1 | 0.1593/0.1816/0.1486 | -0.0514/-0.0629/-0.0006 |  0.0498/ 0.0338/ 0.1172 | -0.0081/-0.0288/ 0.0431
ALBERT-xxlarge-v1 | 0.1926/0.2207/0.1462 | -0.1314/-0.1346/-0.0929 |  0.0629/ 0.0647/ 0.0526 | -0.0563/-0.0535/-0.0475
Table 8: The Pearson correlation coefficients between the ATEs and the factual
knowledge capture metrics.


# Instrument for simultaneous measurement of Seebeck coefficient and thermal
conductivity in the temperature range $300-800$ K with python interfacing

Shamim Sk1, Electronic mail<EMAIL_ADDRESS>Abhishek Pandey1 Sudhir K.
Pandey2, Electronic mail<EMAIL_ADDRESS>1School of Basic Sciences,
Indian Institute of Technology Mandi, Kamand - 175075, India  2School of
Engineering, Indian Institute of Technology Mandi, Kamand - 175075, India

###### Abstract

Fabrication and characterization of instrument for high-temperature
simultaneous measurement of Seebeck coefficient (S) and thermal conductivity
($\kappa$) has been carried out with python automation. The steady-state based
Fourier’s law of thermal conduction is employed for $\kappa$ measurement. The
parallel thermal conductance technique is implemented for heat loss
measurement. Introducing the thin heater and insulating heater base minimize
the heat loss and make the way easier to arrive at high temperature.
Measurement of S is carried out using differential method. Same thermocouples
are used to measure temperature as well as voltage for S measurement. Care of
temperature dependent S of thermocouple has also been taken. Simple design,
small size, lightweight make this instrument more robust. All the components
for making sample holder are easily available in the market and can be
replaced as per the user demand. This instrument can measure samples with
various dimensions and shapes in the temperature range $300-800$ K. The
instrument is validated using different class of samples, such as nickel,
gadolinium, Fe2VAl and LaCoO3. Wide range of S from $\sim-$20 to $\sim$600
$\mu$V/K and $\kappa$ from $\sim$1.1 to $\sim$23.5 W/m-K are studied. The
measured values of S and $\kappa$ are in good agreement with the reported
data.

##  I INTRODUCTION

Fast consumption of natural energy sources (e.g. coal, oil, gas etc) has
always pushed the researchers to find the alternative sources of energy. In
this contest, thermoelectric generator (TEG) is one which can convert the
industrial waste heat into useful electricity.teg1 ; teg2  TEG is
characterized by the means of dimension-less parameter, called figure-of-merit
(ZT),zt

$ZT=\dfrac{S^{2}\sigma T}{\kappa},$ (1)

where, S, $\sigma$ and $\kappa$ are the Seebeck coefficient, electrical
conductivity and thermal conductivity, respectively. T is absolute temperature
of the sample. The efficient TE materials should possess high S and $\sigma$
with low $\kappa$ to attain a high ZT. To calculate ZT, these three parameters
have to be measured. Out of these, $\kappa$ is most difficult quantity to
measure specially at high temperature region due to undefined amount of heat
loss by conduction, convection, and radiation. From Eqn. 1, it is seen that
only the quantity S enters in ZT as square. Therefore, any small error in
measuring S can give rise to the significant change in ZT value. Hence,
precise measurement of S is required to get more accurate ZT.

The S is simply defined as the production of electrical voltage in the
presence of temperature gradient. S can be measured in two ways: integral and
differential. In integral method, the temperature of one end of the sample is
varied, whereas other end of the sample is kept at constant temperature. The
extra cooling system is required to maintain one end of the sample at constant
temperature. This makes the instrument more complex and costly. This method is
also not suitable for nondegenerate semiconductors and insulators.integral1 ;
integral2  In the differential method, S of the sample is calculated by:
$S=-\frac{\Delta V}{\Delta T}+S_{wire}$. Where, $\Delta V$ is the induced
voltage difference across the temperature difference $\Delta T$ within a
sample, and $S_{wire}$ is the Seebeck coefficient of connecting wire. Most of
the seebeck measuring instruments prefer to use this method because of its
simplicity and no additional requirement of cooling system.differential1 ;
differential2 ; differential3 ; differential4  In this method, the connecting
wire and thermocouple should be attached at the identical point of the sample
to measure temperature and induced voltage, which is the main difficulty of
this method.integral2  To overcome this issue, Boor et al.boor  came with the
different appraoch under differential method, where same thermocouple is used
for the measurement of temperature as well as induced voltage. Later, this
method of Boor et al. is used by otherspatel_seebeck ; saurabh_seebeck  to
measure S from 300 K to 620 K. Further, Kumar et al.saurabh_seebeck2  have
utilized this method for measuring S in the temperature range $80-650$ K. The
difficulty in implementation of this method for high temperature application
comes from inserting the thermocouple for measuring output signal. Here, we
have fabricated the instrument using the method proposed by Boor et al.boor
to measure S in the temperature region $300-800$ K. We have fixed the
thermocouple with the help of fire gun and using thin wire of the special
designed material contains 95$\%$ silver and 5$\%$ zinc. Details of welding of
thermocouple is described in Sec. III.

In most general, the $\kappa$ is defined as the ability of materials to
conduct heat. Various methods have been developed for the measurement of
$\kappa$ with their own merit and demerit.tong  The accurate measurement of
$\kappa$ is always a terrible job. To quantify the amount of heat flowing
through the sample is really difficult. The existing $\kappa$ measurement
techniques are broadly classified into two divisions: steady-state and non-
steady-state. In steady-state, measurement is carried out when the temperature
of the material at a paticular position does not vary with time. The non-
steady-state based method relatively takes less time for the measurement as
this method does not need to wait for steady-state. The laser flash method
developed by Parker et al.parker  is most popular among the non-steady-state
based techniques. But, this method has been tested for metals only around room
temperature and proposed that measurement can be done for all types of solid
materials in any temperature range. Another popular technique is 3$\omega$
method, which has been developed to measure the thermal conductivity of thin
film and thin wire mainly.tong ; bourlon ; lu  In this method, a frequency of
$\omega$ is used to excite heater and then $\kappa$ can be measured from the
3$\omega$ response.

The signal analysis in non-steady-state method is quite heavy, while steady-
state method involves easy signal analysis. The available steady-state based
methods are axial heat flow, guarded hot plate, direct electrical heating, hot
wire method etc.tong  In steady-state method, one-dimensional Fourier’s law of
heat conduction is employed to measure $\kappa$. The measurement accuracy of
this method totaly depends on how accurately we measure the amount of heat
passing through the sample. Because, heat is lost through conduction,
convection, and radiation, which becomes more significant at high
temperature.amatya  The undefined amount of heat loss is measured by different
ways. Zawilski et al.zawilski  described a parallel thermal conductance
technique for measuring $\kappa$. In this technique, heat loss is determined
by running the instrument without a sample. They measured $\kappa$ from room
temperature to 12 K. Later, Dasgupta et al.differential2  implemented this
technique to measure $\kappa$ from room temperature to 700 K. Wide variation
of $\kappa$ is reported at room temperature only. They reported the $\kappa$
value of CrSi2 from room temperature to 650 K with the maximum value of
$\sim$2.2 W/m-K. Recently, Patel et al.patel_kappa  fabricated the instrument
for measuring $\kappa$ using the same technique of parallel thermal
conductance. But their instrument is limited to measure $\kappa$ from room
temperature to 620 K. Needless to say that heat loss increases with increase
in temperature and measurement of $\kappa$ at high temperature becomes more
challenging. Keeping this in mind, we have fabricated the instrument by making
very thin heater and insulating heater base to minimize the heat loss.
Instrument is tested from 300 to 800 K for measuring wide range of $\kappa$
with standard samples. The speciality of this instrument is that it can
measure S as well as $\kappa$ simultaneously.

Manual data collection is always a painful job for a researcher. This is more
troublesome when measurement is carried out for a longer time. In this regard,
we can say that manual monitoring of S and $\kappa$ is a hectic and time
taking process when these measurements are performed in steady-state and wide
temperature range. In the present study, our instrument facilitates the
simultaneous measurement of S and $\kappa$ in steady-state and wide
temperature range. Here simultaneous measurement saves time, but it requires
the number of output signals collection by maintaining the thermal stability,
which increases the difficulty for an user. Therefore, to make the job easier,
automation of measurement can be done where these complexities come into
account. One of the popular interfacing tool is LabVIEW. But, LabVIEW is a
commercially paid graphical language. So, to avoid the commercial cost, we
have automated the measurement with the open source ”python” programming
language. Python has vast libraries support which evades the dependency of
external libraries.

In this work, we have designed and fabricated the simple and low cost
instrument for the simultaneous measurement of S and $\kappa$ in the
temperature range $300-800$ K. Differential method has been implemented for
measuring S, whereas parallel thermal conductance technique is used for the
measurement of $\kappa$. Every parts of the sample holder are easily available
in the market and can be replaced as per the user interest. This instrument
can measure samples with various dimensions and shapes. Instrument is tested
by performing the measurement on the standard samples of nickel, gadolinium,
Fe2VAl and LaCoO3. A wide range of S from $\sim-$20 to $\sim 600$ $\mu$V/K and
$\kappa$ from $\sim$1.1 to $\sim$23.5 W/m-K are measured using our instrument.
The measured values give good match with reported data. The instrument is
automated with open source programming language python to minimize the human
effort.

##  II FORMULATIONS OF MEASUREMENT

In this section, we discuss the formulations used for the measurement of S and
$\kappa$ in the present study.

###  II.1 Measurement of Seebeck coefficient

Here, the differential method proposed by the Boor et al.boor  has been
implemented to measure S. In this method, S of the sample is calculated by

$S=-\frac{U_{neg}}{U_{pos}-U_{neg}}S_{TC}(\overline{T})+S_{neg}(\overline{T}),$
(2)

where $U_{pos}$ and $U_{neg}$ are the voltages measured across the positive
legs and negative legs of thermocouple wires, respectively.
$S_{TC}=S_{pos}-S_{neg}$, is the Seebeck coefficient of thermocouple, where
$S_{pos}$ and $S_{neg}$ are the Seebeck coefficient of positive legs and
negative legs of the thermocouple wires, respectively.
$\overline{T}=(T_{1}+T_{2}$)/2 is the absolute temperature, where $T_{1}$ and
$T_{2}$ are hot side and cold side temperature. Two K-type thermocouples are
used to measure $T_{1}$, $T_{2}$, $U_{pos}$ and $U_{neg}$ as shown in Fig. 1.
Here, it is important to note that temperature dependency of $S_{TC}$ and
$S_{neg}$ have been taken care in the present study.

Figure 1: Wiring scheme for the measurement of $T_{1}$, $T_{2}$, $U_{pos}$ and
$U_{neg}$ using thermocouples. Figure 2: Schematic diagram of heat flow: (a)
without sample and (b) with sample.

###  II.2 Measurement of Thermal conductivity

Steady-state based Fourier’s law of thermal conduction is used for the
measurement of $\kappa$. In this method, $\kappa$ of a material is calculated
as

$\kappa=\frac{\dot{Q_{s}}}{A}.\frac{l}{\Delta T},$ (3)

where $\dot{Q_{s}}$ is the amount of heat passing through the cross-sectional
area, $A$ of the sample per unit time. $\Delta T$ is the temperature gradient
across the thickness, $l$ of the sample. The accuracy of the measurement of
$\kappa$ totally depends on how accurately we measure the $\dot{Q_{s}}$. The
difficulty associates with the measurement of $\dot{Q_{s}}$ due to the
undefined amount of heat loss during the heat transfer process. In the present
work, the amount of heat loss is measured by parallel thermal conductance
technique given by Zawilski et al.zawilski  as discussed in Sec. I. To
minimize the heat loss we have made a thin heater which is fixed on insulating
blocks as shown in Fig. 2. At the first step, heat loss is recorded by running
the instrument without a sample. In this case, heat loss occurs by means of
conduction loss through insulator blocks and radiation loss from the surfaces
of copper block as shown in Fig. 2(a). For a given input power, the
temperature initially increases linearly as a function of time because heat
loss is very less. Afterwards, heat loss increases as the temperature
increases. Eventually, at equilibrium both the input power and rate of heat
loss become equal. At this condition temperature is recorded and the
corresponding input power is noted as baseline correction. Then at the next
step, sample is placed on a copper block and power delivered to the heater. In
this case heat generated by the heater is distributed through insulating
blocks as well as through sample as shown in Fig. 2(b). If the same
temperature is maintained in both the steps, then input power difference
between these two steps (with sample $-$ without sample) will give the net
power flows through the sample. Therefore, net heat flow through the sample is
mathematically defined as:

$\dot{Q_{s}}\approx P_{in}-P_{l},$ (4)

where $P_{in}$ is the power delivered to the heater when sample is placed,
$P_{l}$ is the baseline power supplied to the heater with empty sample. This
$\dot{Q_{s}}$ is used in Eqn. 3 to calculate $\kappa$. At this point, it is
important to note that heat loss through side walls of the sample is ignored,
since this loss is normally expected to be negligibly small as compared to
heat flow through the sample.differential4

##  III APPARATUS DESCRIPTION

A schematic diagram of the instrument is shown in Fig. 3. Different components
are labeled by numbers. The sample (5) is placed between the copper blocks
(17) & (18) of cross-sectional area 8 mm $\times$ 8 mm and thickness 3 mm. Two
K-type thermocouples (7) & (8) of 30 swg are welded at the side wall of the
copper blocks. Welding is done with the help of fire gun and using thin wire
of the special designed material for this purpose having mixture of 95$\%$
silver and 5$\%$ zinc. At the time of welding, if heat is provided to copper
block and thermocouple junction for longer time it will make the thermocouple
little brittle and may break after few measurements. So, to make it robust we
need to cover the thermocouple with glass wool or ceramic tube at welding
time. Both the thermocouples are embedded as close as possible to the sample
to minimize the unnecessary creation of $\Delta T$ across the copper blocks.
Here, it should be mentioned that galinstan (GaInSn) liquid metal eutectic
alloy has been used between the copper block and sample to ensure proper
thermal contact. Boiling point of galinstan is $\sim$1800 K, which makes it
suitable for high temperature application.

A thin heater (6) of 24 $\Omega$ is used to heat the sample. This heater is
made by winding the kanthal wire of 40 swg over a mica sheet of dimension 10
mm $\times$ 8 mm $\times$ 0.2 mm (length $\times$ width $\times$ thickness).
This heater is coated with high temperature cement and wrapped by another mica
sheet to avoid electrical contact. Then this is covered by copper sheet of
thickness 0.4 mm and welded to the copper block (17) over which sample is
placed. The copper wire (16) is used to supply the current to the heater. The
heater is placed over two insulating gypsum blocks (9) using high temperature
cement. The thermal conductivity of gypsum block is very low $\sim$0.017 W/m-K
at 300 K and also suitable for high temperature application.gypsum  The
dimension of both the gypsum block is 8 mm $\times$ 2 mm $\times$ 25 mm. A
finite gap between two gypsum blocks has been kept to minimize the conduction
loss through gypsum block. These gypsum blocks have been made to stand on a
brass plate (11). This brass plate is supported with an stainless steel (SS)
rod (19) using screw (15). A mica sheet (4) is laid on the cold side copper
plate (18) to insulate the sample electrically. An SS rod (2) having threads
is used to apply the pressure on the sample, which is screwed with brass plate
(3). This brass plate (3) is supported by an SS rod (20) using screw (21).
Both the SS rods (19) & (20) are attached to the SS flange (14). A circular
O-ring (10) is used for proper vacuum seal of the chamber.

The whole assembly of sample holder mounted on the SS flange (14) is covered
by an SS made cylindrical vacuum chamber (22) using nut bolts through hole
(12). The inner diameter and length of the vacuum chamber are $\sim$10 cm and
$\sim$30 cm, respectively. A port (1) of size KF-25 is used to connect the
sample chamber to the vacuum pump. The electrical feedthrough (13) is used for
electrical communication between inside and outside the vacuum chamber.

Figure 3: Schematic diagram of the measurement setup.

##  IV SAMPLE DIMENSION

For S measurement sample dimension is not necessary. For $\kappa$, proper
sample dimension is required as the thickness and cross-sectional area are
used for $\kappa$ measurement. The cylindrical and rectangular shaped samples
are used in the present study. Top and bottom surfaces of the sample should be
parallel for the purpose of making good contact of the sample with the copper
block. Pellet of the sample should be as compact as possible for the
measurement.

##  V MEASUREMENT PROCEDURE

At first, heat loss measurement has to be done before loading the sample. Once
heat loss measurement is performed, the sample (5) is mounted between the
copper blocks (17) & (18). We should ensure that both the surfaces of the
sample are in well contact with the copper blocks. A small amount of galinstan
are used at the interface of sample and copper block to secure the proper heat
transfer. Then vacuum chamber (22) is evacuated using vacuum pump. The
Keithley sourcemeter 2604B is used to feed the current to the heater. The
various output signals ($T_{1}$, $T_{2}$, $U_{pos}$, $U_{neg}$ etc) are
measured by the Keithley digital multimeter 2002 with the multichannel scanner
card. The sourcemeter and multimeter are interfaced with computer using
IEEE-488B interface bus. It is important to mention that the measurement of S
and $\kappa$ for a given sample are carried out simultaneously.

The whole measurement process is automated with open source programming
language ”python.” Few segments of this program are shown in Fig. 4. PyVISA,
NumPy, matplotlib libraries are utilized in the program as displayed in Fig.
4(a). The user only needs to define the input parameters as shown in Fig. 4(b)
and then physical interaction of user is not required during the measurement.
As per the input power defined, sourcemeter provides current to the heater and
wait to achieve steady-state. Steady-state is defined by monitoring the rate
of change of $T_{h}$. This rate of change of $T_{h}$ is set as 0.006 K which
is checked in every 5 seconds. Program for steady-state is exhibited in Fig.
4(c). Once, the steady-state criteria is fulfilled 5 times successively,
multimeter measures the output signals. When the heating cycle reaches its
defined maximum temperature limit, heating loop breaks and next loop for
cooling starts to cool the sample. During cooling period power in each cycle
decreases with defined step size until the sample again comes back to room
temperature. Data are also collected in cooling cycle like heating process.
Eqns. 2 and 3 are used in the python program to calculate S and $\kappa$,
respectively. Here, it is important to note that the temperature dependence of
$S_{TC}$ and $S_{neg}$ have been implemented in the program using
interpolation method. The raw values of $S_{TC}$ and $S_{neg}$ are taken from
the literatures.boor ; bentley  During the measurement, user can observe all
the necessary parameters graphically, as the real-time plotting has been
employed using matplotlib library. All the measured quantities along with
input parameters are saved to [file name].csv file. The live data for
acquiring steady-state is shown in Fig. 5. Fig. 5(a) represent the increase in
hot side temperature with counts, while Fig. 5(b) exhibits the rate of change
of hot side temperature with counts. Every count takes 5 seconds. Hence, for a
single data point it takes around 15 minutes.

(a)

(b)

(c)

Figure 4: Few segments of python program: (a) Libraries used in python
program, (b) input parameters defined by user and (3) program to achieve
steady-state. Figure 5: Screenshot of live data: (a) Change in hot side
temperature (Th) with counts and (b) rate of change of Th with counts. One
count takes 5 seconds.

##  VI RESULTS AND DISCUSSION

###  VI.1 HEAT LOSS MEASUREMENT

Before doing the measurement with sample, the heat loss measurement is taken
for $\kappa$. Fig. 6 shows the heat loss as a function of hot side temperature
($T_{h}$). The reliability of $\kappa$ value is desperately depend on how
accurately measure the heat loss value. Keeping this in mind, we have made a
thin heater which has been placed on insulating gypsum blocks to minimize the
heat loss and to achieve required temperature easily. Fig. 6 shows heat loss
is quite low at the beginning. As the temperature increases heat loss also
increases non-linearly. This non-linear increment is fitted with polynomial of
quartic degree: Heat loss = $aT+bT^{4}$. Where, values of $a$ and $b$ are
0.959 $mW/K$ and 1.566 $\times$ 10-8 $mW/K^{4}$, respectively. The first term
indicates the convection loss where heat loss depends on temperature linearly.
The quartic term signifies the radiation loss at high temperature region which
confirms by Stefan’s law: $E=\sigma T^{4}$ (E is radiant heat energy and
$\sigma$ is Stefan’s constant). The purpose of this polynomial fit is that (i)
user can get heat loss at any value of temperature in between $300-800$ K,
(ii) the heat loss value can also be calculated at little above 800 K.

Figure 6: Heat loss at different value of hot side temperature.

###  VI.2 SEEBECK COEFFICIENT MEASUREMENT

The instrument is validated by measuring S of different samples in the
temperature region $300-800$ K. In order to check the flexibility of the
instrument, measurement of various class of samples having wide range of S are
carried out. Fig. 7 shows the measured S of nickel, Fe2VAl and LaCoO3 as a
function of temperature. The shape and dimension of these samples are
tabulated in Table 1.

Fig. 7(a) shows the measured S as a function of temperature of high purity
(99.99$\%$) nickel. The value of S at 300 K is found to be $\sim-$19.2
$\mu$V/K, then decreases up to $\sim$490 K with the corresponding value of
$\sim-$24.4 $\mu$V/K. After 490 K, the S increases monotonically till
$\sim$660 K and has the value of $\sim-$18.9 $\mu$V/K. The value of S
continuously decreases above 660 K and reaches $\sim-$22.9 $\mu$V/K at 800 K.
In order to verify, we have compared our S with the reported data of Burkov et
al.burkov  and Ponnambalam et al.ponnambalam  in the same figure. The S value
matches well with the reported data with the maximum deviation of $\approx$ 1
$\mu$V/K in the whole studied temperature range.

Table 1: Shape and dimension of the test samples.

Sample name |  |  | Shape of |  |  | Thickness |  |  | Cross-sectional area
---|---|---|---|---|---|---|---|---|---
|  |  | cross-section |  |  |  |  | (mm) |  |  |  |  |  |  | (mm2)
Nickel |  |  | Circular |  |  |  |  | 2 |  |  |  |  |  |  | 28.27
Gadolinium |  |  | Rectangular |  |  |  |  | 0.7 |  |  |  |  |  |  | 4.86
Fe2VAl |  |  | Rectangular |  |  |  |  | 1.38 |  |  |  |  |  |  | 15
LaCoO3 |  |  | Rectangular |  |  |  |  | 0.5 |  |  |  |  |  |  | 24

Figure 7: Seebeck coefficient of (a) Nickel, (b)Fe2VAl and (c) LaCoO3 as a
function of temperature.

Fig. 7(b) exhibits the temperature dependent S of Fe2VAl. This compound
possesses a deep pseudogap at the Fermi level and due to this it is known to
give the large S and $\sigma$.nishino ; hanada  This compound has been
synthesized by arc melting technique.shamim_mrx  The value of S is observed as
$\sim-$138 $\mu$V/K at 300 K. The S increases monotonically up to the highest
temperature of interest. At 800 K, S is found to be $\sim-$18 $\mu$V/K. Our
result is compared with the reported data by Sk et al.shamim_mrx  and Kurosaki
et al.kurosaki  as shown in the same figure. One can notice that our data has
good agreement with the reported data throughout the temperature range.
However, S of Kurosaki et al.kurosaki  and our data get deviating at low
temperature region where this deviation vanishes as the temperature increases.
This deviation may be due to the different annealing temperature were applied
in synthesizing the compound in the respective works.

The S of LaCoO3 is measured in the temperature region $300-800$ K as displayed
in Fig. 7(c). This compound is synthesized by combustion technique. LaCoO3
gives a large S at 300 Kmaignan ; jirak , which is a good signature for
efficient TE material. At 300 K, S is found to be $\sim$618 $\mu$V/K then
decreases continuously throughout the full temperature range as Fig 7(c)
shows. At 800 K, the observed value of S is $\sim$15 $\mu$V/K. Figure shows
that S is decreasing rapidly from 300 to $\sim$500 K. After 500 K, this
decrement rate falls down and continues up to 800 K. In the temperature
interval $\sim 300-500$ K, the change in S is $\sim$535 $\mu$V/K with
decrement rate of 2.67 $\mu$V/K. From 500 to 800 K, this change in S is
observed as $\sim$68 $\mu$V/K with the decrement rate of 0.23 $\mu$V/K. The
similar trend of S is reported by other groupsjirak ; singh ; behera  also,
which are shown in the same Fig. 7(c). In the full studied temperature range,
our S value give the good match with reported data.

Figure 8: Temperature dependence of thermal conductivity of (a) Gadolinium,
(b)Fe2VAl and (c) LaCoO3.

Among the test samples used here, nickel is a pure metal, Fe2VAl is an inter-
metallic Heusler-type compound and LaCoO3 is an oxide perovskite material.
Hence, measurement of S on various kind of materials are carried out in the
present study. This suggests that our setup can be used to measure S in the
temperature range $300-800$ K for any class of materials having wide range of
S values.

###  VI.3 THERMAL CONDUCTIVITY MEASUREMENT

To examine our setup, we have measured $\kappa$ of gadolinium, Fe2VAl and
LaCoO3 in the temperature range $300-800$ K as shown in Fig. 8. The shape and
dimension of these samples are shown in Table 1. Thickness and cross-sectional
area of the sample are important for the measurement of $\kappa$.

Fig. 8(a) represents the measured $\kappa$ for gadolinium as a function of
temperature. Gadolinium has one of the lowest $\kappa$ among the metallic
compounds.meis  From Fig. 8(a), the $\kappa$ is observed to increase with
increasing temperature. The values are found to be $\sim$6.3 and $\sim$12.2
W/m-K at 300 and 800 K, respectively. Our $\kappa$ is compared with the
reported data of Meis et al.meis  as shown in the same figure. The $\kappa$ of
gadolinium at room temperature is highly controversial. Because around room
temperature gadolinium possesses the critical ferromagnetic (16o C) and
paramagnetic ($25-29^{o}$ C) phases.pascal  In $18-25^{o}$ C, the values of
$\kappa$ measured using different methods are reported from 6.6 to 14 W/m-K in
the different literaure.meis ; legvold ; arajs ; powell  Fig. 8(a) shows that
at lower temperature region both the data are little deviating, while this
deviation diminishes at high temperature region.

Fig. 8(b) illustrates the temperature dependence values of $\kappa$ for
Fe2VAl. The value of $\kappa$ at 300 K is found to be $\sim$ 23.5 W/m-K. From
temperature 300 to $\sim$500 K $\kappa$ decreases drastically then $\kappa$ is
almost constant throughout the remaining temperature range. At 800 K, the
observed value of $\kappa$ is $\sim$ 15.3 W/m-K. In the temperature range
$300-500$ K, the magnitude change in $\kappa$ is 7 W/m-K, whereas this value
is 1 W/m-K in the temperature range $500-800$ K. Our $\kappa$ is well matched
with the reported value of Kurosaki et al.kurosaki  in the whole temperature
region as shown in the figure.

We have also carried out the measurement of $\kappa$ for the oxide sample.
Oxide sample is known for having low $\kappa$. The temperature dependence of
$\kappa$ for LaCoO3 is shown in Fig. 8(c). The value of $\kappa$ is measured
as $\sim$1.2 W/m-K at 300 K, then increases monotonically up to 800 K with the
corresponding value of $\kappa$ is $\sim$2.3 W/m-K. Figure shows that the
$\kappa$ increases almost linear up to $\sim$500 K then get saturated as the
temperature increases. The values of $\kappa$ around room temperature are
reported from 0.38 to 3.2 W/m-K in the different works.cortes ; behera ; kumar
; li ; bousnina ; pillai  As we know that $\kappa$ depends on particle size as
well as porosity of the sample. Therefore, this diference is may be because of
sample has been synthesized by different techniques and $\kappa$ is measured
by various methods in the respective works.cortes ; behera ; kumar ; li ;
bousnina ; pillai  Our $\kappa$ is closely matched with the reported value of
Kumar et al.kumar  as shown in the figure. A small deviation is observed at
high temperature. Maximum deviation of 0.36 W/m-K is found at 800 K.

The various type of samples are tested for $\kappa$ measurement using our
instrument with wide range of $\kappa$ from $\sim$1.1 to $\sim$23.5 W/m-K. Our
instrument is also examined for wide range of S from $\sim-$20 to $\sim$600
$\mu$V/K. Therefore, this study propose that present apparatus can be used for
the simultaneous measurement of S and $\kappa$ in the temperature range
$300-800$ K.

##  VII CONCLUSIONS

In conclusion, we have fabricated the instrument for measuring Seebeck
coefficient and thermal conductivity simultaneously in the temperature range
$300-800$ K. A simple, low-cost and user friendly instrument is fully
automated with open source programming language python. The differential
method is used for Seebeck coefficient measurement, where the same
thermocouple is employed to measure temperature as well as voltage.
Temperature dependent Seebeck coefficient of thermocouple has been taken care.
The parallel thermal conductance technique is implemented for thermal
conductivity measurement. We have made a thin heater and insulating heater
base to minimize the heat loss. The instrument is validated by doing
measurement on various kind of samples with different dimensions and shapes.
The measurement on test samples like nickel, gadolinium, Fe2VAl and LaCoO3 are
carried out. The instrument is tested for a wide range of Seebeck coefficient
from $\sim-$20 to $\sim$600 $\mu$V/K and thermal conductivity from $\sim$1.1
to $\sim$23.5 W/m-K. The measured values are found in good agreement with the
reported values.

Data Availability Statement

Data available on request from the authors.

## References

  * (1) M. Akasaka, T. Iida, A. Matsumoto, K. Yamanaka, Y. Takanashi, T. Imai, and N. Hamada, J. Appl. Phys. 104, 013703 (2008).
  * (2) Y. Pei, H. Wang, and G. J. Snyder, Adv. Mater. 24, 6125 (2012).
  * (3) Y. Pei, X. Shi, A. LaLonde, H. Wang, L. Chen, and G. J. Snyder, Nature 473, 66 (2011).
  * (4) S. R. S. Kumar, and S. A. Kasiviswanathan, Rev. Sci. Instrum. 79, 024302 (2008).
  * (5) C. Wood, A. Chmielewski, and D. Zoltan, Rev. Sci. Instrum. 59, 951 (1988).
  * (6) Z. Zhou, and C. Uher, Rev. Sci. Instrum. 76, 023901 (2005).
  * (7) T. Dasgupta, and A. M. Umarji, Rev. Sci. Instrum. 76, 094901 (2005).
  * (8) A. Muto, D. Kraemer, Q. Hao, Z. F. Ren, and G. Chen, Rev. Sci. Instrum. 80, 093901 (2009).
  * (9) R. Amatya, P. M, Mayer, and R. J. Ram, Rev. Sci. Instrum. 83, 075117 (2012).
  * (10) J. de Boor, and E. Muller, Rev. Sci. Instrum. 84(6), 65102 (2013).
  * (11) A. Patel, and S. K. Pandey, Instrum. Sci. Technol. 45(4), 366 (2017).
  * (12) S. Singh, and S. K. Pandey, Measurement 102, 26 (2017).
  * (13) D. K. A. Kumar, A. Patel, S. Singh, and K. Asokan, Rev. Sci. Instrum. 90(10), 104901 (2019).
  * (14) X. C. Tong, “Characterization methodologies of thermal management materials,” in Advanced Materials for Thermal Management of Electronic Packaging (Springer, New York, 2011), Vol. 30, pp. 59–129.
  * (15) W. J. Parker, R. J. Jenkins, C. P. Butler, and G. L. Abbot, J. Appl. Phys. 32, 1679 (1961).
  * (16) A. J. G. Bourlon (2005) ”Thermal conductivity measurement by the 3$\omega$ method.” Technical Note PR-TN 2005/01035. Philips Electronics, Koninklijke.
  * (17) L. Lu, W. Yi, and D. L. Zhang, Rev. Sci. Instrum. 72, 2996 (2001).
  * (18) R. Amatya, P. M. Mayer, and R. J. Ram, Rev. Sci. Instrum. 83, 075117 (2012).
  * (19) B. M. Zawilski, R. T. Littleton IV, and T. M. Tritt, Rev. Sci. Instrum. 72, 1770 (2001).
  * (20) A. Patel, and S. K. Pandey, Rev. Sci. Instrum. 88, 015107 (2017).
  * (21) S. L. Manzello, S. Park, T. Mizukami, and D. P. Bentz, “Measurement of thermal properties of gypsum board at elevated temperatures,” in Proceedings of the Fifth International Conference on Structures in Fire (SiF08) (Organising Committee, Fifth International Conference Structures in Fire (SiF’08), 2008), pp. 656–665
  * (22) R. E. Bentley, Handbook of Temperature Measurement Vol. 3: The Theory and Practice of Thermoelectric Thermometry. Springer, 1998.
  * (23) A. T. Burkov, A. Heinrich, P. P. Konstantinov, T. Nakama, and K. Yagasaki, Meas. Sci. Technol. 12, 264 (2001).
  * (24) V. Ponnambalam, S. Lindsey, N. S. Hickman, and T. M. Tritt, Rev. Sci. Instrum. 77, 073904 (2006).
  * (25) Y. Nishino, S. Deguchi, and U. Mizutani, Phys. Rev. B 74, 115115 (2006).
  * (26) Y. Hanada, R. O. Suzuki, and K. Ono, J. Alloys Compd. 329, 63 (2001).
  * (27) S. Sk, P Devi, S. Singh, and S. K. Pandey, Mater. Res. Express 6, 026302 (2019).
  * (28) K. Kurosaki, H. Muta, Yu Endo, A. Charoenphakdee, M. Uno, and S. Yamanaka, J. Alloys Compd. 486, 507 (2009).
  * (29) A. Maignan, D. Flahaut, and S. Hebert, Eur. Phys. J. B 39, 145 (2004).
  * (30) Z. Jirak, J. Hejtmnek, K. Knek, and M. Veverka, Phys. Rev B 78, 014432 (2008).
  * (31) S. Singh, and S. K. Pandey, Philos. Mag 97(6), 451 (2017).
  * (32) S. Behera, V. B. Kamble, S. Vitta, A. M. Umarji, and C. Shivakumara, Bull. Mater. Sci. 40(7), 1291 (2017).
  * (33) C. Meis, A. K. Froment, and D. Moulinie, J. Phys. D: Appl. Phys. 26, 560 (1993).
  * (34) P. Pascal, 1958 Nouveau Traite de Chimie Minrale (Paris: Masson et Cie) pp 630-1.
  * (35) S. Legvold, and F. H. Spedding, 1 SC-508 1954.
  * (36) S. Arajs, and R. V. Colvin, J. Appl. Phys. 35, 1043 (1964).
  * (37) R. W. Powell, and B. W. Jolliffe, Phys. Lett. 14, 171 (1965).
  * (38) A. Cortes, J. S. T. Hernandez, J. F. Piamba, W. Lopera, and P. Prieto, J. Supercond. Nov. Magn. 32, 1367 (2019).
  * (39) A. Kumar, K. Kumari, B. Jayachandran, D. Sivaprahasam, and A. D. Thakur, J. Alloys Compd. 749, 1092 (2018).
  * (40) F. Li, J. F. Li, J. H. Li, and F. Z. Yao, Phys. Chem. Chem. Phys. 14, 12213 (2012).
  * (41) M. A Bousnina, R. Dujardin, L. Perriere, F. Giovannelli, G. Guegan, and F. Delorme, J. Adv. Ceram. 7(2), 160 (2018).
  * (42) C. G. S. Pillai, and A. M. George, Int. J. Therm. 4, 183 (1983).