# Towards Verifiable Generation: A Benchmark for Knowledge-aware Language
Model Attribution

Xinze Li1, Yixin Cao2†, Liangming Pan3, Yubo Ma1, Aixin Sun1†

1 S-Lab, Nanyang Technological University
2 Singapore Management University 3 University of California, Santa Barbara
{xinze.li<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Although achieving great success, Large Language Models (LLMs) usually suffer
from unreliable hallucinations. In this paper, we define a new task of
Knowledge-aware Language Model Attribution (KaLMA) that improves upon three
core concerns on conventional attributed LMs. First, we extend attribution
source from unstructured texts to Knowledge Graph (KG), whose rich structures
benefit both the attribution performance and working scenarios. Second, we
propose a new “Conscious Incompetence" setting considering the incomplete
knowledge repository, where the model identifies the need for supporting
knowledge beyond the provided KG. Third, we propose a comprehensive automatic
evaluation metric encompassing text quality, citation quality, and text
citation alignment. To implement the above innovations, we build a dataset in
biography domain BioKaLMA via a well-designed evolutionary question generation
strategy, to control the question complexity and necessary knowledge to the
answer. For evaluation, we develop a baseline solution and demonstrate the
room for improvement in LLMs’ citation generation, emphasizing the importance
of incorporating the "Conscious Incompetence" setting, and the critical role
of retrieval accuracy.

##  1 Introduction

Figure 1: A demonstration of our task set up. Given a question, the system
generates answers attributed from a retrieved knowledge graph. The underlines
in question are the retrieved entities, and the underlines in outputs are the
citations. [NA] is the “Not Applicable Citation”.

Recently, Large Language Models Brown et al. (2020) (LLMs) have exhibited
great capability in open-ended question answering Yang et al. (2019). However,
the generated answers are not always reliable, e.g., “hallucination” Shuster
et al. (2021); Ji et al. (2023). To minimize the negative impacts, researchers
have proposed the task of language attribution Bohnet et al. (2023), which not
only enables users to verify the generated text flexibly but also contributes
to many important applications, such as situation reports Reddy et al. (2023),
academic papers Salvagno et al. (2023), medical diagnosis Zuccon and Koopman
(2023).

Existing works mainly attribute generated outputs to unstructured documents
like web pages Nakano et al. (2021); Menick et al. (2022) or passages Gao et
al. (2023). To verify the answer quality, they typically compare with a human
annotated reference answer for automatic evaluation or conduct human
evaluation. We argue that there are several concerns on such task definition.
Firstly, are documents the only source for attribution? Many real-world
applications have their own knowledge bases or semi-structured reports.
Unstructured documents neglect the informative structures. Secondly, does the
attribution source always include all the required knowledge? It is still
necessary to consider the coverage issue since no perfect repository can
contain all the information in this world for attribution. Thirdly, how to
systematically evaluate the attributed content without references? For open-
ended questions, there are unlimited number of answers and it is difficult to
define a single ground truth.

To address the first challenge, we utilize knowledge graph (KG) as a reliable
source for attribution, namely Knowledge-aware Language Model Attribution
(KaLMA). We show a demonstration of task in Figure 1. KGs efficiently organize
world knowledge in a structured manner and has the potential to unify various
formats of data. For example, databases can be easily converted into KGs, or,
passages and web pages can be represented as a node in KG like Wikipedia.
KaLMA differs from entity linking Sevgili et al. (2022) since the sentences or
phrases are attributed to a knowledge triplet rather than a single entity. For
the second challenge, we tackle the coverage problem by making the model aware
of its limitations. We introduce a new setting “Conscious Incompetence”
Curtiss and Warren (1974), which is the psychological stage that one is aware
of the knowledge gap. During generation, except for references to KG, LLMs
also identify sentences or phrases that require supporting knowledge not
present in the knowledge graph. For the third challenge, we propose a
comprehensive automatic evaluation metric including text quality, citation
quality, and text citation alignment. The entire evaluation process does not
require human annotated ground truth.

To implement the above innovations, we construct a dataset111The codes and
dataset will be released upon acceptance. in biography domain, namely
BioKaLMA, for a benchmark with all-rounded automatic measurements. We choose
biography because it forms a good testset for attribution due to its practical
application and convenient evaluation. Derived from the biographical
database222https://plumaj.github.io/biographical/ Plum et al. (2022) and
WikiData, BioKaLMA contains 1,085 questions and BioKGs. For evaluation, each
question refers to a minimum set of knowledge triples in the corresponding
BioKG to answer, which is obtained via a well-designed Evolutionary Question
Generation strategy. We use GPT-EVAL Liu et al. (2023a) to automatically
evaluate the text. We also design measurement for correctness, precision, and
recall for citations based on the minimum set of knowledge in BioKG.
Addtionally, we determine the alignment between texts and citations employing
NLI Dagan et al. (2005). Our setting “Conscious Incompetence” enables an
attributed LM to recognize the knowledge gaps and allows users to verify
uncertain claims, which enhances trustworthiness.

We conducted extensive experiments and show a room for improvement of the
LLMs’ ability to generate accurate and thorough citations based on provided
knowledge graphs. Our experiments on “Conscious Incompetence” investigate the
capability of current LLMs to identify if there are required knowledge not in
knowledge base. We highlight the necessity of incorporating this setting in
future language attribution works. Furthermore, our ablation studies
demonstrate the crucial role of retrieval accuracy in achieving desirable
generation results.

##  2 Task and Dataset

###  2.1 Task Formulation

We hereby define the task Knowledge-aware Language Model Attribution (KaLMA):
Given a question $q$ and the knowledge graph $G$, the system generates an
output text $t$ that answers the question. The output text consists of a list
of $m$ sentences $s_{1}$, …, $s_{m}$ grounded with a list of $n$ grounded
knowledge $k_{1}$ .. $k_{n}$ where $\\{k_{1}..k_{n}\\}\in G$.

#### Conscious Incompetence

We extend this task setting to include conscious incompetence. Given the same
input, each sentence $s$ in the output text $t$ can map to a Not Applicable
Citation (we use [NA] to represent it) if it includes some knowledge to be
verified, but the knowledge is absent in the knowledge graph $G$. A sentence
can map to both [NA] and a list of sub-graph knowledge it it can be partially
verified by the knowledge graph $G$. [NA] is not a citation on conventional
means, but a indicator of knowledge gap.

###  2.2 Dataset Construction

We construct dataset BioKaLMA for the task. The dataset construction pipeline
consists of three components: Name Pair Extraction, Name Disambiguation, and
Evolutionary Question Generation. While we use the human biography domain as
an example, this method is applicable to all domains.

#### Name Pair Extraction

To ensure that each generated question involves people related to one another,
we select name pairs from the biographical database, which is specifically
designed for the relation extraction (RE) task. Each data entry includes a
short paragraph and a triple of names and relations, such as <William
Shakespeare, Spouse, Anne Hathaway>, extracted from the paragraph. We
specifically choose the human annotated set from the database to ensure high-
quality name pairs. To avoid potential ambiguities, we filter out data if any
name in the triple is incomplete. In practice, we consider a name complete if
it has at least a family name and a surname.

#### Name Disambiguation

Due to the presence of duplicate names (e.g., Anne Hathaway the actress and
Anne Hathaway the wife of William Shakespeare), we perform name disambiguation
to map each name in the triple to a unique entity from the knowledge graph. We
utilize WikiData333https://www.wikidata.org/wiki/Wikidata:Main_Page Vrandečić
and Krötzsch (2014) as the knowledge base and employ SPARQL Pérez et al.
(2009) queries to retrieve all entities associated with the name. Each entity
represents a node in the knowledge graph. Since each triple consists of two
names and one relation, we select the two entities obtained from the query if
they are connected to each other on WikiData. Additionally, the connecting
edge should align with the relation specified in the triple. Subsequently, we
extract the one-hop sub-graph centered around each person node, which provides
properties related to the person, such as gender, birth date, occupation, and
more.

#### Evolutionary Question Generation

We employ an “evolutionary question generation” approach inspired by WizardLM
Xu et al. (2023), where we gradually increase the complexity of the questions
by injecting knowledge through iterations. In each iteration, LLMs extend the
original answer by incorporating the additional knowledge and then formulates
a question based on the expanded answer. After the last iteration, all
injected knowledge form a “minimum knowledge set”, which includes the least
knowledge required to answer the final question (Table 1). We call this set
“BioKG” for the question, which is a controllable set of knowledge that we
choose. To maintain question coherence, we use an “answer-to-question”
paradigm within each iteration, where we generate the answer first and then
construct the question based on the answer. In the first iteration, we utilize
short paragraphs from the biographical database to generate simple questions.
For subsequent iterations, we take the answer and question from the previous
iteration and inject a new knowledge. We leverage the “text-davinci-003” model
for generation with in-context learning. The detailed prompt used is provided
in the appendix C We provide one human written demonstration. Some examples of
full question evolution process are provided in appendix D. In practice, we
employ six iterations to ensure sufficient complexity in the questions without
making them overly tedious.

Question:
---
How did the succession of Chandragupta II, Kumaragupta I, and Skandagupta
contribute to the development of the Gupta Empire, and what can we learn from
their reigns?
BioKG (Minimum required knowledge):
[Q844536, religion, Hinduism]
[Q844536, occupation, monarch]
[Q844536, position held, Gupta emperor]
[Q844536, succeeded by, Kumaragupta I]
[Q844536, spouse, Dhruvadevi]
Q844536 is Chandragupta II’s qid in WikiData
Table 1: An example for the generated question and minimum required knowledge.

###  2.3 Dataset Analysis

With the above efforts, we have obtained 1,085 questions for BioKaLMA. The
dataset includes a wide range of distribution with people from 196 countries
and 949 cities, taking 279 kinds of different occupations. The era of people
ranging from 1950 B.C. to 2001 A.D. Compared existing datasets for document
attribution, BioKaLMA supports a more fine-grained attribution at an entity
level.

##  3 Method

We build a pipeline to enable LLMs to generate knowledge-aware attributed
answers. Following the approach of many retrieval-augmented generation works
Lee et al. (2022); Izacard and Grave (2021), we utilize a pipeline consisting
of three components: retrieval, re-ranking, and generation.

###  3.1 Retrieval

Our baseline retrieval process consists of two parts: named entity recognition
and graph retrieval. We utilize spaCy444https://spacy.io/api/entityrecognizer
to identify the named entities mentioned in the question. Using these
entities, we retrieve entity-centered sub-graphs using SPARQL. For each
retrieved entity, we search for nodes in the graph that match the entity’s
name. We use the named entity recognition (NER) entity type as a simple filter
(e.g., the NER category “person” matches the “human” entity type in WikiData).
Taking each selected node as the center, we retrieve one-hop sub-graphs that
contain properties associated with the entity.

###  3.2 Re-ranking

The re-ranking component plays a crucial role in disambiguating retrieved
entities, as multiple entities may share the same name in the WikiData graph.
Two common scenarios are different individuals with the same name (e.g., Anne
Hathaway the American actress and Anne Hathaway the wife of William
Shakespeare) and different references to the same word (e.g., “Chinese” the
language and “Chinese” the ethnic group). When multiple entities are retrieved
from the graph for a given entity name, we rank the graphs based on the Exact
Match (EM) between the neighboring nodes and the question. We select the
entity with the highest number of matched neighboring nodes.

###  3.3 Generation

Instruction: You answer the question based on your knowledge, with ...
---
Question: Consider the information:
{Q367360, name: Orazio Gentileschi, place of death: London, child: Artemisia
Gentileschi, ...}
{Q212657, name: Artemisia Gentileschi, place of birth: Rome, occupation:
painter, ...}
How did Orazio Gentileschi’s influence on Artemisia’s life and career shape
her development as a Baroque painter?
Answer:  Artemisia Gentileschi was an Italian painter born on July 8, 1596
[NA] in Rome [Q212657, ethnic group: Italians, occupation: painter, place of
birth: Rome] ...
Table 2: An example for the prompt we use for attributed answer generation.

The generation component effectively prompt the LLMs with the retrieved
knowledge graphs (KGs) to generate answers that attribute the KG. To adapt to
the input format of the LLMs, we transform the structured KGs into flat texts.
We preserve the information of the retrieved sub-graphs by mapping each sub-
graph to a set of triples. Each triple consists of two nodes and one edge,
where one node is the centered entity, the other node is its neighbor, and the
edge represents the relationship between them. For example, [Q212657 - place
of birth - Q220] can be translated to [Artemisia Gentileschi - place of birth
- Rome]. In this translation, we use the names of the entities for better
comprehension by both the models and humans, since WikiData utilizes Qids
(e.g., Q220) to represent unique entities. We construct a prompt (Table 2)
which includes 1) instruction to the models to generate attributed answers. 2)
retrieved knowledge graph, and 3) the question. We employ one-shot in-context
learning Brown et al. (2020) by prepending one human written demonstration. In
the one-shot demostration, we use the special token [NA] to represent the “Not
Applicable Citations” for conscious incompetence. We deliberately omit some
knowledge in the demonstration example knowledge graph, and we insert [NA]
tokens in the corresponding sentences that use these knowledge within the
example answer. The detailed prompt is in appendix C.

##  4 Automatic Evaluation

Our benchmark includes measurements for both the generated text and citations.
We also provide evaluations for the alignment between the text and
corresponding citations through both automatic and human evaluation.

###  4.1 Text Evaluation

Since our test-set has no human-written gold answers as references, we do not
utilize comparison-based metrics such as BERTScore Zhang et al. (2019a) or
MAUVE Pillutla et al. (2021). Instead, we employ reference-free NLG evaluator
GPTScore Fu et al. (2023), which defines the following four metrics: 1)
Coherence: whether the generated text is well-structured and well-organized.
2) Consistency: whether the generated text is consistent with the knowledge
provided. 3) Fluency: whether the generated text is well-written and
grammatical. 4) Relevance: how well is the generated text relevant to the
question.

We use the model text-davinci-003 for evaluation, which assigns an integer
score of 1 to 5 for each metric. We follow the prompt provided in G-Eval Liu
et al. (2023b) and customize it based on our task. The full prompts are given
in appendix C.

###  4.2 Citation Evaluation

We evaluate the citation qualities from three aspects: 1) Correctness, which
measures whether the generated knowledge matches the given knowledge from the
knowledge graph, 2) Precision, which determines how much of the generated
citations are helpful to answer the question, and 3) Recall, which measures
how much of the minimum knowledge set are covered by the generated citations.
We also calculate the F1-Score based on the Precision and Recall to reflect
the overall quality of citations.

#### Correctness

We calculate the citation correctness for each citation (0 or 1) and average
over all citations. Each citation comprises a triplet of 1) center entity qid,
2) relation 3) neighbour entity value. If the generated citation is complete
with all three parts, and exactly matches a triplet from the question’s
retrieved KG, correctness = 1.

#### Precision

We calculate citation precision for each citation (0 or 1) and average over
all citations to get micro precision. Precision = 1 for a citation if and only
if 1) it is correct, and 2) it matches one knowledge triplet from mininum
knowledge set of the question. (See Figure 2.)

#### Recall

We calculate citation recall for each knowledge (0 or 1) in mininum knowledge
set, and average over all knowledge to get micro recall. Recall = 1 if and
only if the knowledge if hit by a correct citation. (See Figure 2.)

We average over all citations/knowledge in an answer, and average all answer-
level precision/recall to get macro precision and recall. we calculate micro
and macro F1-Score from corresponding precision and recall.

Figure 2: An illustration of how we evaluate the precision and recall for
generated citations.

###  4.3 Text-Citation Alignment

Other than the text quality and citation quality, we measure whether the
generated citations provide support for the corresponding sentences. A piece
of useful knowledge is not an ideal citation if it is irrelevant to the
sentence it links to. Therefore, we propose the metric “Alignment” which
determines whether the generated citations are aligned to the sentences to
which they belong. We use a state-of-the-art natural language inference (NLI)
model TRUE Honovich et al. (2022), which is a fine-tuned T5-11B Raffel et al.
(2020) model, to check whether the generated sentence entails the generated
citation. Since one sentence could have multiple citations, we run NLI on all
sentence-citation pairs and report the percentage of entailment. Additionally,
we conduct human evaluation in § 5.4 to showcase if the automatic evaluation
is correlated with human judgments.

| Micro | Macro
---|---|---
Model | Align. | Corr. | Prec. | Rec. | F1. | Prec. | Rec. | F1.
GPT-4 (0.5) | 92.0(1.5) | 97.6(0.1) | 36.0(0.6) | 43.6(1.0) | 39.4 | 40.7(1.1) | 43.9(1.0) | 42.3
ChatGPT (0.1) | 85.9(2.5) | 96.1(0.4) | 29.0(0.0) | 50.8(0.3) | 36.9 | 32.7(0.4) | 51.2(0.3) | 39.9
ChatGPT (0.5) | 84.5(1.1) | 94.8(0.2) | 29.9(0.2) | 49.0(0.8) | 37.2 | 34.1(0.5) | 49.4(0.9) | 40.4
ChatGPT (0.9) | 84.1(0.5) | 94.2(0.4) | 28.7(0.2) | 49.0(0.3) | 36.2 | 32.5(0.2) | 49.4(0.3) | 39.2
Alpaca-7B | 46.9(0.9) | 78.9(0.6) | 14.9(1.4) | 19.4(0.2) | 16.8 | 19.8(0.4) | 19.9(0.3) | 19.8
LLaMA-7B | 47.8(0.8) | 70.2(0.2) | 7.7(2.4) | 41.1(0.7) | 13.0 | 11.0(1.9) | 41.4(0.7) | 17.4
LLaMA-13B | 62.1(0.4) | 71.7(1.9) | 10.5(3.3) | 43.7(1.0) | 16.9 | 13.8(2.2) | 43.5(1.0) | 20.9
Vicuna-13B | 66.9(0.1) | 59.0(0.6) | 14.9(0.2) | 16.8(0.0) | 15.8 | 15.1(0.0) | 17.0(0.0) | 16.0
Table 3: Citation Quality OpenAI models and LLaMA family models. The first
five metrics are reported in Micro, and the last three metrics are reported in
Macro. We also report text citation alignment

###  4.4 Conscious Incompetence Evaluation

Figure 3: An illustration of how we evaluate the precision and recall for
conscious incompetence ([NA])

Theoretically, each [NA] should map to a piece of knowledge absent from the
retrieved knowledge graph. However, it is difficult to identify if sentence
requires any absent knowledge since there is no ground truth. Therefore, we
conduct a three-round experiment to manually create ground truth for absent
knowledge. In round 1, we select one knowledge from the minimum knowledge set,
and remove it from the ground-truth knowledge graph. We let the LLMs attribute
to this incomplete knowledge graph to generate answers, whereby the removed
knowledge forms the “absent knowledge ground truth”. In subsequent rounds, we
each remove one additional knowledge from the minimum knowledge set,
simulating a knowledge graph with more serious coverage problem.

We employ the NLI model TRUE Honovich et al. (2022) to measure the alignment
between sentences and knowledge. A sentence with [NA] should be aligned to an
absent knowledge. We calculate precision and recall for [NA].

#### [NA] precision

We calculate [NA] precision for each sentence with [NA] (0 or 1) and average
over all sentences with [NA]. Precision = 1 for a sentence if and only if it
entails one knowledge triplet from absent knowledge set of the question. (See
Figure 3.)

#### [NA] Recall

We calculate [NA] recall for each knowledge (0 or 1) in absent knowledge set
and average over all absent knowledge. Recall = 1 if and only if the knowledge
if entailed by a sentence with [NA]. (See Figure 3.)

##  5 Experiments

We run through the method pipeline described in § 3 on different large
language models and present the results in this section. The implementation
details are reported in appendix A. We report four model baselines from two
different model families:

#### OpenAI Models

We use GPT4 (gpt-4-0314) and ChatGPT (gpt-3.5-turbo-0301) for our experiments.
For ChatGPT, we experiment on temperature of 0.1, 0.5, and 0.9 to obtain
different levels of randomness and creativity in generation.

#### LLaMA

We conduct experiments with LLaMA-7B Touvron et al. (2023) and LLaMA-13B since
they are powerful open-source models that are widely accessible. We have also
conducted human instruction tuned LLaMA models, including Alpaca-7B Taori et
al. (2023) and Vicuna-13B Chiang et al. (2023).

###  5.1 Main Results

#### Citation Quality Evaluation

We present the main results in Table 3. For correctness, we report on a micro
scale. For precision, recall, and F1-Score, we report on both micro and macro
scales. The experimental results are the mean of three runs, and the standard
deviation is reported in brackets.

In general, there is a room of improvement for all models, and OpenAI models
outperform the LLaMA family models in almost all metrics. Fine-tuning LLaMA
models may improve their performances on this task. For ChatGPT, temperature
does not play a significant role. The GPT-4 model achieves the best
performance across almost all metrics, except for recall, since GPT-4 models
tend to generate shorter answers with fewer citations, resulting in higher
precision. While LLaMA is better at Recall by generating long answers with
many citations. The F1-Score of models from the same family are close to one
another, showing that our automatic evaluation metric designed is reliable.

#### Text-Citation Alignment

From Table 3, models with 7B, 13B, 175B (ChatGPT), and trillion level (GPT4)
parameters have an alignment of 40+, 60+, 80+, 92 respectively. The parameter
size may play an important role in generating sentences and citations with
good alignment.

#### Text Quality Evaluation

Model | Coh. | Con. | Flu. | Rel.
---|---|---|---|---
GPT-4 (0.5) | 4.48 | 4.89 | 4.64 | 4.72
ChatGPT (0.1) | 4.57 | 4.94 | 4.69 | 4.83
ChatGPT (0.5) | 4.57 | 4.94 | 4.71 | 4.81
ChatGPT (0.9) | 4.52 | 4.91 | 4.67 | 4.79
Alpaca-7B | 4.10 | 4.46 | 4.23 | 3.76
LLaMa-7B | 3.06 | 3.79 | 3.62 | 2.96
LLaMa-13B | 3.60 | 4.23 | 3.94 | 3.56
Vicuna-13B | 3.67 | 4.50 | 3.96 | 3.64
Table 4: Evaluation on generated text quality.

We present the evaluation of generated text quality in Table 4. From the
results, we find that OpenAI models, in general, have better text quality in
all metrics compared to LLaMA family models, which corresponds to the citation
evaluation results. All models exhibit rather high consistency, indicating
that the LLMs are capable of generating answers that are not contradictory to
the provided knowledge or self-contradictory. However, the relevance is
relatively low for smaller models, indicating the difficulty these models face
in generating answers that are relevant to the questions.

###  5.2 Conscious Incompetence

Removed | Corr. | Prec. | Rec. | F1.
---|---|---|---|---
0 (gold) | 95.5 | 30.1 | 57.1 | 39.4
1 | 94.1 | 26.1 | 42.5 | 32.3
2 | 94.0 | 21.0 | 31.4 | 25.2
3 | 93.9 | 16.3 | 20.4 | 18.1
Table 5: Citation quality evaluation for generated texts using a KG with N
pieces of knowledge removed. Figure 4: Precision, Recall, and F1-Score for
[NA].

We first evaluate citation quality of the generated text with knowledge
removed using method described in § 4.4. From Table 5, the removal of required
knowledge has a minimal impact on correctness, but significantly affects
citation precision and recall. With more knowledge absent from provided
knowledge graph, both precision and recall drops drastically, demonstrating
that the coverage issue poses a considerable challenge to generating answers
with high quality citations.

Next, we evaluate [NA] precision and recall. From Figure 4, The recall is
stable at about 15 regardless of the number of absent knowledge. This
indicates that the current LLMs have ability to identify absent knowledge to a
limited extent. While precision and F1-Score exhibit a clear upward trend,
which shows that with more absent knowledge in KG, [NA] enables generated
outputs to locate absent knowledge more accurately. Therefore, the “Conscious
Incompetence” setting plays an increasingly crucial role when the coverage
problem of knowledge graph is more serious.

###  5.3 Retrieval Analysis

We conduct an ablation study to examine the impact of retrieval accuracy on
the model’s output. The experiment simulates retrieval accuracy from 100 to 20
at intervals of 20. We start with the ground truth knowledge graphs that we
used for question construction. In each subsequent rounds, we randomly replace
additional 20% knowledge graphs with irrelevant knowledge graphs to simulate
retrieving wrong graphs. The results for citation quality are in Figure 5.
Answers are generated using ChatGPT with a temperature of 0.5.

The results show clear downward trends in all metrics as expected when
retrieval accuracy dropped. Among precision and recall, the impact of poor
retrieval quality on recall (green) is much more significant than on precision
(yellow). This indicates that the model has the ability to filter out
incorrect knowledge to a certain extent, resulting in less noticeable impact
on precision compared to recall. The reduction in recall was nearly linear as
retrieval accuracy decreased, which is understandable since a knowledge cannot
be cited if it is not provided. The greatest drop in recall occurred between
the ground truth (57.1) and 80 accuracy (42.5), demonstrating the potential of
the model to generate high-quality citations under perfect retrieval
conditions. In practice, a retrieval accuracy of 80 is closest to the actual
scenario of our experiment (our retrieval accuracy is 75.9). Therefore, when
retrieval accuracy is reasonably high, the correctness of citations is not the
most significant concern compared to recall.

Figure 5: Citation evaluation (Micro) of generated texts using knowledge
graphs with retrieval accuracy 100 (gold), 80, 60,40, and 20.

###  5.4 Human Evaluation

| Alignment | Human Avg.
---|---|---
ChatGPT(0.5) | 84.5 | 82.0
LLaMA-7B | 47.8 | 45.5
Vicuna-13B | 66.9 | 64.5
Table 6: Result of Human Evaluation on text-citation alignment

We conduct human evaluation to verify the correlation between automatic
evaluation and human judgment. We randomly sample 100 sentence-citation pairs
from each of the three baselines: ChatGPT (temperature 0.5), LLaMA-7B, and
Vicuna-13B. We request two proficient English annotators for each baseline to
determine if the citatiom aligns to the sentence and provides support for it.
The reason we choose metric alignment here is in appendix B, with instruction
to annotators and IAA.

The comparison between automatically calculated Alignment and human evaluation
results is shown in Table 6. For all three baselines, the automatic and human
scores are close with a gap of within 2.5, despite the significant differences
among the baselines. This indicates a strong correlation between the
automatically calculated alignment and human judgments. The results
demonstrate that the automatic evaluation serves as a reliable measurement of
the alignment between generated texts and citations.

##  6 Related Work

#### Retrieval-augmented LLMs

KiC Pan et al. (2022) empower models with external memory of multiple formats
including knowledge graph but does not explore attribution. WebGPT Nakano et
al. (2021) outsources document retrieval to Microsoft Bing and fine-tunes GPT3
to answer questions. GopherCite Menick et al. (2022) fine-tunes a Gopher Rae
et al. (2021) model to generate text alongside quotes extracted from Google
search. ALCE Gao et al. (2023) retrieves top-k passages from Wikipedia and
asks LLMs to generate outputs with citations to corresponding supporting
documents. These works attribute LLMs to unstructured documents but not
knowledge graph.

#### Evaluation

Rashkin et al. (2021) define the “Attributable to Identified Sources” (AIS) to
measure whether model-generated statements are supported by underlying
sources. Bohnet et al. (2022) study an automatic metric (AutoAIS) that
formulates evaluation of automated question answering as a NLI task. Yue et
al. (2023) investigate the automatic evaluation of attribution by prompting
LLMs and fine-tuning smaller LMs. Liu et al. (2023a) conduct human evaluation
to audit generative search engines for their citation qualities. ALCE Gao et
al. (2023) evaluates generated answers by comparing with gold answers using
MAUVE, and calculates precision and recall for citations using NLI. To the
best of our knowledge, our evaluation methods are the first framework that
requires no human annotated data.

##  7 Conclusion

We propose KaLMA that comprises a new dataset BioKaLMA, a pipeline for
generating attributed answers by retrieving from KGs, and a set of automatic
evaluation metrics to assess text quality, citation quality, and text-citation
alignment. We introduce the “Conscious Incompetence” setting, enabling LLMs to
identify the knowledge required to support the answers but is absent from the
KG. Through this benchmark, we address three challenges: incorporating diverse
attribution sources, limited attribution source coverage, and the absence of
human annotated ground truth for automatic evaluation.

Our extensive experimental results demonstrate that current LLMs still have
room for improvement when utilizing KGs as attribution sources. We also
highlight the increasing effectiveness of “Conscious Incompetence” setting as
the coverage of attribution source becomes worse. Lastly, we prove the crucial
role of retrieval accuracy in generating high-quality attributed texts.

## Limitations

One limitation is that our work only investigates a simple form of knowledge
graph, where each node is an entity, and each sub-graph is a knowledge triple.
There are more complicated forms of knowledge graph, where each node is a
document. We will explore this setting in future works.

Another limitation lies within the text quality evaluation. We uses ChatGPT as
the model to evaluate texts, which could potentially have a bias if the model
prefers the text style generated by itself. Such bias can be observed from the
abnormal phenomenon that the scores of ChatGPT generated answers are higher
than that of the GPT4 generated answers for all four dimensions. Due to cost
considerations, we do not repeat the text quality evaluation with GPT-4.

## References

  * Bohnet et al. (2023) Bernd Bohnet, Vinh Q. Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni, Lierni Sestorain Saralegui, Tal Schuster, William W. Cohen, Michael Collins, Dipanjan Das, Donald Metzler, Slav Petrov, and Kellie Webster. 2023.  Attributed question answering: Evaluation and modeling for attributed large language models.
  * Bohnet et al. (2022) Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, et al. 2022\.  Attributed question answering: Evaluation and modeling for attributed large language models.  _arXiv preprint arXiv:2212.08037_.
  * Bowman et al. (2015) Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015\.  A large annotated corpus for learning natural language inference.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Clark et al. (2021) Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021.  All that’s ‘human’ is not gold: Evaluating human evaluation of generated text.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7282–7296, Online. Association for Computational Linguistics.
  * Curtiss and Warren (1974) Paul R Curtiss and Phillip W Warren. 1974.  The dynamics of life skills coaching. life skills series.
  * Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005.  The pascal recognising textual entailment challenge.  In _Machine learning challenges workshop_ , pages 177–190. Springer.
  * Fu et al. (2023) Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023.  Gptscore: Evaluate as you desire.  _arXiv preprint arXiv:2302.04166_.
  * Gao et al. (2023) Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023.  Enabling large language models to generate text with citations.
  * Honovich et al. (2022) Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022.  True: Re-evaluating factual consistency evaluation.  _arXiv preprint arXiv:2204.04991_.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 874–880, Online. Association for Computational Linguistics.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38.
  * Khot et al. (2018) Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018.  Scitail: A textual entailment dataset from science question answering.  volume 32.
  * Lee et al. (2022) Haejun Lee, Akhil Kedia, Jongwon Lee, Ashwin Paranjape, Christopher D. Manning, and Kyoung-Gu Woo. 2022.  You only need one model for open-domain question answering.
  * Liu et al. (2023a) Nelson F Liu, Tianyi Zhang, and Percy Liang. 2023a.  Evaluating verifiability in generative search engines.  _arXiv preprint arXiv:2304.09848_.
  * Liu et al. (2023b) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b.  G-eval: Nlg evaluation using gpt-4 with better human alignment.
  * Menick et al. (2022) Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. 2022.  Teaching language models to support answers with verified quotes.  _arXiv preprint arXiv:2203.11147_.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021.  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_.
  * Pan et al. (2022) Xiaoman Pan, Wenlin Yao, Hongming Zhang, Dian Yu, Dong Yu, and Jianshu Chen. 2022\.  Knowledge-in-context: Towards knowledgeable semi-parametric language models.  _arXiv preprint arXiv:2210.16433_.
  * Pérez et al. (2009) Jorge Pérez, Marcelo Arenas, and Claudio Gutierrez. 2009.  Semantics and complexity of sparql.  _ACM Transactions on Database Systems (TODS)_ , 34(3):1–45.
  * Pillutla et al. (2021) Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021.  Mauve: Measuring the gap between neural text and human text using divergence frontiers.  _Advances in Neural Information Processing Systems_ , 34:4816–4828.
  * Plum et al. (2022) Alistair Plum, Tharindu Ranasinghe, Spencer Jones, Constantin Orasan, and Ruslan Mitkov. 2022.  Biographical semi-supervised relation extraction dataset.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 3121–3130.
  * Rae et al. (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021.  Scaling language models: Methods, analysis & insights from training gopher.  _arXiv preprint arXiv:2112.11446_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Rashkin et al. (2021) Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2021\.  Measuring attribution in natural language generation models.  _arXiv preprint arXiv:2112.12870_.
  * Reddy et al. (2023) Revanth Gangi Reddy, Yi R Fung, Qi Zeng, Manling Li, Ziqi Wang, Paul Sullivan, et al. 2023.  Smartbook: Ai-assisted situation report generation.  _arXiv preprint arXiv:2303.14337_.
  * Salvagno et al. (2023) Michele Salvagno, Fabio Silvio Taccone, Alberto Giovanni Gerli, et al. 2023.  Can artificial intelligence help for scientific writing?  _Critical care_ , 27(1):1–5.
  * Schuster et al. (2021) Tal Schuster, Adam Fisch, and Regina Barzilay. 2021.  Get your vitamin C! robust fact verification with contrastive evidence.  pages 624–643.
  * Sevgili et al. (2022) Özge Sevgili, Artem Shelmanov, Mikhail Arkhipov, Alexander Panchenko, and Chris Biemann. 2022.  Neural entity linking: A survey of models based on deep learning.  _Semantic Web_ , (Preprint):1–44.
  * Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021.  Retrieval augmentation reduces hallucination in conversation.  _arXiv preprint arXiv:2104.07567_.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023.  Alpaca: A strong, replicable instruction-following model.  _Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html_ , 3(6):7.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and VERification.  pages 809–819.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: a free collaborative knowledgebase.  _Communications of the ACM_ , 57(10):78–85.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.
  * Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023.  Wizardlm: Empowering large language models to follow complex instructions.
  * Yang et al. (2019) Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019.  End-to-end open-domain question answering with bertserini.  _arXiv preprint arXiv:1902.01718_.
  * Yue et al. (2023) Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. 2023.  Automatic evaluation of attribution by large language models.  _arXiv preprint arXiv:2305.06311_.
  * Zhang et al. (2019a) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019a.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_.
  * Zhang et al. (2019b) Yuan Zhang, Jason Baldridge, and Luheng He. 2019b.  PAWS: Paraphrase adversaries from word scrambling.  pages 1298–1308.
  * Zuccon and Koopman (2023) Guido Zuccon and Bevan Koopman. 2023.  Dr chatgpt, tell me what i want to hear: How prompt knowledge impacts health answer correctness.  _arXiv preprint arXiv:2302.13793_.

##  Appendix A Implementation Details

###  A.1 Main Experiment

For the main experiments, we run reach model with different seeds for three
times. The OpenAI family models are implemented using OpenAI APIs. Running one
round of experiment with ChatGPT model takes approximately 1 hour, and costs
about 3 USD. Running one round of experiment with GPT4 model takes
approximately 1.5 to 2 hours, and costs about 60 USD. Each LLaMA family model
is run on one TESLA V100 GPU, where each run takes about 6 to 8 hours for
Alpaca-7B and Vicuna-13B, and about 12-16 hours for LLaMA-7B and LLaMA-13B.

###  A.2 Text Quality Evaluation

For text quality evaluation, we use the model “text-davinci-003” with
temperature 0 to ensure stability and reproducibility of the results. We
randomly sample 100 outputs from each baseline and take three runs to report
mean. We do not report standard deviation since most of them are mostly
insignificantly small (below 0.1).

###  A.3 NLI

For the automatic evaluation of text citation alignment and evaluation of the
known unknown citations, we implement the TRUE model from HuggingFace555
https://huggingface.co/google/t5_xxl_true_nli_mixture, which was trained on
SNLI Bowman et al. (2015), MNLI Williams et al. (2018), Fever Thorne et al.
(2018), Scitail Khot et al. (2018), PAWS Zhang et al. (2019b), and VitaminC
Schuster et al. (2021). The model uses the prompt of “premise: {PREMISE}
hypothesis: {HYPOTHESIS}”. For each sentence citation pair, we place the
sentence in the “PREMISE”, and the citation to the “HYPOTHESIS”, like the
following: “premise: {Hertwig served as a professor at the University of Jena
for the last 40 years of his career.} hypothesis: {employer: University of
Jena}”

##  Appendix B Human Evaluation

Among text quality evaluation, citation quality evaluation, and text-citation
alignment, we conduct human evaluation on text-citation alignment. Text
quality evaluation is conducted using GPT-EVAL. We acknowledge this is not a
perfect metric, but the human evaluation is conducted in Liu et al. (2023b).
The focus is this paper is not to improve G-Eval. Citation quality evaluation
is conducted with looking for exact match between generated citations and
minimum knowledge set, which is an objective evaluation. The text-citation
alignment evaluation is conducted using NLI, which we are not certain if
entailment means providing support. In addition, whether a knowledge supports
a sentence can be subjective. Therefore, we conduct human evaluation on
alignment.

We present the Human Evaluation Instructions provided to the annotators in
Table 7. We follow the implementation from Clark et al. (2021), and provide
detailed instructions and examples to improve evaluation accuracy.

The Inter-Annotator Agreement for ChatGPT, LLaMA-7B, and Vicuna-13B are
reported as follows: 90%, 97%, and 89% respectively.

Annotation Method:
---
Each evaluation content includes a sentence and a piece of knowledge. Our task
is to determine whether this sentence contains the given knowledge, i.e.,
whether this knowledge provides support for the sentence. If the sentence does
not mention the given knowledge or if the content of the sentence does not
align with the knowledge, it is considered unsupported. We use 1 to indicate
support and 0 to indicate lack of support.
Here are some examples:
Sentence: Stephen Crane was an American writer born on November 1, 1871, in
Newark, and died on June 5, 1900, in Badenweiler.
Knowledge: date of birth: 1871-11-01
Result: 1, because the sentence’s date of birth matches the knowledge’s date
of birth.
Sentence: Merton died on December 10, 1968, in Bangkok, Thailand.
Knowledge: country of citizenship: United States of America
Result: 0, because the sentence does not mention Merton’s nationality.
Table 7: Instruction we provide to the human annotators.

##  Appendix C Prompts

We present the prompts and instructions we used in Table 8, 9, 10, 11, 12, 13,
and 14.

Instruction: I want to you act as a Question Generator. Your objective is to
select relevant knowledge to label the sentence and generate a question
---
sentence: Artemisia Gentileschi was born Artemisia Gentileschi Lomi in Rome on
July 8 1593 although her birth certificate from the Archivio di Stato
indicated she was born in 1590 the eldest child of the Tuscan painter Orazio
Gentileschi and Prudenzia di Ottaviano Montoni.
knowledge: {qid: Q367360, name: Orazio Gentileschi, sex or gender: male, place
of birth: Pisa, place of death: London, instance of: human, occupation:
painter, child: Artemisia Gentileschi, described by source: The Great Theatre
of Dutch Painters, notable works: Diana the Huntress, given name: Orazio,
topic’s main category: Category:Orazio Gentileschi, surname: Gentileschi,
genre: portrait, languages spoken: Italian, movement: mannerism, work
location: Rome, ethnic group: Italians, date of birth: 1563-07-19, date of
death: 1639-02-07}
{qid: Q212657, name: Artemisia Gentileschi, sex or gender: female, place of
birth: Rome, place of death: Naples, instance of: human, occupation: painter,
member of: Accademia delle Arti del Disegno, father: Orazio Gentileschi,
described by source: The Great Theatre of Dutch Painters, notable works:
Judith Slaying Holofernes, topic’s main category: Category:Artemisia
Gentileschi, movement: Caravaggisti, ethnic group: Italians, work location:
Florence, depicted by: Artemisia, field of work: painting, surname:
Gentileschi, genre: portrait, languages spoken: Italian, position held: court
painter, student of: Orazio Gentileschi, spouse: Pierantonio Stiattesi, given
name: Artemisia, mother: Prudenzia di Ottaviano Montoni, date of birth:
1596-07-08, date of death: 1654-01-01}
Generated Answer:  Artemisia Gentileschi [qid: Q212657, name: Artemisia
Gentileschi] was born Artemisia Gentileschi Lomi in Rome [qid: Q212657, place
of birth: Rome] on July 8 1593 [qid: Q212657, date of birth: 1596-07-08]
although her birth certificate from the Archivio di Stato indicated she was
born in 1590 the eldest child of the Tuscan painter Orazio Gentileschi [qid:
Q212657, father: Orazio Gentileschi] [qid: Q367360, name: Orazio Gentileschi,
occupation: painter] and Prudenzia di Ottaviano Montoni.
Generated Question: When was Artemisia Gentileschi born?
Table 8: Prompts for initial round of evolutionary question construction.
Instruction: I want to you act as a Question Rewriter. Your objective is to
extend the question and answer, so the they include the additional knowledge.
There should be only one question when possible, if not, make sure the
question is a few as possible.
---
answer: Artemisia Gentileschi [qid: Q212657, name: Artemisia Gentileschi] was
born Artemisia Gentileschi Lomi in Rome [qid: Q212657, place of birth: Rome]
on July 8 1593 [qid: Q212657, date of birth: 1596-07-08] although her birth
certificate from the Archivio di Stato indicated she was born in 1590 the
eldest child of the Tuscan painter Orazio Gentileschi [qid: Q212657, father:
Orazio Gentileschi] [qid: Q367360, name: Orazio Gentileschi, occupation:
painter] and Prudenzia di Ottaviano Montoni.
question: When was Artemisia Gentileschi born?
knowledge: {qid: Q212657, name: Artemisia Gentileschi, notable works: Judith
Slaying Holofernes}
Generated Answer:  Artemisia Gentileschi [qid: Q212657, name: Artemisia
Gentileschi] was born Artemisia Gentileschi Lomi in Rome [qid: Q212657, place
of birth: Rome] on July 8 1593 [qid: Q212657, date of birth: 1596-07-08]
although her birth certificate from the Archivio di Stato indicated she was
born in 1590 the eldest child of the Tuscan painter Orazio Gentileschi [qid:
Q212657, father: Orazio Gentileschi] [qid: Q367360, name: Orazio Gentileschi,
occupation: painter] and Prudenzia di Ottaviano Montoni. Under the influence
of her father, Artemisia Gentileschi created her iconic painting Judith
Slaying Holofernes [qid: Q212657, notable works: Judith Slaying Holofernes]
when she was around twenty years old.
Generated Question: When was Artemisia Gentileschi born and under what
circumstance did she create her representative work?
Table 9: Prompts for evolutionary question construction for round 2 to 6.
Instruction: You answer the question based on your knowledge, with the given
information for annotation, following the given format. Use [NA] for claims
that need annotation but is unprovided.
---
Question: Considering the information:
{name: Orazio Gentileschi, place of death: London, child: Artemisia
Gentileschi, notable works: Diana the Huntress, given name: Orazio, surname:
Gentileschi, languages spoken: Italian, movement: mannerism, work location:
Rome, ethnic group: Italians, date of birth: 1563-07-19, date of death:
1639-02-07, qid: Q367360}
{name: Artemisia Gentileschi, place of birth: Rome, place of death: Naples,
occupation: painter, member of: Accademia delle Arti del Disegno, father:
Orazio Gentileschi, notable works: Judith Slaying Holofernes, movement:
Caravaggisti, ethnic group: Italians, work location: Florence, depicted by:
Artemisia, surname: Gentileschi, languages spoken: Italian, student of: Orazio
Gentileschi, given name: Artemisia, mother: Prudenzia di Ottaviano Montoni,
date of death: 1654-01-01, qid: Q212657}
How did Orazio Gentileschi’s influence on Artemisia’s life and career shape
her development as a prominent Baroque painter, despite facing significant
challenges as a female artist in a male-dominated field?
Answer:  Artemisia Gentileschi was an Italian painter born on July 8, 1596
[NA] in Rome [Q212657, ethnic group: Italians, occupation: painter, place of
birth: Rome]. She was a member of the Accademia delle Arti del Disegno and is
best known for her work Judith Slaying Holofernes [Q212657, member of:
Accademia delle Arti del Disegno, notable works: Judith Slaying Holofernes].
She was the eldest child of the Tuscan painter Orazio Gentileschi [Q212657,
father: Orazio Gentileschi]. Orazio Gentileschi was an Italian painter [NA]
born in 1563 and died in 1639 [Q367360, ethnic group: Italians, date of birth:
1563-07-19, date of death: 1639-02-07]. He was born in Pisa and died in London
[Q367360, place of death: London]. Orazio’s influence on Artemisia’s
development as a prominent Baroque [NA] painter can be seen in her highly
naturalistic portrayal of figures, dramatic scenes and the use of chiaroscuro
technique [NA]. He also provided her with the opportunity to study with him
and learn from his experience and expertise. She became an important second-
generation proponent of Caravaggio’s dramatic realism [Q212657, movement:
Caravaggisti].
Table 10: Full prompt for answer generation with citaion. Instruction: You
will be given one question and answer. Your task is to rate the answer on one
metric. Please make sure you read and understand these instructions carefully.
Please keep this document open while reviewing, and refer to it as needed.
---
Evaluation Criteria:
Coherence (1-5) - the collective quality of all sentences. We align this
dimension with the DUC quality question of structure and coherence whereby the
answer should be well-structured and well-organized. The answer should not
just be a heap of related information, but should build from sentence to
sentence to a coherent body of information about a topic.
Evaluation Steps:
1\. Read the questions carefully and identify the main topic and key points.
2\. Read the answer and compare it to the question. Check if the answer covers
the main topic and key points of the question, and if it presents them in a
clear and logical order.
3\. Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest
and 5 is the highest based on the Evaluation Criteria.
Table 11: Instruction for text evaluation with GPT-EVAL - Coherence
Instruction: You will be given one question and answer. Your task is to rate
the answer on one metric. Please make sure you read and understand these
instructions carefully. Please keep this document open while reviewing, and
refer to it as needed.
---
Evaluation Criteria:
Consistency (1-5) - the answer should be consistent with the given knowledge.
The answer should also be self-consistent, without any contradiction to
itself.
Evaluation Steps:
1\. Read the question and knowledge carefully.
2\. Read the answer and compare it to the knowledge. Check if the answer is
consistent with the give knowledge.
3\. Assign a score for consistency on a scale of 1 to 5, where 1 is the lowest
and 5 is the highest based on the Evaluation Criteria.
Table 12: Instruction for text evaluation with GPT-EVAL - Consistency
Instruction: You will be given one question and answer. Your task is to rate
the answer on one metric. Please make sure you read and understand these
instructions carefully. Please keep this document open while reviewing, and
refer to it as needed.
---
Evaluation Criteria:
Fluency (1-5) - the answer should be written in fluent language. The answer
should use appropriate vocabulary, grammar, and sentence structures that
enable readers or listeners to comprehend the content effortlessly.
Evaluation Steps:
1\. Read the question carefully.
2\. Read the answer and check if the language in the answer is fluent.
3\. Assign a score for fluency on a scale of 1 to 5, where 1 is the lowest and
5 is the highest based on the Evaluation Criteria.
Table 13: Instruction for text evaluation with GPT-EVAL - Fluency Instruction:
You will be given one question and answer. Your task is to rate the answer on
one metric. Please make sure you read and understand these instructions
carefully. Please keep this document open while reviewing, and refer to it as
needed.
---
Evaluation Criteria:
Relevance (1-5) - the answer should be relevant to the question. The answer
should directly answers the question, without providing any irrelevant
information.
Evaluation Steps:
1\. Read the question carefully.
2\. Read the answer and compare with the question to check if it fully answers
the question and have no redundancies.
3\. Assign a score for relevance on a scale of 1 to 5, where 1 is the lowest
and 5 is the highest based on the Evaluation Criteria.
Table 14: Instruction for text evaluation with GPT-EVAL - Relevance

##  Appendix D Evolutionary Question Generation

We provide an example of evolutionary question generation in Table 15.

Round 1:
---
Paragraph: Abanindranath Tagore was born in Jorasanko Calcutta British India
to Gunendranath Tagore and Saudamini Tagore.
Question: When and where was Abanindranath Tagore born?
Round 2:
Input Knowledge: [qid: Q691796, name: Abanindranath Tagore, employer:
University of Calcutta]
Paragraph: Abanindranath Tagore was born in Jorasanko Calcutta British India
to Gunendranath Tagore and Saudamini Tagore. Abanindranath Tagore was a
prominent Indian artist and the founder of the Bengal School of Art . When he
was young, he was greatly influenced by the Mughal and Rajput paintings, which
later became the inspiration for his own works. Abanindranath Tagore’s
artistic career was marked by his efforts to revive traditional Indian art
forms and techniques. He was a teacher at the University of Calcutta.
Question: Can you provide more information about Abanindranath Tagore’s
artistic career?
Round 3:
Input Knowledge: [qid: Q691796, name: Abanindranath Tagore, notable works:
Bharat Mata]
Paragraph: Abanindranath Tagore was born in Jorasanko Calcutta British India
to Gunendranath Tagore and Saudamini Tagore. Abanindranath Tagore was a
prominent Indian artist and the founder of the Bengal School of Art . When he
was young, he was greatly influenced by the Mughal and Rajput paintings, which
later became the inspiration for his own works. Abanindranath Tagore’s
artistic career was marked by his efforts to revive traditional Indian art
forms and techniques. He was a teacher at the University of Calcutta and his
notable works include "Bharat Mata" , "The Passing of Shah Jahan" and "The
LastDays of Shah Jahan"
Question: Can you provide more information about Abanindranath Tagore’s
artistic career and notable works?
Round 4:
Input Knowledge: [qid: Q691796, name: Abanindranath Tagore, date of birth:
1987-08-07]
Paragraph: Abanindranath Tagore was born on August 7, 1871 in Jorasanko
Calcutta British India to Gunendranath Tagore and Saudamini Tagore.
Abanindranath Tagore was a prominent Indian artist and the founder of the
Bengal School of Art. When he was young, he was greatly influenced by the
Mughal and Rajput paintings, which later became the inspiration for his own
works. Abanindranath Tagore’s artistic career was marked by his efforts to
revive traditional Indian art forms and techniques. He was a teacher at the
University of Calcuttaand his notable works include "Bharat Mata", "The
Passing of Shah Jahan" and "The Last Days of Shah Jahan". Abanindranath
Tagore’s works were characterized by their use of Indian themes and motifs,
and his style was a fusion of Indian and Western art techniques. His works
were highly influential in the development of modern Indian art, and he is
considered one of the most important artists of the Bengal School
Question: Can you introduce Abanindranath Tagore, his artistic career, notable
works, and his influence on modern Indian art?
Table 15: An example of the generated questions after each round in
evolutionary question construction

##  Appendix E Examples

We show examples of the attributed answers generated by the LLMs in Table 16
and 17.

Question:
---
What were some of Stephen Crane’s notable works that reflected his views on
religion and the harsh realities of life, and how did his experiences at
Syracuse University influence his writing? Also, what was the name of the
sports team he was a member of at Syracuse University?
Retrieved Knowledge:
[qid: Q206534, name: Stephen Crane, sex or gender: male, place of birth:
Newark, place of death: Badenweiler, country of citizenship: United States of
America, cause of death: Tuberculosis, place of burial: Evergreen Cemetery,
alma mater: Syracuse University, instance of: human, given name: Stephen,
medical condition: Tuberculosis, occupation: writer, described by source:
Great Soviet Encyclopedia (1969â€“1978), manner of death: natural causes,
movement: literary realism, notable works: The Red Badge of Courage, surname:
Crane’, "topic’s main category": ’Category:Stephen Crane, religion: atheism,
languages spoken: English, mother: Mary Helen Peck Crane, sport: baseball,
member of sports team: Syracuse Orange baseball, father: Jonathan Townley
Crane, name: Stephen Crane, date of birth: 1871-11-01, date of death:
1900-06-05]
Generated Output:
Stephen Crane was a male writer born on November 1, 1871, in Newark, United
States of America, and died on June 5, 1900, in Badenweiler due to
tuberculosis [Q206534, sex or gender: male, place of birth: Newark, place of
death: Badenweiler, country of citizenship: United States of America, cause of
death: Tuberculosis, medical condition: Tuberculosis, date of birth:
1871-11-01, date of death: 1900-06-05]. Crane was a prominent literary realist
writer [Q206534, movement: literary realism] who wrote about the harsh
realities of life and his atheistic views on religion [Q206534, religion:
atheism]. His notable works include "The Red Badge of Courage" [Q206534,
notable works: The Red Badge of Courage].
Crane attended Syracuse University and was a member of the Syracuse Orange
baseball team [Q206534, alma mater: Syracuse University, sport: baseball,
member of sports team: Syracuse Orange baseball]. His experiences at Syracuse
University influenced his writing, and he wrote several stories about college
life, including "The Eighty-Yard Run," which was based on his own experiences
as a football player at Syracuse [NA].
Table 16: An example of the attributed answer generated by ChatGPT
(temperature 0.5) Question:
---
What were some of Stephen Crane’s notable works that reflected his views on
religion and the harsh realities of life, and how did his experiences at
Syracuse University influence his writing? Also, what was the name of the
sports team he was a member of at Syracuse University?
Retrieved Knowledge:
[qid: Q206534, name: Stephen Crane, sex or gender: male, place of birth:
Newark, place of death: Badenweiler, country of citizenship: United States of
America, cause of death: Tuberculosis, place of burial: Evergreen Cemetery,
alma mater: Syracuse University, instance of: human, given name: Stephen,
medical condition: Tuberculosis, occupation: writer, described by source:
Great Soviet Encyclopedia (1969â€“1978), manner of death: natural causes,
movement: literary realism, notable works: The Red Badge of Courage, surname:
Crane’, "topic’s main category": ’Category:Stephen Crane, religion: atheism,
languages spoken: English, mother: Mary Helen Peck Crane, sport: baseball,
member of sports team: Syracuse Orange baseball, father: Jonathan Townley
Crane, name: Stephen Crane, date of birth: 1871-11-01, date of death:
1900-06-05]
Generated Output:
Stephen Crane was an American writer born on November 1, 1871, in Newark, and
died on June 5, 1900, in Badenweiler [Q206534, date of birth: 1871-11-01, date
of death: 1900-06-05, place of birth: Newark, place of death: Badenweiler]. He
was a part of the literary realism movement and is best known for his notable
work, The Red Badge of Courage [Q206534, movement: literary realism, notable
works: The Red Badge of Courage]. Crane’s views on religion were atheistic
[Q206534, religion: atheism]. His works often reflected the harsh realities of
life, such as war, poverty, and human suffering [NA].
Crane attended Syracuse University [Q206534, alma mater: Syracuse University],
where his experiences likely influenced his writing. He was exposed to a
diverse range of ideas and perspectives, which may have contributed to his
realistic approach to literature [NA].
While at Syracuse University, Stephen Crane was a member of the Syracuse
Orange baseball team [Q206534, member of sports team: Syracuse Orange
baseball].
Table 17: An example of the attributed answer generated by GPT-4