# Large language models as oracles for instantiating ontologies
with domain-specific knowledge

Giovanni Ciatto<EMAIL_ADDRESS>Andrea Agiollo
<EMAIL_ADDRESS>Matteo Magnini<EMAIL_ADDRESS>Andrea Omicini
<EMAIL_ADDRESS>
###### Abstract

Background. Endowing intelligent systems with semantic data commonly requires
designing and instantiating ontologies with domain-specific knowledge.
Especially in the early phases, those activities are typically performed
_manually_ by human experts possibly leveraging on their own experience. The
resulting process is therefore time-consuming, error-prone, and often biased
by the personal background of the ontology designer. Objective. To mitigate
that issue, we propose a novel domain-independent approach to automatically
instantiate ontologies with domain-specific knowledge, by leveraging on large
language models (LLMs) as oracles. Method. Starting from  _(i)_ an initial
schema composed by inter-related classes and properties and  _(ii)_ a set of
query templates,  our method queries the LLM multiple times, and generates
instances for both classes and properties from its replies. Thus, the ontology
is _automatically_ filled with domain-specific knowledge, compliant to the
initial schema. As a result, the ontology is quickly and automatically
enriched with manifold instances, which experts may consider to keep, adjust,
discard, or complement according to their own needs and expertise.
Contribution. We formalise our method in general way and instantiate it over
various LLMs, as well as on a concrete case study. We report experiments
rooted in the nutritional domain where an ontology of food meals and their
ingredients is semi-automatically instantiated from scratch, starting from a
categorisation of meals and their relationships. There, we analyse the quality
of the generated ontologies and compare ontologies attained by exploiting
different LLMs. Finally, we provide a SWOT analysis of the proposed method.

###### keywords:

ontology population , large language models , nutrition , automation , domain-
specific knowledge

††journal: Knowledge-Based Systems

[disi] organization=Department of Computer Science and Engineering (DISI),
Alma Mater Studiorum – Università di Bologna, addressline=via dell’Università
50, city=Cesena, FC, postcode=47522, country=Italy

##  1 Introduction

Nowadays, the demand for intelligent systems capable of understanding,
reasoning, and interacting with complex information is becoming paramount.
However, the possibility of truly intelligent systems hinges upon the
incorporation of _machine-interpretable_ knowledge representations that can
capture humans’ _domain-specific_ knowledge from diverse domains.

The rapid proliferation of data and data representation formats, coupled with
the need for intelligent systems to comprehend and contextualise ambiguous,
vague, and often incomplete information, brings about the question “how can we
precisely represent domain-specific information in a machine-understandable
manner?”, whose answer is rooted in the notion of _ontology_ [1].

Regardless of the particular technological reification of choice, ontologies
are formal, extensional representations of knowledge following the principles
of description logics. There, concepts and relationships among concepts are
formally defined and expressed in a machine-readable format, facilitating
computational systems to reason about the world in a manner akin to human
cognition. In other words, ontologies provide a means to convey human
knowledge in a machine-understandable way.

Unlike sub-symbolic approaches that rely on distributed or loosely organised
data [2], ontologies provide a structured framework that defines concepts,
relationships, and properties within a specific domain. Such a structured
knowledge representation is a key enabling factor in bridging the semantic
gap, pushing AI systems beyond superficial pattern recognition and possibly
grasp the underlying meaning of the data.

Hence, from the human perspective, ontologies provide a shared vocabulary and
an unambiguous understanding of domain-specific knowledge. This comes at the
cost of a meticulous and time-consuming process of ontology creation, which
should be performed in such a way to guarantee some degree of adherence among
the ontology and the real-world domain it represents.

Currently, ontologies are populated either manually, by domain experts or
communities, or through semi-automatic extraction from data [3]. On the one
hand, manual ontology population is time- and effort-consuming, and possibly
affected by humans’ errors and biases. However, if the population process is
long and inclusive enough, it may easily yield higher-quality (in terms of
adherence to reality) results on the long run. On the other hand, data-driven
approaches offer speed and scalability but might yield lower-quality
ontologies. In this case, low quality may be due to biases in the data or to
the characteristic of the extraction procedure (e.g. data may be under-
representing some aspects of the domain, or the procedure may trade off
completeness for speed).

Arguably, the ideal population procedure would involve both human- and data-
driven approaches: the latter should provide “mass” to the ontology, while the
former should refine and revise its details. Along this line, this paper
introduces KGFiller: a novel approach that aims at amalgamating the strengths
of both human- and data-driven ontology population methods.

Leveraging the observation that large language models (LLM) are trained on
various data from the entire web, we hypothesise that these models encapsulate
a substantial amount of domain-specific knowledge. Based on this premise, we
propose an automated procedure, KGFiller, to extract domain-specific knowledge
from LLM and use it for populating ontologies. Starting from  _(i)_ an initial
schema composed by inter-related classes and properties, and  _(ii)_ a set of
query templates  our method queries the LLM multiple times, and it generates
instances for both classes, relationships, and properties from its replies.
Further queries are performed to refine the ontology, and to balance the
instances in the class hierarchy. In this way, the ontology is _automatically_
filled with domain-specific knowledge from the LLM, while remaining compliant
to the initial schema. As result, the ontology is quickly and automatically
enriched with manifold instances, which experts may consider to keep, adjust,
discard, or complement according to their own needs and expertise.

While data-driven, our method comes with several peculiarities w.r.t. other
state-of-the-art approaches. First, it is not tailored upon any specific
dataset, but rather it uses LLM as oracles to generate data. Second, it
supports not only the population of the ontology, but its refinement as well.
Third, it is incremental, in the sense that it can be applied to already-
instantiated ontologies, hence enriching or populating them even more.
Finally, it is general-purpose, in the sense that it can be applied to
virtually any domain, and to different LLM.

To validate our approach, we provide a Python implementation of KGFiller, and
we let it populate a custom OWL ontology of ours – tailored to the nutritional
domain –, and we evaluate the quality of the resulting ontology. The
evaluation involves a thorough inspection of the populated instances,
assessing the meaningfulness and correct placement of LLM-generated
individuals within the ontology structure. We repeat the experiment with
different LLMs, and we compare the results.

Broadly speaking, our method contributes to the ongoing discourse on ontology
population methods, presenting a promising hybrid approach that harnesses the
power of LLMs to enhance the efficiency and accuracy of the ontology creation
process. Along this line, to deepen the discussion about the strenghts,
weaknesses, opportunities, and threats of our method, we dedicate an entire
section to a SWOT analysis of our approach.

Accordingly, the remainder of this paper is organised as follows. Section 2
provides background on ontologies, LLMs, and related work. Section 3 presents
the KGFiller method. Section 4 introduces a case study where KGFiller is
validated, and results from different LLM families are compared. After Section
5 reports a SWOT analysis of our method, Section 6 concludes the paper.

##  2 Background

In this section, we recall basic theoretical and practical notions about
ontologies, and we summarise the state of the art in ontology population.
Then, we provide an overview on large language models.

###  2.1 Ontologies, Description Logics, and Semantic Web

An ontology is “a formal explicit specification of a shared conceptualisation
of a domain of interest” [1]. It describes the entities involved in a specific
domain, and their properties, as well as the relationships among them. Inside
an ontology, information is organised and represented in such a way that it
can be interpreted by both humans and machines.

More formally, in the remainder of this paper, we will follow the syntactical
convention, as well as the semantics, of the well-known $\mathcal{ALC}$
description logics [4]. In $\mathcal{ALC}$, one may express an ontology by
means of a logic theory, i.e. a set of axioms aimed at specifying:

classes

(a.k.a. concepts) i.e. groups of entities of the same sort. Mathematically
speaking, classes are sets. We denote classes either by their mnemonic names,
written in capital italics – e.g.: $\mathit{{A}nimal}$ may represent the class
of all animals –, or by expressions composing simpler classes via algebraic
operations such as union ($\sqcup$), intersection ($\sqcap$), or negation
($\neg$).

instances

(a.k.a. individuals), i.e. entities of interest. Mathematically speaking,
instances are constant values representing entity identifiers. We denote
instances by mnemonic names too, written in lowercase, monospaced font, e.g.:
$\mathtt{{t}om}$. Entities may be part of one or more classes: we use operator
$(\cdot:\cdot)$ to denote the “is a” relation. So, for instance, we write
$\mathtt{{t}om}:\mathit{{C}at}$ to denote the statement “Tom is a cat”.

properties

(a.k.a. roles), i.e. relationships among pairs of instances. Mathematically
speaking, roles are named, binary relations. We denote properties by mnemonic
names too, written in lowercase, sans-serif font, e.g. $\mathsf{{e}ats}$. As
binary relations, properties require specifying which pair of concepts they
are linking together. So, for instance, we write
$\mathsf{{e}ats}\subseteq\mathit{{A}nimal}\times\mathit{{E}dible}$ to denote
one animal may, in general, eat something edible. In that case, we say that
$\mathit{{A}nimal}$ is the _domain_ of $\mathsf{{e}ats}$, and
$\mathit{{E}dible}$ is its _range_. For any two actual instances in the
relation, we use the prefix, functional notation to denote that the
relationship holds among those two instances, e.g.
$\mathsf{{e}ats}(\mathtt{{t}om},\mathtt{{j}erry})$.

Both concepts and roles may be subject to the subsumption (a.k.a. subclass)
relation ($\sqsubseteq$), which essentially corresponds to set inclusion. So,
for instance, $\mathit{{C}at}\sqsubseteq\mathit{{A}nimal}$ states that all
cats are also animals, whereas
$\mathsf{{p}redatorOf}\sqsubseteq\mathsf{{e}ats}$ means that, for any couple
of animals, if one eats another, then the former is a predator for the latter.

As far as subsumpiton is concerned, two special concepts are introduced in the
$\mathcal{ALC}$ logic, namely $\top$ and $\bot$—i.e., the set of all instances
and the set with no instances, respectively. These are used to denote the most
general and the most specific concepts, respectively. Thanks to subsumption,
concepts (and properties) can be organised hierarchically, forming a
taxonomy—i.e. a graph where nodes are concepts and arcs are subsumption
relations.

Summarising, any $\mathcal{ALC}$ theory would consist of  _(i)_
_terminological_ axioms (a.k.a. TBox), aimed at defining concepts and
properties, as well subsumption or equivalence relations among them, and
_(ii)_ _assertion_ axioms (a.k.a. ABox), aimed at assigning instances to
either classes or properties.  When it comes to practice, and a new ontology
is being defined from scratch, the TBox is usually defined first, in the
_modelling_ phase, and the ABox is defined next, in the _population_ phase.
Both phases could be iterated multiple times until the ontology is deemed
complete and satisfactory, but the key point is: there is a modelling phase,
where concepts and properties are defined, and a population phase, where
instances are assigned to them.

Technologically speaking, ontologies are most often expressed by means of
Semantic Web [5] languages, such as the Web Ontology
Language111https://www.w3.org/OWL/ (OWL) or the Resource Description
Framework222https://www.w3.org/2001/sw/wiki/RDF (RDF). These languages come
along with concrete syntaxes for representing ontologies, as well as with
software tools facilitating their manipulation, visualisation, and processing.
Among these, we rely upon Protégé [6] for the manual editing and inspection of
ontologies, and Owlready2333https://github.com/pwin/owlready2 [7] for
automating ontology manipulation by means of the Python language.

###  2.2 Related works on Ontology Population

The problem of filling ontologies with instances is well-known in the Semantic
Web community [8]. Put it simply, the problem consist of expanding an ontology
with further assertions (i.e., ABox axioms), compliant with the concepts and
properties therein defined (i.e., TBox axioms). Such problem is commonly
referred to in the literature as “ontology population”—or “learning”, when the
authors want to stress that the ontology is being populated via some
(semi-)automatic procedure based on data processing [3]. Some authors have
also used the expression “ontology learning” to refer to the process of
(semi-)automatically inferring the whole ontology – there including TBox
axioms too –, hence why we stick to the “ontology population” nomenclature.

The need of a (semi-)automatic procedure to populate ontologies is motivated
by the fact that _manual_ population activities are time-consuming and error-
prone, and therefore inherently costly [9].

Several methods for structured data extraction from text have been proposed in
the literature to serve the purpose of semi-automatic ontology population.
Broadly speaking, these methods can be classified into two main categories:
_linguistics-based_ and _machine-learning_ (ML) approaches [10].

Linguistics-based approaches [11, 12, 13] are early techniques, including, but
not limited to: analysis of the syntactic structure of sentences, pattern-
based extraction, part-of-speech tagging, dictionaries, etc. Because these
approaches are not fully automatic, the main problem is that they still need
lots of human effort even if they partially reduce it (e.g., corpora
selection, supervision, revision, parameter fine-tuning, etc.). Moreover,
these methods are not easy to use out of the example domain provided in the
articles due to domain-dependant characteristics of the algorithms (e.g.,
different corpora might lead to different statistical information for the same
terms, therefore different thresholds may be required).

ML approaches may either rely on statistic-based shallow ML, or deep learning.
Statistic-based approaches attempt to convert free text into numerical data –
by relying on classical NLP techniques such as: term frequency-inverse
document frequency, co-occurrence, point-wise mutual information, etc. – which
is later used to feed shallow data-mining algorithms—such as: support vector
machines, decision trees, random forest, etc. The main limitation of methods
in this category is that they may struggle to capture contextual information
from the text. The interested reader may find examples of methods from this
category in the following works [14, 15, 16, 17, 18, 19, 20].

Recent trends in ontology population involve the application of deep learning
techniques [21, 22, 23]. Differently than shallow ML approaches, which rely on
hand-crafted, statistical representations of text, deep learning approaches
attempt to learn such representations automatically, from the data, by relying
on deep neural networks. Networks of these kind are trained on large amounts
of textual data, and they are capable of learning complex representations of
text, which are then able to capture contextual information.

Overall, all the approaches mentioned so far require a large corpus of textual
documents to be available, from whichconcepts, instances, and properties are
extracted. This is a major limitation, because it requires the ontology
designer to either manually collect the documents, or to rely on third-party
document sources. In both cases, the corpus of documents may be biased,
incomplete, non-representative of the domain, or simply lacking.

The reliance on large amounts of data also implies that state-of-the-art
population methods are very domain-sensitive, and poorly incremental. Here,
domain-sensitivity refers to the availability of documents for the domain of
interest: if the domain is too narrow, it may be difficult to find enough
relevant documents to populate the ontology; if the domain is too broad, it
may be difficult to avoid the inclusion of irrelevant information in the
ontology. Poor incrementality, instead, refers to the fact that, once the
ontology is populated from a given corpus of documents, it may be difficult to
add further instances to the ontology, without having to re-populate it from
scratch.

###  2.3 Large Language Models

Large-language models are deep NN with enormous number of weights – from BERT
[24] with 340 millions weights to GPT-3 [25] with 175 billions weights –,
which are specialised on NLP tasks, and, in particular, text generation. In
this subsection, we provide a brief overview of LLM, referencing relevant
literature on the topic and outlining the most relevant aspects that we
exploit in our work.

##### The role of pretraining

As further detailed in [26], one key aspect of modern LLM is that they are
_pre-trained_ on very large corpi of textual data, in order to learn _general-
purpose_ language models. Language models are sub-symbolic, probabilistic
representations of natural language, which allow computational agents to grasp
the underlying meaning of words in texts. Put it simply, meaning is
approximated by estimating how much likely words may co-occur in a given
context.

Before LLM were introduced, language models were trained on small, domain-
specific datasets, resulting in neural networks which were only able to
operate in that domain. Changing the domain implied re-training the model from
scratch, on a possibly different dataset. This is not the case for LLM, which
are trained on a very large corpus of textual data, in order to learn a
reusable language model, which is in principle-domain agnostic—while still
supporting fine-tuning on specific domains of interest via additional
training.

From an engineering perspective, pre-training is highly beneficial, as it
implies a paradigm shift [26]. Instead of restarting the whole data-
provisioning, pre-processing, and training pipeline for every new domain, LLM
allow the factorisation of the efforts, by training the model once, for the
general use case—and by then specialising it via fine-tuning upon need.

##### Learning from the Web

One key requirement for the paradigm shift is the availability of large
amounts of documents for pre-training. As the Web is a huge source of textual,
publicly-available data, this is not a problem. In fact, one common practice
in state-of-the-art LLM technologies is to train the model on tons of Web
pages.

For instance, the early BERT model [24, sec. 3.1] was trained on the well-
known BooksCorpus dataset – that is “a corpus of 11,038 books from the web”
(cf. [27, sec. 3]) –, and on dump of the English Wikipedia of the time.

Similarly, according to the seminal paper by Carlini et al. [28, sec. 2.1],
models from the GPT family [29, 25] were trained on data scraped from the
public Internet, and in particular on a dataset attained by “following
outbound links from the social media website Reddit” and by cleaning up the
corresponding HTML pages.

Models following the RoBERTa [30] approach were trained on  _(i)_ the same
datasets as BERT, plus  _(ii)_ a sample of the
CommonCrawl444https://commoncrawl.org News dataset (a dump of news articles
from the Web collected in 2016) including 63 million English documents, plus
_(iii)_ OpenWebTextCorpus [31], i.e. an open-source attempt to reproduce the
aforementioned Web-crawling procedure stemming from Reddit, plus  _(iv)_ the
Stories dataset [32], i.e. a corpus of roughly 1 million documents describing
stories about common-sense situations.

The LLAMA model family [33, 34] relies on a composition of datasets, including
“50% dialogs data from public forums; 12.5% documents from the Colossal Clean
Crawled Corpus (C4) [35]; 12.5% code documents from sites related to
programming like Q&A sites, tutorials, etc; 12.5% Wikipedia (English); 6.25%
English web documents; and 6.25% Non-English web documents”. There, the C4
dataset is a “data set consisting of hundreds of gigabytes of clean English
text scraped from the Web”, which has been finely pruned to exclude noisy,
glibberish, or boiler-plate text.

Finally, according to their Web site555https://mistral.ai/news/mixtral-of-
experts, MistralAI’s Mixtral-of-Experts model is “pre-trained on data
extracted from the open Web”.

##### Oracles for data generation

Summarising, thanks this paradigm shift, LLM are subject to intensive
training, involving textual information from virtually any area of expertise.
Our hypothesis is that Web-wise pre-training lets LLM accumulate substantial
amounts of domain-specific knowledge, on manifold domains. Such knowledge may
then be extracted by querying the LLM with adequate prompts, aimed at letting
the LLM generate relevant text for the domain at hand. In the perspective of
the user, this implies that LLM may be exploited as _oracles_ for information
retrieval tasks.

Using LLM as oracles may be beneficial, in particular, when synthetic data
needs to be generated while taking domain specific knowledge into account. In
that case, users may just need to engineer prompts to induce the LLM to
generate the desired data, and then convert the generated text into the
desired format. This is essentially the approach that we prove feasible in
this paper.

##### Hallucinations

It is worth mentioning that, at the current state of technology, LLM are not
perfect oracles, and they may generate text that is not coherent with the
context of the conversation, or simply factually wrong. Such phenomenon is
known as _hallucination_ [36], and it is a major issue to take into account
when using LLM as oracles—as it implies that the LLM cannot be trusted
blindly.

##### Temperature

Arguably, hallucinations may be more or less impactful depending on the
situation at hand: they may be critical if information retrieval is performed
by inexperienced users willing to learn from it, but it may be tolerable if
the goal is data generation—as some degree of error may be acceptable. In any
case, some degree of control may be desirable to tolerate the issue.

In order to mitigate the hallucination issue, some LLM technologies – such as
GPT-based models – come with a parameter called _temperature_ , which
regulates the randomness of the response. In the intention of designers, this
should allow users to control the “creativity” of the LLM, and hence to
mitigate hallucination. For instance, in GPT, the temperature parameter is a
real value between 0 and 1, where lower values implies “more determinism” and
higher values implies “more randomness”66footnotemark: 6.

##### The role of prompts

Other mitigation techniques may be applied to the hallucination issue, such as
_prompt engineering_. In fact, while of course writing clear and unambiguous
prompts may help the LLM in matching users’ expectations, it has been observed
that the prompt may be tweaked in finer ways. For instance, the authors of
[37] observed that explicitly telling the LLM to put itself in the shoes of a
domain expert when asking a question may lead to more accurate answers.
Similarly, the authors of [38] observed that the LLM may be induced to
generate more accurate procedural descriptions by simply asking it to “take a
deep breath and work on this problem step-by-step”.

##### Emergent capabilities

In the future however, we expect LLM performance to improve, and hallucination
issues to become more and more controllable—if not negligible.

In fact, recent research has shown that LLM may be capable of _emergent_
capabilities, which go beyond the data they have been trained upon, and the
tasks they have been trained for. According to Wei et al. [39], LLM
capabilities are emergent if they are not present in smaller models, but they
are in larger models.

Empirically, emergent capabilities have been observed in “few-shot prompted
tasks”, i.e. tasks where a few input-output examples are provided as prompts
to trained LLM, before asking them to perform the task at hand. Researchers
observed that LLM having an above-threshold number of parameters perform well
in such tasks, even if they have not been trained for them—whereas performance
drops if the number of parameters is below the threshold.

Despite some authors hypothesise that emergent capabilities may just be the
result of non-linear evaluation metrics [40], and other authors [41] observed
performance degradation in counterfactual tasks – i.e. tasks whose actual
setting is different from most common settings for the same tasks, e.g. base-9
arithmetic –, most researchers and practitioners are currently optmistic about
future performance improvements of LLM.

##### Pricing and rating plans

However, currently, using LLM implies having access to considerable
computational resources.

Operating LLM – even if in inference-only mode – is currently unfeasible on
commodity hardware, and it requires the use of specialised hardware, such as
GPUs or TPUs—i.e. ad-hoc investments. This is maybe a bottleneck in industry-
or research-oriented activities involving the automation of queries to LLM.

To overcome this limitation, most LLM technologies nowadays are available “as-
a-Service”, via Web API. In other words, LLM providers expose LLM-querying
capabilities via Web services having well-defined interfaces, which can be
exploited by client applications written in virtually any programming
language. Most commonly, such Web interfaces are made even easier to exploit
in mainstream programming languages, as they are wrapped in dedicated client
libraries, which are then distributed via package managers.

Two notable examples of LLM technologies available as-a-Service are OpenAI
Platform’s API777https://platform.openai.com/ – which provides remote access
to GPT-* models –, and Hugging Face888https://huggingface.co/—that is, a Web
community where LLM models can be published, and possibly queried via a
unified Web API. Both these solutions come with Python client libraries999 cf.
packages openai and hugchat on PyPI. , which make it easy to automate LLM
queries.

One key limitation implied by LLM as Web services open to the general audience
is guaranteeing fair access to the service—as well as providers covering their
operational costs. For these reasons, most providers will either require users
to pay for the service, or be subject to some rate limitations when using the
service.

For instance, OpenAI Platform comes with a pricing
model101010https://openai.com/pricingwhich depends on  _(i)_ the model being
queried,  _(ii)_ the amount of input tokens in the query, and  _(iii)_ the
amount of output tokens generated by the LLM.  There, tokens are essentially
words or pieces of words, and the final price is proportional to the total
amount of tokens. Such pricing model poses an implicit incentive on the user
side, towards minimising the amount of tokens in the query, and in the
response. Similarly, OpenAI Platform imposes rate
constraints111111https://platform.openai.com/docs/guides/rate-limitswhich
depend on  _(i)_ the model being queried,  _(ii)_ on the sort of client
querying it (paying or free),  _(iii)_ on the amount of tokens per request,
and  _(iv)_ requests per time unit from the same client.  Such rate-limiting
model forces the user to artificially slow down any client application
querying the LLM in automatic ways.

Hugging Face comes with similar pricing121212https://huggingface.co/pricing
and rate-limiting models131313https://huggingface.co/docs/api-
inference/faq#rate-limits, which are however more flexible, as they include a
free – yet very slow and limited – option.

##  3 Filling Ontologies with KGFiller

In this section, we present KGFiller, a framework for semi-automatic ontology
population exploiting LLM as oracles. We start by introducing the abstract
framework, and we then discuss a number of technical aspects which makes it
effective in practice.

###  3.1 Abstract Framework

At the abstract level, the core functioning of KGFiller is very simple.
Stemming from a partially instantiated ontology including at least class and
property definitions, a set of query templates, and a trained LLM oracle,
KGFiller generates questions from the templates, with the purpose of querying
the oracle in order to populate the ontology from its answers.

##### Input ontology

KGFiller assumes the ontology to be populated is already partially
initialised. In particular, the ontology must contain class and property
definitions, and classes must be organised hierarchically, so that one may
distinguish between super-classes and sub-classes for each class in the
ontology. A class can have multiple super-classes and multiple sub-classes.
All classes have at least one super-class (namely, $\top$) and at least one
sub-class (namely, $\bot$).

To perform ontology population, KGFiller needs to generate individuals for all
the classes in the aforementioned ontology. For this to work properly, classes
must be named after the concepts they represent. The name of the classes must
be meaningful, short, and unambiguous to avoid problems in later stages of the
algorithm (e.g., generation of queries for the LLM oracle). A similar argument
holds for the property names in the ontology.

Under such hypotheses, KGFiller will generate individuals for all the classes
in the ontology, and it will associate individuals by means of the properties
in the ontology. In doing so, it will guarantee that generated individuals are
represented by meaning names, and they are placed in the most specific class
available in the ontology.

##### Query templates

As part of its functioning, KGFiller needs to generate questions to be asked
to the LLM oracle. To do so, it relies on a set of query templates, which are
pieces of text of arbitrary length (i.e. strings), containing named
placeholders. Placeholders are meant to be eventually replaced with actual
string, by means of substitutions. Substitutions are assignments of
placeholders names to actual strings. They can be applied to templates to
generate concrete strings.

For example, the template $t$ = "What is the capital of $\mathit{\langle
c\rangle}$?" may be instantiated by replacing placeholder $\mathit{\langle
c\rangle}$ with the string "Italy", resulting in the concrete query
$t/\\{\mathit{\langle c\rangle}\mapsto\texttt{"Italy"}\\}=\texttt{"What is the
capital of Italy?"}$.

As far as KGFiller is concerned, we distinguish among query templates of four
sorts, namely:

individual seeking templates

aim at retrieving instances of a given class. As such, they contain one
placeholder for the class name, i.e. $\mathit{\langle class\rangle}$. For
example, they may consist of query patterns similar to "examples of
$\mathit{\langle class\rangle}$?".

relation seeking templates

aim at retrieving individuals related to some given individual via a given
property. As such, they contain two placeholders, one for the property name,
i.e. $\mathit{\langle property\rangle}$, and one for the reference individual
name, i.e. $\mathit{\langle individual\rangle}$. For example, they may consist
of query patterns similar to "examples of $\mathit{\langle property\rangle}$
for $\mathit{\langle individual\rangle}$?".

best match templates

aim at identifying the best class for a given individual, given a set of
candidate classes. As such, they contain two placeholders, one for the
individual name, i.e. $\mathit{\langle individual\rangle}$, and one for the
set of candidate classes, i.e. $\mathit{\langle classes\rangle}$. For example,
they may consist of query patterns similar to "what class is best for
$\mathit{\langle individual\rangle}$ among $\mathit{\langle
classes\rangle}$?".

individuals merging templates

aim at identifying if two instances – belonging to the same class – that are
syntactically similar are also semantically identical and thus should be
merged together. As such, they contain one placeholder for the class name,
i.e. $\mathit{\langle class\rangle}$, and two placeholders for the individual
names to be compared. For example, they may consist of query patterns similar
to "in $\mathit{\langle class\rangle}$, are the instances $\mathit{\langle
ind_{1}\rangle}$ and $\mathit{\langle ind_{2}\rangle}$ the same?".

##### LLM oracle

We do not impose any particular constraint on the nature of the LLM oracle,
nor on the sort of model it is based on. In fact, KGFiller is agnostic to the
LLM oracle, and it can be used with any LLM oracle available on the market, as
far as it supports the generation of textual reponses from textual prompts.
Both the prompt and the repsonse are assumed to consist of strings of
arbitrary length, representing text in (some) natural language. The natural
language is assumed to be the same one used to name classes and properties in
the ontology. Without loss of generality, we shall focus on the English
language in the remainder of this paper.

Finally, one key assumption is that the LLM oracle is already trained on a
large corpus of textual data – also covering text in the natural language of
choice –, and that it absorbed a substantial amount of domain-specific
knowledge. Under such hypothesis, the goals of KGFiller is to extract that
knowledge and to bring in into structured form, in a way which is consistent
with the ontology to be populated.

##### Problem statement

Accordingly, the ontology population problem solved by KGFiller can be
formalised as follows. Stemming from:

  1. 1.

a partially instantiated ontology
$\mathcal{{O}}=\mathcal{{C}}\cup\mathcal{{P}}\cup\mathcal{{X}}$, including a
non-empty set of class definitions $\mathcal{{C}}\neq\varnothing$, a non-empty
set of property definitions $\mathcal{{P}}\neq\varnothing$, and a possibly
empty set of individuals and their relationships $\mathcal{{X}}$ (matching the
aforementioned definitions);

  2. 2.

a subsumption relation $\sqsubseteq$ among the classes in $\mathcal{{C}}$;

  3. 3.

a set of query templates
$\mathcal{{T}}=\mathcal{{T}}_{I}\cup\mathcal{{T}}_{R}\cup\mathcal{{T}}_{B}\cup\mathcal{{T}}_{M}$,
where $\mathcal{{T}}_{I}$ is a set of individual seeking templates,
$\mathcal{{T}}_{R}$ is a set of relation seeking templates,
$\mathcal{{T}}_{B}$ is a set of best match templates, and $\mathcal{{T}}_{B}$
is a set of individuals merging templates;

  4. 4.

and a trained LLM oracle $\mathcal{{L}}$, encapsulating domain-specific
knowledge about $\mathcal{{O}}$;

KGFiller aims at generating a set of individuals and relationships
$\mathcal{{X}}^{\prime}$ such that
$\mathcal{{X}}\subseteq\mathcal{{X}}^{\prime}$, and all novel individuals and
relationships in $\mathcal{{X}}^{\prime}$ are consistent with the class and
property definitions in $\mathcal{{C}}$ and $\mathcal{{P}}$, respectively.

In particular, KGFiller guarantees that, at the end:  _(i)_ every individual
is associated with the most specific concept available in $\mathcal{{C}}$ –
w.r.t. the subsumption relation $\sqsubseteq$ –, and  _(ii)_ for every
property in $\mathcal{{P}}$, every individual in that property’s domain is
associated with several individuals in that property’s range.  So, for
instance, if $\mathcal{{C}}$ includes classes
$\mathit{{C}at},\mathit{{M}ouse}\sqsubset\mathit{{A}nimal}$ and
$\mathcal{{P}}$ includes property
$\mathsf{{e}ats}\subseteq\mathit{{A}nimal}\times\mathit{{A}nimal}$, then
KGFiller puts individual $\mathtt{{t}om}$ into class $\mathit{{C}at}$ and
individual $\mathtt{{j}erry}$ into class $\mathit{{M}ouse}$, and associates
$\mathtt{{t}om}$ with $\mathtt{{j}erry}$ via property $\mathsf{{e}ats}$.

##### Phases

To compute $\mathcal{{X}}^{\prime}$, KGFiller encompasses four major phases,
namely:  _(i)_ the population phase, where novel individuals are identified
for each class in $\mathcal{{C}}$,  _(ii)_ the relation phase, where novel
relationships are identified for each property in $\mathcal{{P}}$,  _(iii)_
the redistribution phase, where the individuals identified in the previous
phases are redistributed among the classes in $\mathcal{{C}}$, in such a way
that each individual is put in the best (i.e. most specific) class available,
_(iv)_ the merge phase, where the individuals of each class in $\mathcal{{C}}$
are checked to detect semantic duplicates.  It is of paramount importance that
phases follow this order of execution, as the population phase generates
individuals that are then used in the relation phase, and the relation phase
may then generate novel individuals; hence, the redistribution phase is needed
to ensure that all individuals are put in the best class available, while the
merge phase is required to reduce the amount of semantically-duplicated
entries.

##### Ancillary functions

Algorithm 1 Adds an individual to a class, unless it is already assigned to a
more specific class

1:$\mathcal{{O}}=\mathcal{{C}}\cup\mathcal{{P}}\cup\mathcal{{X}}$: partially
populated ontology

2:$\mathtt{{i}}$ is an individual

3:$\mathit{{R}}\in\mathcal{{C}}$: root concept $\mathtt{{i}}$ should be
assigned to

4:$\mathcal{{X}}^{\prime}$ contains the (possibly indirect) assignment of
$\mathtt{{i}}$ to $\mathit{{R}}$

5:function AddToClass($\mathcal{{O}},\mathtt{{i}},\mathit{{R}}$)

6: if
$\exists\mathit{{C}}~{}\text{s.t.}~{}(\mathtt{{i}}:\mathit{{C}})\in\mathcal{{X}}^{\prime}\wedge\mathit{{C}}\sqsupset\mathit{{R}}$
then

7: return
$\mathcal{{X}}\cup\\{\mathtt{{i}}:\mathit{{R}}\\}-\\{\mathtt{{i}}:\mathit{{C}}\\}$

8: else if
$\not\exists\mathit{{C}}~{}\text{s.t.}~{}(\mathtt{{i}}:\mathit{{C}})\in\mathcal{{X}}^{\prime}$
then

9: return $\mathcal{{X}}\cup\\{\mathtt{{i}}:\mathit{{R}}\\}$

10: return $\mathcal{{X}}$

In the remainder of this subsection, we delve into the details of each phase,
and we propose pseudo-code formalisations for each of them. To make this
possible, we rely on a number of shared ancillary functions, whose semi-formal
definitions are provided here.

Functions GetRange and GetDomain are used to retrieve the domain and range of
a given property, respectively. So, for instance, for property
$\mathsf{{p}}\subseteq\mathit{{D}}\times\mathit{{R}}$,
GetDomain($\mathsf{{p}}$) returns $\mathit{{C}}$, whereas
GetRange($\mathsf{{p}}$) returns $\mathit{{R}}$.

Function AskOracle is used to query LLM oracles, whereas functions
ExtractBinary and ExtractNames aim to extract binary answers, or relevant
individuals’ or concepts’ names from their textual responses, respectively.
Their internal functioning is non-trivial, and we discuss it in section 3.2.
Here, we just focus on their syntax and expected semantics. Accordingly,
function AskOracle($\mathcal{{L}},q$) accepts as input the LLM oracle
$\mathcal{{L}}$ to query and the query string $q$ to be submitted to the
oracle, and it returns a string as response $text$. Both the input ($q$) and
output ($text$) strings are assumed to contain natural language text, and to
be of arbitrary length. Function ExtractNames($text$) accepts as input the
response string $text$ from an LLM oracle, and it returns a set of strings
$names$, to be interpreted as names of individuals or concepts, depending on
the concept. Function ExtractBinary($text$) performs a similar operation, but
it returns a boolean value (true/false, yes/no, _et similia_).

Finally, function AddToClass is used to add individuals to classes. Despite
the simplicity of its intuitive definition, its actual operation is non-
trivial, hence why we detail pseudo-code in algorithm 1. In particular, we
write AddToClass($\mathcal{{O}},\mathtt{{i}},\mathit{{C}}$) to denote the
assignment of individual $\mathtt{{i}}$ to class $\mathit{{C}}$ in ontology
$\mathcal{{O}}$. However, there may be corner cases in doing so, such as the
case where $\mathtt{{i}}$ is already assigned to some sub-class
$\mathit{{S}}\sqsubset\mathit{{C}}$. In that case, no actual edit is performed
on the ontology. Vice versa, if $\mathtt{{i}}$ is already assigned to some
super-class $\mathit{{R}}\sqsupset\mathit{{C}}$, then $\mathtt{{i}}$ is
assigned to $\mathit{{C}}$, and it is unassigned from $\mathit{{R}}$. In any
case, the function returns a novel set of assertions $\mathcal{{X}}^{\prime}$,
which is possibly different from the input one (denoted by $\mathcal{{X}}$).

####  3.1.1 Population phase

Algorithm 2 Populates the given ontology with novel individuals queried from
an LLM oracle

1:$\mathcal{{O}}=\mathcal{{C}}\cup\mathcal{{P}}\cup\mathcal{{X}}$: partially
populated ontology

2:$\mathcal{{T}}_{C}$: individual seeking query templates

3:$\mathcal{{L}}$: LLM oracle

4:$\mathit{{R}}\in\mathcal{{C}}$: root concept to be populated

5:$\mathcal{{X}}^{\prime}$ contains novel individuals, assigned to the classes
in $\mathcal{{C}}$

6:function
Populate($\mathcal{{O}},\mathcal{{T}}_{C},\mathcal{{L}},\mathit{{R}}$)

7: $\mathcal{{X}}^{\prime}\leftarrow\mathcal{{X}}$

8: for all
$\mathit{{C}}\in\mathcal{{C}}~{}\text{s.t.}~{}\mathit{{C}}\sqsubset\mathit{{R}}\wedge\mathit{{C}}\neq\bot$
do

9:
$\mathcal{{O}}^{\prime}\leftarrow\mathcal{{C}}\cup\mathcal{{P}}\cup\mathcal{{X}}^{\prime}$

10:
$\mathcal{{X}}^{\prime}\leftarrow\textsc{{Populate}}(\mathcal{{O}}^{\prime},\mathcal{{T}}_{C},\mathcal{{L}},\mathit{{C}})$

11: for all $\mathit{t}\in\mathcal{{T}}^{C}$ do

12: $\mathit{q}\leftarrow\mathit{t}/\\{\mathit{\langle
class\rangle}\mapsto\mathit{{R}}\\}$

13: $\mathit{text}\leftarrow\textsc{{AskOracle}}(\mathcal{{L}},\mathit{q})$

14: for all $\mathtt{{i}}\in\textsc{{ExtractNames}}(\mathit{text})$ do

15:
$\mathcal{{O}}^{\prime}\leftarrow\mathcal{{C}}\cup\mathcal{{P}}\cup\mathcal{{X}}^{\prime}$

16:
$\mathcal{{X}}^{\prime}\leftarrow\textsc{{AddToClass}}(\mathcal{{O}}^{\prime},\mathtt{{i}},\mathit{{R}})$

17: return $\mathcal{{X}}^{\prime}$

The population phase relies on the Populate function, defined in algorithm 2.
As the name suggests, this function aims at populating a partially
instantiated ontology $\mathcal{{O}}$ with novel instances, queried to some
LLM oracle $\mathcal{{L}}$, via a set of instance seeking query templates
$\mathcal{{T}}_{I}$.

Starting from some root class of choice $\mathit{{R}}\in\mathcal{{O}}$, the
function recursively explores the class graph spawned by the subsumption
relation $\sqsubseteq$, following a depth-first, post-order exploration
strategy. In other words, it visits all direct and indirect sub-classes of
$\mathit{{R}}$, before visiting $\mathit{{R}}$ itself, giving priority to most
specific classes. Under the assumption that concepts in $\mathcal{{O}}$ are
connected by the subsumption relation $\sqsubseteq$ to form a directed acyclic
graph (DAG), the exploration shall eventually visit all sub-classes of
$\mathit{{R}}$, and therefore terminate.

For each visited sub-class $\mathit{{C}}\sqsubseteq\mathit{{R}}$, the function
generates as many queries as the templates in $\mathcal{{T}}_{I}$, by
replacing the placeholder $\mathit{\langle class\rangle}$ in each template
with the name of $\mathit{{C}}$. Then, it submits each query to the LLM oracle
$\mathcal{{L}}$, and it extracts the names of the individuals from the
response. The amount of individual generated per query is unbounded, and it
depends on the LLM oracle $\mathcal{{L}}$, and on other technicalities
discussed in section 3.2.

It may happen that the same individual is generated by multiple queries, or
that the same individual is generated for multiple different sub-classes of
$\mathit{{R}}$. None of these situations is problematic, as individual
addition is performed by means of the AddToClass function, which does not
duplicate individuals, nor assignments to classes, and in case of multiple
assignments, will always prioritise the most specific class. In keeping
individuals assignment as specific as possible, the _post-order_ exploration
strategy is crucial. By visiting most specific classes first, the function
ensures that the most specific concepts are populated first, and that less
specific concepts are directly populated only when no more specific concepts
are available in the ontology.

Notably, in order to populate the whole ontology $\mathcal{{O}}$ one may
simply invoke the function as follows:
Populate($\mathcal{{O}},\mathcal{{T}}_{C},\mathcal{{L}},\top$). In this case,
the starting point is the top class ($\top$), which implies that all classes
in $\mathcal{{O}}$ are eventually visited, and therefore populated.

####  3.1.2 Relation phase

Algorithm 3 Populates the given ontology with novel relationships queried from
an LLM oracle

1:$\mathcal{{O}}=\mathcal{{C}}\cup\mathcal{{P}}\cup\mathcal{{X}}$: partially
populated ontology

2:$\mathcal{{T}}_{R}$: relation seeking query templates

3:$\mathcal{{L}}$: LLM oracle

4:$\mathsf{{p}}\in\mathcal{{P}}$: property to be populated

5:$\mathcal{{X}}^{\prime}$ contains novel relationships, involving individuals
in $\mathcal{{X}}$

6:function
Relate($\mathcal{{O}},\mathcal{{T}}_{R},\mathcal{{L}},\mathsf{{p}}$)

7: $\mathcal{{X}}^{\prime}\leftarrow\mathcal{{X}}$

8: $\mathit{{D}}\leftarrow\textsc{{GetDomain}}(\mathsf{{p}})$

9: $\mathit{{R}}\leftarrow\textsc{{GetRange}}(\mathsf{{p}})$

10: for all $(\mathtt{{i}}:\mathit{{D}})\in\mathcal{{X}}$ do

11: for all $\mathit{t}\in\mathcal{{T}}^{R}$ do

12: $\mathit{q}\leftarrow\mathit{t}/\\{\mathit{\langle
individual\rangle}\\!\mapsto\\!\mathtt{{i}},\mathit{\langle
property\rangle}\\!\mapsto\\!\mathsf{{p}}\\}$

13: $\mathit{text}\leftarrow\textsc{{AskOracle}}(\mathcal{{L}},\mathit{q})$

14: for all $\mathtt{{i}}^{\prime}\in\textsc{{ExtractNames}}(\mathit{text})$
do

15:
$\mathcal{{O}}^{\prime}\leftarrow\mathcal{{C}}\cup\mathcal{{P}}\cup\mathcal{{X}}^{\prime}$

16:
$\mathcal{{X}}^{\prime}\leftarrow\textsc{{AddToClass}}(\mathcal{{O}}^{\prime},\mathtt{{i}}^{\prime},\mathit{{R}})$

17:
$\mathcal{{X}}^{\prime}\leftarrow\mathcal{{X}}^{\prime}\cup\\{\mathsf{{p}}(\mathtt{{i}},\mathtt{{i}}^{\prime})\\}$

18: return $\mathcal{{X}}^{\prime}$

The relation phase relies on the Relate function, defined in algorithm 3. As
the name suggests, this function aims at populating a partially instantiated
ontology $\mathcal{{O}}$ with novel relationships between the individuals
therein contained (and, possibly, novel individuals as well), queried to some
LLM oracle $\mathcal{{L}}$, via a set of relation seeking query templates
$\mathcal{{T}}_{R}$.

Focussing upon the some property $\mathsf{{p}}\in\mathcal{{O}}$, the function
first retrieves the domain $\mathit{{D}}$ and range $\mathit{{R}}$ of
$\mathsf{{p}}$, by means of the GetDomain and GetRange functions,
respectively. Then, it iterates over all individuals $\mathtt{{i}}$ in
$\mathit{{D}}$, and it queries the LLM oracle for each of them, via the many
templates in $\mathcal{{T}}_{R}$. In particular, for each individual, and for
each query template $t\in\mathcal{{T}}_{R}$, the function produces a query $q$
by replacing the placeholders $\mathit{\langle property\rangle}$ and
$\mathit{\langle individual\rangle}$ in $t$ with the names of $\mathsf{{p}}$
and $\mathtt{{i}}$, respectively. Each query $q$ is then submitted to the LLM
oracle $\mathcal{{L}}$, and the names of the individuals in the response – as
extracted by the ExtractNames function – are considered as individuals to be
added to the ontology—and, in particular, to the range $\mathit{{R}}$ of
$\mathsf{{p}}$. Accordingly, for each individual $\mathtt{{i}}^{\prime}$
returned by the LLM oracle, the function shall add both the individual
$\mathtt{{i}}^{\prime}$ and the relationship
$\mathsf{{p}}(\mathtt{{i}},\mathtt{{i}}^{\prime})$ to the ontology.

Of course, in order to populate relationships for all properties
$\mathsf{{p}}\in\mathcal{{O}}$ in the ontology, one may simply invoke the
function once per property, as follows:
Relate($\mathcal{{O}},\mathcal{{T}}_{R},\mathcal{{L}},\mathsf{{p}}$).

It is worth highlighting that the Relate function may generate novel
individuals, as a byproduct of its operation. When this is the case, there
could be two relevant situations, depending on whether the generated
individuals are already present in the ontology or not. In the former case,
the exploitation of the AddToClass function ensures that the already-present
individuals are reused, and that they are assigned to the most specific class
available. In the former case, there is no such guarantee, and the function
may assign the generated individuals the range class of some property. Despite
the range class is not necessarily the most specific class in the ontology,
this is not a problem, as the redistribute phase (see section 3.1.3) will
eventually assign the generated individuals to the most specific class
available.

####  3.1.3 Redistribution phase

Algorithm 4 Redistributes individuals from the given ontology’s classes, in
such a way that each individual is assigned to the most specific class
available

1:$\mathcal{{O}}=\mathcal{{C}}\cup\mathcal{{P}}\cup\mathcal{{X}}$: partially
populated ontology

2:$\mathcal{{T}}_{B}$: relation seeking query templates

3:$\mathcal{{L}}$: LLM oracle

4:$\mathit{{R}}\in\mathcal{{C}}$: root concept within which redistribution
should occur

5:$\mathcal{{X}}^{\prime}$ contains the different assignments of individuals
to classes

6:function
Redistribute($\mathcal{{O}},\mathcal{{T}}_{B},\mathcal{{L}},\mathit{{R}}$)

7: $\mathcal{{X}}^{\prime}\leftarrow\mathcal{{X}}$

8:
$\mathcal{{S}}\leftarrow\\{\mathit{{S}}\in\mathcal{{C}}\mid\mathit{{S}}\sqsubset\mathit{{R}}\\}$

9: for all $(\mathtt{{i}}:\mathit{{R}})\in\mathcal{{X}}^{\prime}$ do

10: $\mathit{{B}}\leftarrow\mathit{{R}}$

11: for all $\mathit{t}\in\mathcal{{T}}^{B}$ do

12: $\mathit{q}\leftarrow\mathit{t}/\\{\mathit{\langle
individual\rangle}\mapsto\mathtt{{i}},\mathit{\langle
classes\rangle}\mapsto\mathcal{{S}}\\}$

13: $\mathit{text}\leftarrow\textsc{{AskOracle}}(\mathcal{{L}},\mathit{q})$

14: for all $\mathit{{C}}\in\textsc{{ExtractNames}}(\mathit{text})$ do

15: $\mathit{{B}}\leftarrow\mathit{{C}}$

16: break going to line 17

17:
$\mathcal{{O}}^{\prime}\leftarrow\mathcal{{C}}\cup\mathcal{{P}}\cup\mathcal{{X}}^{\prime}$

18:
$\mathcal{{X}}^{\prime}\leftarrow\textsc{{AddToClass}}(\mathcal{{O}}^{\prime},\mathtt{{i}},\mathit{{B}})$

19: for all
$\mathit{{C}}\in\mathcal{{C}}~{}\text{s.t.}~{}\mathit{{C}}\sqsubset\mathit{{R}}\wedge\mathit{{C}}\neq\bot$
do

20:
$\mathcal{{O}}^{\prime}\leftarrow\mathcal{{C}}\cup\mathcal{{P}}\cup\mathcal{{X}}^{\prime}$

21:
$\mathcal{{X}}^{\prime}\leftarrow\textsc{{Redistribute}}(\mathcal{{O}}^{\prime},\mathcal{{T}}_{B},\mathcal{{L}},\mathit{{C}})$

22: return $\mathcal{{X}}^{\prime}$

The redistribution phase relies on the Redistribute function, defined in
algorithm 4. As the name suggests, this function aims at redistributing
individuals from an instantiated ontology $\mathcal{{O}}$, in such a way that
each individual is assigned to the most specific class available, according to
the subsumption relation $\sqsubseteq$. To do so, the function queries the LLM
oracle $\mathcal{{L}}$ via a set of best match query templates
$\mathcal{{T}}_{B}$.

This phase is necessary because at the end of the population and relation
phases, the ontology may still contain individuals assigned to less specific
classes, even if more specific classes are available. This situation may be
due to the fact that the ontology may have been instantiated with individuals
assigned to less specific classes, or to the relation phase—which may have
generated under-optimal assignments to classes.

Starting from some root class of choice $\mathit{{R}}\in\mathcal{{O}}$, the
function recursively explores the class graph spawned by the subsumption
relation $\sqsubseteq$, following a depth-first, _pre_ -order exploration
strategy. In other words, it visits all direct sub-classes of $\mathit{{R}}$,
after visiting $\mathit{{R}}$ itself, and then it does the same, recursively,
for each sub-class. Like in the population phase, there are no loops in the
class graph, the exploration shall eventually visit all sub-classes of
$\mathit{{R}}$, and therefore terminate.

For each visited class $\mathit{{C}}\sqsubseteq\mathit{{R}}$, the function
attempts to determine whether $\mathit{{C}}$ is actually the best class for
all individual therein contained. For some individuals, it might be the case
that $\mathit{{C}}$ is not the best class, as some of its sub-classes is more
indicated. Consider for instance the case where the classes
$\mathit{{C}at}\sqsubset\mathit{{A}nimal}$ are available, yet $\mathtt{{t}om}$
is assigned to $\mathit{{A}nimal}$ rather than to $\mathit{{C}at}$.

To determine whether $\mathit{{C}}$ is the best class (among it, and all its
sub-classes) for some individual $\mathtt{{i}}:\mathit{{C}}$, the function
generates a query for each template $t\in\mathcal{{T}}_{B}$, by replacing the
placeholders $\mathit{\langle individual\rangle}$ and $\mathit{\langle
classes\rangle}$ in $t$ with the name of $\mathtt{{i}}$ and the concatenation
of names of all direct sub-classes of $\mathit{{C}}$, respectively. Each query
$q$ is then submitted to the LLM oracle $\mathcal{{L}}$, and the names of the
individuals in the response – as extracted by the ExtractNames function – are
considered as class names, i.e. as candidate classes for $\mathtt{{i}}$. Of
these, the function selects the first one ($B$, for “best”) and re-assigns
$\mathtt{{i}}$ to $B$.

As the function is recursive, each individual may be reassigned multiple
times, until eventually reaching the most specific class available for it in
the ontology. To make this possible, the depth-first, pre-order exploration
strategy is crucial, as it lets individuals “move down” in the class hierarchy
quickly, while keeping the size of queries relatively small.

Notably, in order to redistribute _all_ individuals from an ontology
$\mathcal{{O}}$ one may simply invoke the function as follows:
Redistribute($\mathcal{{O}},\mathcal{{T}}_{C},\mathcal{{L}},\top$), which
implies that all individuals in $\mathcal{{O}}$ are eventually visited, and
therefore assessed and possibly redistributed.

####  3.1.4 Merge phase

Algorithm 5 Merges syntactically and semantically similar individuals from the
given ontology’s classes

1:$\mathcal{{O}}=\mathcal{{C}}\cup\mathcal{{P}}\cup\mathcal{{X}}$: partially
populated ontology

2:$\mathcal{{T}}_{M}$: indivuals merging query templates

3:$\mathcal{{L}}$: LLM oracle

4:$\mathit{{R}}\in\mathcal{{C}}$: root concept within which merge should occur

5:$\mathcal{{X}}^{\prime}$ contains no duplicated individuals

6:function Merge($\mathcal{{O}},\mathcal{{T}}_{M},\mathcal{{L}},\mathit{{R}}$)

7: $\mathcal{{X}}^{\prime}\leftarrow\mathcal{{X}}$

8: for all
$\mathit{{C}}\in\mathcal{{C}}~{}\text{s.t.}~{}\mathit{{C}}\sqsubset\mathit{{R}}$
do

9: $\mathcal{D}\leftarrow\emptyset$

10: for all
$\mathtt{{i}},\mathtt{{j}}:\mathit{{C}}~{}\text{s.t.}~{}(\mathtt{{i}},\mathtt{{j}})\in\mathcal{{X}}^{\prime}\times\mathcal{{X}}^{\prime}$
do

11: if
$\textsc{{Syn\\-tac\\-ti\\-cal\\-lySi\\-mi\\-lar}}(\mathtt{{i}},\mathtt{{j}})$
then

12:
$\mathcal{D}\leftarrow\mathcal{D}\cup\\{\\{\mathtt{{i}},\mathtt{{j}}\\}\\}$

13: for all $\\{\mathtt{{i}},\mathtt{{j}}\\}\in\mathcal{D}$ do

14: for all $\mathit{t}\in\mathcal{{T}}_{M}$ do

15: $\mathit{q}\leftarrow\mathit{t}/\\{\mathit{\langle
ind_{1}\rangle}\\!\mapsto\\!\mathtt{{i}},\mathit{\langle
ind_{2}\rangle}\\!\mapsto\\!\mathtt{{j}},\mathit{\langle
class\rangle}\\!\mapsto\\!\mathit{{C}}\\}$

16: $\mathit{text}\leftarrow\textsc{{AskOracle}}(\mathcal{{L}},\mathit{q})$

17: if $\textsc{{ExtractBinary}}(text)$ then

18: $\mathcal{{X}}^{\prime}\leftarrow\textsc{{MergeIndividuals}}(i,j)$

19: break going to line 14

The merge phase relies on the Merge function, defined in Algorithm 5. As the
name suggests, this function aims at merging possible duplicated individuals
from an instantiated ontology $\mathcal{{O}}$.

By ‘duplicated individuals’ we mean individuals with slightly different names,
which are actually semantically identical. We observe empirically that these
could appear in the ontology as a by-product of the previous phases. Consider
for instance the case where the two semantically identical individuals (say
$\mathtt{{m}aincoone\\_cat}$ and $\mathtt{{m}aincoone}$) are added to the
class $\mathit{{C}at}\sqsubset\mathit{{A}nimal}$. Those individuals would be
considered as different just because of the _syntactical_ differences in their
names. To mitigate this issue, the Merge function queries the LLM oracle to
identify _semantically_ similar individuals, and it merges them together into
a single individual.

To serve its purpose, the Merge function scans an instantiated ontology
$\mathcal{{O}}$, looking for _pairs_ of syntactically similar individuals.
Whenever any such pair is found, the function queries the LLM oracle
$\mathcal{{L}}$ via a set of individuals merging templates $\mathcal{{T}}_{M}$
to determined whether to merge them into a single individual.

More precisely, starting from some root class of choice
$\mathit{{R}}\in\mathcal{{O}}$, the function recursively explores the class
graph spawned by the subsumption relation $\sqsubseteq$, following a depth-
first, _pre_ -order exploration strategy. In other words, it visits all direct
sub-classes of $\mathit{{R}}$, after visiting $\mathit{{R}}$ itself, and then
it does the same, recursively, for each sub-class. For each visited class
$\mathit{{C}}\sqsubseteq\mathit{{R}}$, the function first determines the set
$\mathcal{D}$ of _candidate_ duplicate pairs, by means of the
SyntacticallySimilar function. Then, for each pair, it decides whether to
merge the two individuals or not, by means of the ExtractBinary function—which
interprets the response from the LLM oracle as a boolean value. In case of a
positive response, the MergeIndividuals function is exploited to perform the
actual merge. It simply works by transferring the values of an instance to the
other and removing one of the two, hence returning the updated ontology
$\mathcal{{X}}^{\prime}$.

In this context, we let the SyntacticallySimilar function decide whether two
instances $\mathtt{{i}}$ and $\mathtt{{j}}$ must be considered _enough_
similar at the syntactic level to be considered for merging. The
implementation of this function has a relevant impact on the overall
performance of the Merge function, as we further discuss in section 3.2.

As far as LLM queries are concerned, these rely on the templates in
$\mathcal{{T}}_{M}$. For each pair of candidate duplicates, the Merge function
generates a query for each template $t\in\mathcal{{T}}_{M}$, by replacing the
placeholders $\mathit{\langle ind\\_1\rangle}$, $\mathit{\langle
ind\\_2\rangle}$, and $\mathit{\langle class\rangle}$ in $t$ with the name of
the instances $\mathtt{{i}}$ and $\mathtt{{j}}$, and of class $\mathit{{C}}$,
respectively. The first template producing a positive response is considered
as definitive, otherwise further templates are queried. Most notably, to avoid
issues that may arise from the LLM’s creativity, we force-set temperature to
0.0 when querying the LLM oracle in the Merge function.

Notably, in order to detect and merge _all_ duplicated individuals from an
ontology $\mathcal{{O}}$, one may simply invoke the function as follows:
Merge($\mathcal{{O}},\mathcal{{T}}_{M},\mathcal{{L}},\top$), which implies
that all classes in $\mathcal{{O}}$ are eventually visited, and therefore any
semantic duplicate therein contained is eventually tackled.

###  3.2 Practical Aspects

At the practical level, a number of aspects should be considered in order for
KGFiller to be effective. Most notably, the abstract formulation from section
3.1 models interaction with LLM oracles by means of functions AskOracle and
ExtractNames, abstracting technicalities away. However, in practice, the
implementation of these functionalities is non-trivial, and it requires taking
into account technicalities involving  _(i)_ the engineering of prompts to be
submitted to the LLM oracle,  _(ii)_ the mining of relevant individual or
concepts names from the textual responses of the LLM oracle, as well as
_(iii)_ corner cases involving empty or inconclusive responses, and  _(iv)_
other relevant aspects such as costs and rate limitations.

Accordingly, we here discuss each of these aspects in detail, following a
means-oriented approach.

####  3.2.1 Prompt and response engineering

Listing 1: Example of response from ChatGPT to the query "can you generate
examples of cats?"

⬇

Certainly! While I can’t physically generate images, I can certainly describe
or provide information about different types of cats. Here are a few examples:

1. Domestic Shorthair:

Description: A common and popular cat breed known for its short, sleek coat.

Characteristics: Versatile in colors and patterns, friendly, and adaptable.

2. Siamese Cat:

Description: Elegant and slender cat with distinctive color points.

Characteristics: Vocal, social, and known for their striking blue almond-
shaped eyes.

$\ldots$

10. Russian Blue:

Description: Short-haired cat with a bluish-gray coat and striking green eyes.

Characteristics: Reserved but affectionate, known for its plush double coat.

These are just a few examples, and there are many more cat breeds, each with
its own unique characteristics and appearance.

Generally speaking, KGFiller performs automatic queries to LLM oracles, with
the purpose of inducing them to generate individuals or concept names.
However, LLM come with no guarantee on the structure of the generated text,
nor on its relevance w.r.t. the ontology: the generated text may be long,
unstructured, and it may contain both relevant and irrelevant chunks of
information. For instance, submitting the query "can you generate examples of
cats?" to ChatGPT may produce a reponse similar to the one in LABEL:lst:cats.
This may complicate the automatic extraction of relevant information from the
textual responses of the LLM oracle, as irrelevant information should be
filtered out, while only relevant information should be retained. In the
example from LABEL:lst:cats the relevant information is the list cats’ races’
names, whereas the irrelevant information is anything else.

There exists a delicate trade-off among the complexity of the prompt and the
complexity of the response. As a consequence, instead of engineering complex
text extraction procedures aimed at distilling relevant information from
arbitrary text, one may try to engineer the prompt in order to induce the LLM
oracle to generate more concise or more structured responses. In other words,
a trade-off exists between the length / intricacy of query templates and the
algorithmic complexity of function ExtractNames.

In the following paragraphs, we discuss a number of strategies to engineer the
prompt in order to induce the LLM oracle to generate more concise or more
structured responses, and then we delve into the details of the ExtractNames.

Listing 2: Example of response from ChatGPT to the query "list of cat races,
names only"

⬇

Certainly, here’s a list of cat breeds with names only:

1. Persian

2. Siamese

$\ldots$

10. Domestic Shorthair

##### Getting concise and structured responses from oracles

One may induce an LLM oracle to generate concise and structured responses by
explicitly asking for these features in the query. For instance, at the time
of writing, it is sufficient to ask for a "concise list" of something to let
the LLM generate text formatted as a list in Markdown
format141414https://www.markdownguide.org/basic-syntax/#lists-1.

Moreover, we observe that complementing the query with the "names only"
requirement, induces the LLM to generate a very compact list of names, with no
additional information.

Finally, we also observe that the word "concise" may be omitted when combining
the two requirements. We speculate that this is possible because the "names
only" requirement already implies the maximum level of conciseness.

As an example, consider again the case of LABEL:lst:cat-names. There, the
response was attained by submitting the query: "list of cat races, names
only".

Along this line, we recommend engineering query templates in such a way to
include the aforementioned requirements. So, for instance, individual seeking
templates should be of the form "list of examples for $\mathit{\langle
class\rangle}$, names only", whereas relation seeking templates should be of
the form "list of examples of $\mathit{\langle property\rangle}$ for
$\mathit{\langle individual\rangle}$, names only". Similarly, best match
templates should be of the form "best class for $\mathit{\langle
individual\rangle}$ among $\mathit{\langle classes\rangle}$, be concise".

##### Avoiding out-of-context responses

One may induce an LLM to generate context-aware responses by explicitly
providing context in the query. But what is context, precisely?

In the particular case of KGFiller applied to a domain-specific ontology, the
context is always (at least) two-folded. In fact, the LLM oracle should be
aware that queries may contain references to  _(i)_ the ontology’s domain, and
_(ii)_ ontologies in general (e.g. ontologic jargon such as “class” or
“individual”).

In our running example about animals, the domain could be for instance “a
zoologist willing to populate an ontology describing sorts of animals”. This
could be made explicit in the query by prepending a statement such as "you’re
a zoologist" or "zoological context".

The takeaway here is that query templates should contain explicit references
to the ontology’s domain, and possibly explicitly clarify when they are
seeking for individuals or class names.

Going back to our running example, consider for instance case of the query
"can you name a few examples of cats?". This is very ambiguous, as no context
is provided. In fact, at the time of writing, ChatGPT replies with a list of
common names for domestic cats, including; "Whisker", "Luna", "Oliver",
"Cleo", "Simba", etc. However, simply prepending the string "zoological
context" to the query may induce ChatGPT to reply with a list of members of
Felidae family (e.g. "African Lion (Panthera leo)", "Tiger (Panthera tigris)",
etc.) plus a warning that "the term ‘cats’ generally refers to members of the
family ‘Felidae’". An even more precise result can be attained via the query
"zoological context. can you name a few instances of class ‘cat race’?", where
the ontological jargon is explicitly adopted.

In the specific case of KGFiller, context information may be simply provided
by means of ad-hoc text in query templates. Contextual text may be the same
for all templates, or it may be slightly different for each template. It could
be simply prepended or appended to each template (e.g. "zoological context.
list of examples for list of examples for $\mathit{\langle class\rangle}$,
names only").

Another key aspect to consider is that query templates are meant to be filled
with class or property names from the initial ontology definition. This may be
problematic, if the name of the class or property is ambiguous outside the
context of the ontology. To avoid this inconvenience, we assume that each
class and property in the ontology may come with an optional attribute, namely
$\mathsf{{f}ancyName}$, which is meant to be used in queries instead of the
actual name of the class/property.

Consider for instance the case of the aforementioned $\mathit{{C}at}$ class
which may be present in the ontology, yet ambiguous in the zoological domain.
Let us also assume that, for some technical / administrative reason, the class
name cannot be changed. When this is the case, the ontology designers may add
the attribute $\mathsf{{f}ancyName}$ to $\mathit{{C}at}$, and set its value to
some zoologically-correct name, such as "feline species" or "felidae family".
In this way, queries would be generated in an unambiguous way, despite the
ontology contains a compact class name.

##### Mining relevant information from textual responses

Listing 3:  Example of LLM response containing a numbered list of fake cat
names. Some items in the list actually contain more than one name. Some names
are composed by more than one word. The response also contains text which is
not part of the list. Relevant names are highlighted in blue and cyan
(alternating colours).

⬇

Certainly, here’s a list of cat breeds with names only:

1. Persian or Siamese

2. Maine Coon

3. Bengal, Caracal

4. Sphynx

$\ldots$

9. Domestic Shorthair

These are just a few examples, and there are many more cat breeds, each with
its own unique characteristics and appearance.

Assuming that query templates are engineered in such a way to force LLM
responses to contain a list of names (spanning either a single line or
multiple ones) KGFiller relies on function ExtractNames for the localisation
of those names in the response text. To do so, the function attempts to parse
LLM responses against a fairly flexible context-free grammar, tailored on the
localisation of relevant chunks of text.

Before delving into the details of the grammar and the parsing procedure, let
us exemplify the typical situation. Both in the population and relation
phases, the LLM oracle is asked to produce a list of names, to be interpreted
as individuals. Conversely, in the redistribution phase, the LLM oracle is
asked to produce a single name, to be interpreted as a class, among a set of
candidate names which are selected by the algorithm. So, in the general case,
LLM is asked to produce a multitude of names, in the form of a list.
Empirically, we observed the LLM oracles tend to produce either multi-line
lists or single-line lists. Multi-line lists are typically formatted as
bulleted or numbered lists in Markdown format, whereas single-line lists are
typically sentences containing commas, semicolons, or words such as "and" or
"or". It is worth highlighting that responses containing singleton names, can
be considered a trivial case of single-line lists. Finally, we observed that
LLM oracles may sometime include irrelevant (w.r.t. KGFiller) information in
the response, such as pleasantries, contextualisation text, or warnings about
the fact that the response may be incomplete. In LABEL:lst:fake-cat-names, we
report a handcrafted response aimed at exemplifying one complex case, where
inline and multi-line lists are mixed, as well as irrelevant text.

The actual implementation of the ExtractNames function is based on the
assumption that meaningful response consists of text (containing some sub-
string) matching the following context-free grammar:

$\small\begin{array}[]{rcl}\mathit{\langle Response\rangle}&:=&\mathit{\langle
SingleLine\rangle}\mid\mathit{\langle MultiLine\rangle}\\\ \mathit{\langle
MultiLine\rangle}&:=&\mathit{\langle Item\rangle}(\mathit{\langle
Whitespace\rangle}\mathit{\langle Item\rangle})*\\\ \mathit{\langle
Item\rangle}&:=&\mathit{\langle Bullet\rangle}\mathit{\langle
SingleLine\rangle}\\\ \mathit{\langle SingleLine\rangle}&:=&\mathit{\langle
Relevant\rangle}(\mathit{\langle Separator\rangle}\mathit{\langle
Relevant\rangle})*\\\ \mathit{\langle Relevant\rangle}&:=&\mathit{\langle
Word\rangle}(\mathit{\langle Whitespace\rangle}\mathit{\langle
Word\rangle})*\\\ \mathit{\langle Word\rangle}&:=&\text{any combination of
letters,}\\\ &&\text{punctuation, or digits}\\\
&\mid&\texttt{"}\mathit{\langle
Word\rangle}\texttt{"}\mid\texttt{`}\mathit{\langle Word\rangle}\texttt{'}\\\
\mathit{\langle Whitespace\rangle}&:=&\text{any combination of spaces,}\\\
&&\text{tabulations, or newlines}\\\ \mathit{\langle
Separator\rangle}&:=&\texttt{and}\mid\texttt{or}\mid\texttt{,}\mid\texttt{;}\\\
\mathit{\langle
Bullet\rangle}&:=&\texttt{-}\mid\texttt{*}\mid\texttt{+}\mid\texttt{\\#}+\mid\mathit{\langle
Index\rangle}\texttt{.}\\\ \mathit{\langle
Index\rangle}&:=&\texttt{1}\mid\texttt{2}\mid\texttt{3}\mid\dots\\\
\end{array}$ (1)

In the grammar, relevant snippets of text are denoted by the non-terminal
symbol $\mathit{\langle Relevant\rangle}$, which may consist of one or more
words, possibly separated by white space. Different relevant snippets may be
separated by a $\mathit{\langle Separator\rangle}$ – which may be one of the
following strings: "and" or "or", or "," –, or they may be spanning multiple
items of a bulleted or numbered list.

To extract relevant names from any given text, the ExtractNames function would
look for the longest sub-string matching the grammar from eq. 1, and, if any
such sub-string exists, it would build a parse tree for it. For instance, for
the response in LABEL:lst:fake-cat-names, the parse tree would look like the
one in fig. 1. In the parse tree, any node corresponds to non-terminal symbols
in the grammar (rectangles in the figure), whereas leaves correspond to
terminal symbols (ellipses in the figure). To select relevant names from the
parse tree, the function would simply select all the subtrees rooted in a
$\mathit{\langle Relevant\rangle}$ node (coloured subtrees in the figure), and
it would concatenate the leaves of each subtree to form one relevant name to
be output. Notably, this procedure would automatically exclude irrelevant text
from the response, there including any prologue or epilogue which does not
contain a list, as well as any irrelevant text in the middle of the list.

Technically, other fine-grained constraints may be imposed on the grammar, in
order to make the extraction procedure more robust. For instance, one may
require that the indexes of a numbered list are continuous and monotonically
increasing, or that bullets in bullet lists are all the same. In the
particular case of queries for the redistribution phase, one may also filter
out from the response any relevant name which is not included in the query (as
that would imply the LLM is providing an answer which is not among the
admissible ones).

Figure 1:  Parse tree for the response in LABEL:lst:fake-cat-names, parsed
according to the grammar in eq. 1. Colours highlight relevant sub-trees of the
parse tree. Blue and cyan (alternating colours) ellipses highlights relevant
names output by the ExtractNames function when fed with that response.

As far as the ExtractBinary function is concerned, information mining is far
less articulated. In that case, the function simply looks for positive (‘yes’,
‘true’, etc.) or negative (‘no’, ‘false’) words in the textual answer, in a
case-insensitive way. Combined with query templates which explicitly invite
the LLM to be sharp in its answer, this approach provides a simple yet
effective way to interpret LLM responses as boolean values.

##### Handling empty or inconclusive responses

Put it simply, the ExtractNames function outputs a set of sentences, each one
to be interpreted as a relevant name. Despite all the precautions adopted in
the design of the templates and in the parsing of responses, the function may
still output names that require post-processing. For instance, it may happen
that the LLM oracle politely refuses to provide an answer to the query. When
this is the case, the answer is not empty, but rather it contains a sentence
such as "I’m sorry, but I’m a large language model trained by OpenAI, and I
$\mathit{\langle motivation\ here\rangle}$". However, our parsing procedure
would interpret this sentence as an inline list of names, such as "I’m sorry",
"but I’m a large language model trained by OpenAI", etc., which are clearly
meaningless. This is the reason why any robust implementation of the
ExtractNames function should include a post-processing phase, where irrelevant
names are filtered out. This can be done by means of a set of regular
expressions, which are meant to match irrelevant snippets of text in a
sentence. For instance, sentences containing sub-strings "I’m sorry", "I’m a
large language model", etc. may be filtered by the list of relevant names
output by the ExtractNames function.

As a consequence of this post-processing phase, the ExtractNames function may
output an empty list of names. To mitigate this issue, users of KGFiller may
consider asking the same question multiple times in slightly different way.
This is the reason why each phase of KGFiller relies on _a set_ of query
templates, rather than a single one. Providing more than one query template
for each phase is the way by which users can let KGFiller ask the same
question multiple times, in slightly different ways. To serve this purpose, it
is quintessential that users of KGFiller design query templates accordingly.

##### Regulating the creativity of the LLM oracle

Despite the many tricks described so far, it is worth recalling that LLM
oracles may still allow for a degree of randomness when generating their
responses. Randomness is, to some extent, a desirable feature in this context
as it allows for the generation of novel names, yet it may also imply higher
chances of generating meaning-less names. Most commonly, LLM providers provide
a way to regulate the randomness of their responses, by means of a parameter
called _temperature_ (cf. section 2.3).

As far as KGFiller is concerned, the temperature parameter can be set by the
user when starting the algorithm, and that value will be used for all queries
submitted to the LLM oracle during a run of the algorithm. It is worth
mentioning that the algorithm will set temperature to 0.0 when querying the
LLM oracle in the merging phase, regardless of what the temperature the user
has set.

##### About the syntactic similarity of individuals

The merging phase heavily relies on the SyntacticallySimilar function, which
is in charge of determining whether two individuals names are similar enough
to be considered for merging. The implementation of this function is critical
for the success of the merging phase, as it is in charge of selecting
candidates for merging. Making it too permissive may lead the algorithm to do
too many LLM queries, whereas making it too strict may lead the algorithm to
miss some duplicates.

We rely on a simple implementation where
SyntacticallySimilar$(\mathtt{{i}},\mathtt{{i}})$ returns true if (and only
if) the names of the two instances $\mathtt{{i}}$ and $\mathtt{{j}}$ share an
identical substring longer than $\lambda$ characters. There, $\lambda$ is a
hyper-parameter of KGFiller, that we empirically set to $4$.

####  3.2.2 Minimising costs and handling rate limitations

As LLM text generation functionalities are currently provided ‘as a service’,
providers may charge users for the queries they submit to the LLM oracle, or
they may limit the rate at which queries can be submitted. On the KGFiller
side, this may be problematic, as the number of queries submitted to the LLM
oracle may be very high. Hence why any practical implementation of KGFiller
should be robust w.r.t. abrupt service unavailability, and it should minimise
the financial cost of the queries submitted to the LLM oracle. In the
following paragraphs, we discuss a number of strategies to achieve these
goals.

##### Automatic query rate regulation

Rate limitations are typically perceived one the user side as service
disruptions: the LLM service provider may simply reject queries because too
many queries have been submitted in a short time span—or maybe because too
many tokens have been consumed/produced by the previous queries and the
corresponding answers.

To mitigate this issue, practical implementations of KGFiller should be able
to automatically regulate the rate at which queries are submitted to the LLM
oracle. One simple yet effective strategy to achieve this goal is the so-
called _exponential back-off_ strategy—which is commonly used in communication
protocols to avoid congestion.

The exponential back-off strategy is based on the assumption that a failure to
submit a query to the LLM oracle may most likely due to a rate-limitation or
service-overload situation, and that retrying the query after a short delay
may be sufficient to overcome the issue. If it is not, the same strategy could
be applied again, with a longer delay.

For instance, as LLM provides’ rate limitations are typically expressed in
terms of queries or tokens per minute, one may consider setting the initial
delay to $30\text{s}$ and the back-off factor to $150\%$. In this way, the
first time a query is submitted, there is no delay, the second attempt would
be delayed of 30 seconds, the third one of 45 seconds, the fourth one of
roughly more than one minute, and so on. Of course, one may also consider
limiting the amount of retries, or to set a maximum cumulative delay.

In KGFiller, the exponential back-off strategy is implemented by means of the
AskOracle function, which is responsible for submitting queries to the LLM
oracle.

##### Minimising costs

Minimising financial costs can either be achieved by minimising the number of
queries submitted to the LLM oracle, or by minimising the amount of tokens
involved in each query. Notice that the second option may imply the first
option: fewer tokens produced per query implies fewer names being generated by
KGFiller, which implies fewer individuals are added to the ontology, which
implies fewer queries are submitted in the relation and redistribution phases.
Accordingly, we here discuss how to minimise the amount of tokens consumed and
produced by each query.

Our first hint is to design query templates in such a way to suggest the LLM
to generate short and structured responses, as already discussed in section
3.2.1. However, that would only reduce the _produced_ tokens. To minimised the
_consumed_ tokens, one may consider making query templates, class names,
property names, and fancy names as concise as possible. So, for instance, the
query template "list of examples for $\mathit{\langle class\rangle}$, names
only" is sub-optimal, as it contains as it could be shrank to "examples list
for $\mathit{\langle class\rangle}$ names only", with no relevant change in
the generated response.

A more direct way to impose token limitations is to exploit ad-hoc API options
provided by LLM providers, if any. For example, the OpenAI allows clients to
specify the maximum amount of tokens in the response via the query parameter
‘max_tokens’151515https://platform.openai.com/docs/api-
reference/chat/create#max_tokens. . When such parameter is set, the LLM oracle
will stop generating text as soon as the maximum amount of tokens is reached.
This may result in truncated responses, but it may give full control on the
amount of tokens consumed by each query—therefore making financial planning
easier.

It is worth remarking that truncated responses may require further post-
processing, especially as far as names extraction is concerned. When
truncation is in place, there is no guarantee that the last extracted name is
complete, as therefore it should be dropped—in order to avoid introducing
corrupted data into the ontology.

Another feature that may be exploited to minimise costs in face of trials and
errors – which are almost unavoidable when fine-tuning query templates on a
new ontology – is _caching_. By caching queries (along with their parameters
such as temperature and max tokens) and the corresponding answers, one may
avoid costs in case the same query is submitted again. This is not really
useful within the single run of KGFiller, but it may be useful to avoid costs
when re-running KGFiller on the same ontology, or when re-running KGFiller on
a similar ontology. Moreover, it is possible to exploit caches for debugging
purposes, e.g. for retrospectively analysing an unsatisfying ontology
population attempt. This could also be useful to compare different runs of
KGFiller on the same ontology, or on different ontologies.

##  4 Case Study and Experiments

In this section, we present a case study we designed to validate KGFiller
empirically.

The experimental setup is as follows. We design a non-trivial ontology (cf.
section 4.1), by defining a class hierarchy and a set of properties, and we
populate it by means of KGFiller. In doing so, we fine-tune ad-hoc query
templates (cf. section 4.2), we exploit several LLM oracles, from different
families and technologies (cf. section 4.3), hence producing a set of
populated ontologies, all sharing the same class hierarchy and properties. We
then analyse and compare the populated ontologies, with the purpose of
assessing the performance of KGFiller, and the impact of the LLM family of
choice on the quality of the population process. To do so, we define a
taxonomy of errors that may occur during the population process, as well as a
set of metrics (cf. section 4.4) based on those errors, to measure the quality
of the populated ontologies. We then _manually_ compute the corresponding
scores, for each experiment, by analysing the generated individuals one-by-one
(cf. section 4.5).

Technically speaking, experiments are conducted by means of our Python
implementation of KGFiller, which is freely available on
GitHub161616https://github.com/Chistera4-Expectation/kg-filler. The
experimental results are available as well on GitHub, on a dedicated
repository171717https://github.com/Chistera4-Expectation/knowledge-
graphs/branches. There, each branch corresponds to a different experiment,
whereas the main branch contains the initial ontology—only containing class
and property definitions, and no individuals.

###  4.1 Ontology

As dictated by the needs of the Expectation project [42], (i.e., one of the
funding sources of this work), we design an ontology in the _nutritional_
domain. In Expectation, the ontology would serve as a basis for a nutritional
recommender system, aimed at suggesting recipes to users based on their
dietary needs and preferences (cf. [43]).

Despite such recommender system lays outside the scope of this work, we
believe this detail may help the reader in understanding the ontology design
choices. We also stress that, as far as this work is concerned, the ontology
is just one of many possible case studies, and it was designed by only
leveraging on the authors’ common-sense. Designing a scientifically accurate
nutritional ontology would require the expertise of a nutritionist, and this
lays outside the scope of this work.

The ontology aims at collecting _recipes_ and their _ingredients_. To do so,
it categorises _edible_ items into several sub-classes, which should allow a
recommender system to categorise edible items in such a way to match all
possible dietary needs and preferences.

Figure 2:  Class hierarchy of the case study ontology. Notice that the
hierarchy is not really a tree, but rather a DAG. The asterisk (*) denotes
classes having multiple super-classes (they are depicted once per super-class
for the sake of readability).

The ontology consists 54 classes organised as in fig. 2. The root class is
$\mathit{{E}dible}$ and it has several direct or indirect subclasses. One
relevant sub-class of $\mathit{{E}dible}$ is $\mathit{{R}ecipe}$, which is the
root class of various sorts of cuisines (e.g. $\mathit{{I}talian}$,
$\mathit{{C}hinese}$, etc.). Some classes have an annotation property,
($\mathsf{{f}ancyName}$) that better describes the meaning of the class (cf.
section 3.2.1). Another relevant property is
$\mathsf{{i}ngredientOf}:\mathit{{E}dible}\times\mathit{{R}ecipe}$, which is
used to connect recipes to their edible ingredients. Of course, the ontology
is just a skeleton, and it initially contains no individuals.

###  4.2 Query Templates

To query the LLM oracle, we fine-tune query templates as follows.

We define the set of individual seeking templates ($\mathcal{{T}}_{I}$), as
follows181818 we denote by $(A|B)$ a template coming in two variants: one with
$A$ and one with $B$; and by $(A)?$ a template with two variants: one with $A$
and one without it. : "$($instances$\mid$examples$)$ list for class
$\mathit{\langle class\rangle}$$($, names only$)?$"

We define the set of relation seeking templates ($\mathcal{{T}}_{R}$) as the
singleton: "ingredient list for $\mathit{\langle individual\rangle}$, names
only". We only use a template with hard-coded property name, to make the
sentence look more natural.

We define the set of best match templates ($\mathcal{{T}}_{B}$) as the
singleton: "most adequate class for $\mathit{\langle individual\rangle}$
among: $\mathit{\langle classes\rangle}$. concise".

Finally, we define the set of individual merging templates
($\mathcal{{T}}_{M}$) as the singleton: "in the $\mathit{\langle
class\rangle}$ class, should instances $\mathit{\langle ind_{1}\rangle}$ and
$\mathit{\langle ind_{2}\rangle}$ be merged together as semantic and ontologic
duplicates? yes or no answer only".

###  4.3 LLM Oracles

To test the impact of the LLM quality on the KGFiller ontology population
process, we consider integrating several different state-of-the-art LLMs into
the KGFiller pipeline. More in detail, we consider 8 different LLMs aiming at
covering the almost totality of the state-of-the-art. We consider both open
source – e.g., OpenChat, Llama, etc. – and closed models – e.g., GPT –, as
well as Mixture of Experts (MoE) models—e.g., Mixtral.

To integrate querying open source models into the KGFiller pipeline, we rely
on the Hugging Chat API191919https://github.com/Soulter/hugging-chat-api,
which represents a third party library of the Hugging chat service built by
the HuggingFace team202020https://huggingface.co. Leveraging the Hugging Chat
API it is possible to define multiple simoultaneous conversations with
different LLMs relying on the Hugging Chat
platform212121https://huggingface.co/chat/. The most popular HuggingFace open
source LLMs can be seamingly integrated into KGFiller and queried
automatically using the Hugging Chat
API222222https://github.com/Soulter/hugging-chat-api. Therefore, we select the
following open source models for our evaluation:

Llama 2

representing probably the most popular family of open source LLMs alternative
to OpenAI’s GPT [34]. Throughout our experiments we select the 70 billion
parameters model of this family as it allows for seamingly integration with
the Hugging Chat API.

OpenChat 3.5

a 13 billion parameters model fine-tuned from the Llama LLM family and
especially designed to be optimised over mixed-quality data [44].

Mistral 7B

a relatively small (7 billion parameters) LLM developed by the Mistral AI
team232323https://mistral.ai and sponsored as the most powerful language model
for its size [45].

Gemma

a family of lightweight, state-of-the-art open models from Google, built from
the same research and technology used to create the Gemini models [46].
Throughout our experiments we select the 7 billion parameters model of this
family.

Similarly, we rely on the same Hugging Chat API to integrate few MoE models
available from the HuggingFace platform into our analysis. MoE models combine
multiple LLMs – possibly optimised on different data and having different
sizes – using ensemble techniques like boosting and bagging to achieve higher
generalisation and reasoning performance. More in detail, we consider the
following MoE LLM agents:

Mixtral

a pretrained generative sparse MoE – developed by the Mistral AI
team242424https://mistral.ai – built aggregating 8 different LLMs each having
7 billion parameters [47].

Nous Hermes

the new flagship Nous Research252525https://nousresearch.com model trained
over GPT-4 generated data – as well as other high quality data –, achieving
state of the art performance on a variety of tasks. Similalrly to Mixtral,
this MoE model is built from the ensemble of 8 different 7 billion parameters
LLMs.

Finally, concerning closed source models, we consider to rely on the OpenAI
API262626https://openai.com/blog/openai-api to query two different versions of
the GPT family of models, namely:

GPT 3.5 Turbo

the most popular LLM, showcasing groundbraking performances against its
predecessors and counterparts [25].

GPT 4 Turbo

the latest version of the the GPT models family [48]. Its exact number of
parameters is unknown, but few sources report the GPT 4 model to be a MoE.

The LLM size is well-known to impact greatly the performance of the model,
thus affecting the overall quality of the pipeline relying on the LLM.
Similarly, the experiments setup may alter the performance of the different
models selected. Therefore, we report in Table 1 the LLM number of parameters
along with few relevant hyperparameters of our experiments. Here, _max tokens_
represents limit of the LLM’s output length, while _max retries_ is used to
limit the number of times the API is called in case of failure. The _back-off
time_ parameter is used to specify the time to wait between two consecutive
API calls in case of failure. Finally, as LLMs are proven to be influenced by
their _temperature_ parameter which represents a cumbersome value to finetune
exactly, we consider to leave it untouched to its default value – available
either through the HuggingFace or the OpenAI API – thorughout all our
experiments. Table 1 highlights a relevant difference in terms of model size
between open-source and close-source LLMs, with GPT 4 being almost 200 times
larger than the largest open source model available (Llama 2). We expect this
difference to profoundly impact the quality of the constructed ontology and
delve more into this issue in Section 4.5.

Table 1: Size (number of parameters) and experiments setup of the different
LLMs used in KGFiller. Values reported with ∗ are an educated estimate – not
confirmed –, as the corresponding models have not be fully disclosed to the
public.

LLM | size [B] | max tokens | max retries | back-off time [s] | temperature
---|---|---|---|---|---
GPT 3.5 Turbo [25] | 375∗ | 1000 | 2 | 30 | 0.7
GPT 4 Turbo [48] | 1500∗ | 1000 | 2 | 30 | 0.7
Openchat [44] | 13 | 1000 | 2 | 30 | 0.1
Llama2 [34] | 70 | 1000 | 2 | 30 | 0.1
Mistral [45] | 7 | 1000 | 2 | 30 | 0.1
Gemma [46] | 7 | 1000 | 2 | 30 | 0.1
Mixtral [47] | 56 | 1000 | 2 | 30 | 0.1
Nous Hermes | 56 | 1000 | 2 | 30 | 0.1

###  4.4 Performance Metrics

Measuring the quality of ontology construction and population processes is not
trivial, as several components come into play, such as the quality of the
added instances, the reliability of the relations or their lack there-of.
Therefore, we here consider to analyse and present the possible issues that we
should take into account when analysing the KGFiller’s outcome, and define the
corresponding quantitative performance metrics.

####  4.4.1 Types of Errors

When relying on sub-symbolic black-box models for populating ontologies,
several issues may arise depending on the quality of the oracle’s output and
the quality of the output post-processing procedure. LLMs are famously prone
to hallucinations, thus making their suggestions often unreliable. Moreover,
their output’s structure is often unpredictable, depending on how the model
interprets the given query, thus the post-processing procedure is complex and
not always infallible. Therefore, we here define several types of errors that
may arise in the ontology population process.

##### Misplacement error ($E_{mis}$)

The individual is part of the ontology, but it is assigned to the wrong class.
This error represents the most common type of issue when dealing with large
ontologies characterised by several classes. More in detail, we can define two
subclasses of misplacement errors, namely when  _(i)_ an individual is
assigned to a class which is too general – i.e., there exists a better suited
sub-class for it –; or  _(ii)_ an individual is assigned to a class which is
semantically wrong.  Throughout our investigation, we consider these errors as
quantitatively identical, meaning that the occurrence of a misplacement is
counted independently of the misplacement type. However, it is relevant
noticing how the first class of misplacement errors represents a less
impactful error, as it may not be linked with semantic errors.

##### Incorrect individual ($E_{ii}$)

The individual makes no sense for the ontology, yet it has a meaningful name.
This error occurs either when  _(i)_ the LLM oracle model produces an
hallucinated response to a query, suggesting that an out-of-scope meaningful
instance should be added to the ontology – e.g., suggesting that “ _dog_ ”
represents an instance of the class “ _Edible meat_ ” –; or  _(ii)_ the LLM
output processing procedure fails, extracting a meaning yet out-of-context
instance from a correct LLM suggestion—e.g., wrongly postprocessing the
suggestion “ _risotto alla milanese_ ” may result in adding the instance “
_milanese_ ” to the class “ _Recipe_ ”, while it indicates a person from
Milan.  As these errors are generally related to LLM’s hallucination issues,
they represent a relevant class of errors which require careful consideration.

##### Meaningless individual ($E_{mi}$)

The individual name makes no sense at all, and should not be considered as a
part of the ontology. These errors occur when the LLM output processing
procedure fails to identify negative response from the query’s output or fails
to identify the correct cluster of words that refer to the suggested instance.
For example, failing to interpret the message “ _As an AI language model I do
not have access to a pre existing list of instances for class milk that
contains names only_ ” as a negative reply would create a meaningless instance
to added to the ontology.

##### Class-like individuals ($E_{ci}$)

The individual has the same or very similar semantic value of the class it
belongs. Broad answers from the LLM oracle cause this issue, translating to
vague instances that should be removed from a class. For example, when
querying for “ _meat_ ” instances, it is not uncommon for LLM oracles to
suggest the same “ _meat_ ” to be part of the class itself.

##### Duplicate individuals ($E_{di}$)

Two or more individuals have the same or very similar semantic value. This
issue may occur either from the LLM oracle suggesting multiple times very
similar instances – e.g., apple, apples, apple slices, etc. – or from a
wrongful postprocessing of the LLM outputs. Similarly, as the proposed
ontology filling process is iterative, requiring the completion of multiple
consecutive steps, it is possible that different queries outputs produce
semantically identical instances to be added to the ontology. In KGFiller, we
aim at overcoming such issue with the latest _merge phase_ (Section 3.1.4),
where we sample possible duplicates amongst the ontology class and query the
LLM to assess if they need to be merged or not. However, this process is not
infallible and duplicate individuals may persist in the final ontology.

##### Wrong relation ($E_{wr}$)

The relation connecting two individuals is not valid. In KGFiller’s relation
phase (Section 3.1.2), we aim at populating a partially instantiated ontology
$\mathcal{{O}}$ with novel relationships between the individuals therein
contained. This process is performed via querying the LLM oracle. Therefore,
the relationships contained in the ontology may suffer issues related to the
quality of the LLM’s output – e.g., hallucinations –, translating to wrong or
invalid relationships between individuals. For example, an hallucinated LLM
may suggest that “ _onions_ ” are an ingredient of the recipe “ _carbonara
spaghetti_ ”, thus generating an inconsistent relation in the ontology.
Measuring these errors is fundamental to identify the quality of the generated
ontology and connect it to the quality of the LLM’s suggestions.

#####

Throughout our experiments, we measure the error occurences, aiming at
identifying the most relevant shortcomings of relying on LLMs for ontology
population.

####  4.4.2 Measures

As KGFiller relies on sub-symbolic models to populate the given ontology, we
need to asses quantitatively the quality of the population process. However,
up to our knowledge there exist no well-established quantitative measures that
analyse the quality of a constructed ontology. Therefore, we define the
following exhaustive metrics, focusing on various aspects of the individuals
and relations quality:

  1. 1.

the total amount of generated individuals $\mathit{TI}$, so as to evaluate
KGFiller’s ability to generate large ontologies with various individuals for
each class

  2. 2.

the minimum (resp. maximum) class weight $\mathit{minCW}$ (resp.
$\mathit{maxCW}$), which is defined as the minimum (resp. maximum) amount of
indivduals generated per class. This measure allows to check if – and to what
extent – the populated classes are unbalanced

  3. 3.

the total number of individuals belonging to leaf classes $\mathit{TL}$,
gauging how much KGFiller can identify specific classes for the generated
individuals

  4. 4.

the total amount of individuals affected by errors $\mathit{TE}$—where each
individual can count as either zero or one error

  5. 5.

the relative individual error $\mathit{RIE}$, defined as the ratio between the
number of individuals affected by errors and the total number of individuals
$\mathit{RIE}=\dfrac{\mathit{TE}}{\mathit{TI}}$

  6. 6.

the total amount of errors for each class of errors identified in Section
4.4.1—each individual may count for zero, one, or more sorts of errors

  7. 7.

the total amount of generated relations between individuals $\mathit{TR}$

  8. 8.

the relative relation error $\mathit{RRE}$, which is simply defined as the
ratio between the number of wrongful relation and the total number of
relations $\mathit{RRE}=\dfrac{E_{wr}}{\mathit{TR}}$

###  4.5 Results

Table 2: Performance of KGFiller over different state-of-the-art LLMs. For
each performance metric (column) we denote with † and ‡ the best and second
best performing model respectively.

LLM | TI | minCW | maxCW | TL | TE | RIE | $E_{mis}$ | $E_{ii}$ | $E_{mi}$ | $E_{ci}$ | $E_{di}$ | TR | $E_{wr}$ | RRE
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
GPT 3.5 Turbo [25] | 511 | 4 | 40 | 495 | 44† | 0.0861† | 37† | 1† | 0† | 13† | 3† | 736 | 51† | 0.0693†
GPT 4 Turbo [48] | 735 | 5 | 56 | 727 | 91‡ | 0.1238 | 40‡ | 5 | 18 | 19 | 8 | 1061 | 96‡ | 0.0905‡
Openchat [44] | 997 | 9 | 105 | 970 | 244 | 0.2447 | 105 | 65 | 11 | 23 | 32 | 1329† | 440 | 0.3311
Llama2 [34] | 665 | 6 | 55 | 609 | 119 | 0.1789 | 84 | 8 | 4‡ | 21 | 7 | 1062 | 197 | 0.1855
Mistral [45] | 1176† | 11† | 110 | 1113† | 255 | 0.2168 | 133 | 14 | 60 | 32 | 72 | 1211 | 284 | 0.2345
Gemma [46] | 433 | 1 | 40 | 407 | 107 | 0.2471 | 70 | 2‡ | 21 | 16‡ | 3† | 989 | 290 | 0.2932
Mixtral [47] | 841 | 3 | 111‡ | 819 | 228 | 0.2711 | 94 | 22 | 78 | 22 | 15 | 960 | 174 | 0.1813
Nous Hermes | 1121‡ | 11† | 121† | 1082‡ | 123 | 0.1097‡ | 59 | 12 | 11 | 28 | 11 | 1222‡ | 227 | 0.1858

Table 2 presents the results of the performances obtained by KGFiller relying
on the different LLMs considered in Section 4.3. Overall, the obtained results
highlight how it is possible to build quite reliably ontologies from the
implicit knowledge that LLMs attain during their training process. Indeed, at
least 75% of the instances and relations added to the initial empty ontology
are valid for most of the given LLMs. Moreover, few different models achieve
up to 91% of valid instances and 93% of valid relations, encouragingly showing
the reliability of our approach. The types of errors occured in the ontology
population phase also represents a valuable insight, as they highlight how
most issues are related to the relative position of the added instances,
rather than highlighting the prevalence of nonsensical entries. These results
represent an encouraging finding as they confirm the feasibility of relying on
large general-knowledge black-box LLMs to extract a structured and precise
definition of the knowledge concerning a specific topic—such as the food
domain in our case study.

Depending on the specific performance metric considered, different models
achieve the highest performance. For example, _Mistral_ constructs one of the
largest ontology, being capable of adding 1176 instances and 1211 relations to
the final version of the ontology. However, the large number of instances and
relations brings issues concerning the quality of the added components,
resulting in a high relative error measurement. On the other hand, GPT-based
solutions achieve low relative error metrics, with _GPT 3.5 Turbo_ achieving
an encouraging $0.0861$ relative individual error and an even more impressive
relative relation error of $0.0693$. However, relying on these models result
in the construction of relatively small ontologies, as the _GPT 3.5 Turbo_
version has almost half of the instances of several counterparts.

As expected, if we focus on the minimisation of erroneous instances and
relations, we identify GPT-based solutions – and especially GPT 3.5 – as the
best approach. The large difference in the size of the GPT-based models
against the selected counterparts represents a clear advantage whenever the
LLM is presented with the complex task of classifying correctly instances into
the corresponding semantic classes or listing ingredients of recipes without
incurring into hallucinations or slightly unreliable recipes. Indeed, the
ontologies constructed relying on the GPT family also present the least
relevant amount of worrying mistakes or hallucinations (see Section 4.5.1).
However, our results also show that relying on closed source models does not
represent the only viable approach, as few open source models achieve
similarly acceptable level of performance. The _Llama 2_ model constructs a
rather small, but quite effective ontology, keeping low both the number of
incorrect – $E_{ii}$ – and meaningless – $E_{mi}$ – individuals, thus
highlighting its potential. Moreover, relying on the MoE process, the _Nous
Hermes_ model allows a large ontology to be constructed – almost twice as big
as the GPT 3.5 version –, while having almost 90% of the added instances
correct—surpassing GPT 4 Turbo. Once again, these results confirm the
correlation between the dimensionality of the LLM model and the quality of the
constructed ontology, as _Llama 2_ and _Nous Hermes_ represent the largest
open source models (see Table 1).

Finally, one may notice quite a strong proportionality between the relative
individual error and the relative relation error measures. Indeed, LLMs
achieving small $\mathit{RIE}$ tend to achieve a smaller $\mathit{RRE}$ as
well. So, the ability of the underlying LLM to suggest instances of a specific
class or to classify correctly random instances as belonging to a specific
class looks like proportional to the LLM’s ability of building correct
relations between instances, without proposing meaningless associations. This
finding represents a relevant discovery as it suggests the generality of the
LLMs capability of dealing with instances and relations between instances of
an ontology.

####  4.5.1 Mistakes and Hallucinations Examples

We here consider to present and analyse few relevant examples of mistakes and
hallucinations we found throughout our investigation. We rank these examples
from least to most relevant, helping to shed some light on the characteristics
of KGFiller’s shortcomings.

##### Minor mistakes

Most recipes of most models – especially smaller open source models – have
troubles with the identification of the proper spices that compose a recipe.
These types of mistakes can not be considered as the product of the LLM
hallucinating, as they do not resemble the LLM leaving the context of the
recipe, nor can really be considered as blatantly wrong information, as the
concept of a recipe varies a lot depending on personal preferences. Therefore,
we consider similar mistakes as relevant, yet minor mistakes, which we
consider as quickly identifiable after the ontology filling process is
complete. Similar mistakes include the identification of sauces as toppings
and vice-versa, or the misclassification of an instance which is fuzzy by
itself—e.g., is peanut butter butter?

##### Mistakes due to the ontology structure

Depending on the context and the capability of human designers, the skeleton
composing the initial empty ontology may be more or less complete. This may
cause issues whenever KGFiller deals with fuzzy instances that are not easy to
classify directly into one of the available classes. This may be the case of
uncommon LLM suggestions or peculiar items. For example, crocodile and
alligator are not the most common type of consumed meat, however it is true
that there exists few recipes based on these meat cuts. Therefore, whenever
the LLM oracle suggests adding the crocodile and aligator instances to the
ontology, it is not perfectly clear where they should be added. This results
in the case of crocodile and alligator to be classified as instances of the
class poultry-derived food by the _Nous Hermes_ model. While the incidence
frequency can be decreased via a rigorous and complete definition of the
starting empty ontology classes, the fuzzyness of the LLM suggestions should
be better taken into account when dealing with ontology population as the
boundary between reasonable and unreasonable suggestion is tight.

##### LLM’s hallucinations

LLMs are known to suffer hallucinations – i.e., the generation of text that is
not coherent with the context of the conversation, or simply factually wrong
–, which may affect the suggestion of instances and relations to be added to
the ontology under construction. For example, in our experiments we found the
_chemotherapy infusion_ keyword to be suggested by the _Mistral_ model as an
instance for the infusion class. This represents an hallucination of the
underlying LLM, which when prompted for a list of infusions in the food domain
– and also having the dietician imposition from its context –, produced the
chemotherapy infusion shot as a possible suggestion. The automatic post-
processing procedure in KGFiller is unable to identify the proposal as a
wrongful instance, as it respects grammar principles and it is also
semantically valid, therefore adding it to the ontology under construction.
Similarly, _pizza_ is confidently suggested as an American recipe by the
_Mixtral_ model, although it is Italian. Dealing with such issues is complex,
as it is not really possible to check reliably if all of the suggestions of
the queried LLM are to be considered as valid or as hallucinations. Therefore,
the complete removal of these instances is almost impossible. One may think to
query multiple times the LLM to check if the predicted suggestion persists or
not – relying on the hypothesis that hallucinations are rare and that the
model is not confidently predicting an hallucinated suggestion – and add to
the ontology only the confirmed predictions. However, this would require
querying multiple times the LLM only to check the validity of each suggestion,
thus rendering the overall ontology filling process very inefficient.

##### Worrying hallucinations

Normally, hallucinations cause the output to be not coherent with the given
context, thus creating instances that should not belong to the ontology to be
constructed. However, there exists the possibility for LLMs to provide
factually false information – under the form of hallucinations –, which
however, belongs to the given context. This probably represents the worst
possible scenario for the ontology construction process, since it results in
false instances which are very complex to detect and may sometimes be
dangerous. For example, in our experiments we found that, in the edible
mushrooms class _Gemma_ ’s first suggestion is the _Amanita muscaria_ which is
a poisonous mushroom. Being a mushroom, the Amanita muscaria does not
represent an out-of-context suggestion for the LLM, but it still represents a
completely false information and the Amanita’s poisonousness should be known
to the LLM. We also confirm that the LLM was trained to recognise the Amanita
muscaria as a poisonous mushroom separately asking for information about it
and confirming that the model does in fact recognise the Amanita muscaria as
poisonous.

##  5 Discussion and SWOT Analysis

In this section, we propose an analysis of the strengths, weaknesses,
opportunities, and threats that characterise the proposed approach.

###  5.1 Strengths

##### Automation

By relying on LLM oracles – which are trained on large amounts of general-
knowledge data _once and for all_ –, KGFiller overcomes the limitations of
state-of-the-art approaches for ontology population, which require large
_corpi_ of textual documents to be available, from which to extract concepts,
instances, and properties. Therefore, KGFiller represents the first truly
automatic ontology population procedure, which does not require the user to
provide any prior form of domain knowledge representation to populate any
given ontology.

##### Generality

Furthermore, LLMs make KGFiller a general-purpose mechanism to be applied to
virtually any domain, independently of data/domain knowledge availability.
Arguably, KGFiller is poorly affected by the narrowness of the domain, as LLM
oracles are usually trained on large amount of data gathered from the Web,
which cover most of the humankind’s written knowledge. On the other hand,
state-of-the-art ontology population approaches heavily suffer the domain-
sensitivity issue, as the corpus of documents they rely upon may be
incomplete, non-representative, or simply lacking. If the domain is too
narrow, it may be difficult to for such methods to find enough relevant
documents to populate the ontology; if the domain is too broad, it may be
difficult to avoid the inclusion of irrelevant information in the ontology.

Exploiting LLM’s domain specific knowledge acquired from Web resources allows
for KGFiller to reduce the occurrence of bias issues. The inclusion of few
text documents, or the reliance on relatively few domain experts may incur in
biased knowledge whenever state-of-the-art ontology population mechanisms
focus on narrow domains. Conversely, LLM are trained on huge amount of text,
possibly carrying multiple perspectives on any relevant domain.

##### Incrementality

It is worth remarking that KGFiller does not require the input ontology to be
empty. In other words, KGFiller supports not only the population of the
ontology, but its refinement as well. In this sense, we say that KGFiller is
an _incremental_ approach, which can be applied to already-instantiated
ontologies, aiming at enriching or populating them even more.

Along this line, one may simply think to execute KGFiller multiple times,
starting from an empty ontology and populating it incrementally. Each new run
of the algorithm would generate novel instances, until reaching one run where
the amount of novel instances is negligible. Despite we do not consider this
aspect in our experiments – as we focus on the population of an _empty_
ontology –, one might consider exploring the re-iterated approach when
computational and financial costs must be kept under control. Most notably,
other state-of-the-art approaches for ontology population are not really well-
suited for incrementality, as their reliance on a _predefined_ pool of input
documents limits the amount of novel information that they can extract for
each execution.

##### Open future

One last _potential_ strength of KGFiller comes from the _emergent_
capabilities of LLMs. As LLMs have been demonstrated capable of going beyond
the data they are trained on, and they are expected to spontaneously develop
more emergent capabilities as they grow in size and complexity, we expect that
the quality of the ontology population process will increase as well, by
simply relying on the latest LLM models available in the future. If this is
the case, KGFiller may be able to extract knowledge from LLM oracles, even if
that knowledge was not present in the data those oracles were trained upon.

###  5.2 Weaknesses

##### Completeness

Similarly to state-of-the-art approaches, KGFiller does not ensure any
completeness of the final output ontology. Arguably, this is inherent in any
(semi-)automatic data-driven ontology population approach.

Put simply, incompleteness is due to the fact that KGFiller is not designed to
extract _all_ the knowledge the LLM oracle holds. It rather provides means to
limit the amount of instances extracted (to save time or money). And, even if
it was designed to extract all the knowledge, there is no guarantee that the
LLM oracle has acquired all mankind’s knowledge in the first place. However,
we estimate the ontology obtained via KGFiller to be less incomplete with
respect to the one achievable via state-of-the-art approaches relying on fixed
documents sets, as the source training data is much wider—and widening even
more as LLM undergo further training cycles.

##### Sampling bias

Being completely dependent on what LLM oracle has learnt from the Web,
KGFiller may be subject to sampling bias issues. Put simply, the algorithm can
only extract what the LLM oracle has learnt (or what is inferrable from that
knowledge). But, if the LLM did not learn about a specific topic, and that
topic is not inferrable from what the LLM knows, then KGFiller will not be
able to extract that topic. As LLMs are trained on the Web, underrepresented
topics on the Web may be less likely to be extracted by KGFiller, thus leading
to a possibly unbalanced or biased ontology.

In other words, when using KGFiller one should account for the potential
presence of biases in the LLM oracle of choice. Although we estimate this
situation as unlikely, a sampling bias may anyhow exists. To mitigate this
issue, we argue that further research is needed to understand if, and to what
extent, KGFiller can be extended to support the application of LLM to domain-
specific _corpi_ of text – possibly provided by ontology designers on the fly
–, without re-training the LLM.

##### Correctness

Finally, similarly to any (semi-)automatic data-driven ontology population
approach, KGFiller does not provide any guarantee that the extracted
information is correct. In this context, ‘correct’ is intended as the
truthfulness of the information added to the ontology. In fact, as exemplified
in Section 4.5.1, LLM may mistakenly assign poisonous mushrooms to some class
of edible mushrooms, which is of course incorrect. This is due to the fact
that LLMs are prone to hallucinations and to providing false information—a
well-known issue in the LLM community [36]. Thus, caution is necessary
ontology population is delegated to (semi-)automatic procedures, as the final
ontology may contain false information. Along this line, it is worth recalling
that the lack of truth guarantees is common to all state-of-the-art baselines.

While in the future this issue may become a less relevant one, – as LLM
precision and reliability are expected to grow significantly –, for now we
mitigate it by using Git272727https://git-scm.com for tracking ontology
changes in our implementation of KGFiller. This makes the detection of issues
and their post-hoc analysis easier, other than rollbacking to previous
versions of the ontology, as well as the selective removal of specific
instances or relations.

###  5.3 Opportunities

##### Cold start

The development workflow of any _novel_ data-driven or knowledge-based
software system shall eventually face the cold start issue: the system is
well-engineered and operative, but for it to be effective there must be some
data to operate upon. But to gather data, the system should have users, and to
have users, the system should be effective.

This sort of circular dependency issues are very common, especially in
research projects and start-ups, where the system is novel and the user base
is not yet established. In such situations, relying on KGFiller’s automatic
ontology population procedure can help mitigating the cold start issue, by
providing a quick, cheap, and reliable way to generate domain-specific data
for the system to start being effective.

For instance, in the Expectation project [42], we needed to develop a
nutritional recommender system (cf. [43]), but we were lacking data about food
ingredients and recipes. That fuelled the idea of KGFiller.

##### Hierarchy Construction

As defined in this paper, KGFiller focuses on ontology population. As such, it
assumes the existence of a pre-defined ontology schema. However, extensions of
the algorithm where LLM oracles are queried to construct the ontology schema
itself are possible. Approaches of this sort have been proposed in the
literature, e.g. [49], where the task is referred to as _hierarchy
construction_.

Along this line, we envision further extensions of KGFiller, where hierarchy
construction and ontology population are intertwined. For instance, the
algorithm may be extended to identify lacking concepts in the input ontology
in order to better assign the individuals generated by the LLM oracle. The
result would be an intriguing continuous learning setup, where domain
knowledge is repetitively extended and refined.

##### Procedural Knowledge Extraction

Another intriguing opportunity may arise from the application of KGFiller to
the extraction of _procedural_ knowledge. By procedural knowledge, we refer to
the structured data representing the steps required to complete a process, or
attain a specific goal. In the fields of automated planning, as well as multi-
agent systems, procedural knowledge is the way by which agents’ _plans_ are
encoded. One key aspects of plans is that agents can _execute_ them, to
reproduce the process or to attain the goal.

What if KGFiller is exploited to extract plans from LLM oracles? This would
pave the way towards the possibility, for a software agent, to learn novel
behaviours autonomously, by simply asking an LLM _how_ to do something, and
letting KGFiller translate that in structured – hence executable – form.

###  5.4 Threats

##### LLM performance degradation

Web-based, conversational agents backed by LLM technology – especially
OpenAI’s ChatGPT – have been shown to suffer from performance _instability_
over time [50]. This phenomenon is probably linked with the continuous
optimisation process these agents periodically undergo. Being trained not only
on the Web, but on previous interactions too, the quality of their training
data varies over time, and the same is true for their performance.

Such instability potentially affects KGFiller’s performances as well.
Depending on the moment when the algorithm is executed, the quality of the
generated responses may vary significantly, possibly hindering the reliability
of the final ontology.

Similarly, since conversational agents are often re-trained on novel Web data,
the publication of a buggy (i.e. incorrect) ontology may reinforce the
probability the same incorrect information is re-generated in future runs of
the algorithm. This situation may occur, for instance, when the buggy ontology
is the result of an unsupervised execution of KGFiller: incorrect instances or
relations may become public, and therefore become training data for future
generations of LLM, potentially reinforcing the errors that caused them in the
first place.

##### Lock-in effect

Finally, one relevant threat to the use of KGFiller in a real-world project
comes from the business model of most LLM technologies. As for any other
technology provided ‘as a Service’ by the very few private players who can
afford the infrastructure to train and maintain LLMs, there is a chance for
the lock-in effect [51] to arise. Once an ontology is populated using a
specific LLM, it may be difficult to consider leveraging a different LLM to
refine or modify the created ontology, as the knowledge of the second LLM
would be (slightly?) different. Therefore, users might end up in a situation
where there is an over-reliance on a specific LLM technology, a no other
different technology could be exploited unless the whole ontology is re-
populated from scratch, which might be very expensive.

Combined with performance instability, the lock-in effect may also affect the
applicability of KGFiller as a whole. This may happen in the periods of time
when the LLM oracle is not performing well: in that period locked-in users may
have no choice but to wait for the LLM to be re-trained, and hope for the
best.

##  6 Conclusions

Ontologies provide a structured framework that defines concepts,
relationships, and properties within a specific domain, defining an
unambiguous and understandable representation of domain-specific knowledge.
However, ontologies come along with the cost of a meticulous and time-
consuming population process, which is usually manual and possibly affected by
humans’ errors and biases. In this paper, we hypothesise that LLMs incorporate
a relevant amount of domain-specific knowledge obtained from their domain-
agnostic training process, involving huge amount of data gathered from the
web. Accordingly, we propose KGFiller, a novel automatic ontology population
approach leveraging LLMs. More in detail, starting from  _(i)_ an initial
schema composed by inter-related classes and properties, and on  _(ii)_ a set
of query templates,  KGFiller queries the LLM multiple times, and generates
instances for both classes, relationships, and properties from its replies.
Several other queries are performed to refine the ontology, balancing the
instances in the class hierarchy and avoiding the occurence of duplicates.

KGFiller’s validity is assesed through a case study on the food domain, where
the ontology population requires gathering food ingredients – over multiple
categories – and recipes. We test KGFiller over multiple state-of-the-art LLMs
– open source, closed source and MoEs –, measuring an array of metrics that
analyse the quality of the constructed ontology. The obtained results
highlight KGFiller’s ability of reliably populating a given ontology across
several LLMs, reaching instances and relations accuracy of up to $0.91$ and
$0.93$ for the GPT 3.5 model, respectively. As expected, larger LLMs achieve
better performance, while we highlight an interesting correlation between
erroneous instances and misplaced relations. Finally, we analyse the various
degrees of error severity, showcasing how LLMs hallucinations may negatively
impact the automatic ontology population process. Overall, the obtained
results are encouraging, as they show how the complex process of constructing
ontologies can be – at least partially – automated, while still requiring few
caveats and minor post-processing efforts.

## References