copyrightbox

# CBench: Towards Better Evaluation of Question Answering Over Knowledge
Graphs

Abdelghny Orogat  Carleton University<EMAIL_ADDRESS>,  Isabelle
Liu  Carleton University<EMAIL_ADDRESS>and  Ahmed El-Roby  Carleton
University<EMAIL_ADDRESS>
###### Abstract.

Recently, there has been an increase in the number of knowledge graphs that
can be only queried by experts. However, describing questions using structured
queries is not straightforward for non-expert users who need to have
sufficient knowledge about both the vocabulary and the structure of the
queried knowledge graph, as well as the syntax of the structured query
language used to describe the user’s information needs. The most popular
approach introduced to overcome the aforementioned challenges is to use
natural language to query these knowledge graphs. Although several question
answering benchmarks can be used to evaluate question-answering systems over a
number of popular knowledge graphs, choosing a benchmark to accurately assess
the quality of a question answering system is a challenging task.

In this paper, we introduce CBench, an extensible, and more informative
benchmarking suite for analyzing benchmarks and evaluating question answering
systems. CBench can be used to analyze existing benchmarks with respect to
several fine-grained linguistic, syntactic, and structural properties of the
questions and queries in the benchmark. We show that existing benchmarks vary
significantly with respect to these properties deeming choosing a small subset
of them unreliable in evaluating QA systems. Until further research improves
the quality and comprehensiveness of benchmarks, CBench can be used to
facilitate this evaluation using a set of popular benchmarks that can be
augmented with other user-provided benchmarks. CBench not only evaluates a
question answering system based on popular single-number metrics but also
gives a detailed analysis of the linguistic, syntactic, and structural
properties of answered and unanswered questions to better help the developers
of question answering systems to better understand where their system excels
and where it struggles.

##  1\. Introduction

Recent years witnessed an unprecedented growth in the number of knowledge
graphs (KGs) (Auer2007, ; Bollacker2008, ; carlson2010toward, ;
matuszek2006introduction, ; Suchanek2007, ; vrandevcic2014wikidata, ). These
knowledge graphs contain a plethora of information that can be potentially
used for question answering (QA). However, finding answers in a KG is not an
easy task. The user is required to have a detailed knowledge of the KG, and a
structured query language to describe their questions in a structured format
that can be used to find matches in the KG. This requirement limits the
ability to ask questions to power users who have the necessary skills to write
syntactically and semantically correct queries to accurately represent their
information needs. The number of such users represents a tiny fraction of a
potentially large userbase. To overcome this challenge, a large number of QA
systems that let users describe their information needs using natural language
were developed (Cabrio2012, ; Cui2017, ; Dong2007, ; Fader2014, ;
Kaufmann2006, ; Lopez2011, ; Unger2012, ; Yahya2013, ; Zenz2009, ; Zheng2018,
; Zhou2007, ). In fact, over 62 QA systems have been developed since 2010
(Hoeffner2017, ).

As a result of the popularity of QA over KGs, several benchmarks were
introduced to evaluate QA systems (Azmy2018, ; Berant2013, ; Cabrio2013, ;
Su2016, ; trivedi2017lc, ; Unger2011, ; Unger2014, ; Unger2015, ; Unger2016, ;
Usbeck2017, ). These benchmarks typically include questions described in
natural language, answers to the questions from the KG targeted by the
benchmark, and possibly structured queries that return the previously
mentioned answers. To evaluate a newly developed QA system, its developers
need to choose from a large number of benchmarks (at least 17 at the time of
writing this paper) to evaluate their system. Without a quantitative
comparison that highlights the differences between these benchmarks, choosing
a subset of them to evaluate a new QA system is mainly motivated by the ease
of comparison to existing systems in the literature 111Out of 20 QA systems
that were developed in the past 5 years, only 6 are open-sourced, and only 3
systems are accessible via a functional web interface. rather than by how
effective a benchmark is in evaluating a QA system. In fact, existing
benchmarks differ significantly from each other with respect to the following
three points:

  * $\bullet$

How the benchmarks were created: Some benchmarks are manually created by human
experts based on heuristics (abujabal-etal-2019-comqa, ; bordes2015large, ;
Unger2011, ; cabrio2012qakis, ; Cabrio2013, ; Unger2014, ; Unger2015, ;
Unger2016, ; Usbeck2017, ; Usbeck2018, ; ngomo20189th, ). Other benchmarks are
automatically generated from the KG (Su2016, ; trivedi2017lc, ; Azmy2018, ).

  * $\bullet$

Metadata: This includes what KGs that the benchmark target, and the number of
questions in the benchmark. Most benchmarks target a limited number of KGs
(All benchmarks that we are aware of target only 5 KGs). Also, the number of
questions in each benchmark varies significantly. Out of the benchmarks we are
aware of, the benchmark with the smallest number of questions includes 150
questions, while the benchmark with the largest number of questions includes
108,442 questions.

  * $\bullet$

Linguistic, syntactical, and structural properties: The natural language
questions have linguistic properties. Their corresponding queries (if they
exist in the benchmark) also have syntactical and structural properties. In
this paper, we reveal that existing benchmarks vary significantly with respect
to these properties, which is reflected on the reported quality scores
depending on the benchmark used.

Another issue with existing benchmarks is their limited usability. Today,
benchmarks are used in the following fashion: The user parses the benchmark
file, extract the questions and utilizes the QA system to find answers in the
targeted KG, then compare the returned answers to the answers extracted from
the benchmark file to calculate multiple evaluation scores like micro, macro,
and global F-1 scores (discussed in Section 7). The user then examines the
questions that the QA system failed to answer correctly and debug their code
to identify why the QA system struggles with these questions. In that sense,
the benchmark is used as a dataset that helps in producing the aforementioned
scores with a lost potential of being more informative to its user by giving
more details on the fine-grained properties of the processed questions, which
will help the user better understand how the QA system behaves.

In this paper, we introduce CBench 222https://github.com/aorogat/CBench, an
extensible fine-grained benchmark suite that overcomes the aforementioned
challenges facing accurate evaluation of QA systems. CBench can be used in two
modes: _Benchmark Analysis Mode_ and _QA Evaluation Mode_. CBench can be used
in the first mode to perform a fine-grained analysis on the natural language
questions and queries in a set of benchmarks selected by the user. CBench
includes a total of 17 benchmarks targeting 5 KGs, and can be easily extended
with new benchmarks and KGs. For the structured queries, CBench analyzes
several syntactical properties of structured queries like the type of the
query, the operators used, and the query size. CBench also analyzes the
structural properties (shapes) of the queries. For the natural language
questions, CBench analyzes several linguistic properties like the type of
question, and part-of-speech (PoS) tags of each question token of the natural
language question. In our analysis, we surprisingly quantify high-degree
variations with respect to the linguistic, syntactical, and structural
properties of questions and queries among different benchmarks in the
literature, and experimentally show that these differences result in an
inconsistent assessment of QA systems. These findings motivate further
research in building better benchmarks that address as much fine-grained
properties as possible to have a good coverage of real-world questions that
the QA systems will encounter in real-world deployments.

Until such benchmarks are available, CBench facilitates evaluating QA systems
in the second mode (_QA Evaluation Mode_), in which the user can choose the
benchmarks they wish to use to evaluate their QA system. Then, CBench
interacts with the QA system using a set of well-defined APIs to send/recieve
questions/answers. In addition to reporting micro, macro, and global F-1
scores, CBench also analyzes all the questions in the chosen benchmarks and
their corresponding structured queries (if available). Specifically, CBench
returns (1) a detailed analysis of the properties of the queries that the
evaluated QA system processed, and (2) linguistically-similar natural language
questions for any question of interest (e.g., a question that the QA system
failed to answer). Using the two aforementioned types of output, the QA user
can either (1) identify common properties between questions that the QA system
struggles with (e.g., most of the questions have a specific query shape), or
(2) identify obvious inconsistencies in the processed questions. For example,
using CBench, we were able to quickly identify that one of the QA systems we
evaluated was able to answer the question “What is the capital of Cameroon?”
correctly, while it incorrectly answered “What is the capital of Canada?”,
which highlights an overfitting problem in their entity recognition and
relation mapping approaches. Being able to quickly identify commonalities or
inconsistencies will help the QA system developers to quickly identify the QA
system component that they need to improve. Based on the insights provided by
CBench, the user can also use it in a _Debugging Mode_ within the _QA
Evaluation Mode_ , where they can control CBench’s output questions based on
any of the linguistic, syntactical, or structural properties of all the
questions and queries in CBench to better understand how their QA system
behaves in several controlled situations. For example, the user can specify
that CBench only outputs temporal questions whose queries have a star-shape.

Our contributions in the CBench suite are:

  * $\bullet$

To the best of our knowledge, We are the first to introduce the concept of
fine-grained analysis of questions and queries in QA benchmarks.

  * $\bullet$

Using our fine-grained analysis, we surprisingly identify a high degree of
variations among existing benchmarks with respect to several linguistic,
syntactical, and structural properties of the natural language questions and
their corresponding queries in the benchmarks.

  * $\bullet$

To demonstrate the effects of such variations, we evaluate six QA systems
using CBench in the _QA Evaluation Mode_ and show that their quality scores
vary significantly.

  * $\bullet$

We give important insights to QA systems’ researchers on benchmark selection
and QA evaluation metrics.

The rest of this paper is organized as follows: Section 2 presents the
preliminaries of this paper. Section 3 discusses the architecture of CBench.
Section 4 gives an overview of the benchmarks in CBench. Section 5 presents
how CBench analyzes the structured queries in the benchmarks. Section 6
presents how CBench analyzes the natural language questions. Section 7
discusses our experiments on six QA systems using CBench. Section 10 concludes
the paper.

##  2\. Preliminaries

Knowledge Graph (_KG_): A knowledge graph is a directed graph $KG=\\{V,E\\}$
consisting of a set of vertices $V$ that represent entities, types, and
literals, and a set of labeled edges $E$ that connect these vertices. RDF
(RDF, ) is a popular representation model for KGs, which organizes data as a
set of triples in the form _$\langle s,p,o\rangle$_ where _$s$_ refers to
_subject_ , _$p$_ refers to _predicate_ , and _$o$_ refers to _object_ , such
that $s,o\in V$, and $p\in E$. SPARQL (SPARQL, ) is the structured query
language for querying RDF. Figure 1 visualizes a sample subgraph from DBpedia
(Auer2007, ).

$dbr:Lenovo$$dbr:Electronics$$dbr:Beijing$$dbo:Company$$dbr:India$$dbr:Indian\\_Railways$$dbr:New\\_Delhi$$dbo:Country$$60000$$1335000$$dbo:numberOfEmployees$
$dbo:foundationPlace$$dbo:industry$
$rdf:type$$rdf:type$$dbo:location$$dbo:capital$$rdf:type$$dbo:numberOfEmployees$
Figure 1. A Subgraph from the DBpedia.

Graph Pattern (_gp_): A Graph Pattern is a set of triple patterns. A triple
pattern can be generated from a triple by replacing the $s$, $p$, or $o$ by a
variable from the universal set of variables $U$. A query can include multiple
graph patterns.

Query (_q_): A query is represented as the pair ($GP$, $SM$), where $GP$ is
the set of graph patterns in the query and $SM$ is the set of solution
modifiers. The query returns an answer set $A$ that (1) matches the given $GP$
from a knowledge graph $KG$, and (2) modified to conform to the the solution
modifiers $SM$ (e.g., select, ordered by, distinct, limit and offset, etc.).
In SPARQL, a query with type $Select$ is used to select all, or a subset of,
the variables bound in the set of subgraphs that matches $GP$. If type is
$Ask$ instead of $Select$, it returns a Boolean answer; $true$ if $GP$ can be
matched ($A$ is a non-empty set).

Natural Language Question (_nlq_): A natural language question is represented
as the list of all tokens of the question excluding white spaces and
punctuation marks. Formally, $nlq=[token_{1},token_{2},\ldots,token_{m}]$,
where $m$ is the number of tokens in the question.

Benchmark (_B_): A benchmark $B=\\{NLQ,Q,G\\}$ consists of a set of natural
language questions $NLQ$, an optional set of formal queries $Q$, and a set of
gold standard answers $G$ such that $g_{i}$ is the set of gold standard
answers for the question $nlq_{i}$, whose corresponding query is $q_{i}$.

###### Example 1.

Following is an example of a natural language question that can be found in a
benchmark ($nlq_{i}$): $[$Which, companies, have, more, than, 1, million,
employees, or, founded, in, Beijing$]$. The query333The prefix dbr is bound to
http://dbpedia.org/resource/
The prefix dbo is bound to http://dbpedia.org/ontology/
The prefix rdf is bound to http://www.w3.org/2000/01/rdf-schema# ($q_{i}$)
that is associated with this question is:

    SELECT DISTINCT ?uri WHERE {
        ?uri a dbo:Company {
            ?uri dbo:numberOfEmployees ?n .
            FILTER ( ?n > 1000000 )
        } UNION {
            ?uri dbo:foundationPlace dbr:Beijing.
        }
    }

The answers ($g_{i}$) to this query based on the subgraph in Figure 1 are
$\\{$“Lenovo”, “Indian Railways”$\\}$.

##  3\. Overview of CBench

Figure 2. CBench Architecture.

Figure 2 shows the architecture of CBench, which can be used in two modes: (1)
The Benchmark Analysis Mode, where CBench can be used to perform a fine-
grained analysis on the structured queries and the natural language questions
on a set of benchmarks selected by the user, and (2) the QA Evaluation Mode,
where CBench can be used to evaluate a QA system over the user-selected
benchmarks providing deeper insights on how the QA system is performing.
Benchmark Analysis Mode: CBench includes 17 benchmarks from which the user can
choose a subset for analysis. The user can also upload their own benchmarks to
be included in the analysis. The Benchmark Builder passes the selected
benchmarks (and the uploaded ones, if any) to the Q-Analyzer which carries out
the syntactical and structural analysis of the queries (Section 5), and the
linguistic analysis (Section 6). Finally, the Q-Analyzer returns the analysis
report to the user.
QA Evaluation Mode: Just like the previous mode, the user selects a set of
benchmarks and/or upload their own to evaluate the QA system. In addition, the
user provides CBench with a URL for an endpoint that CBench can query. Other
configuration parameters that are used in the evaluation (e.g., thresholds for
calculating quality scores) are also chosen by the user prior to evaluation.
To avoid the scenario where the selected benchmarks target different versions
of the same KG (discussed in Section 4), the Benchmark Builder updates the
answers of the queries in the selected benchmarks through the Benchmark
Updater module, which queries the used KG that will be used for the evaluation
to retrieve the updated answers. The updated benchmarks are then passed to the
System Evaluator. The System Evaluator carries out three tasks: (1)
Communicating with the QA system to collect the answers to the questions from
the selected benchmarks, (2) calculating the micro, macro, and global F-1
scores (discussed in Section 7), and (3) retrieving the fine-grained analysis
of the processed questions from the Q-Analyzer. The System Evaluator then
outputs an interactive report that includes the scores and the analysis of the
processed questions to the user. The user can choose to focus on specific
questions to view all their fine-grained properties. CBench also finds other
questions that are linguistically _similar_ to the selected questions
(discussed in Section 6). Using this feature, the user is able to quickly
identify either common features or inconsistencies of unanswered or
incorrectly answered questions, which will help the user to understand which
components of their QA system to improve. The user can also use CBench in the
Debugging Mode, in which they can group questions/queries based on specific
properties to evaluate their QA system in specific scenarios. For example, the
user can choose to evaluate their QA system only on aggregate questions (e.g.,
How many) whose queries have a star-shape to investigate how their system
deals with such questions.

The details of the configurations of CBench, how to add a new benchmark, and
the API’s used for communication with the QA systems can be found in the
system’s repository444https://github.com/aorogat/CBench.

##  4\. Benchmarks in CBench

Table 1. The benchmarks used in CBench. DB refers to DBpedia, FB refers to
Freebase, MB refers to MusicBrainz, WK refers to Wikidata, and LS refers to
LinkedSpending. Benchmarks annotated with $\star$ do not include queries, and
benchmarks annotated with $\dagger$ target only single-factoid questions.
Benchmarks | #Qs | KG | Version
---|---|---|---
_QALD-1_ (Unger2011, ) | 199 | DB, MB | 3.6
_QALD-2_ (cabrio2012qakis, ) | 344 | DB, MB | 3.7
_QALD-3_ (Cabrio2013, ) | 397 | DB, MB | 3.8
_QALD-4_ (Unger2014, ) | 321 | DB | 3.9
_QALD-5_ (Unger2015, ) | 334 | DB | 2014
_QALD-6_ (Unger2016, ) | 431 | DB, LS | 10-2015
_QALD-7_ (Usbeck2017, ) | 530 | DB, WD | 04-2016
_QALD-8_ (Usbeck2018, ) | 315 | DB, WD | 10-2016
_QALD-9_ (ngomo20189th, ) | 408 | DB | 10-2016
_LC-QuAD_ (trivedi2017lc, ) | 4,998 | DB | 04-2016
_WebQuestions_ (Berant2013, ) | 5,810 | FB | 09-08-2015
_GraphQuestions_ (Su2016, ) | 5,166 | FB | 06-2013
_SimpleQuestions_ $\star\dagger$ (bordes2015large, ) | 108,442 | FB | FB2M, FB5M
_SimpleDBpediaQA_ $\star\dagger$ (Azmy2018, ) | 43,086 | DB | 10-2016
_TempQuestions_ $\star$ (jia2018tempquestions, ) | 1,271 | FB | 09-08-2015
_ComplexQuestions_ $\star$ (Abujabal2017, ) | 150 | FB | 09-08-2015
_ComQA_ $\star$ (abujabal-etal-2019-comqa, ) | 11,214 | Wikipedia | -
Table 2. QALDs Overlaps. #Qs is the total number of all questions. #SU is the number of Same questions but associated with Updated queries. #SS is the number of Same questions and associated with the Same queries. #New is the number of New questions and associated with New queries. |  #Qs |  #SU |  #SS |  #New
---|---|---|---|---
_QALD-1_ |  199 |  - |  - |  199
_QALD-2_ |  344 |  150 |  0 |  194
_QALD-3_ |  397 |  236 |  52 |  109
_QALD-4_ |  321 |  146 |  6 |  169
_QALD-5_ |  334 |  27 |  193 |  114
_QALD-6_ |  431 |  317 |  0 |  114
_QALD-7_ |  530 |  262 |  57 |  211
_QALD-8_ |  315 |  174 |  100 |  41
_QALD-9_ |  408 |  231 |  171 |  6

Before discussing the analysis of queries and natural language questions, we
give an overview of the benchmarks that are included in CBench. CBench can
also be augmented with other benchmarks provided by the user. Table 1 shows
the benchmarks along with the number of questions in each and the KGs they
target. _QALD_ 555http://qald.aksw.org is an annual evaluation campaign for
question answering that started in 2011. Therefore, it includes 9 benchmarks
(_QALD-1_ to _QALD-9_).  There are overlaps between these versions of QALD. As
each question in any of the QALD benchmarks is associated with a query, there
are cases where the natural language question is in a more recent benchmark
but associated with a different query. Table 2 shows the number of repeated
natural language questions for each benchmark and how many of those have
repeated queries (i.e., no change at all in the benchmark entry). The second
column shows the total number of questions in each benchmark. While the third
and fourth columns show the number of questions borrowed from previous
versions, the fifth column shows how many questions were inserted for the
first time in the corresponding version of QALD. For example, QALD-9 has in
total 408 questions. Only 6 questions are new. The other 402 questions are
borrowed from all older versions, where 231 of the 402 questions use the same
natural language questions but are associated with updated queries. The
remaining 171 questions use the same natural language questions and the same
queries. It is interesting to see that some questions are used in some
version, left in a newer version and then reused again in a more recent
version.

_LC-QuAD_ is a semi-automated question generation dataset. SPARQL templates
are automatically generated and are converted into natural question templates.
These general templates are manually transformed into natural language
questions. _GraphQuestions_ is a set of questions that are generated in two
steps. First, generating a set of graph-structured logical patterns from the
KG, then transforming them into natural questions with the help of human
annotators. _WebQuestions_ is a set of questions obtained from non-experts.
These questions are collected based on suggestions from Google Suggest API.
The questions with answers from Freebase are taken and annotated by Amazon
Mechanical Turk workers then converted to SPARQL queries by experts.
_SimpleQuestions_ is generated by shortlisting the set of facts from Freebase
that can be converted to informative questions. Then, these elected facts were
sampled and passed to annotators to manually generate natural language
questions whose answers are the entities in these facts. _SimpleDBpediaQA_ is
a subset of the _SimpleQuestions_ benchmark dataset created by mapping
entities and predicates from Freebase to DBpedia. _TempQuestions_ consists
exclusively of temporal questions. These questions are extracted from
_Free917_ (cai-yates-2013-large, ), _WebQuestions_ and _ComplexQuestions_.
_ComplexQuestions_ is multi-constraints question-answer pairs that have some
questions from _WebQuestions_ (596 questions) and some manually labelled
questions. _ComQA_ is a large benchmark that includes real questions taken
from the WikiAnswers platform and present various challenging aspects such as
compositionality, temporal reasoning, and comparisons.

##  5\. Analysis of Structured Queries

Most of the benchmarks discussed in Section 4 include structured queries that
can be used to obtain the answers to the natural language questions in the
benchmark. Inspired by prior work on the analysis of query logs of endpoints
(Bonifati2017, ), we discuss how the Query Analyzer processes the SPARQL
queries by focusing on both their syntactically and structural properties to
present them in the final report shown to the user. We use CBench’s Benchmark
Analysis Mode to process _all_ queries from the 12 benchmarks that include
SPARQL queries to highlight the high-degree variations among them, then we
give our insights in light of these variations. In Section 7, we use CBench’s
QA Evaluation mode to experimentally show the effects of these variations on
the accurate assessment of QA systems.

###  5.1. Syntactical Analysis of Queries

In this section, we study the syntactical properties of the queries, which
include the frequency of query keywords, the number of triple patterns in the
query, and the frequency usage of operators.

Table 3. Percentage of keyword occurrences in queries for each benchmark. Element | QALD | LC-QuAD | Web | Graph
---|---|---|---|---
_Select_ | 91.63% | 91.52% | 100.00% | 100.00%
_Ask_ | 8.37% | 8.48% | 0.00% | 0.00%
_Distinct_ | 76.65% | 91.52% | 99.98% | 0.00%
_Limit_ | 6.51% | 0.00% | 0.02% | 0.00%
_Offset_ | 3.93% | 0.00% | 0.00% | 0.00%
_Order By_ | 5.99% | 0.00% | 0.02% | 0.00%
_And_ | 51.65% | 70.67% | 37.65% | 41.75%
_Filter_ | 10.33% | 0.00% | 99.62% | 100.00%
_Union_ | 6.10% | 0.00% | 0.36% | 0.00%
_Optional_ | 5.37% | 0.00% | 0.00% | 0.00%
_Not Exists_ | 0.21% | 0.00% | 0.00% | 0.00%
_Minus_ | 0.21% | 0.00% | 0.00% | 0.00%
_Aggregators_ | 5.27% | 0.00% | 0.00% | 20.17%
_Group By_ | 5.27% | 0.00% | 0.00% | 13.74%
_Having_ | 1.34% | 0.00% | 0.00% | 0.00%

####  5.1.1. Query Keywords

We count the frequency of query keywords in the benchmarks. The results are
shown in Table 3. Due to the lack of space, we combine all the queries from
_QALD-1_ to _QALD-9_ and report them under _QALD_. In the case of repetitive
questions, we consider the most recent query. After deduplication, the _QALD_
discussed here include 959 questions. We find that even within the 9 _QALD_
benchmarks, variations still exist. We will highlight these variations later
in this section. The first block in Table 3 reports the query types. In
general, the majority of queries use the _Select_ keyword across all
benchmarks (at least 91% of queries). _Ask_ queries whose answers are either
true (if a solution matches the graph pattern in the query) or false
(otherwise) represent 8.37% and 8.48% of the queries in _QALD_ and _LC-QuAD_ ,
respectively. _GraphQuestions_ and _WebQuestions_ do not include queries that
use the keyword _Ask_.

The second block in Table 3 includes the keywords used as solution modifiers
in the queries. We ignored the Reduced modifier as it is never used in all
benchmarks. We notice that the majority of the queries use the _Distinct_
keyword in all benchmarks except _GraphQuestions_ , which is the only
benchmark that does not use any of the four solution modifiers. The _Limit_ ,
_Offset_ and _Ordered-By_ keywords are not frequently used in _QALD_ , and
almost non-existent in other benchmarks.

The third block in Table 3 has keywords used to describe the graph patterns as
described in Section 2. The _And_ operator, which represents the conjunctions
of triple patterns, is often used across all benchmarks. However, there is a
large variation in the percentage of queries that use the keyword, where the
minimum percentage is 37.65% (_WebQuestions_), and the maximum percentage is
70.67% (_LC-QuAD_). The _Filter_ keyword demonstrates an interesting case,
where it is used in almost all questions in _WebQuestions_ and
_GraphQuestions_ , but with a much smaller percentage in _QALD_ and not used
at all in _LC-QuAD_. The remaining keywords (_Union_ , _Optional_ , _Not
Exists_ , and _Minus_) are used in a small percentage of queries across all
benchmarks. But again, there is a high degree of variation across all
benchmarks in their usage for these keywords.

Lastly, _Aggregators_ (e.g., _count_ , _max_ , etc.) and _Group-By_ keywords
are used only in _QALD_ and _GaphQuestions_. The keyword _Having_ is used in
_QALD_ only.

Table 4. Percentage of keyword occurrences in queries for each QALD benchmark.
Element | QALD-1 | QALD-2 | QALD-3 | QALD-4 | QALD-5 | QALD-6 | QALD-7 | QALD-8 | QALD-9
---|---|---|---|---|---|---|---|---|---
_Select_ | 90.48% | 89.26% | 90.54% | 91.13% | 91.78% | 92.77% | 88.04% | 89.17% | 90.61%
_Ask_ | 9.52% | 10.74% | 9.46% | 8.87% | 8.22% | 7.23% | 11.96% | 10.83% | 9.39%
_Distinct_ | 38.69% | 80.00% | 86.25% | 91.13% | 89.14% | 86.51% | 77.20% | 67.83% | 76.40%
_Limit_ | 4.17% | 4.44% | 4.30% | 8.06% | 9.21% | 8.19% | 7.45% | 11.15% | 9.64%
_Offset_ | 0.00% | 1.85% | 2.01% | 6.05% | 8.55% | 7.71% | 4.29% | 5.41% | 6.09%
_Order By_ | 4.17% | 4.44% | 4.30% | 8.06% | 9.21% | 8.19% | 7.45% | 9.55% | 8.88%
_And_ | 79.76% | 64.81% | 60.74% | 50.81% | 50.33% | 46.75% | 39.28% | 35.99% | 42.89%
_Filter_ | 39.29% | 54.44% | 9.17% | 8.06% | 6.58% | 4.34% | 9.71% | 3.18% | 4.82%
_Union_ | 7.74% | 7.04% | 5.73% | 8.87% | 9.87% | 8.19% | 4.97% | 1.59% | 6.60%
_Optional_ | 38.10% | 50.00% | 0.86% | 0.40% | 0.33% | 0.24% | 6.09% | 0.32% | 0.25%
_Not Exists_ | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.48% | 0.45% | 0.64% | 0.00%
_Minus_ | 0.00% | 0.00% | 0.00% | 0.81% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00%
_Aggregators_ | 2.98% | 3.33% | 4.30% | 1.61% | 2.63% | 6.02% | 5.64% | 5.10% | 4.82%
_Group By_ | 2.98% | 3.33% | 4.30% | 1.61% | 2.63% | 6.02% | 5.64% | 5.10% | 4.82%
_Having_ | 0.60% | 2.22% | 2.29% | 0.81% | 0.66% | 0.72% | 0.68% | 0.32% | 0.76%

With respect to the 9 _QALD_ benchmarks, variations also exist although they
come from the same organization as shown in Table 4. For example, the
_Distinct_ keyword occurs in only 38.69% of the _QALD-1_ queries , whereas in
the other _QALD_ benchmarks, the _Distinct_ keyword occurs in between 67% to
91% of the queries. The _Limit_ keyword occurrences steadily increase from
4.17% in _QALD-1_ to 9.64% in _QALD-9_. The _Offset_ keyword is approximately
never used in _QALD-1_ to _QALD-3_ , but occurs more frequently in the other
_QALD_ benchmarks (from 4% to 8%). The _Filter_ and _Optional_ keywords occur
frequently in _QALD-1_ and _QALD-2_ (39.29% and 54.44% of the queries,
respectively), while this percentage is significantly declined in other _QALD_
benchmarks (does not exceed 11%).

####  5.1.2. Number of Triple Patterns

Figure 3. Percentage of queries exhibiting different number of triple patterns
for each benchmark.

The Query Analyzer also counts the number of triple patterns in all the graph
patterns of the queries as a measure of the size of the queries in the
benchmarks. The total number of triple patterns has been computed and
categorized from 1 to 11+ triple patterns for every benchmark. Figure 3 shows
that the queries with a low number of triple patterns (from 1 to 3) are
dominant in all benchmarks except the _GraphQuestions_ benchmark, in which
there are no queries with 1 triple pattern. Noticeably, the earlier versions
of _QALD_ and the _GraphQuestions_ benchmarks include longer queries when
compared to the other benchmarks.

####  5.1.3. Query Operators

Table 5. The frequency of the operators used in queries: Filter (F), And (A), Optional (O), and Union (U). Operators | QALD | LC-QuAD | Web | Graph
---|---|---|---|---
$none$ | 42.25% | 29.33% | 0.09% | 0.00%
$F$ | 0.00% | 0.00% | 62.19% | 58.25%
$A$ | 42.87% | 70.67% | 0.17% | 0.00%
$A,F$ | 4.65% | 0.00% | 37.19% | 41.75%
$CPF$ | 89.77% | 100.00% | 99.64% | 100.00%
$O$ | 0.00% | 0.00% | 0.00% | 0.00%
$O,F$ | 2.58% | 0.00% | 0.00% | 0.00%
$A,O$ | 0.10% | 0.00% | 0.00% | 0.00%
$A,O,F$ | 1.45% | 0.00% | 0.00% | 0.00%
$CPF+O$ | +4.13% | +0.00% | +0.00% | +0.00%
$U$ | 2.48% | 0.00% | 0.07% | 0.00%
$U,F$ | 0.10% | 0.00% | 0.00% | 0.00%
$A,U$ | 1.96% | 0.00% | 0.05% | 0.00%
$A,U,F$ | 0.31% | 0.00% | 0.24% | 0.00%
$CPF+U$ | +4.86% | +0.00% | +0.36% | +0.00%

We also study the co-occurrences of the _Filter_ , _And_ , _Union_ , and
_Optional_ operators in the queries. The results can be found in Table 5. The
first block of the table shows the queries with graph patterns that have
triple patterns with no operators (with a single triple pattern without
filter), with _Filter_ only, with _And_ only, or with both _Filter_ and _And_
operators. The next row has their subtotal (_c onjunctive patterns with
filters_ or _CPF_). Most of the queries (from 89.77% to 100.00%) in all
benchmarks are _CPF_ queries. However, the distribution of queries using the
combinations of the two operators varies.

The second block in the table shows the co-occurrences of the _Optional_
operator with different types of _CPF_ queries. This operator is only used in
the _QALD_ benchmarks with an increase of +4.13% in the relative size.

Table 6. The frequency of the operators used in QALDs queries: Filter (F), And
(A), Optional (O), and Union (U).
Operators | QALD-1 | QALD-2 | QALD-3 | QALD-4 | QALD-5 | QALD-6 | QALD-7 | QALD-8 | QALD-9
---|---|---|---|---|---|---|---|---|---
$None$ | 7.14% | 10.37% | 35.53% | 44.76% | 44.41% | 48.43% | 53.50% | 63.06% | 53.05%
$F$ | 0.00% | 0.37% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00%
$A$ | 50.60% | 35.19% | 50.14% | 39.92% | 40.46% | 39.76% | 32.96% | 32.17% | 36.04%
$A,F$ | 3.57% | 3.70% | 7.74% | 6.05% | 4.93% | 3.37% | 3.61% | 2.87% | 4.06%
$CPF$ | 61.31% | 49.63% | 93.41% | 90.73% | 89.80% | 91.57% | 90.07% | 98.09% | 93.15%
$O$ | 0.60% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00%
$O,F$ | 7.74% | 20.37% | 0.57% | 0.40% | 0.33% | 0.24% | 4.51% | 0.32% | 0.25%
$A,O$ | 1.79% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00%
$A,O,F$ | 20.83% | 22.96% | 0.29% | 0.00% | 0.00% | 0.00% | 0.45% | 0.00% | 0.00%
$CPF+O$ | +30.95% | +43.33% | +0.86% | +0.40% | +0.33% | +0.24% | +4.97% | +0.32% | +0.25%
$U$ | 0.60% | 0.00% | 3.15% | 3.63% | 4.28% | 4.34% | 1.58% | 0.64% | 3.55%
$U,F$ | 0.00% | 0.00% | 0.00% | 0.40% | 0.66% | 0.24% | 0.00% | 0.00% | 0.25%
$A,U$ | 0.00% | 0.00% | 2.01% | 3.63% | 4.28% | 3.13% | 2.26% | 0.96% | 2.54%
$A,U,F$ | 0.00% | 0.37% | 0.57% | 1.21% | 0.66% | 0.48% | 0.00% | 0.00% | 0.25%
$CPF+U$ | +0.60% | +0.37% | +5.73% | +8.87% | +9.87% | +8.19% | +3.84% | +1.59% | +6.60%

Finally, The third block of the table shows that the _Union_ operator is not
used in _GraphQuestions_ and _LC-QuAD_ , and is rarely used in _WebQuestions_
, while it is more frequently used in _QALD_. For the QALD benchmarks, there
are other combinations that are rare and therefore not included in this table.
For example, (O,U), (O,F,U), (A,O,U), etc.. Table 6 shows the results of each
QALD version separately.

###  5.2. Structural Analysis

In addition to the syntactical analysis of the queries in the benchmarks, we
also analyze the structural shapes of the queries of the following types of
queries: 1. Conjunctive queries that can use only the _And_ operator, denoted
by _CQ_. 2. Conjunctive queries that can use both _And_ and _Filter_
operators, denoted by _CQ F_. 3. Conjunctive queries that can use _And_ ,
_Filter_ and _Optional_ operators, denoted by _CQ OF_.

Figure 4. The different shapes recognized by CBench  Table 7. Cumulative shape
analysis of CQ, CQF, CQOF across all benchmarks.
| QALD | LC-QuAD | Web | Graph
---|---|---|---|---
Shape | CQ | CQF | CQOF | CQ | CQF | CQOF | CQ | CQF | CQOF | CQ | CQF | CQOF
Single-Edge | 50.82% | 46.88% | 45.13% | 29.35% | 29.35% | 29.35% | 40.00% | 63.29% | 63.29% | 0% | 0.00% | 0.00%
Chain | 83.77% | 80.79% | 80.20% | 72.93% | 72.91% | 72.91% | 80.00% | 90.98% | 90.98% | 0% | 59.56% | 59.56%
Chain-Set | 83.90% | 81.60% | 80.97% | 72.93% | 72.91% | 72.91% | 80.00% | 90.98% | 90.98% | 0% | 59.56% | 59.56%
Star | 14.84% | 15.86% | 16.59% | 27.05% | 27.06% | 27.06% | 20.00% | 8.61% | 8.61% | 0% | 30.70% | 30.70%
Tree | 99.75% | 97.80% | 97.90% | 100.00% | 100.00% | 100.00% | 100.00% | 99.66% | 99.66% | 0% | 97.74% | 97.74%
Forest | 100.00% | 99.54% | 99.56% | 100.00% | 100.00% | 100.00% | 100.00% | 99.93% | 99.93% | 0% | 97.74% | 97.74%
Cycle | 0.13% | 0.12% | 0.11% | 0.00% | 0.00% | 0.00% | 0.00% | 0.32% | 0.32% | 0% | 0.00% | 0.00%
Flower | 99.87% | 98.15% | 98.23% | 100.00% | 100.00% | 100.00% | 100.00% | 99.98% | 99.98% | 0% | 99.33% | 99.33%

CBench identifies eight different shapes of queries. Figure 4 illustrate these
shapes666Examples of questions, their corresponding queries, and their
structural shapes can be viewed in the repository.. The _Single-Edge_ shape
has only one edge. The _Chain_ shape with length $n$ is a series of edges
{$x_{0}$,$x_{1}$}, {$x_{1}$,$x_{2}$}, $\ldots$ {$x_{n-1}$,$x_{n}$}. The
_Cycle_ shape is like the _Chain_ shape except that the first node in the
chain is the same as the last node. The _Chain-Set_ shape is a set of one or
more unconnected chains. The _Tree_ shape can have any connected nodes keeping
only one path between any two nodes. The _Star_ shape is a special case of the
_Tree_ shape where there exists exactly one node with more than 2 neighbors.
The _Flower_ shape is the graph that has a node that is connected to at least
one attachment that could have any of the following three shapes: _Chain_ ,
_Tree_ and _Petal_ , where the _Petal_ is a graph with two or more disjoint
paths between a source node and a destination node. The _Forest_ shape
includes a set of unconnected _trees_. It is worth noting that some shapes
subsume other shapes. For example, the _Tree_ shape subsumes the _Chain_
shape, which subsumes the _Single-Edge_ shape.

The set-shapes questions (e.g., _Chain-Set_ and _Forest_) are used in NLQ
where a comparison between 2 variables is required. Figure 5 is an example
from QALD-9 which visualizes the Chain-Set query:

    SELECT DISTINCT ?uri WHERE {
        res:Burj_Khalifa dbo:floorCount ?burj .
        ?uri rdf:type dbo:Building .
        ?uri dbo:floorCount ?proj
        FILTER ( ?proj < ?burj )
    } ORDER BY DESC(?proj) LIMIT 1

This query corresponds to the NLQ: Which building after the Burj Khalifa has
the most floors? As shown in the query, there are two chains. The first chain
(a single triple pattern) finds the number of floors of Burj Khalifa. The
second chain (two triple patterns) finds all buildings and their number of
floors.

$?burj$$res:Burj\\_Khalifa$$?uri$$dbo:Building$$?proj$$dbo:floorCount$$rdf:type$$dbo:floorCount$
Figure 5. A Chain-Set example from QALD-9.

Table 7 shows the distribution of query shapes across all benchmarks. Again,
due to the lack of space, we combine all 9 benchmarks of _QALD_ into one
column. The table shows that queries with the _Single-Edge_ shape represent at
least 45.13% of the queries in almost all benchmarks. This shape corresponds
to simple-factoid questions. The only exception is the _GraphQuestions_
benchmark, which does not include any _Single-Edge_ queries.

It is interesting to see that a high percentage of the queries have the
_Chain_ shape. With the exception of _GraphQuestions_ , this shape includes at
least 72% of queries in all benchmarks. This observation along with the
information from Figure 3 indicate that a high percentage of the queries have
a _Chain_ shape with a relatively short length of the chain. In contrast, the
_Chain-Set_ shape distribution has almost the same percentage as that of the
_Chain_ shape, which indicates that queries that have the _Chain-Set_ shape,
but not the _Chain_ shape are rare in all benchmarks. The same observation
applies to the _Tree_ and _Forest_ shapes, indicating rare occurrences of
_Forest_ shapes that are not trees. The _Star_ shape distribution ranges
between 8.61% and 30.7% across all benchmark. The _Cycle_ shape is poorly
represented in all benchmarks as every benchmark has from one to two queries
or no queries for this shape.

For the different versions of QALD, Tables 8, 9, and 10 show the distribution
of the discussed shapes for $CQ$, $CQ_{F}$, and $CQ_{OF}$ queries
respectively.

Table 8. Cumulative shape analysis of $CQ$ across all versions of QALD benchmarks. Shape | QALD-1 | QALD-2 | QALD-3 | QALD-4 | QALD-5 | QALD-6 | QALD-7 | QALD-8 | QALD-9
---|---|---|---|---|---|---|---|---|---
_Single-Edge_ | 13.64% | 24.56% | 42.91% | 53.92% | 53.15% | 55.71% | 63.03% | 66.78% | 60.53%
_Chain_ | 50% | 54.39% | 78.89% | 92.16% | 90.94% | 91.64% | 89.89% | 91.1% | 91.81%
_Chain-Set_ | 50% | 54.39% | 78.89% | 92.16% | 90.94% | 91.64% | 90.16% | 91.1% | 91.81%
_Star_ | 47.73% | 42.11% | 19.03% | 7.35% | 8.27% | 7.8% | 9.31% | 8.56% | 7.89%
_Tree_ | 98.86% | 100% | 99.65% | 99.51% | 99.21% | 99.72% | 99.73% | 100% | 99.71%
_Forest_ | 98.86% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100%
_Cycle_ | 0% | 0% | 0.35% | 0.49% | 0.39% | 0.28% | 0% | 0% | 0.29%
_Flower_ | 100% | 100% | 100% | 100% | 99.61% | 100% | 99.73% | 100% | 100%
Table 9. Cumulative shape analysis of $CQ_{F}$ across all versions of QALD benchmarks. Shape | QALD-1 | QALD-2 | QALD-3 | QALD-4 | QALD-5 | QALD-6 | QALD-7 | QALD-8 | QALD-9
---|---|---|---|---|---|---|---|---|---
_Single-Edge_ | 11.65% | 21.64% | 38.04% | 49.11% | 49.63% | 53.17% | 59.7% | 63.93% | 57.02%
_Chain_ | 49.51% | 51.49% | 75.77% | 88.84% | 87.5% | 89.15% | 86.65% | 89.18% | 89.53%
_Chain-Set_ | 50.49% | 51.49% | 76.99% | 89.73% | 88.24% | 89.68% | 87.91% | 89.51% | 89.81%
_Star_ | 47.57% | 40.3% | 19.33% | 8.48% | 9.93% | 8.99% | 10.33% | 9.18% | 8.82%
_Tree_ | 98.06% | 95.52% | 96.63% | 97.32% | 97.43% | 98.41% | 97.48% | 98.69% | 98.35%
_Forest_ | 99.03% | 97.76% | 99.08% | 100% | 100% | 100% | 100% | 100% | 100%
_Cycle_ | 0% | 0% | 0.31% | 0.45% | 0.37% | 0.26% | 0% | 0% | 0.28%
_Flower_ | 99.03% | 96.27% | 97.55% | 97.77% | 97.79% | 98.68% | 97.48% | 98.69% | 98.62%
Table 10. Cumulative shape analysis of $CQ_{OF}$ across all versions of QALD benchmarks. Shape | QALD-1 | QALD-2 | QALD-3 | QALD-4 | QALD-5 | QALD-6 | QALD-7 | QALD-8 | QALD-9
---|---|---|---|---|---|---|---|---|---
_Single-Edge_ | 7.74% | 11.55% | 38.3% | 49.33% | 49.82% | 53.3% | 57.04% | 64.05% | 57.14%
_Chain_ | 42.58% | 51% | 75.68% | 88.89% | 87.55% | 89.18% | 86.87% | 89.22% | 89.56%
_Chain-Set_ | 43.87% | 51% | 76.9% | 89.78% | 88.28% | 89.71% | 88.07% | 89.54% | 89.84%
_Star_ | 52.26% | 42.23% | 19.45% | 8.44% | 9.89% | 8.97% | 10.26% | 9.15% | 8.79%
_Tree_ | 96.13% | 96.02% | 96.66% | 97.33% | 97.44% | 98.42% | 97.61% | 98.69% | 98.35%
_Forest_ | 99.35% | 98.8% | 99.09% | 100% | 100% | 100% | 100% | 100% | 100%
_Cycle_ | 0% | 0% | 0.3% | 0.44% | 0.37% | 0.26% | 0% | 0% | 0.27%
_Flower_ | 97.42% | 96.41% | 97.57% | 97.78% | 97.8% | 98.68% | 97.61% | 98.69% | 98.63%

###  5.3. Insights on Benchmark Selection (1)

It is out of the scope of our study to determine which benchmark is
quantitatively and qualitatively best in evaluating QA systems. However, given
the previous fine-grained properties of queries in the discussed benchmarks,
and until further research on benchmarking QA over KGs is done, we can give
insights on what the user can expect when using the discussed benchmarks in QA
evaluation. In this section, and without loss of generality, we discuss
evaluating QA on the two most common KGs in the literature (DBpedia and
Freebase), excluding simple-factoid questions.
QA over DBpedia: _QALD-1_ to _QALD-9_ , and _LC-QuAD_ target DBpedia with a
mix of simple and complex queries. If the user intends to evaluate their QA
system over DBpedia, we recommend using all benchmarks from _QALD_ in addition
to _LC-QuAD_ , which is not a common practice in the literature. Based on the
previously discussed properties, we notice that the _QALD_ benchmarks have
better coverage of all possible different cases of using query keywords (Table
3) and combination of operators (Table 5), while the _LC-QuAD_ queries
exclusively use only the Distinct along with conjunctions of triple patterns.
Also, the _QALD_ queries include a non-negligible number of queries that have
more than three triple patterns (differs from one _QALD_ benchmark to
another), whereas 100% of the queries in _LC-QuAD_ have at most three triple
patterns (Figure 3). This is an indication that the _QALD_ benchmarks include
more complex queries, which may prove more challenging for QA systems. This is
demonstrated by the more complex query shapes in _QALD_ that do not exist in
_LC-QuAD_ (i.e., Forest, Cycle, and Flower). However, since the _QALD_
benchmarks are manually created, they include 959 questions in total, whereas
_LC-QuAD_ includes almost 5000 questions. This indicates better linguistic and
vocabulary coverage in _LC-QuAd_. Indeed, further investigations reveal that
_LC-QuAD_ ’s queries span 6 times more resources (entities) and 2 times more
predicates than _QALD_.
QA over Freebase: The two benchmarks targeting Freebase that include queries
are _WebQuestions_ and _GraphQuestions_. We recommend using _GraphQuestions_
if the user intends to evaluate their QA system over Freebase. While the
differences are minor in terms of the operators used in the benchmarks
(Table5), other syntactical and structural properties showcase the
differences. _GraphQuestions_ include aggregate queries, while _WebQuestions_
do not (Table 3). Also, _WebQuestions_ queries tend to be shorter (over 60% of
queries are single-triple-pattern and almost 100% have no more than three
triple patterns), whereas _GraphQuestions_ have longer and more complex
queries (more than 35% of queries have more than three triple patterns
reaching 10 triple patterns) as shown in Figure 3. This is further showcased
by the percentage of simpler query shapes for both benchmarks, where most of
the queries in _WebQuestions_ are chain queries, while _GraphQuestions_ have
higher percentage of more complex shapes. _WebQuestions_ can be considered a
better alternative to evaluate simple questions over Freebase because it
includes single-triple-pattern queries, while _GraphQuestions_ do not include
such queries.

We will discuss our insights on benchmark selection with respect to the
benchmarks that do not include queries in Section 6.

##  6\. Analysis of Natural Language Questions

SPARQL queries are used to represent the graph patterns, whose matches are the
desired answers in the KG. Therefore, analyzing SPARQL queries gives deeper
insights into the structure of the questions in the benchmarks. However, not
all benchmarks include SPARQL queries that correspond to the natural language
questions (as previously shown in Table 1). In CBench, we also analyze the
natural language questions via the NLQ Analyzer to provide linguistic-based
insights on the questions in the benchmarks. The analysis of the natural
language questions in CBench focuses on the type of the natural language
questions (Section 6.1). This analysis gives the user insights on benchmark
selections (Section 6.2). In addition, to support the user with linguistic-
based insights on the performance of the evaluated QA, CBench provides the
user with the most linguistically-similar questions to any question of
interest. In order to support this feature, CBench converts all natural
language questions in the benchmarks to their vector space (Section 6.3). This
feature is demonstrated in Section 7.

###  6.1. Type of Questions

In CBench, the questions are categorized into:

  * $\bullet$

Wh-questions: the type of questions that starts with a _wh_ -pronoun (i.e.,
_What_ , _When_ , _Where_ , _Who_ , _Whom_ , _Which_ , and _Whose_). For
example, _Where was the first ford motor company located?_ (_WebQuestions_) is
a _wh_ question. This excludes questions starting with the keyword _Why_ ,
which targets non-factoid questions that are out of scope for QA over KGs.

  * $\bullet$

How-questions: Questions starting with the keyword _how_ followed by _many_ or
an adjective, such as _How many different currencies are used in the places
governed by the president of France?_ (_LC-QuAD_) or _How tall is Michael
Jordan?_ (_QALD-9_). Other non-factoid _How_ questions are not included.

  * $\bullet$

Yes/No questions: Questions that can be answered with _yes_ or _no_ : _Is
Michelle Obama the wife of Barack Obama?_ (_QALD-9_).

  * $\bullet$

Requests: Direct requests, such as _Can you name all the states of the US?_
(_GraphQuestions_). Such a question can be considered as a rephrasing of a
_What_ question. However, we put these questions in a different category
because they are usually handled differently in QA systems.

  * $\bullet$

Topicalized questions: Questions in which an entity or prepositional phrase is
topicalized for the purpose of emphasis: _Adobe pdf supports how many
computing platforms?_ (_GraphQuestions_).

Table 11 shows the percentage of questions of each question type across all
benchmarks. We break the _wh_ -questions into their sub-types. For _wh_
-questions, we also include many questions that start with a preposition
followed by a _wh_ -word, such as _in which_ or _to where_, because the single
preposition can be easily moved to the end without changing the rest of the
sentence. From these results, we observe that there is no dominant question
type across the benchmarks and the distribution of query types vary widely.

Table 11. Question frequency percentages (%) by type for all benchmarks.
|  QALD-1 |  QALD-2 |  QALD-3 |  QALD-4 |  QALD-5 |  QALD-6 |  QALD-7 |  QALD-8 |  QALD-9 |  QALD |  LC-QuAD |  Web |  Graph |  Simple |  SimpleDB |  Temp |  Complex |  ComQA
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
What | 6.53 | 6.10 | 7.30 | 10.59 | 11.08 | 13.92 | 11.70 | 18.10 | 13.48 | 10.80 | 53.44 | 55.32 | 33.08 | 60.73 | 57.19 | 29.35 | 32.00 | 47.13
When | 9.55 | 6.40 | 6.80 | 3.12 | 3.59 | 4.18 | 5.09 | 4.76 | 3.43 | 6.00 | 0.00 | 4.12 | 0.07 | 0.01 | 0.00 | 22.03 | 8.00 | 10.66
Where | 0.50 | 0.58 | 0.50 | 0.93 | 1.80 | 2.32 | 2.26 | 2.54 | 2.94 | 1.88 | 9.96 | 18.57 | 1.10 | 7.37 | 10.48 | 4.48 | 0.67 | 4.19
Which | 31.66 | 26.74 | 25.44 | 35.83 | 27.54 | 25.99 | 26.60 | 20.95 | 25.74 | 27.25 | 13.30 | 1.81 | 18.28 | 13.20 | 12.51 | 9.44 | 29.33 | 6.96
Who | 14.07 | 14.24 | 14.61 | 12.15 | 15.87 | 16.71 | 19.62 | 19.05 | 16.91 | 15.68 | 11.97 | 19.82 | 8.52 | 11.52 | 12.09 | 33.52 | 30.00 | 21.27
Whom | 0.00 | 0.29 | 0.76 | 0.00 | 0.00 | 0.23 | 0.19 | 0.32 | 0.00 | 0.34 | 0.12 | 0.00 | 0.17 | 0.01 | 0.03 | 0.00 | 0.00 | 0.09
Whose | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.22 | 0.00 | 0.07 | 0.06 | 0.05 | 0.00 | 0.00 | 0.04
How | 10.55 | 12.79 | 13.35 | 7.79 | 10.48 | 10.44 | 11.51 | 13.02 | 10.78 | 12.60 | 1.26 | 0.36 | 9.27 | 0.69 | 0.41 | 1.02 | 0.00 | 0.25
Yes/No | 9.55 | 9.88 | 9.07 | 7.48 | 7.78 | 6.96 | 10.38 | 10.48 | 9.07 | 7.63 | 2.09 | 0.00 | 0.14 | 1.20 | 1.48 | 0.00 | 0.00 | 0.01
Requests | 17.59 | 22.67 | 21.91 | 21.18 | 21.26 | 18.79 | 11.13 | 10.79 | 17.65 | 16.88 | 5.63 | 0.00 | 9.92 | 3.31 | 3.99 | 0.00 | 0.00 | 0.98
Topical | 0.00 | 0.29 | 0.25 | 0.93 | 0.60 | 0.46 | 1.51 | 0.00 | 0.00 | 0.94 | 2.01 | 0.00 | 19.38 | 1.90 | 1.77 | 0.16 | 0.00 | 8.42

###  6.2. Insights on Benchmark Selection (2)

In Section 5.3, we discussed our insights on benchmark selection based on our
analysis of benchmarks that include queries. In this section, we enrich this
discussion on these benchmarks and add new insights on the benchmarks that do
not include queries. Again, and without loss of generality, we discuss
evaluating QA on the two most common KGs in the literature (DBpedia and
Freebase), excluding simple-factoid questions.
QA over DBpedia: We previously mentioned that both _QALD_ and _LC-QuAD_ need
to be used to evaluate a QA system over DBpedia. This recommendation is
solidified by our observations in this section. Both benchmarks vary in terms
of the types of questions, where each benchmark covers the types that are not
covered by the other. For example, _LC-QuAD_ does not include _When_
-questions but _QALD_ does, and _QALD_ does not include _Whose_ -questions but
_LC-QuAD_ does.
QA over Freebase: Among the benchmarks that target Freebase, _GraphQuestions_
is the most comprehensive being the only benchmark that covers all question
types. In contrast, the other benchmarks (_WebQuestions_ , _TempQuestions_ ,
_ComQA_ , and _ComplexQuestions_) miss between 4 and 6 types. Another
advantage of using _GraphQuestions_ is that it includes different utterances
of the same question, which further challenges a QA system. For example, “What
celebrities are imitated by Will Ferrell?”, “Who did Will Ferrell
impersonate?”, etc. It is interesting to notice that _GraphQuestions_ include
yes/no questions. However, the queries and answers do not reflect this type of
questions by not using the SPARQL’s _ASK_ keyword and having non-binary
answers. For example, the question ”Are there any digital cameras with the
maximum aperture reaching 5.2?” is answered by a list of these cameras instead
of yes/no answer. Prior to CBench, the user needed to adjust for this scenario
by changing the answer in the benchmark to either Yes or No. CBench handles
this scenario for the user. _TempQuestions_ focuses on questions that include
temporal aspects. So, it can be used as an auxiliary benchmark to another main
benchmark (e.g., _GraphQuestions_). The number of questions in
_ComplexQuestions_ (150) and the number of types not included (6) are the main
disadvantages of using it.

###  6.3. Linguistically-Similar Questions

Each token in the question is assigned a tag based on both the token itself
and its context in the question. This process is known as Part-of-Speech (PoS)
tagging. There have been numerous works in the literature on PoS tagging
(choi-etal-2015-depends, ). State-of-the-art approaches use a bidirectional
recurrent neural network (BiLSTMs) and a subsequent conditional random field
(CRF) decoding layer in combination with word embeddings that are trained over
a large corpus of text (akbik2018, ; ma2016, ). In CBench, we utilize a pre-
trained model (honnibal-johnson:2015, ) to annotate each token in a question
with its predicted PoS tag. However, CBench can be used with any PoS tagger.
We define the _tag dictionary_ $td=\\{t_{1},t_{2},\ldots,t_{n}\\}$ as the set
of all possible tags (of size $n$) that can be assigned to a question token.
We use two types of tags: (1) The Universal POS
(UPOS)777https://universaldependencies.org/u/pos/, and (2) the more
comprehensive Penn Treebank tag set (Weischedel_2013, ). The simple UPOS tags
are the common representations for the word types, such as _NOUN_ , _VERB_ ,
and _ADJ_ , and do not code for any morphological features (i.e. the structure
of words such as stems, root words, prefixes, and suffixes). On the other
hand, the Penn Treebank tags are more specific. For example, the _VERB_ tag is
further divided into _MD_ \- auxiliary, _VB_ \- base form, _VBD_ \- past
tense, _VBG_ \- present participle, _VBN_ \- past participle, _VBP_ \- non-3rd
person singular present, and _VBZ_ \- 3rd person singular present. We denote
the tag dictionary for UPOS as $td_{UPOS}$, where $|td_{UPOS}|=17$, and the
tag dictionary for Penn Treebank as $td_{Penn}$, where $|td_{Penn}|=36$. In
section 7, we qualitatively evaluate both choices of tag types in CBench.

1

input : List of tokens that represent the natural language question $tokens$,
tag dictionary $td$, PoS tagger $tagger$

output : Fixed-size vector representation $pos\\_freq[|td|]$

2

3Initialize $pos\\_tag=$ [$i$] = 0;

4 for  _$token$ in $tokens$_ do

5 $pos\\_freq\left[indexOf(tagger(token))\right]$++;

6

7 end for

8Return $pos\\_freq$;

9

10

Algorithm 1 Vectorization of natural language questions using PoS tagging

Due to the richness of natural language, it is not expected that a sequence of
PoS tags will reoccur frequently in the limited set of questions in the
benchmarks. Therefore, for this linguistic feature (PoS tags), we use a
function $f_{tag}$ that embeds the previously defined natural language
representation (list of tokens) into a fixed-size vector space whose number of
dimensions $n$ is equal to the size of the tag dictionary used. Algorithm 1
illustrates this function. The function uses the PoS tagger to tag each token
in the question, then updates the frequency of the tag in the final vector
representation. This representation not only captures the definition and the
context of each token but also indirectly captures the length of the question
by counting the frequencies of the occurrences of the tags.

We use the obtained vector representation to calculate how _syntactically
similar_ a pair of questions is. In CBench, we use the euclidean distance
measure to represent how dissimilar two questions are. Formally, the distance
between two vector representations of questions $p$ and $q$ is calculated as
$d(p,q)=\sqrt{\sum_{i=1}^{n}(p_{i}-q_{i})^{2}}$, where $n=|td|$. This
definition of distance is used to find the $k$ most syntactically close
questions to a question of interest (e.g., a question that is incorrectly
answered). The $k$ questions are shown to the user marked with whether they
are correctly or incorrectly answered. To quickly find the $k$ nearest
questions to a question of interest, we pre-compute a distance matrix between
each pair of questions in the benchmark. This one-time process takes less than
a minute.

It is worth noting that in addition to PoS tagging, we also implemented a
similar approach based on dependency parse trees of the questions in the
benchmark to map the questions to their corresponding vectorized
representations before calculating the pair-wise euclidean distances. We did
not observe major changes in the output (discussed in Section 7).

##  7\. Evaluation

In this section, we experimentally evaluate six QA systems over different
benchmarks using CBench. We show how the variations in the previously
discussed properties of the questions/queries affect the evaluation of QA
systems (Section 7.1). We then discuss our insights on evaluating the QA
systems using CBench (Section 7.2). We also use CBench to perform a fine-
grained analysis of the questions processed by the QA systems (Section 7.3).
Finally, we qualitatively evaluate the different settings for highlighting
linguistically-close questions to a question of interest (Section 7.4).

Evaluation Environment: All experiments were run on a machine having an
Intel(R) Core(TM) i9-9900K<EMAIL_ADDRESS>and 16 MB cache, 32 GB RAM, running on
Ubuntu 18.04.

QA Systems: We evaluate six QA systems (diefenbach2017wdaqua, ;
hu2017answering, ; singh2018reinvent, ; liang2021querying, ; dubey2016asknow,
; tanon2018platypus, ). Three systems are remotely located and accessed via
web services and three systems are run locally on the same computer as CBench.
All systems are evaluated over the most recent versions of DBpedia and
Wikidata.

  * $\bullet$

WDAqua (diefenbach2017wdaqua, ): WDAqua is accessed remotely888http://qanswer-
frontend.univ-st-etienne.fr/ and targets DBpedia and Wikidata. It ignores the
syntax of the questions and focuses only on the semantics of the words in the
question.

  * $\bullet$

gAnswer (zou2014natural, ; hu2017answering, ): gAnswer is accessed
remotely999http://ganswer.gstore-pku.com/ and targets DBpedia only. It is
based on finding a semantic graph to represent the question’s intention and
then maps the problem to a subgraph matching problem over the KG.

  * $\bullet$

Qanary (singh2018reinvent, ; singh2018frankenstein, ): Qanary is a modular QA
framework that breaks the QA pipeline into independent modules that can be
easily replaced to improve the overall quality of QA. Although there are many
open-source modules are shared101010https://github.com/WDAqua/Qanary, most of
them are not functioning. We selected three working components for this
experiment which are NED-tagme (for the entity recognition module),
Diambiguation-Property-OKBQA (for the relation mapping module) and Query
Builder (for the query generation module). This QA pipeline is evaluated
locally over DBpedia.

  * $\bullet$

QAsparql (liang2021querying, ): QAsparql uses machine learning to classify the
question into one of three types: list, count or Boolean questions. After
that, it detects entities and relations and builds the queries based on the
question type. The generated queries are then ranked by a module based on
Tree-LSTM (tai2015improved, ) to get the final query that is used to retrieve
the system answer. QAsparql is oped-sourced111111https://github.com/Sylvia-
Liang/QAsparql and is evaluated locally over DBpedia.

  * $\bullet$

AskNow (dubey2016asknow, ): AskNow first normalizes the NLQ into an
intermediary canonical syntactic form, named Normalized Query Structure, and
then renders it into SPARQL queries. This QA system is open-
sourced121212https://github.com/AskNowQA/AskNowNQS and is evaluated locally
over DBpedia.

  * $\bullet$

AskPlatypus (tanon2018platypus, ): AskPlatypus is a multilingual QA system
over Wikidata that is mainly based on query templates and slot filling. It is
evaluated remotely131313https://askplatyp.us using questions targeting
Wikidata from _QALD-7_ and _QALD-8_.

Benchmarks: The benchmarks included in CBench are shown in Table 1. For the
fine-grained query analysis, we exclude the benchmarks that do not include
SPARQL queries.

Evaluation Metrics: We evaluate the QA system using three metrics: Micro,
macro, and global F-1 scores. First, we define precision, recall, and F-1 for
an individual question $q_{i}$. If we assume that the set of answers in the
gold standard for $q_{i}$ (correct answers) is $G_{q_{i}}$, and the set of
answers from the QA system for this question is $A_{q_{i}}$, then we define
the precision as $P_{q_{i}}=\frac{|G_{q_{i}}\bigcap A_{q_{i}}|}{|A_{q_{i}}|}$,
recall as $R_{q_{i}}=\frac{|G_{q_{i}}\bigcap A_{q_{i}}|}{|G_{q_{i}}|}$, and
F-1 as $F_{q_{i}}=\frac{2P_{q_{i}}R_{q_{i}}}{P_{q_{i}}+R_{q_{i}}}$. If there
are $n$ questions in the benchmark, then micro F-1 is calculated as follows:
$P_{\mu}=\frac{\sum_{i=1}^{n}|G_{i}\bigcap A_{i}|}{\sum_{i=1}^{n}|A_{i}|}$,
$R_{\mu}=\frac{\sum_{i=1}^{n}|G_{i}\bigcap A_{i}|}{\sum_{i=1}^{n}|G_{i}|}$,
and $F_{\mu}=\frac{2P_{\mu}R_{\mu}}{P_{\mu}+R_{\mu}}$. The macro F-1 is
defined by calculating the F-1 scores per question and averaging the values
for all questions. Formally, $F_{\Sigma}=\frac{\sum_{i=1}^{n}F_{q_{i}}}{n}$.
The global F-1 score evaluates the overall quality of the QA system in terms
of the question it can answer correctly. If we assume that the set of
questions in a benchmark is $Q$, the set of questions that are processed by
the QA system (non-empty answers) is $S$, and the set of questions that is
answered correctly is $C$, then we define the global precision as
$P_{G}=\frac{|C|}{|S|}$, global recall as $R_{G}=\frac{|C|}{|Q|}$, and global
F-1 as $F_{G}=\frac{2P_{G}R_{G}}{P_{G}+R_{G}}$. The latter definition of the
global scores is strict because a question is considered to be answered
correctly only if $F_{q_{i}}=1$. Therefore, in the literature, the notion of
what is considered a correctly answered question is relaxed, which is referred
to as _partially correct answered questions_. A question is considered
partially correctly answered if $0<F_{q_{i}}<1$. In this paper, we also study
the effect of changing the $F_{q_{i}}$ value used to consider a question to be
partially correctly answered on the reported $F_{G}$.

###  7.1. Evaluation of QA Systems

Table 12. Evaluation of QA Systems over benchmarks targeting DBpedia/Wikidata.
Benchmarks annotated with $\star$ include questions that target Wikidata.
|  |  | Qanary(singh2018reinvent, ; singh2018frankenstein, ) |  |  |
---|---|---|---|---|---|---
| WDAqua(diefenbach2017wdaqua, ) | gAnswer(zou2014natural, ; hu2017answering, ) | (TM+DP+QB) | QAsparql(liang2021querying, ) | AskNow(dubey2016asknow, ) | AskPlatypus(tanon2018platypus, )
Basis | FG | F$\mu$ | F$\Sigma$ | FG | F$\mu$ | F$\Sigma$ | FG | F$\mu$ | F$\Sigma$ | FG | F$\mu$ | F$\Sigma$ | FG | F$\mu$ | F$\Sigma$ | FG | F$\mu$ | F$\Sigma$
QALD-1 | 0.31 | 0.27 | 0.14 | 0.44 | 0.18 | 0.24 | 0.00 | 0.00 | 0.00 | 0.02 |  $\approx$0.00 | 0.01 | 0.12 |  $\approx$0.00 | 0.07 | - | - | -
QALD-2 | 0.32 | 0.17 | 0.16 | 0.41 | 0.08 | 0.21 | 0.00 | 0.00 | 0.00 | 0.03 |  $\approx$0.00 | 0.01 | 0.14 |  $\approx$0.00 | 0.10 | - | - | -
QALD-3 | 0.21 | 0.23 | 0.11 | 0.28 | 0.11 | 0.16 | 0.05 |  $\approx$0.00 | 0.02 | 0.12 | 0.01 | 0.06 | 0.19 |  $\approx$0.00 | 0.13 | - | - | -
QALD-4 | 0.21 | 0.17 | 0.12 | 0.30 | 0.13 | 0.16 | 0.03 |  $\approx$0.00 | 0.01 | 0.16 | 0.02 | 0.08 | 0.13 | 0.05 | 0.08 | - | - | -
QALD-5 | 0.31 | 0.19 | 0.18 | 0.36 | 0.10 | 0.20 | 0.04 |  $\approx$0.00 | 0.02 | 0.23 | 0.01 | 0.12 | 0.29 | 0.11 | 0.09 | - | - | -
QALD-6 | 0.36 | 0.15 | 0.24 | 0.39 | 0.09 | 0.25 | 0.05 |  $\approx$0.00 | 0.02 | 0.29 | 0.01 | 0.17 | 0.30 | 0.09 | 0.09 | - | - | -
QALD-7$\star$ | 0.39 | 0.19 | 0.29 | - | - | - | 0.07 | 0.02 | 0.06 | 0.30 | 0.14 | 0.17 | 0.37 | 0.14 | 0.15 | 0.15 |  $\approx$0.00 | 0.08
QALD-8$\star$ | 0.43 | 0.17 | 0.33 | - | - | - | 0.09 | 0.01 | 0.04 | 0.46 | 0.12 | 0.30 | 0.33 | 0.10 | 0.13 | 0.11 |  $\approx$0.00 | 0.06
QALD-9 | 0.43 | 0.20 | 0.32 | 0.44 | 0.10 | 0.30 | 0.08 |  $\approx$0.00 | 0.07 | 0.32 | 0.02 | 0.19 | 0.26 | 0.07 | 0.08 | - | - | -
Mean | 0.33 | 0.19 | 0.21 | 0.36 | 0.12 | 0.20 | 0.05 |  $\approx$0.00 | 0.03 | 0.21 | 0.04 | 0.12 | 0.24 | 0.06 | 0.10 | 0.13 |  $\approx$0.00 | 0.07
Std | 0.08 | 0.04 | 0.09 | 0.06 | 0.04 | 0.04 | 0.03 |  $\approx$0.00 | 0.03 | 0.15 | 0.05 | 0.09 | 0.09 | 0.05 | 0.03 | 0.03 |  $\approx$0.00 | 0.01
LC-QuAD | 0.20 | 0.03 | 0.15 | - | - | - | 0.02 | 0.01 | 0.01 | 0.46 | 0.14 | 0.34 | 0.16 | 0.01 | 0.11 | - | - | -
Mean | 0.32 | 0.18 | 0.20 | 0.36 | 0.12 | 0.20 | 0.04 | 0.01 | 0.03 | 0.24 | 0.05 | 0.15 | 0.23 | 0.06 | 0.10 | 0.13 |  $\approx$0.00 | 0.07
Std | 0.09 | 0.06 | 0.08 | 0.06 | 0.04 | 0.04 | 0.03 | 0.01 | 0.02 | 0.16 | 0.06 | 0.11 | 0.09 | 0.05 | 0.03 | 0.03 |  $\approx$0.00 | 0.01
Table 13. Evaluation of QA Systems over benchmarks targeting Freebase. ComplexQuestion is annotated with $\star$ because it includes 150 questions only. | WDAqua (diefenbach2017wdaqua, )
---|---
| (200 Questions)
Benchmark | FG | F$\mu$ | F$\Sigma$
GraphQuestions |  $\approx$0.00 |  $\approx$0.00 |  $\approx$0.00
WebQuestions | 0.22 | 0.12 | 0.12
TempQuestions | 0.13 | 0.05 | 0.06
ComQA | 0.16 | 0.02 | 0.07
ComplexQuestions$\star$ | 0.32 | 0.11 | 0.19
Mean | 0.17 | 0.06 | 0.09
Std | 0.12 | 0.05 | 0.07

Table 12 shows the six systems and their scores for the 3 metrics used in
CBench with $F_{q_{i}}>0$ for considering a question to be considered
partially correctly answered. The effect of changing the value of the
$F_{q_{i}}$ threshold will be discussed later in this section. gAnswer
experiments on _QALD-7_ and _QALD-8_ are excluded since they include questions
that target Wikidata, which is not supported by gAnswer. The table shows that
although the _QALD_ benchmarks come from the same source, the six systems have
considerably large variations in their scores. We calculate the average and
standard deviation for all scores. The standard deviation values range from
0.0 to 0.15, which are considered to be high for variations in F-1 scores.
When adding the _LC-QuAD_ scores to the results, we notice a slight increase
in the standard deviation of the global and micro scores and a slight decrease
in the macro score. We will address the differences between the F-1 scores in
Section 7.2

Even though Freebase is deprecated since 2015, several benchmarks still target
evaluating question answering over it. The following experiment is inspired by
the recent trend of migrating Freebase to Wikidata (pellissier2016freebase, ).
We evaluate WDAqua using the benchmarks that originally target Freebase, but
over Wikidata. The challenge to do this experiment is to guarantee that a
chosen question from any of the benchmarks can be answered using Wikidata. We
randomly sample questions from all the benchmarks and manually examine if the
question can be answered using Wikidata by identifying the entities in the
questions and the answers in the benchmark in Wikidata and finding a path that
connects them, which resembles how a query should be written to retrieve the
answers. We continue sampling until we find 200 Wikidata-answerable questions
for each benchmark (with the exception of _ComplexQuestions_ , which includes
only 150 questions). The results for this experiment are shown in Table 13.
WDAqua performs considerably better for questions from _WebQuestions_ and
_ComplexQuestions_ compared to the other benchmarks. This especially affects
the standard deviation of $F_{G}$ (0.12), which is significant for F-1 scores.

Figure 6. Evaluation of QA systems over _QALD-9_ using different thresholds
for the $F_{q_{i}}$.

We also investigate the effects of using different thresholds for the
$F_{q_{i}}$ for a question to be considered partially correct. For this
analysis, we evaluate WDAqua and gAnswer over _QALD-9_ (the benchmark both
systems perform best in). Figure 6 shows that the two systems are
significantly affected by the change of the $F_{q_{i}}$ value used. Although
the two systems start at nearly the same value for $F_{G}$, where a question
is considered partially correct if $F_{q_{i}}>0$, WDAqua is more sensitive to
stricter thresholds. Overall, there is a drop-in $F_{G}$ of 0.12 in WDAqua,
where as gAnswer’ score drops by 0.09. Both drops are considered significant
for F-1 scores, in general. The drop in WDAqua’s score is higher than that of
gAnswer because WDAqua produces less-quality answers for individual questions
compared to gAnswer. Therefore, a more restrictive quality threshold results
in such a drop. It is also noticeable that the first step (from $F_{q_{i}}\geq
0$ to $F_{q_{i}}\geq 0.1$) has the most significant impact as $F_{G}$ drops by
0.04 for both systems, which is considered significant given the slightly
stricter threshold.

###  7.2. Insights on Evaluation Metrics and Benchmarks

CBench includes the three evaluation metrics that have been used in the
literature: Micro F-1 ($F_{\mu}$), Macro F-1 ($F_{\Sigma})$, and Global F-1
($F_{G}$). It is interesting to see that previous works on QA over KG usually
use only a subset of the three metrics. It is also common to use only the
Global F-1. Our evaluation suggests that using a subset of the metrics is
misleading in evaluating QA systems. Table 12 shows that the choice of a
metric changes the order of the superior system. For example, if we consider
the Global F-1 scores and use any of the _QALD_ benchmarks for evaluation,
gAnswer is superior to WDAqua. However, if we consider the Micro F-1 scores,
WDAqua is superior. This means that the quality of the answer of individual
questions is better in WDAqua (better Micro F-1 scores). However, gAnswer is
able to answer more questions correctly or partially correctly. Interestingly,
the same observation applies for the same QA system in some cases. For
example, when evaluating WDAqua using the benchmarks _QALD-1_ to _QALD-5_ ,
the Micro F-1 score is larger than the Macro F-1 score. However, using
_QALD-6_ to _QALD-9_ , the Macro F-1 score is larger. This means that the
questions WDAqua struggles with in _QALD-6_ to _QALD-9_ have long answers.
This results in a worse hit on the micro scores than on the macro scores
(average of scores of individual questions). The order also changes based on
the benchmark used, suggesting that the choice of a set of benchmarks in
evaluation can be misleading in evaluating QA systems. For example, if we
consider the Global F-1 scores and use _QALD-7_ to evaluate QAsparql and
AskNow, we notice that AskNow is superior. However, if we use _QALD-8_ , a
benchmark from the same authors but one year younger, QAsparql is superior.
These variations pose questions on how the benchmarks are created. It is
interesting to see that these variations do not exist only for the
automatically-generated benchmarks, but also for benchmarks that are manually
created. This leads us to conclude that more research effort needs to be put
to better understand how to sufficiently represent a good balance between all
the combinations of the properties that we proposed in our work to have good
coverage of as many cases as possible that a QA system could face in practice.
Until such research is done, we recommend taking a comprehensive approach in
evaluating QA systems by considering all evaluation metrics over the
benchmarks that we recommended using in Sections 5.3 and 6.2. Such a
comprehensive approach is now facilitated by CBench.

###  7.3. Fine-Grained Evaluation

Figure 7. The properties of the _Chain_ -shaped queries of the queries from
_QALD-9_ answered by WDAqua. T represents the number of triple patterns.
Figure 8. The properties of the _Star_ -shaped queries of the questions from
_QALD-9_ answered by WDAqua. T represents the number of triple patterns

In light of the results in Section 7.1, we analyze the queries of the
questions processed by the QA systems. Due to the lack of space, the
experiments in this section focus on WDAqua, over _Chain_ -shaped and _Star_
-shaped queries of _QALD-9_ , using a threshold $F_{q_{i}}>0.8$ to consider a
question to be partially correct. We highlight the effectiveness of using
CBench’s Debugging Mode to obtain the fine-grained results for these specific
category of queries. Other settings (shapes, benchmarks, thresholds, and
gAnswer) show similar results. It is also worth noting that these specific
categories can be navigated to through the interactive report shown to the
user if the user decides to evaluate the QA system using all the questions in
a benchmark since we do a two-level grouping of all questions processed by the
QA system when presenting them to the user. The first level of grouping is by
shape, and the second level is by combinations of query properties.

Figures 7 and 8 show the fine-grained analysis for the the aforementioned
categories (_Chain_ -shaped and _Star_ -shaped queries, respectively). For
each shape, each combination of properties shows how many questions were
answered correctly or incorrectly by the QA system. Figure 7 shows that there
are 17 different combination of properties for all the _Chain_ queries in
_QALD-9_. Most of the _Chain_ queries are either conjunctive queries that use
the keywords _select_ and _distinct_ and have one triple pattern (56 questions
are correctly answered and 97 are incorrectly answered), or conjunctive
queries that use the same two keywords and have two triple patterns (28
questions are correctly answered and 49 are incorrectly answered). The user
can choose any of the categories in the figure to examine individual
questions, in which case, they can use the natural language analysis features
of CBench (will be qualitatively evaluated in Section 7.4). It is worth noting
that the _Chain_ shape subsumes the _Single-Edge_ shape, which is actually
represented in this figure through the top five combinations of properties
(i.e., T = 1).

Figure 8 shows the _Star_ queries of _QALD-9_. These are considered to be more
complex queries, with at least three triple patterns. The figure shows that
WDAqua answered only 8 out of 38 questions whose query shape is _Star_. It is
also worth noting that the union of the results in the two figures is the
expected results for the _Tree_ -shaped queries. The user can also navigate
through the figures representing different shapes through the subsumption
hierarchy of shapes.

Debugging the QA System Using Fine-Grained Analysis: The main three components
in any QA system are (1) Entity Recognizer, (2) Relation Mapper and the (3)
Query Builder. For example, given the question What is the capital of Canada?
discussed in Table 15, a good QA system’s Entity Recognizer would recognize
the token _Canada_ as the corresponding resource (i.e. _dbr:Canada_ in
DBpedia). The Relation Mapper maps the sequence of tokens _is the capital of_
to the corresponding predicate (i.e. _dbo:capital_ in DBpedia). Lastly, the
Query Builder produces the following formal query that returns the answers
from the targeted KG: _SELECT ?x{dbr:Canada dbo:capital ?x.}_. These modules
can be either dependent/independent on/of each other. The Query Builder mainly
depends on a predefined/learned set of query templates as well as the entities
and relations detected by the Entity Recognizer and the Relation Mapper.
CBench can help the QA developers to learn more about the query templates that
the QA does not handle using its fine-grained analysis. For example, Figure 7
shows that WDAqua fails to answer all the queries with properties like _(CQ,
select-distinct-limit-offset-and, T=2)_ and _( $CQ_{F}$, select-distinct-
filter-and, T=2)_. This indicates that the query builder of WDAqua may not be
generating the query templates: _SELECT DISTINCT ?x{ … } OFFSET m LIMIT n_ and
_SELECT DISTINCT ?x{ … FILTER(…)}_. It is possible, however, that there is an
issue with the Entity Recognizer and Relation Mapper for all the questions
that were incorrectly answered. A more obvious example is when the fine-
grained analysis shows that some questions whose queries have specific
properties were answered correctly and some were not. The user can choose any
of these questions for further investigations (discussed next).

###  7.4. Qualitative Evaluation of Linguistically-Similar Questions

Table 14. Closest 5 question to the incorrectly answered question ”What
currency should I take to Dubai?”, based on UPOS and Penn Treebank tag. The
marks ✓/✗denote correctly/incorrectly answered questions. Linguistically
Closest Questions - UPOS
---
Answer | Question
✓ | What currency should I take to Mauritius?
✓ | What currency should I take to Jamaica?
✓ | What currency should I take to Mexico?
✓ | What currency should you take to Morocco?
✗ | What prompted Shakespeare to write poetry?
Linguistically Closest Questions - Penn Treebank
Answer | Question
✓ | What currency should I take to Jamaica?
✓ | What currency should I take to Mauritius?
✓ | What currency should you take to Morocco?
✓ | What currency should I take to Mexico?
✓ | What kind of money should I take to Jamaica?
Table 15. Closest 5 question to the incorrectly answered question ”What is the
capital of Canada?”, based on UPOS and Penn Treebank tag. The marks ✓/✗denote
correctly/incorrectly answered questions. Linguistically Closest Questions -
UPOS
---
Answer | Question
✗ | What is the nickname of Edinburgh?
✓ | What is the capital of Cameroon?
✗ | What is the meaning of Heydar ?
✗ | What is the currency of Chile?
✗ | What is the capital of Venezuela?
Linguistically Closest Questions - Penn Treebank
Answer | Question
✗ | What is the title of Kakae?
✓ | What is the currency of Rhodesia?
✓ | What is the capital of Cameroon?
✗ | What is the capital of Venezuela?
✗ | What is the origin of Xynisteri?

In this section, we assume that the user navigates through the fine-grained
analysis figures and investigates a question of interest that is not correctly
answered by the QA system. For benchmarks that do not include SPARQL queries,
the user can directly choose any question from the benchmark. CBench finds the
$k$ linguistically closest questions to this chosen question based on the
approach discussed in Section 6. This feature is especially beneficial in
debugging the Entity Recognizer and the Relation Mapper components in the QA
system. In this experiment, we use gAnswer as the QA system. Other
configurations do not reflect major changes to the top-5 questions. Changing
the value of the thresholds for $F_{q_{i}}$ to consider a question to be
partially correctly answered results in possible changes in the label of the
correctness of the question.

Table 14 shows the linguistically-closest 5 questions to the incorrectly
answered question “What currency should I take to Dubai?”. There is
considerable overlap between the two tag dictionaries (4 out of 5 questions
match). In both cases, we note that almost all of the similar questions are
correctly answered, unlike the question of interest, possibly highlighting
that the question is correctly answered if the entity is a country (one-hop
connection to currency in the KG) rather than a city (two-hops connection to
currency).

Table 15 shows a different interesting scenario for the incorrectly answered
question “What is the capital of Canada?”, where for the top part (using
UPOS), most of the similar questions are also incorrectly answered. We note
that all the questions have the template “What is the ¡place_holder¿ of
¡place_holder¿?”. This indicates that the QA system struggles with this type
of questions with only few exceptions. However, the question “What is the
capital of Cameroon?” is answered correctly. This indicates that the QA system
either could not identify _Canada_ as an entity, or _capital of_ as the
predicate _dbo:capital_. Indeed, investigating gAnswer reveals that the
Relation Mapper (referred to in gAnswer as the Relation Recognition step)
mistakenly mapped the relationship in the question to the _type_ predicate due
to the sparsity of the relation mention dictionary that is built based on a
large text corpus, which is considered an overfitting problem in learning the
textual patterns.

##  8\. Discussion

In light of our work, we would like to highlight the following observations
that we believe should be taken into consideration for future research work in
the area of QA over KGs:

  * $\bullet$

Variations Among Benchmarks: Although all benchmarks target the same problem,
there are high-degree variations with respect to several linguistic features
in the natural language questions, and syntactical and structural features in
the queries (Sections 5 and 6). These variations pose questions on how the
benchmarks are created. It is interesting to see that these variations do not
exist only for the automatically-generated benchmarks, but also for benchmarks
that are manually created. More research effort needs to be put to better
understand how to sufficiently represent a good balance between all the
combinations of the features that we proposed in our work to have good
coverage of as many cases as possible that a QA system could face in practice.

  * $\bullet$

Correctness of Benchmarks: We noticed that the automatically-generated
benchmarks may include some erroneous entries, which may affect the reported
accuracy of the evaluated QA system. For example, it is interesting to notice
that _GraphQuestions_ include yes/no questions. However, the queries and
answers do not reflect this type of questions by not using the SPARQL’s _ASK_
keyword and having non-binary answers.

  * $\bullet$

Staleness: KGs are continuously evolving, while benchmarks are static by
nature. Most of the KGs targeted by the benchmarks in CBench now have more
recent versions than those that were used to generate/create the benchmark. In
our work, we proposed a simple solution by updating the benchmark entries
based on the queried KG. However, this approach will not work if the KG’s
evolution touches its ontology or vocabulary. In fact, we noticed such
behaviour in our benchmarks. Some queries no longer produce any results
because the ontology of DBpedia changed. It is interesting to also notice that
some other queries in earlier QALD benchmarks are no longer syntactically
correct due to changes in the standard query language (SPARQL). This
observation is especially concerning for benchmarks that do not include SPARQL
queries, where it is not even possible to automatically update the answers in
the benchmark and will require daunting human efforts to make the benchmark
useful.

  * $\bullet$

KG-Related Issues: In our work, we encountered some interesting cases where
the query in the benchmark is unnatural in the sense that it would have been
written in a different way but for the representation of the answer in the KG.
For example, the following query (from QALD-9) is associated with the question
_“What airlines are part of the SkyTeam alliance?”_

        SELECT DISTINCT ?uri WHERE {
        ?uri a dbo:Airline {
            ?uri dbo:alliance dbr:SkyTeam
        } UNION {
            ?uri dbo:Alliance dbr:SkyTeam
        }
    }

Normally, the query should only include one graph pattern that consists of two
triple patterns. However, due to the co-existence of the two predicates in the
query above (dbo:alliance versus dbo:Alliance), the query in the benchmark had
to be written in that way. This issue in DBpedia was later corrected and the
predicate dbo:Alliance was removed.

##  9\. Related Work

Retrieving data from databases using non-structured languages has been a topic
of interest since the emergence of databases (baseball, ). More recent works
focused more on information retrieval (IR)-based QA systems, which were
designed to handle free text. Then for the last two decades, many QA over KGs
systems emerged due to the increasing amount of new structured data on the web
that is in the form of KGs (Hoeffner2017, ). Moreover, hybrid QA systems that
retrieve data from both free text and KGs have also emerged (e.g. (hawk, ;
Beaumont2015, )).

Due to the popularity of QA over KGs, several evaluation datasets emerged
(Unger2011, ; cabrio2012qakis, ; Cabrio2013, ; Unger2014, ; Unger2015, ;
Unger2016, ; Usbeck2017, ; Usbeck2018, ; ngomo20189th, ; trivedi2017lc, ;
Su2016, ; Berant2013, ; bordes2015large, ; Azmy2018, ; cai-yates-2013-large, ;
jia2018tempquestions, ; Abujabal2017, ; abujabal-etal-2019-comqa, ) targeting
multiple popular KGs like DBpedia (Auer2007, ), Freebase (Bollacker2008, ),
Wikidata (vrandevcic2014wikidata, ), and others. This variety of evaluation
datasets (or benchmarks) poses a choice challenge when it comes to evaluating
a new QA system.

Closest to our work is _GERBIL_ (gerbil, ), which allows uniform evaluation on
diverse benchmarks. It focuses on question answering and semantic entity
annotation. GERBIL is an extension of an earlier entity annotation system
(usbeck2015evaluating, ). With the new functionalities, GERBIL can be used to
report micro and macro F-1 scores, along with execution time for multiple
DBpedia-based benchmarks.

In CBench, we support using any benchmark over any KG. CBench also allows the
user to add their own benchmarks over their own KG. In addition to providing
the user with a report that includes single-numbered scores that indicates the
overall performance with respect to the number of questions processed and
correctly answered by the system, CBench also provides a fine-grained report
that groups processed questions per several syntactical and structural
features of their corresponding SPARQL queries, and per the user request,
highlights common characteristics or inconsistencies of the processed
questions chosen by the user through contrasting them with the linguistically
closest questions. This detailed information provided by CBench helps the user
identify strong and weak points of their QA system rather than just viewing
single-number scores.

##  10\. Conclusion

In this paper, we highlight the high-degree variations in the benchmarks used
to evaluate question answering over knowledge graphs. We experimentally show
that these variations affect the reported scores for multiple question
answering systems. To overcome the effects of such variations, we introduce
CBench, a fine-grained benchmark suite that comes with a prepacked set of
popular benchmarks that target multiple popular knowledge graphs. CBench is
easy to use and extensible. It does not only provide the user with the
traditionally known quality scores, but it also gives a fine-grained analysis
of the processed questions and their corresponding queries based on multiple
features. This analysis can be used by the user to quickly identify the strong
and weak points of their evaluated question answering system.

## References

  * [1] SPARQL 1.1 query language. http://www.w3.org/TR/sparql11-query/, 2013.
  * [2] RDF 1.1 concepts and abstract syntax. http://www.w3.org/TR/2014/REC-rdf11-concepts-20140225/, 2014.
  * [3] A. Abujabal, R. Saha Roy, M. Yahya, and G. Weikum.  ComQA: A community-sourced dataset for complex factoid question answering with paraphrase clusters.  In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HL), 2019.
  * [4] A. Abujabal, M. Yahya, M. Riedewald, and G. Weikum.  Automated template generation for question answering over knowledge graphs.  In Proceedings of the International Conference on World Wide Web (WWW), 2017.
  * [5] A. Akbik, D. Blythe, and R. Vollgraf.  Contextual string embeddings for sequence labeling.  In Proceedings of the International Conference on Computational Linguistics, 2018.
  * [6] S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives.  DBpedia: A nucleus for a web of open data.  In Proceedings of the International Semantic Web Conference (ISWC). 2007.
  * [7] M. Azmy, P. Shi, J. Lin, and I. Ilyas.  Farewell freebase: Migrating the simplequestions dataset to dbpedia.  In Proceedings of the International Conference on Computational Linguistics, pages 2093–2103, 2018.
  * [8] R. Beaumont, B. Grau, and A.-L. Ligozat.  SemGraphQA@ QALD5: LIMSI participation at QALD5@ CLEF.  In CLEF Working Notes Papers, 2015.
  * [9] J. Berant, A. Chou, R. Frostig, and P. Liang.  Semantic parsing on Freebase from question-answer pairs.  In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2013.
  * [10] K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor.  Freebase: a collaboratively created graph database for structuring human knowledge.  In Proceedings of the ACM SIGMOD International Conference on Management of Data, 2008.
  * [11] A. Bonifati, W. Martens, and T. Timm.  An analytical study of large SPARQL query logs.  Proceedings of the VLDB Endowment, 11(2), 2017.
  * [12] A. Bordes, N. Usunier, S. Chopra, and J. Weston.  Large-scale simple question answering with memory networks.  arXiv preprint arXiv:1506.02075, 2015.
  * [13] E. Cabrio, P. Cimiano, V. Lopez, and S. Walter.  Qald-3: Multilingual question answering over linked data.  In CLEF Working Notes Papers, 2013.
  * [14] E. Cabrio, J. Cojan, A. P. Aprosio, B. Magnini, A. Lavelli, and F. Gandon.  QAKiS: an open domain QA system based on relational patterns.  In Proceedings of the International Semantic Web Conference (ISWC), 2012.
  * [15] E. Cabrio, J. Cojan, B. Magnini, F. Gandon, and A. Lavelli.  Qakis@ qald-2.  2012\.
  * [16] Q. Cai and A. Yates.  Large-scale semantic parsing via schema matching and lexicon extension.  In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2013.
  * [17] A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E. R. Hruschka, and T. M. Mitchell.  Toward an architecture for never-ending language learning.  In Proceedings of the AAAI conference on artificial intelligence, 2010.
  * [18] J. D. Choi, J. Tetreault, and A. Stent.  It depends: Dependency parser comparison using a web-based evaluation tool.  In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) and the International Joint Conference on Natural Language Processing (IJCNLP), 2015.
  * [19] W. Cui, Y. Xiao, H. Wang, Y. Song, S.-w. Hwang, and W. Wang.  KBQA: Learning question answering over QA corpora and knowledge bases.  Proceedings of the VLDB Endowment (PVLDB), (5), 2017.
  * [20] D. Diefenbach, K. Singh, and P. Maret.  Wdaqua-core0: A question answering component for the research community.  In Semantic Web Evaluation Challenge, 2017.
  * [21] X. Dong and A. Halevy.  Indexing dataspaces.  In Proceedings of the ACM SIGMOD International Conference on Management of Data, 2007.
  * [22] M. Dubey, S. Dasgupta, A. Sharma, K. Höffner, and J. Lehmann.  Asknow: A framework for natural language query formalization in sparql.  In European Semantic Web Conference (ESWC), 2016.
  * [23] A. Fader, L. Zettlemoyer, and O. Etzioni.  Open question answering over curated and extracted knowledge bases.  In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2014.
  * [24] B. F. Green Jr, A. K. Wolf, C. Chomsky, and K. Laughery.  Baseball: an automatic question-answerer.  In The western joint IRE-AIEE-ACM computer conference, 1961.
  * [25] K. Höffner, S. Walter, E. Marx, R. Usbeck, J. Lehmann, and A.-C. Ngonga Ngomo.  Survey on challenges of question answering in the semantic web.  Semantic Web, 8(6), 2017.
  * [26] M. Honnibal and M. Johnson.  An improved non-monotonic transition system for dependency parsing.  In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015.
  * [27] S. Hu, L. Zou, J. X. Yu, H. Wang, and D. Zhao.  Answering natural language questions by subgraph matching over knowledge graphs.  IEEE Transactions on Knowledge and Data Engineering (TKDE), 30(5), 2017.
  * [28] Z. Jia, A. Abujabal, R. Saha Roy, J. Strötgen, and G. Weikum.  Tempquestions: A benchmark for temporal question answering.  In Proceedings of the International Conference on World Wide Web (WWW), 2018.
  * [29] E. Kaufmann, A. Bernstein, and R. Zumstein.  Querix: A natural language interface to query ontologies based on clarification dialogs.  In Proceedings of the International Semantic Web Conference (ISWC), 2006.
  * [30] S. Liang, K. Stockinger, T. M. de Farias, M. Anisimova, and M. Gil.  Querying knowledge graphs in natural language.  Journal of Big Data, 8(1), 2021.
  * [31] V. Lopez, M. Fernández, E. Motta, and N. Stieler.  PowerAqua: supporting users in querying and exploring the semantic web.  Semantic Web, (3), 2011.
  * [32] X. Ma and E. Hovy.  End-to-end sequence labeling via bi-directional lstm-cnns-crf.  In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2016.
  * [33] C. Matuszek, M. Witbrock, J. Cabral, and J. DeOliveira.  An introduction to the syntax and content of cyc.  UMBC Computer Science and Electrical Engineering Department Collection, 2006.
  * [34] T. Pellissier Tanon, D. Vrandečić, S. Schaffert, T. Steiner, and L. Pintscher.  From Freebase to Wikidata: The great migration.  In Proceedings of the International Conference on World Wide Web (WWW), 2016.
  * [35] K. Singh, A. Both, A. Sethupat, and S. Shekarpour.  Frankenstein: a platform enabling reuse of question answering components.  In European Semantic Web Conference (ESWC), 2018.
  * [36] K. Singh, A. S. Radhakrishna, A. Both, S. Shekarpour, I. Lytra, R. Usbeck, A. Vyas, A. Khikmatullaev, D. Punjani, C. Lange, et al.  Why reinvent the wheel: Let’s build question answering systems together.  In Proceedings of the World Wide Web Conference (WWW), 2018.
  * [37] Y. Su, H. Sun, B. Sadler, M. Srivatsa, I. Gur, Z. Yan, and X. Yan.  On generating characteristic-rich question sets for qa evaluation.  In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016.
  * [38] F. M. Suchanek, G. Kasneci, and G. Weikum.  Yago: a core of semantic knowledge.  In Proceedings of the International World Wide Web Conference (WWW), 2007.
  * [39] K. S. Tai, R. Socher, and C. D. Manning.  Improved semantic representations from tree-structured long short-term memory networks.  arXiv preprint arXiv:1503.00075, 2015.
  * [40] T. P. Tanon, M. Dias de Assuncao, E. Caron, and F. Suchanek.  Platypus – A Multilingual Question Answering Platform for Wikidata.  Technical report, 2018.
  * [41] P. Trivedi, G. Maheshwari, M. Dubey, and J. Lehmann.  Lc-quad: A corpus for complex question answering over knowledge graphs.  In International Semantic Web Conference (ISWC), 2017.
  * [42] C. Unger, L. Bühmann, J. Lehmann, A.-C. Ngonga Ngomo, D. Gerber, and P. Cimiano.  Template-based question answering over RDF data.  In Proceedings of the International World Wide Web Conference (WWW), 2012.
  * [43] C. Unger, P. Cimiano, V. Lopez, and E. Motta.  Question answering over linked data (QALD-1).  In Workshop on Question Answering Over Linked Data (QALD-1), 2011\.
  * [44] C. Unger, C. Forascu, V. Lopez, A.-C. N. Ngomo, E. Cabrio, P. Cimiano, and S. Walter.  Question answering over linked data (QALD-4).  In CLEF Working Notes Papers, 2014.
  * [45] C. Unger, C. Forascu, V. Lopez, A.-C. N. Ngomo, E. Cabrio, P. Cimiano, and S. Walter.  Question answering over linked data (QALD-5).  In CLEF Working Notes Papers, 2015.
  * [46] C. Unger, A.-C. N. Ngomo, and E. Cabrio.  6th open challenge on question answering over linked data (qald-6).  In Semantic Web Challenges, 2016.
  * [47] R. Usbeck, R. H. Gusmita, M. Saleem, and A.-C. N. Ngomo.  9th challenge on question answering over linked data (qald-9).  Joint Workshop on Natural Language Interfaces for Web of Data (NLIWoD) and Question Answering over Linked Data challenge, 2018.
  * [48] R. Usbeck, A.-C. N. Ngomo, L. Bühmann, and C. Unger.  Hawk–hybrid question answering using linked data.  In European Semantic Web Conference, pages 353–368. Springer, 2015\.
  * [49] R. Usbeck, A.-C. N. Ngomo, F. Conrads, M. Röder, and G. Napolitano.  8th challenge on question answering over linked data (qald-8).  Joint Proceedings of the International Workshop on Benchmarking Linked Data and Natural Language Interfaces for the Web of Data (NLIWoD), 2018\.
  * [50] R. Usbeck, A.-C. N. Ngomo, B. Haarmann, A. Krithara, M. Röder, and G. Napolitano.  7th open challenge on question answering over linked data (qald-7).  In Semantic Web Challenges, 2017.
  * [51] R. Usbeck, M. Röder, M. Hoffmann, F. Conrads, J. Huthmann, A.-C. Ngonga-Ngomo, C. Demmler, and C. Unger.  Benchmarking question answering systems.  Semantic Web, 10(2), 2019.
  * [52] R. Usbeck, M. Röder, and A.-C. N. Ngonga.  Evaluating entity annotators using gerbil.  In European Semantic Web Conference (ESWC), 2015.
  * [53] D. Vrandečić and M. Krötzsch.  Wikidata: a free collaborative knowledgebase.  Communications of the ACM, 57(10), 2014.
  * [54] R. Weischedel.  OntoNotes Release 5.0 LDC2013T19.  Linguistic Data Consortium, 2013.
  * [55] M. Yahya, K. Berberich, S. Elbassuoni, and G. Weikum.  Robust question answering over the web of linked data.  In Proceedings of the ACM International Conference on Information and Knowledge Management (CIKM), 2013.
  * [56] G. Zenz, X. Zhou, E. Minack, W. Siberski, and W. Nejdl.  From keywords to semantic queries-Incremental query construction on the semantic web.  Journal of Web Semantics, 7(3), 2009.
  * [57] W. Zheng, J. X. Yu, L. Zou, and H. Cheng.  Question answering over knowledge graphs: question understanding via template decomposition.  Proceedings of the VLDB Endowment (PVLDB), (11), 2018.
  * [58] Q. Zhou, C. Wang, M. Xiong, H. Wang, and Y. Yu.  Spark: Adapting keyword query to semantic search.  In Proceedings of the International Semantic Web Conference (ISWC), 2007.
  * [59] L. Zou, R. Huang, H. Wang, J. X. Yu, W. He, and D. Zhao.  Natural language question answering over RDF: a graph data driven approach.  In Proceedings of the ACM SIGMOD International Conference on Management of Data, 2014.