# GRAMMAR: Grounded and Modular Methodology for Assessment of Domain-Specific
Retrieval-Augmented Language Models

Xinzhe Li School of IT, Deakin University, Australia
{lixinzhe<EMAIL_ADDRESS>Ming Liu School of IT, Deakin
University, Australia
{lixinzhe<EMAIL_ADDRESS>Shang Gao School of IT, Deakin
University, Australia
{lixinzhe<EMAIL_ADDRESS>
###### Abstract

Retrieval-augmented Generation (RAG) systems have been actively studied and
deployed across various industries to query on domain-specific knowledge base.
However, evaluating these systems presents unique challenges due to the
scarcity of domain-specific queries and corresponding ground truths, as well
as a lack of systematic approaches to diagnosing the cause of failure
cases—whether they stem from knowledge deficits or issues related to system
robustness. To address these challenges, we introduce GRAMMAR (GRounded And
Modular Methodology for Assessment of RAG), an evaluation framework comprising
two key elements: 1) a data generation process that leverages relational
databases and LLMs to efficiently produce scalable query-answer pairs. This
method facilitates the separation of query logic from linguistic variations
for enhanced debugging capabilities; and 2) an evaluation framework that
differentiates knowledge gaps from robustness and enables the identification
of defective modules. Our empirical results underscore the limitations of
current reference-free evaluation approaches and the reliability of GRAMMAR to
accurately identify model vulnerabilities.

GRAMMAR: Grounded and Modular Methodology for Assessment of Domain-Specific
Retrieval-Augmented Language Models

Figure 1: Evaluation Framework of GRAMMAR Model Performance: This figure
illustrates the dual-phase evaluation approach for the GRAMMAR model. The left
section outlines the data generation process for creating sets of
hypothetically robust ($\mathcal{D_{\text{robust}}}$) and non-robust
($\mathcal{D_{\text{non-robust}}}$) data to test model hypotheses. The right
section details the methodology for identifying defective modules. This
grouping step accumulates queries with identical semantics and delineates
three group types for identifying detective modules. Gap Groups indicate a
complete lack of correct responses, reflecting deficiencies in the model’s
knowledge or capabilities. Robust Groups show the model’s comprehensive
understanding, with correct responses to all queries in the subset. Non-robust
Groups highlight the robustness issue, where the model correctly answers some
but not all text forms with identical query logic.

##  1 Background

##  2 Introduction

The emergent ability of Large Language Models (LLMs) has advanced a active
research and deployment of Retrieval-Augmented Generation (RAG) systems for
domain-specific question answering. This trend underscores the critical need
for reliable evaluation of these systems, a task that remains intricate and
largely unaddressed.

One of the foremost challenges is the scarcity of domain-specific data.
Typically, evaluators resort to using data designed for human assessment, as
referenced in studies such as Santurkar et al. (2023); Wang et al. (2022);
Zhong et al. (2022, 2023); Hendrycks et al. (2021); Choi et al. (2023), which
mainly focuses on evaluating commonsense and broad world knowledge. Yet,

The assessment of industrial RAG systems demands knowledge specific to
particular domains, such as details about company projects and employees.
While user queries can be gathered at the time of system deployment, acquiring
accurate ground truths poses a significant challenge. Recent studies have
explored the use of reference-free LLM evaluators in the absence of ground
truths (Chern et al., 2023; Min et al., 2023; Es et al., 2023). Nonetheless,
the reliability of such reference-free evaluation methods remains
questionable. Our analysis of two representative methods convincingly
demonstrates their false assessment across accurate and inaccurate
predictions: only RAGAS demonstrates reliability in identifying correct
predictions. In this paper,  an automatic data generation methodology is
proposed to ensure the accessibility of ground truths (Contribution 1). The
methodology leverages relational databases and LLMs to enable the extraction
of ground truths via SQL queries.

A second significant challenge relates to the robustness of models.
Specifically, we observe inconsistencies in the models’ responses to queries
that, while semantically equivalent, differ in their linguistic presentation,
as highlighted in Shen et al. (2023). Such inconsistencies pose a direct
challenge to the crucial requirement for consistent performance. Researchers
often hypothesize about the factors influencing the robustness or
susceptibility of retrieval and language models to certain attributes,
questioning what impedes these models from responding accurately to queries.
Yet, the current body of research, including works on adversarial robustness
(Alzantot et al., 2018; Li et al., 2020, 2019), lacks the mechanisms to
effectively test these hypotheses. To address this gap, we introduce a
evaluation workflow, as domonstrated in Figure 1: 1) utilizing the proposed
data generation process that creates hypothetically robust data
$\mathcal{D_{\text{robust}}}$ and non-robust data $\mathcal{D_{\text{non-
robust}}}$ according to a proposed hypothesis, 2) identifying defective
modules based on a grouping mechanism, contextual sufficiency analysis and a
reliable robustness metric that minimizes knowledge gap interference. Our
empirical study highlights the reliability of GRAMMAR to verify the hypothesis
and further identify the defective modules.

#### Retrieval-augmented Generation

RAG systems augment LLMs with retrieval for domain-specific use where
extensive databases can potentially provide necessary information. A retrieval
can be used for retrieving relevant passages or documents. A dense retrieval
(Mialon et al., 2023; Lewis et al., 2020; Borgeaud et al., 2022) consists of
an embedding model to produce a vector $v_{q}$ for any given query $q$. This
vector is then used to identify pertinent text segments normally via dot
product calculations with the vectors of text chunks. The most relevant chunks
are aggregated to form an additional context $c$, which serves as the factual
basis for the response generation, until adding another chunk would surpass
the maximum allowable context length. Subsequently, an LLM will process this
aggregated context alongside the original query to generate a predicted answer
$\hat{a}$ for $q$. In contrast, sparse retrieval computes similarity without
parametric embedding models. Keyword matching . For example, sparse retrieval
techniques operate without the need for parametric embedding models. For
instance, keyword-matching approaches (Manning, 2009) select documents that
share the highest number of common words for context retrieval. Additionally,
methods like TF-IDF (Chen et al., 2017) factor in the inverse document
frequency, emphasizing the significance of less common words which are likely
to be more indicative of pertinent content.

#### Knowledge-intensive Evaluation

Many evaluation protocols of RAG and LLMs focus on open-domain knowledge. Both
the queries $Q$ and their ground-truth answers $A$ in the open domains can be
easily accessed via online forums, public surveys (Santurkar et al., 2023) and
qualification and admission exams that were originally designed for humans
Wang et al. (2022); Zhong et al. (2022, 2023); Hendrycks et al. (2021); Choi
et al. (2023). Since LLMs are originally trained on open-domain knowledge,
these evaluation protocols on RAG focuses more on whether LLMs’ factuality can
be improved by retrieval. However, this paper focuses on industrial scenarios
where retrieval mostly provides new knowledge to LLMs and ground-truth answers
are hardly available. The reference-free evaluation protocol is commonly
proposed for this situation, e.g., ARES (Saad-Falcon et al., 2023) and RAGAS
(Es et al., 2023). For example, instead of the correctness of answers against
the users’ queries, Saad-Falcon et al. (2023) assess answer faithfulness
against the context from retrieval modules by profiling LLMs to generate
evaluation scores. The results can be skeptical due to the reliability of
retrieval and the fairness of LLMs as evaluators. Reference-free evaluation is
also utilized to assess LLMs’ open-domain knowledge (Kadavath et al., 2022).
SelfCheck (Manakul et al., 2023) relies on the stochastic nature of LLMs for
validations, while FactScore (Min et al., 2023) and FacTool (Chern et al.,
2023) use Retrieval+LLM as a validation system. FactScore and FactTool are
naturally not suitable for evaluating RAG systems that already include
retrieval as an inherent component.

##  3 GRAMMAR: Grounded and Modular Methodology for Assessment of RAG

In this section, we explore our evaluation framework’s design. Our goal is to
create scalable, grounded data clusters for comparison (see Algorithm 1),
establish a fair evaluation protocol that separates knowledge gaps from
robustness assessment and analyze modular errors effectively.

The data generation framework spans from schema definition through SQL
templates and text templates to the final formulation of text queries. The
development of SQL templates is driven by two key objectives: ensuring
scalability and facilitating access to the ground truths of queries, as
discussed in sections 3 and 3.3, respectively.

#### Scaled Data Generation

There exists significant combinatorial potential udeer the propsed data
generation process, driven by a range of factors. Specifically, a manually
crafted database with a modest number of attributes and rows suffices for
testing purposes, given the extensive variety of query permutations made
possible through the combinatorial approach. A comprehensive analysis of these
concepts is presented in Appendix D and C.

Algorithm 1 Grounded Data Generation Process.

1: SQL Template Generator $g_{\mathrm{sql}}$, Text Template Generator
$g_{\mathrm{t}}$, Semantic criteria for SQL template generation
$\mathcal{C}_{\mathrm{sql}}$, Linguistic criteria for text template generation
$\mathcal{C}_{\mathrm{t}}$, Database $\mathcal{D}$, Database Schema
$\mathcal{S}=\left\\{S_{1},S_{2},\ldots,S_{n}\right\\}$, where each $S_{i}$ is
a schema for a table

2:$Q\leftarrow\emptyset$ $\triangleright$ Used to store final evaluation data

3:$\mathcal{S}_{\text{target}}\in\mathcal{S}$ $\triangleright$ Step 1: Define
schema of a domain (§3.1)

4:$\\{tpl_{\text{sql}}\\}\leftarrow
g_{\mathrm{sql}}(\mathcal{S}_{\text{target}},\mathcal{C}_{\mathrm{sql}})$
$\triangleright$ Step 2: Generate SQL templates applying semantic criteria
(§3.1)

5:for $tpl_{\mathrm{sql}}\in\\{tpl_{\text{sql}}\\}$ do

6: $\\{tpl_{\text{t}}\\}\leftarrow
g_{\mathrm{t}}(tpl_{\mathrm{sql}},\mathcal{C}_{\mathrm{t}})$ $\triangleright$
Step 3: Generate text templates applying linguistic criteria (§3.2))

7: $P\leftarrow$ extract placeholders from $tpl_{\mathrm{sql}}$

8: for each $p\in P$ do

9: $C\leftarrow$ extract column from $p$

10: $V_{p}\leftarrow$ query $\mathcal{D}$ for distinct values of $C$

11: end for

12: $Comb\leftarrow$ Cartesian product of $\\{V_{p}:p\in P\\}$

13: for each $comb\in Comb$ do

14: $q_{\mathrm{sql}}\leftarrow$ substitute $comb$ into $tpl_{\mathrm{sql}}$
$\triangleright$ Step 4: Generate SQL queries (§3.3)

15: $a\leftarrow$ query $\mathcal{D}(q_{\mathrm{sql}})$ $\triangleright$ Step
5: Query DB for answers (§3.3)

16: $\\{q_{t}\\}\leftarrow$ substitute $comb$ into
$tpl_{\mathrm{t}}\in\\{tpl_{\text{t}}\\}$ $\triangleright$ Step 6: List of
text queries (§3.3)

17: $Q\leftarrow Q\cup\\{(\\{q_{t}\\},a)\\}$

18: end for

19:end for

20:return $Q$

###  3.1 Generating SQL Templates

A SQL template $tpl_{\text{sql}}$ essentially represents a query logic. It can
be instantiated into text templates, each denoted as $tpl_{t}$, by linguistic
features and then transformed into text queries by specifying the details of
entities.

#### Database Schema

Database schema serves as the blueprint that defines how classes of entities
are organized and the relations among them. Appendix A demonstrates the
reasons why we use database schema for knowledge representation. Let a
database schema with a collection of $N$ tables
$S=\left\\{S_{1},S_{2},\ldots,S_{N}\right\\}$, where each $S_{i}$ is a schema
for a table. A table schema $S_{i}$ consists of the table name $T_{i}$, a set
of attributes $\left\\{A_{1},A_{2},\ldots,A_{M}\right\\}$ and a set of
constraints, i.e.,
$S_{i}=\left(T_{i},\left\\{A_{1},A_{2},\ldots,A_{m}\right\\}\right.,C)$, where
the constraint $C$ demonstrates the primary key $PK$ and foreign keys $FK$.
Specifically, $FK\left(T_{i}.A_{k}\rightarrow T_{j}.A_{h}\right)$ indicates a
foreign key $A_{k}$ in table $T_{i}$ referencing the primary key $A_{h}$ in
table $T_{j}$. The schema defines the knowledge structure of an entity, which
can be enough to generate query logic that is not tied to any specific
linguistic expressions and sensitive data.

#### Generation of SQL Templates

The creation of SQL templates can be undertaken manually, with the formulation
of various query logics through the application of relational algebra
operators. In our study, we concentrate on three fundamental operators:

  * •

The Select $(\sigma)$ operator determines the attributes for querying, such as
requesting specific attributes of an entity or seeking entities that meet
certain predicates;

  * •

The Project $(\pi)$ operator, analogous to the WHERE clause in SQL, sets the
conditions for selection;

  * •

The Join $(\bowtie)$ operator facilitates the expression of query logic
involving multiple types of entities.

Furthermore, we have developed an automated SQL template generator $g_{sql}$,
powered by a generative language model. Specifically, when provided with a
target schema $\mathcal{S}_{\text{target}}$ and a verbal description of the
query logic $\mathcal{C}{\mathrm{sql}}$, the template generator $g_{sql}$ is
tasked with producing a set of SQL query templates that align with the
established criteria, drawing from the relevant domain (refer to Appendix B
for details).

###  3.2 Generating Text Templates

Upon acquiring SQL query templates denoted as $tpl_{\mathrm{sql}}$, the next
phase involves the generation of corresponding text templates, represented as
$tpl_{t}$. This process, driven by the text template generator
$g_{\mathrm{t}}$, leverages linguistic criteria $\mathcal{C}_{\mathrm{t}}$ to
transform $tpl_{\mathrm{sql}}$ into natural language forms that align with
specified linguistic characteristics, e.g., complexity, length and stylistic
nuances. The text template generator $g_{\mathrm{t}}$ is also operationalized
using a sophisticated language model, such as GPT-4, which is adept at
producing diverse linguistic variations of the same semantic content.

$\\{tpl_{\text{t}}\\}=g_{\text{t}}(tpl_{\text{sql}},\mathcal{C}_{\text{t}})$

, where $\\{tpl_{\text{t}}\\}$ signifies the resulting list of text templates.

###  3.3 From Templates To Evaluation Data

Utilizing a database $\mathcal{D}$ that aligns with the predefined schema, the
framework samples a diverse range of queries along with their corresponding
ground-truth answers. This dataset, derived from SQL and text templates filled
with database content, provides a rich source for evaluation.

#### Generating Queries Via Placeholder Fill-in

The placeholder fill-in step populates the templates with actual data from the
rows of tables in $\mathcal{D}$. Specifically, the approach employs a SELECT
query format: “SELECT DISTINCT {column_name} FROM {table_name};” to ensure
variety in the data points. It handles SQL query templates with multiple
placeholders by employing combinations of placeholders and using Cartesian
products to generate multiple query permutations. This approach leads to a
comprehensive set of SQL queries ($q_{\mathrm{sql}}$) and their natural
language equivalents, the text queries ($q_{t}$).

#### Generating Answers

Answer generation is integral to completing our dataset. Each SQL query
($q_{\mathrm{sql}}$) is executed against $\mathcal{D}$ to match with a factual
answer $a$. Each answer is then paired with the relevant text queries
$\\{q_{t}\\}$, forming the basis of our evaluation data.

Figure 2: Scaled Generation of Query-answer Pairs With Combinatorial Power.
The step 3 and step 4 are independent of each other and depend only on the SQL
templates.

###  3.4 Modular Evaluation

Assessing an RAG model’s overall performance may obscure insights into the
robustness or weaknesses of its constituent modules (refer to Section 4.2). It
is essential, therefore, to evaluate the contributions of the retrieval and
language modeling (LM) components individually.

We introduce three group types to distinguish the knowledge gap and the
robustness issue, as shown in Figure 1. All textual queries $Q$ generated from
a specific SQL logic $q_{\mathrm{sql}}$ can be grouped semantically.111Refer
to Appendix E for an illustration. Depending on the model $M$’s performance,
we categorize $q_{\mathrm{sql}}$ (and consequently, the group $Q$) into three
principal categories:

  * •

Gap Groups emerge when the model $M$ consistently provides incorrect answers
for every text query within a subset $S\subseteq Q$ produced through the data
generation process. This reflects a deficiency in $M$ ’s knowledge or
capability. Formally, this scenario is described as:

$\forall q_{t}\in S,\neg M\left(q_{t}\right)$

wherein $\neg M\left(q_{t}\right)$ signifies an inaccurate response from $M$
to the text query $q_{t}$.

  * •

Robust Groups are established when model $M$ accurately responds to all text
queries within the subset $S$, evidencing a robust comprehension of the SQL
logic. Formally, this is articulated as:

$\forall q_{t}\in S,M(q_{t})$

.

  * •

Non-robust Groups are characterized by model $M$’s exhibited capability to
potentially address the query logic, yet there exists at least one query
$q_{t}$ within $S$ that $M$ inaccurately predicts. This is formally denoted
as:

$\left(\exists q_{t}\in S,M(q_{t})\right)\land\left(\exists q_{t}\in S,\neg
M(q_{t})\right)$

.

#### Defective Modules Responsible for The Robustness Issue

Gap groups mostly reflect the limited coverage of the retrieval database. The
analysis of the semantics of queries in gap groups can help expand the
database. However, defective modules responsible for the robustness issue need
to be further identified by answering the following questions: Does the
retrieval module source and provide the sufficient context needed for a given
query? With the sufficient context at its disposal, is the LM capable of
drawing accurate inference? Utilizing a “context comparison” methodology
enables pinpointing the LM’s role in generating erroneous predictions when
adequate context is provided. By contrasting the document indices
$\text{idx}_{c}$ of each non-robust instance against indices from accurate
predictions within analogous semantic groups against those linked to correct
predictions within the same semantic groups
$I=\text{idx}_{1},\text{idx}_{2},\dots$, we evaluate the sufficiency of
context. This assumes the LM’s reliance on external information for query
responses, mirroring our scenario and prevalent industrial applications.
Should any index in $I$ match $\text{idx}_{c}$, we reclassify the outcome as a
successful retrieval yet an LM shortfall. This strategy allows for precise
identification of LM-related non-robust instances. Nevertheless, the absence
of such matches does not conclusively indicate retrieval failures. Given our
experimental setup with a synthetic database comprising 10 documents, this
“context comparison” method effectively identifies all document combinations
across a moderate query range (3 to 6) within semantically related groups.
This framework lays the groundwork for future explorations into more intricate
cases.

A reliable robustness metric is necessary to allow precise identification of
detective modules. We quantify robustness as the fraction of instances
correctly answered, excluding those in gap groups:

$\begin{split}R=\frac{\text{\\# with Correct Predictions}}{\text{Total
\\#}-\text{Total \\# in Gap Groups}}\end{split}$ (1)

, where “#” represents “the number of instances”. By omitting gap groups, this
approach ensures that the metric reflects true robustness, distinct from
knowledge gaps. Consequently, accuracy Acc is articulated as:

$\begin{split}\text{Acc}=R\times\frac{\text{Total \\#}-\text{Total \\# in Gap
Groups}}{\text{Total \\#}}\end{split}$ (2)

This formulation reveals how the proportion of instances within gap groups
$\lambda$ influences the efficacy of accuracy as a measure of robustness.
Specifically, a higher $\lambda$ value in the robust compared to the non-
robust cluster may obscure the model’s non-robust characteristics. Section 4
will empirically demonstrate the crucial importance of excluding gap groups
for accurately assessing system vulnerabilities.

##  4 Case Study: Question Answering on Industrial Projects

This section presents a case study focusing on question answering about
industrial projects. It serves to critically evaluate both existing reference-
free evaluation frameworks and the newly proposed evaluation framework

#### System Overview

Our study incorporates two distinct systems: 1) Dense Retrieval + GPT3.5
222OpenAI’s gpt-3.5-turbo-16k: It is deployed in a commercial context with
secure access to the retrieval module via the Qdrant API, ensuring data
privacy. The system employs an OpenAI embedding model for document embedding,
providing a basis to assess the reference-free evaluation framework’s
reliability (§4.1). 2) Simple Sparse Retrieval + GPT3.5: The absence of
comprehensive research into the assured robustness of dense retrieval systems
makes it difficult to pinpoint distinct robust or vulnerable characteristics
with confidence. Consequently, we employ a straightforward keyword-matching
retrieval approach. Although this method is less prevalent in real-world
applications, this method’s inherent shortfall in handling lengthy queries
makes it exceptionally ideal to validate our evaluation framework in
conducting reliable hypothesis testing (§4.2).

#### Synthetic Knowledge Base

Concurrently, a synthetic dataset has been created, consisting of project
documents from the fictitious company, Aurp, serving as a retrieval database.
It facilitates a precise evaluation of the retrieval component and the
language model, since the knowledge contained within the synthetic data is not
present in the training datasets of large language models. In scenarios where
retrieval fails to procure adequate context, the language models are
definitely unable to accurately respond to queries. Additional information on
this methodology can be found in Appendix G.

#### Mapping Domain Knowledge into Database Schema

Our approach distills domain knowledge into a database schema centered around
three pivotal entity types in project management: Project, Client, and
Employee. 333This triad are reflective of the core components outlined in the
Project Management Body of Knowledge (PMBOK), a widely recognized standard in
the field of project management (Guide, 2001).This schema encapsulates the
quintessential entity types found within a corporate setting. With the
scalability of our framework, the limited amounts of entity types and
associated attributes and entities lead to large amounts of instances for
evaluation: 198 examples for the commercial case (with totally 12 attributes
and 11 rows) and 942 examples for the synthetic scenario (with 14 attributes
and 49 rows in total). The details of the database schema and the amounts of
columns and rows for each table are shown in Appendix F and H), respectively.

###  4.1  Is Reference-free Evaluation Reliable?

Leveraging ground truths, our evaluation framework sets a benchmark for
reference-free assessments. This section delves into the reliability of
reference-free evaluation methods, utilizing examples generated for the
commercial case. The focus is to discern the reliability of such evaluation
approaches: How trustworthy are these reference-free protocols? From what
perspectives can they be relied upon in domain-specific contexts?

#### Reference-free Evaluation Protocols

We assess two reference-free evaluation protocols: SelfCheck and RAGAS. RAGAS-
Fact utilizes the context-query-response triplets to assess the veracity of
responses. It evaluates the faithfulness of a response by calculating the
ratio of claims grounded on the context to the total claims made. This
process, termed as Ragas-Fact, involves identifying statements that hold
atomic facts, following the methodology outlined by Chern et al. (2023).
SelfCheck, as introduced by Manakul et al. (2023), relies on stochastically
generating responses from the model under the premise that incorrect answers
are unlikely to repeatedly yield identical outcomes through random sampling.
This principle, initially applied to LLMs, is adapted for our analysis of RAG
systems. In a departure from its original application, we enhance the
evaluation prompt to include not only stochastic responses but also the query
itself. Refer to Appendix I for details. To generate four stochastic samples,
we adjust the temperature setting to 1.0, contrasting with a temperature of
0.0 used to procure the primary response. A response is deemed correct if it
aligns consistently across all stochastic samples.

#### RAGAS’ Reliable Assessment of Correct Predictions Amidst Broader
Unreliability in Reference-Free Evaluation

To gauge reliability, we devise two metrics: the ratio of predictions judged
as accurate among all those classified as correct by the evaluation, and the
ratio of predictions deemed accurate within the set of instances that were
indeed correctly predicted. When considering reference-free evaluation as a
binary classification task, these metrics align with precision and recall,
respectively. In this context, the positive class represents correct
predictions, and the negative class represents incorrect ones. The ground
truth corresponds to the actual accuracy of model predictions, while the
evaluation task is to determine the correctness of these predictions. As shown
in Table 1, RAGAS demonstrates effectiveness in assessing accurately predicted
instances, yet it achieves only a 19% accuracy rate across all evaluations,
highlighting a deficiency in correctly identifying erroneous predictions. This
trend suggests a propensity for inflating model performance assessments,
likely due to the RAG system producing responses that, while contextually
relevant, fail to directly address the intended query. SelfCheck performs even
worse than RAGAS in both cases. SelfCheck’s performance falls short of
RAGAS’s, underperforming in both precision and recall. The preliminary insight
suggests that reference-free evaluation methods are inherently prone to
inaccuracies, underscoring the critical need for a robust, reference-based
evaluation framework to ensure dependable model assessment.

| Precision | Recall
---|---|---
Ours (Upper Bound) | 1 | 1
RAGAS-Fact | 19% | 92%
SelfCheck | 15% | 50%
Table 1: Reliability of reference-free evaluation protocols.  |  | Robust | Non-robust | Success?
---|---|---|---|---
Accuracy | Baseline: Unbalanced Sampling (US) | 0.31 | 0.43 | No
| US+Modular Evaluation (Retrieval) | 0.36 | 0.46 | No
| Stratified Sampling (SS) | 0.31 | 0.29 | Yes
| SS+Modular Evaluation (Retrieval) | 0.38 | 0.35 | Yes
Robustness | Baseline: Unbalanced Sampling (US) | 0.84 | 0.85 | No
| US+Modular Evaluation (Retrieval) | 0.99 | 0.93 | Yes
| Stratified Sampling (SS) | 0.78 | 0.74 | Yes
| SS+Modular Evaluation (Retrieval) | 0.94 | 0.87 | Yes
Table 2: Evaluating the efficacy of accuracy and designed robustness metrics
in distinguishing between robust and non-robust clusters. Modifications
include retrieval-specific adjustments by excluding LM-induced non-robust
examples.

###  4.2 Empirical Validation of GRAMMAR

This section will validate design choices of identifying defective modules in
the proposed framework.

#### The Importance of Modular Evaluation: Avoiding The Masking Effect of
Comprehensive Assessments

Analysis presented in Table 2 underscores the value of segmenting the
evaluation to specifically examine the retrieval process. The non-robust
cluster actually receives higher scores (better performance) than the robust
cluster regarding both the accuracy and robustness metrics. However, under the
modular evaluation, the evaluation result correctly reflects the non-robust
cluster. This underscores the importance of evaluating the retrieval function
independently; failing to do so may allow language model inaccuracies to
obscure the true performance metrics, potentially masking retrieval robustness
issues. Although instances causing such distortions were not dominant in our
dataset, they highlight a critical evaluation nuance.

#### The Importance of Excluding Gap Groups for Robustness Measurment

Highlighting the necessity of omitting gap groups, our ablation study
implements two distinct approaches: 1) Comparing the Robustness Metric with
Accuracy; 2) Comparative Analysis of Stratified Sampling and Unbalanced
Sampling. Stratified sampling maintains a consistent $\lambda$ across both
robust and non-robust clusters. By prompting the text template generator, an
equal number of text queries are generated for each query logic. To explore
the effects of unbalanced sampling, that can occur in the real world, we
randomly reduce examples from query groups related to “clients” and
“employees”.

Table 2 demonstrates that traditional accuracy metrics falter in
distinguishing clusters based on vulnerability attributes, particularly when
$\lambda$ varies between the hypothesis-driven cluster and its counterpart
(Row 2 under the “Accuracy” category). Stratified sampling ameliorates this
issue, enabling a precise identification of both robust and non-robust
clusters (Row 4 under the “Accuracy” category). Importantly, the
implementation of a specifically designed robustness metric ensures consistent
and accurate identification of the robust cluster, irrespective of $\lambda$
variations (Row 2 and 4 under the “Robustness” category). This approach,
integrating stratified sampling with our bespoke robustness metric, providing
a reliable analysis.

##  5 Conclusions

In our work, we have developed an evaluation framework tailored for Retrieval-
augmented Generation (RAG) systems within an industrial context, enhancing the
reliability and interpretability of evaluations. Additionally, we have exposed
the limitations of current reference-free evaluation methods, particularly
their ineffectiveness in accurately assessing incorrect predictions.

#### Reproducibility

The entire evaluation framework, encompassing both the code and the prompts
for the LLM, will be made open-source and available to the academic and
industrial community. This initiative aims to facilitate the wider use and
adaptation of our framework for research and real-world implementations. In
terms of the systems and data evaluated to enable result replication, we plan
to publish the model, including the documents intended for retrieval, as well
as all templates and queries developed for the hypothetical Aurp scenario.
However, the model designated for commercial use and its associated knowledge
base will remain confidential and unpublished to honor non-disclosure
agreements and protect proprietary information.

##  6 Limitations

The challenges related to the presence of multiple correct answers (refer to
Appendix J) and the constraints in expressiveness posed by database schema and
SQL (refer to Appendix K) ultimately result in difficulties in generating
queries that demand multi-step reasoning and free-form responses. Nonetheless,
as demonstrated in the case study, straightforward queries prove to be
sufficient for identifying and debugging the intrinsic robustness issues
within the models.

## References

  * Alzantot et al. (2018) Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018.  Generating natural language adversarial examples.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2890–2896, Brussels, Belgium. Association for Computational Linguistics.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022.  Improving language models by retrieving from trillions of tokens.  In _International conference on machine learning_ , pages 2206–2240. PMLR.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading Wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics.
  * Chern et al. (2023) I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. 2023.  Factool: Factuality detection in generative ai–a tool augmented framework for multi-task and multi-domain scenarios.  _arXiv preprint arXiv:2307.13528_.
  * Choi et al. (2023) Jonathan H Choi, Kristin E Hickman, Amy Monahan, and Daniel Schwarcz. 2023.  Chatgpt goes to law school.  _Available at SSRN_.
  * Es et al. (2023) Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023.  Ragas: Automated evaluation of retrieval augmented generation.
  * Guide (2001) A Guide. 2001.  Project management body of knowledge (pmbok® guide).  In _Project Management Institute_ , volume 11, pages 7–8.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations_.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022.  Language models (mostly) know what they know.  _arXiv preprint arXiv:2207.05221_.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 9459–9474. Curran Associates, Inc.
  * Li et al. (2019) Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. 2019.  Textbugger: Generating adversarial text against real-world applications.  In _26th Annual Network and Distributed System Security Symposium, NDSS 2019, San Diego, California, USA, February 24-27, 2019_. The Internet Society.
  * Li et al. (2020) Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020.  BERT-ATTACK: Adversarial attack against BERT using BERT.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6193–6202, Online. Association for Computational Linguistics.
  * Liu et al. (2023) Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S Yu. 2023.  A comprehensive evaluation of chatgpt’s zero-shot text-to-sql capability.  _arXiv preprint arXiv:2303.13547_.
  * Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023.  Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.
  * Manning (2009) Christopher D Manning. 2009.  _An introduction to information retrieval_.  Cambridge university press.
  * Mialon et al. (2023) Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Roziere, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023.  Augmented language models: a survey.  _Transactions on Machine Learning Research_.  Survey Certification.
  * Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023.  Factscore: Fine-grained atomic evaluation of factual precision in long form text generation.  _arXiv preprint arXiv:2305.14251_.
  * Saad-Falcon et al. (2023) Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2023.  Ares: An automated evaluation framework for retrieval-augmented generation systems.
  * Santurkar et al. (2023) Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023.  Whose opinions do language models reflect?  _arXiv preprint arXiv:2303.17548_.
  * Shen et al. (2023) Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2023.  In chatgpt we trust? measuring and characterizing the reliability of chatgpt.  _arXiv preprint arXiv:2304.08979_.
  * Wang et al. (2022) Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. 2022.  From lsat: The progress and challenges of complex reasoning.  _IEEE/ACM Trans. Audio, Speech and Lang. Proc._ , 30:2201–2216.
  * Zhong et al. (2023) Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023.  Agieval: A human-centric benchmark for evaluating foundation models.  _arXiv preprint arXiv:2304.06364_.
  * Zhong et al. (2022) Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Yining Chen, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan. 2022.  Analytical reasoning of text.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 2306–2319, Seattle, United States. Association for Computational Linguistics.

##  Appendix A Knowledge Structure and Representation

Commonly, the knowledge is formalized by entities along with their relations,
e.g., a knowledge graph, ontology and a database schema. Instead of using
knowledge graphs containing concrete entities and their relations, we use the
schema-based definition, where the schema is characterized by entity types
(classes of entities). An entity type is defined by its name, its attributes
and relations with other entity types that will be concretized as components
in a database, i.e., the table name, columns and foreign keys.

The definition of entity types can provide combinatorial expansion of data
generation (See §3), protecting the leak of private information (normally
stored as rows of tables) during the data generation process and the ability
of retrieving ground-truths via Structured Query Language (SQL). The three
advantages are the reasons why we represent knowledge in database schema
rather than knowledge graphs. It disentangles the private data stored in the
database with universal metadata expressed in the database schema. Since
schemas and templates contain only meta-data for structuring the real data,
there is no private issue to use commercial large language models for this
process.

You are a SQL query Template Generator: Generate ACCEPTABLE SQL query
templates with placeholders according to the give data schema and
requirements. A simple example of an acceptable SQL query template is: SELECT
Industry FROM Company WHERE Name = ’[Company.Name]’; You must follow the basic
criteria below except for other requirements: ##CRITERIA## \- The placeholder
format should be a combination of a table name and a column name, enclosed
within square brackets, e.g., ’[User.Name]’. \- Use only ’SELECT’ queries. \-
Select specific column(s) instead of using ’*’. Avoid projecting attributes
that appear in the predicate. \- The selected and condition columns in the
query MUST BE MEANINGFUL and DESCRIPTIVE to ensure the queries are easily
understood by non-technical users. \- Avoid using technical column names that
don’t clearly signify the nature of the entities or objects involved, e.g.,
column for semantically void record identifiers. \- Do not create redundant or
semantically duplicated queries when translated into natural language. \- Each
query must contain at least one parameter placeholder in the WHERE clause. \-
Ensure the query yields a specific and singular answer to avoid multiplicity
issues, thus facilitating accurate chatbot evaluation. {SPECIFIC_REQUIREMENTS}
\- If no acceptable SQL template can be generated with the given table and
column information, do not generate any text. ##RESPONSE FORMAT## \- Output
each SQL template as a single line, without any prefix or suffix. \- Do not
include any other text in your response, even something like ##RESPONSE_END##.
##DATA SCHEMA## {GIVEN_SCHEMA} ##RESPONSE_START##
---
Table 3: A prompt template designed to guide the LLM in functioning as a
controllable SQL template generator

##  Appendix B Utilizing LLMs for Generating SQL Templates

We utilize GPT-4 as a SQL template generator for its state-of-the-art
capabilities in SQL generation benchmarks (Liu et al., 2023).

#### Profile Prompt

The profile prompt defines its purpose and establishes criteria for generating
valid SQL templates. These criteria limit the types of templates that can be
generated, such as requiring the selection of attributes that are
understandable to humans, as detailed in Table 3.

#### Limitations: Scope of Use for Automated SQL Generators

The effectiveness of queries generated by LLMs hinges on the meaningfulness of
table and column names to accurately reflect the essence of the entities or
concepts they represent. If these names lack contextual clarity, the resulting
queries may be impractical. To mitigate this, practitioners can conduct an
informal assessment or consult established benchmarks that evaluate LLMs’
proficiency in domain knowledge and commonsense reasoning, for instance,
(Zhong et al., 2023; Hendrycks et al., 2021).

##  Appendix C Breakdown of Scalable Data Generation

The proposed data generation process of generating queries via LLM, which
includes both the creation of templates and the formulation of queries, can be
succinctly represented by the formula (see Figure 2 for a visual
representation):

$\text{Total Query Variations}=M\times N\times Q,$ (3)

where “M” denotes the array of SQL templates that can be extracted from a
given schema, “N” refers to the vast number of text templates that can be
produced from a single SQL template, showcasing the flexibility of natural
language, and “Q” accounts for the broad spectrum of text queries that can be
generated from a single text template, with this diversity arising from
different combinations of unique values filling the placeholders.

Let’s break down each step of the proposed data generation process:

  * •

1 schema $\Rightarrow$ M SQL Templates: The number of possible SQL templates
(M) that can be generated from a defined schema is influenced by the SQL
operators used, the number of tables, and the columns in the database schema.
Each combination of tables and columns, along with different SQL operators
(like SELECT, WHERE, JOIN), can lead to a unique SQL query template, e.g., the
exponential complexity of the SELECT operator, and the combinatorial increase
of possible predicates (See Appendix D for details). Note that while the
theoretical maximum is high, practical and meaningful queries will be a
smaller subset, as specified in the case study and Appendix B.

  * •

1 SQL Template $\Rightarrow$ N Text Templates: For each SQL template, there
can be an arbitrary number (N) of textual expressions. This variation arises
from the different ways to linguistically express the same SQL query due to
the flexibility and richness of natural language, e.g., the variation of
synonyms and sentence structures.

  * •

1 Text Template $\Rightarrow$ Q Text Queries: Each text template can lead to a
number of text queries (Q), depending on the unique values available for each
placeholder. With more than one placeholder, the potential for growth in the
number of text queries is combinatorial, barring semantic conflicts (context
conflict), where certain combinations of column values may not be semantically
valid. Overall, this framework demonstrates a combinatorial expansion at each
transition stage, especially notable in steps involving natural language due
to its inherent variability. However, this growth is tempered by practical
constraints such as the meaningfulness of queries (semantic validity) and the
actual data distribution in the database. The exponential increase is most
pronounced in the transition from text templates to text queries, where the
permutations of placeholders can lead to a vast array of unique query
possibilities.

##  Appendix D Possibilities Of SQL Templates

#### One Table/Entity

Given a schema with only one table, the possibilities for SQL templates can be
analyzed by considering the basic SQL operators like SELECT, WHERE. Here’s a
breakdown:

  * •

Variation in Selected Attributes (SELECT): The number of SQL templates varies
based on the combination of columns selected. If the “Company” table has $n$
columns, then theoretically, there are $2^{n}-1$ possible combinations of
columns for selection (excluding the case where no column is selected).

  * •

Conditions in Queries (WHERE): Each SQL query can include zero or more
conditions in the WHERE clause. The number of possible conditions is
determined by the number of columns, the type of each column (text, numeric,
date, etc.), and the range of operators applicable to these types (like =, <,
>, LIKE, IN for text columns; =, !=, <, >, BETWEEN for numeric columns). The
complexity increases combinatorially with multiple conditions combined using
AND/OR.

#### Two Entities

The relations between tables further amplifies the number of possible
templates. For example, with two entities, “Company” and “Project,” and their
associative table “Company_Project”, the possibilities for SQL templates
expand significantly due to the introduction of joins and more complex WHERE
clauses. Let’s break down the possibilities.

  * •

Selection Variations (SELECT): The number of SQL templates grows with the
combination of columns selected across the three tables: “Company”, “Project”
and “Company_Project”. If “Company” has n columns, “Project” has m columns,
and “Company_Project” has p columns, the possible combinations for selection
are $(2^{n}-1)\times(2^{m}-1)\times(2^{p}-1)$.

  * •

Join Conditions (JOIN): The introduction of the associative table
“Company_Project” allows for meaningful JOIN operations between “Company” and
“Project.” Templates can include joins like Company JOIN Company_Project ON
condition and Project JOIN Company_Project ON condition, or a multi-table join
linking all three. The variety of JOIN conditions adds another layer of
complexity to the possible templates.

  * •

WHERE Clause Complexity: With more tables, the WHERE clause can include a
wider range of conditions, potentially involving attributes from any of the
three tables. The complexity increases with the number of columns and their
types across all tables, and combinations of these conditions.

##  Appendix E Semantic Group

Textual queries that originate from a specific SQL logic $q_{\text{sql}}$ are
organized into a semantic group because of their underlying shared semantics.
For instance, the SQL query “SELECT C.address FROM COMPANY C WHERE
C.CompanyName = ’Carlton Innovation Precinct’“ results in the creation of a
semantic group $Q=\\{q_{t1},q_{t2},\ldots\\}$, with each $q_{t}$ representing
a distinct textual interpretation. Examples include $q_{t1}$ as “What is the
address of ’Carlton Innovation Precinct’?” and $q_{t2}$ as “Where is ’Carlton
Innovation Precinct’ located?”.

##  Appendix F Database Schemas Overview

Figures 3 and 4 illustrate the database schemas used for an actual industrial
context and a fabricated scenario, respectively, within our research.

Figure 3: Entity-Relationship Diagram (ERD) showcasing the schema.  Figure 4:  Entity-Relationship Diagram (ERD) depicting the database schema for a simulated setting.  |  Attribute # | Template # |  Row # | Queries #
---|---|---|---|---
|  SQL |  SQL (Extra Constraints) |  Text |  SQL |  Text
Project |  5 |  7 |  3 |  18 |  4 |  13 |  78
Employee |  5 |  12 |  5 |  30 |  3 |  14 |  84
Client |  2 |  6 |  2 |  12 |  4 |  6 |  36
Total |  / |  / |  / |  / |  / |  33 |  198
Table 4: Statistics of DB Rows And Columns And Templates/Queries.  |  Attribute # | Template # |  Row # | Queries #
---|---|---|---|---
|  SQL |  Text |  SQL |  Text
Project |  6 |  5 |  15 |  10 |  50 |  300
Employee |  5 |  4 |  12 |  29 |  87 |  522
Client |  3 |  2 |  6 |  10 |  20 |  120
Total |  / |  / |  / |  / |  157 |  942
Table 5: Statistics of Aurp DB Rows And Columns And Templates/Queries.

##  Appendix G Synthetic Knowledge Base for Evaluating Retrieval-Augmented
Language Models

The process of generating synthetic data for a synthetic environment involves
creating comprehensive, detailed, and interconnected datasets for a fictitious
company named Aurp, which specializes in structural engineering consultancy
services. This multi-step procedure includes:

  1. 1.

Generating Company Profile: Initially, a company profile for Aurp is created,
detailing its foundation year, headquarters location, CEO, number of
employees, and the services it offers, such as bespoke architectural
solutions, sustainable urban planning, and structural health monitoring.

  2. 2.

Generating Organizational Structure: Next, a project-oriented organizational
structure is established, naming key positions and employees within the
company, similar to real-world firms. This includes a wide range of roles from
executive positions to specialized engineers and support staff.

  3. 3.

Generating Employee Information: For each employee listed in the
organizational structure, detailed job titles, departments, and direct
supervisors or managers are fabricated, creating a network of relationships
and reporting lines within the company.

  4. 4.

Generating Client Information: The process creates a list of ten clients
across various industries—ranging from technology and real estate to
hospitality and healthcare—each with specified locations, thereby illustrating
the company’s diverse portfolio.

  5. 5.

Generating Projects: Specific projects are then devised, including names,
locations, start and end dates, clients, project directors, and project
managers. This step integrates previously generated data (client and employee
information) to create realistic project scenarios.

  6. 6.

Generating Project Reports: This step involves synthesizing data from the
previous steps to produce detailed analyses, updates, and outcomes of the
various projects, which may or may not contain detailed information for
clients and employees.

This elaborate and interconnected data creation method is tailored for
evaluating retrieval-augmented language models. It serves as a critical tool
for exploring how these models handle data that is entirely fictional and thus
not present in their training datasets. The synthetic nature of this
environment provides a unique testbed for separating and assessing the
distinct capabilities of the retrieval component and the language model
component in isolation. Because the knowledge in the synthetic data does not
appear in the training data of large language models.

##  Appendix H Assessing Scalability with Limited Entity Types, Rows, and
Attributes

Our scalability assessment focused exclusively on the utilization of the
SELECT operator and the WHERE clause for each table schema. As illustrated in
Table 4 and 5, imposing constraints to generate relevant queries significantly
reduces the potential quantity of SQL templates based on attribute count.
Despite these restrictions, a minimal set of attributes and rows still yields
hundreds of viable query-answer pairs.

##  Appendix I Prompt Templates for LLM-Based Evaluation

For SelfCheck, given that our responses are typically concise, we opt for
assessing the overall response rather than rather than performing a sentence-
by-sentence validation as in the original implementation, as shown in Table 6.

SelfCheck from (Manakul et al., 2023)
---
Context: {context}
Sentence: {sentence}
Is the sentence supported by the context above?
Answer Yes or No:
SelfCheck-QA
Query: {query}
Answer A: {answer}
Answer B: {stochastic_answer}
Do both answers address the query with equivalent meaning?
Use only “Yes” or “No” for your evaluation:

Ragas from (Es et al., 2023)
Natural language inference. Use only ’Yes’ (1), ’No’ (0) and ’Null’ (-1) as
verdict.
context: {context}
statement: {statement}
verdict:

Ours
Evaluate the accuracy of the given response in relation to the true answer for
the specified query. After evaluating, provide a judgement as either “Correct”
or “Incorrect” based on whether the ##Given Response## accurately matches the
##True Answer##.
##Query##: {query}
##True Answer##: {true_answer}
##Given Response##: {given_response}
##Judgement##:
Table 6: Prompt templates for LLM-based evaluation.

##  Appendix J Presence of Multiple Correct Answers

In the evaluation of question-answering (QA) models, a unique challenge arises
from the existence of multiple valid answers to a single query, which
necessitates a nuanced approach to assessing model performance. Consider the
question: “Get the name of the client associated with the project named
Innovation Precinct” For such a question, a set of correct responses could
include any combination of names from a predefined list, such as {Apple,
Amazon, Meta, Facebook}. This multiplicity of correct answers underscores the
complexity of evaluating QA models, as it requires the assessment mechanism to
recognize and validate the full spectrum of possible correct answers rather
than comparing the model’s output against a single ’gold standard’ answer.
This scenario demands a more flexible method for answer validation that can
accommodate the variability in correct responses. Also, evaluation metrics may
be required to effectively measure the performance of QA models in handling
diverse and equally valid answers. Such metrics must account for the
exhaustive set of correct answers and evaluate the model’s ability to retrieve
any or all valid responses within the context of the query, thereby ensuring a
comprehensive assessment of the model’s understanding and response generation
capabilities.

#### Simple Queries to Avoid Answer Multiplicity

Generally, encompassing all valid responses within the database is not
feasible. Hence, rather than let the non-robust results affect the accuracy of
evaluation, we look for an approach to avoid it, i.e., only generating queries
ensuring one ground-truth answer. Specifically, the evaluation of certain SQL
queries, such as “SELECT Name, Department, BusinessAddress FROM Employee WHERE
JobTitle = ’designer”’, requires a complete and thorough listing of multiple
answers that can be dynamically changed. To solve the issue, the solution
involves adding a specific criterion to the prompt for generating SQL queries.
This criterion, “Ensure the query yields a specific and singular answer”, aims
to produce queries that result in a single, clear answer, thereby avoiding the
complexities of multiple possible correct answers. For example, with this
criterion, the query “SELECT Industry FROM Company WHERE Name =
’[Company.Name]’;” is preferred as it’s likely to yield a singular answer
about a company’s industry based on a specific company name. In contrast,
without this criterion, queries like “SELECT Name FROM Company WHERE Industry
= ’[Company.Industry]’;” are acceptable but may result in multiple names,
leading to evaluation difficulties due to data completeness and query
multiplicity issues.

##  Appendix K Queries Beyond SQL Expressiveness

While SQL and relational algebra offer a wide range of operations enabling the
formulation of numerous user queries, certain semantic nuances exceed the
expressive capabilities of SQL. The following examples illustrate such
limitations:

  * •

Real-world queries often contain ambiguity and subjective interpretations that
SQL struggles to accommodate. For instance, the term “major” in the query
“What are some major rail projects we’ve been involved with?” does not
directly translate into SQL criteria without additional interpretative steps.

  * •

SQL queries are typically structured to elicit specific, predefined responses,
contrasting with the open-ended nature of many real-world inquiries that seek
exploratory or comprehensive answers.


Experimental validation of Fastcat kV and MV cone beam CT (CBCT) simulator
Jericho O’Connell, Clayton Lindsay, and Magdalena Bazalova-Carter
University of Victoria, 3800 Finnerty Road, Canada
Version typeset

Author to whom correspondence should be addressed. email<EMAIL_ADDRESS>

###### Abstract

Purpose: To experimentally validate the Fastcat cone beam CT (CBCT) simulator
against kV and MV CBCT images acquired with a Varian Truebeam linac.
Methods: kV and MV CBCT images of a Catphan 504 phantom were acquired using a
100 kVp beam with the on-board imager (OBI) and a 6 MV treatment beam with the
electronic portal imaging device (EPID), respectively. The kV Fastcat
simulation was performed using detailed models of the x-ray source, bowtie
filter, a high resolution voxelized virtual Catphan phantom, anti-scatter
grid, and the CsI scintillating detector. Likewise, an MV Fastcat CBCT was
simulated with detailed models for the beam energy spectrum, flattening
filter, a high resolution voxelized virtual Catphan phantom, and the GOS
scintillating detector. Experimental and simulated CBCT images of the phantom
were compared with respect to HU values, contrast to noise ratio (CNR), and
dose linearity. Detector modulation transfer function (MTF) for the two
detectors were also experimentally validated. Fastcat’s dose calculations were
compared to MC dose calculations performed with Topas.
Results: For the kV and MV simulations, respectively: Contrast agreed within
14 and 9 HUs and detector MTF agreed within 4.2% and 2.5%. Likewise, CNR had a
root mean squared error (RMSE) of 2.6% and 1.4%. Dose agreed within 2.4% and
1.6% of MC values. The kV and MV CBCT images took 71 and 72 seconds to
simulate in Fastcat with 887 and 493 projections, respectively.
Conclusions: We present a multi energy experimental validation of a fast and
accurate CBCT simulator against a commercial linac. The simulator is open
source and all models found in this work can be downloaded from
https://github.com/jerichooconnell/fastCAT.git.

###### Contents

  1. I. Introduction
  2. II. Materials and Methods
    1. II.A. Experimental Data Acquisition
      1. II.A.1. kV CBCT
      2. II.A.2. MV CBCT
    2. II.B. Fastcat Overview
      1. II.B.1. Phantom
    3. II.C. Fastcat CBCT Simulations
    4. II.D. Additional Fastcat Modelling
      1. II.D.1. Bowtie and Flattening Filters
      2. II.D.2. Anti-scatter Grid
      3. II.D.3. Virtual Phantom
    5. II.E. Dose Comparison
    6. II.F. Validation Metrics
      1. II.F.1. Detector MTF
      2. II.F.2. CBCT Contrast and CNR
  3. III. Results
    1. III.A. CBCT Image Comparison
    2. III.B. Contrast and MTF
    3. III.C. CNR and Dose
    4. III.D. Dose and Speed
  4. IV. Discussion
  5. V. Conclusion
  6. VI. Acknowledgments
  7. References

##  I. Introduction

Cone beam computed tomography (CBCT) 1, 2 is used extensively to visualize
patient anatomy in the planning and administration of radiotherapy treatment.
As such, CBCT that provides more accurate and realistic images can improve
radiotherapy patient positioning and beam alignment. To this end CBCT is being
constantly ameliorated, active research is being conducted end to end in the
imaging process. To evaluate the efficacy of new research, two approaches are
often used. New CBCT imaging methods can be evaluated based on an experimental
phantom of known composition and dimensions with reproducible results but
limited relation to real patient data. Conversely, evaluations can be
performed using real patient data of unknown composition and dimensions with
limited reproducibility. The ideal case, to evaluate new research on a
realistic patient model with known composition, dimensions and robust
reproducibility is an active area of study and the focus of this work.

With the introduction of new anthropomorphic virtual phantoms 3, 4, 5, 6, 7, a
new approach to evaluating x-ray imaging research has become viable. Realistic
anthropomorphic phantoms allow simulations to be performed with realistic
patient anatomy without the patient dose and uncertain ground truth associated
with clinical data. Numerous simulators exist that are compatible with these
phantoms.

These CT simulators generally use Monte Carlo (MC) platforms 8, 9, 10. These
MC platforms, while used primarily for dose calculation in radiation therapy,
have superior accuracy due to precise simulation of the particle transport
underlying CBCT. However, MC methods suffer from long compute times for
simulating CBCT. This is due to a variety of factors, e.g. the number of
projections needed, the handling of a high resolution voxelized phantom, and
the transport of optical photons in the simulator, all of which can be
detrimental to computation speed. For example, in a study by Blake et al. 11
simulation of one projection image with a scintillating detector and 107
primary x-rays took 3000 CPU-hours. Likewise, MC methods suffer from increased
load times, compute times, and memory constraints as the resolution of an
anthropomorphic phantom is increased. For example, in the work of Yeom et al.
12, which examines the performance of Geant4, MCNP, and PHITS with respect to
dose calculation speeds in voxelized anthropomorphic phantoms. Simulation
times were increased by an order of magnitude in Geant4 when decreasing voxel
size from 1 mm3 to 0.1 mm3, while MCNP nor PHITS could simulate a 0.1 mm3
phantom due to the number of phantom voxels exceeding the maximum limit of the
simulation codes.

Other simulation approaches are currently available to improve simulation
time. Graphical processing unit (GPU) MC is a current method that shows speed
up factors of 27-90 over traditional CPU-based MC 13, 8. Likewise, ray-tracing
methods are available for CPU and GPU 14, 15. These methods are very fast but
cannot account for scatter in phantoms and detector response. A variety of
hybrid methods also exist 16, 17, 18, 19. Hybrid methods have been successful
in some applications. For example, gDDR simulations 16 showed agreement with
experimental noise amplitude in CBCT with a relative difference 3.8%. However,
these methods have some limitations. The methods were either not validated
against commercial CBCT scanners for realistic voxelized phantoms or take
hours to generate a CBCT image without a high performance computing cluster.

Here we experimentally validate Fastcat, an efficient hybrid CBCT simulator of
both kV and MV CBCT imaging, previously validated by means of MC simulations
20. This simulation platform is capable of simulating both kV and MV
commercial CBCT scanners and takes a few minutes to run a CBCT simulation with
the same number of projections as a commercial scanner on a desktop computer
with a single GPU. A comprehensive evaluation of image quality metrics is
performed to demonstrate the accuracy of the method. This platform enables
rapid evaluation of new CBCT protocols.

##  II. Materials and Methods

###  II.A. Experimental Data Acquisition

Figure 1: A schematic of the Truebeam linac and CTP404 module. The kV and MV
imaging systems are shown in blue and black, respectively. Dotted lines
outline where the beams would be generated. The phantom displayed is used to
represent the phantom geometry. Thus, the phantom colormap does not correspond
to any material property. The schematic is not to scale.

Fastcat was validated for two different experimental CBCT data sets acquired
on a Truebeam STx linac (Varian Medical Systems, Palo Alto, CA).

####  II.A.1. kV CBCT

An experimental kV CBCT image was acquired using the Varian Truebeam kV Planar
Imager (which we also refer to as the OBI). The kV CBCT image was acquired
using a Varian GS-1542 x-ray tube with a 100 kVp x-ray beam. The spectra was
filtered by 2.7 mm aluminum inherent filtration as well as a 0.89 mm titanium
beam hardening filter. An aluminum bowtie filter was also used with a minimum
thickness of 1.53 mm and a maximum thickness of 27.42 mm. The detector was a
PaxScan 4030CB (Varex Imaging Corporation) flat panel detector with an anti-
scatter grid (ASG). The ASG was the default model with a grid ratio of 10,
septal thickness of 0.036 mm, and line density of 60 lines/cm. The detector
itself was a 0.6 mm columnar CsI detector with a fill factor of 70% and an
amorphous silicon readout pitch of 194 $\mu$m. The scan was acquired in the
flouro-mode with pixels binned to 388 $\mu$m. Thus, the detector had a pixel
matrix of 1024$\times$768, with a total detector size of 397$\times$298 mm. A
total of 887 views of the phantom were acquired at equally spaced angles over
a 360 degree rotation. The scan CTDIvol was 21.1 mGy. The CBCT was acquired in
developer mode and the reconstruction was performed using the FDK algorithm 21
with a ram-lak filter. The reconstruction kV CBCT image size was
512$\times$512 pixels with isotropic voxel dimensions of 0.391 mm3.

####  II.A.2. MV CBCT

The MV CBCT was acquired using the 6 MV therapy beam with the default tungsten
target and a flattening filter. The detector used was a Varian as1200 GOS
detector with a pixel pitch of 0.392 mm. Optical properties of the detector
are described in more detail in the work of Shi et al. 22. The detector had a
pixel matrix of 1024$\times$768, with a total detector size of 401$\times$301
mm. A total of 493 views of the phantom were taken at equally spaced angles
over a 360 degree rotation. The scan was acquired with a dose of 300 MU. A
high dose was used to ensure adequate image quality from the low quantum
efficiency GOS detector. The CBCT was acquired in developer mode and the
reconstruction was performed using the FDK algorithm with a ram-lak filter.
The reconstructed CBCT image size was 512 $\times$ 512 pixels with isotropic
voxel dimensions of 0.391 mm3. An fft-wavelet ring artifact reduction was used
to reduce artifacts in the presented image 23. Ring artifact reduction was not
used in the analysis since the dampening of high wavelet frequencies can
reduce image noise.

###  II.B. Fastcat Overview

The goal of Fastcat is to simulate CBCT with the highest computational
efficiency without compromising accuracy. Fastcat’s methodology is to separate
a MC simulation into primary radiation and scatter. The primary radiation is
attenuated analytically while the scatter is simulated in MC. The scatter is
then curve fit to remove noise. Scatter is only calculated once per phantom
and stored in Fastcat. To allow for arbitrary polyenergetic beams the primary
and secondary radiation are both simulated at 18 discrete energies between 10
keV to 6 MeV. To form the image the primary radiation and scatter are weighted
and combined. These weights are determined by the input spectrum and the
detector energy response. A detailed description of Fastcat can be found in
the previous work of O’Connell and Bazalova-Carter 20. Here we will focus on
the descriptions of additional Fastcat features that were included in this
work.

Fastcat simulations were performed on a linux desktop computer with 16 GB
memory a Nvidia GeForce RTX 2070 GPU and eight Intel Skylake CPUs.

####  II.B.1. Phantom

The phantom used in this study was the CTP404 sensitometry module from a
Catphan 504 phantom (The Phantom Laboratories, Salem, NY). The 20-cm diameter
sensitometry module contains 1.2-cm diameter inserts filled with air, acrylic,
low-density polyethylene (LDPE), teflon, polystyrene, delrin and
polymethylpentene (PMP). The phantom was positioned with the center of the
CTP404 module at isocenter.

###  II.C. Fastcat CBCT Simulations

The kV Fastcat simulation used as input an analytical x-ray source generated
in xpecgen 24 with an anode angle of 14∘ and a tube voltage of 100 kVp. The
source was subject to additional analytical filtration of 2.7 mm of aluminum
and a 0.89 mm titanium beam hardening filter. The detector modelled in the
simulation was a columnar CsI detector with optical properties based on the
work of Freed et al. 25. The columnar CsI was 0.6 mm thick with a fill factor
of 70% and a pixel pitch of 384 $\mu$m.

The MV Fastcat simulation source was a 6 MV Truebeam phase-space file provided
by Varian. This phasespace was binned by photon energy to provide a photon
spectra for input into Fastcat. The detector used in the simulation was
modelled as a Varian as1200 detector using the optical properties from Shi et
al 22. This detector is the same as described in previous work 20. The GOS
scintillator was 0.29 mm thick with a pixel pitch of 392 $\mu$m.

###  II.D. Additional Fastcat Modelling

####  II.D.1. Bowtie and Flattening Filters

Additional models in Fastcat are applied to model some features of the scans.
The kV bowtie filter was known to have a minimum thickness of 1.53 mm aluminum
and a maximum thickness of 27.3 mm aluminum. The proprietary bowtie filter
shape was unknown and estimated from the shape of a kV OBI air scan. With
Fastcat’s model of the kV imager spectra and the energy response of the
detector, a weighted attenuation coefficient of aluminum was calculated for
the kV OBI. This attenuation coefficient was then used to estimate the
thickness of aluminum filtration present at each pixel. While an additional
thickness of 1.53 mm of aluminum was added to each pixel to account for the
base thickness of the bowtie filter. The maximum thickness of aluminum was
calculated to be 28.3 mm using this method. This thickness is seen to be
nearly equivalent to the specified thickness plus an additional factor
accounting for a non-perpendicular path through the filter. Fastcat’s kV beam
was then filtered by the calculated amount of aluminum at each pixel during
simulation. Additionally, the MC scatter for the kV image was recalculated to
include the bowtie filter’s effect on scatter. To account for the heel effect
in the beam, tungsten filtration in the shape of a linear ramp increasing in
the cathode-anode direction with a maximum height of 0.1 mm was added.

The thickness of the MV flattening filter was also estimated. In this case,
the Fastcat beam was attenuated by a gaussian tungsten flattening filter. The
height and standard deviation were chosen such that a profile through the
reconstructed CBCT best matched experimental CBCTs.

####  II.D.2. Anti-scatter Grid

To correctly model the kV OBI a model for the anti-scatter grid (ASG) was
introduced. The model is based on measurements made by Wiegert et al. 26
Primary transmission factor and scatter transmission factors from the 44r10
ASG discussed in the paper were used. These factors are 0.76 and 0.37,
respectively. Primary fluence reaching the detector in the Fastcat simulation
was multiplied by the primary transmission factor. The scatter transmission
factor was used to separate the scatter into two portions handled by the
simulation as either accepted or attenuated, respectively. The first portion
was accepted by the detector while the second portion was assumed to be
incident on the ASG at a 12 degree angle. Where 12 degrees was the mean angle
of scatter incidence from Weigert et al.’s work. This second portion was
filtered by 173 $\mu$m of lead, which is the path length through the 36 $\mu$m
lamella at an angle of 12∘. The number of particles in these two portions were
calculated such that the total Fastcat scatter transmission factor agreed with
the theoretical.

####  II.D.3. Virtual Phantom

A virtual version of the Catphan phantom was created. The virtual phantom was
voxelized into 1024 pixels horizontally (h) 1024 pixels vertically (v) with
voxel dimensions of 0.195 mm (h), (v) and a slice thickness of 0.313 mm.
Materials linear attenuation coefficients were calculated using the material
compositions and densities in the Catphan documentation. Densities of Delrin
and the inner and outer body materials of the phantom were estimated as data
was not available on their exact composition. Delrin’s composition was
estimated to match the attenuation coefficient available on the manufacturer
site. The two body materials were estimated using the composition of acrylic
and densities that would give the materials the relative electron densities
reported in Star-Lack et al. 27.

###  II.E. Dose Comparison

A validation of dose linearity was performed. Four CBCTs at increasing doses
were acquired experimentally for both the kV and the MV setups. Doses were
measured using machine CTDI for the kV CBCTs with CTDIs of 5.27, 7.03, 10.55,
and 21.1. Four MV CBCTs were acquired with machine MU values which were 75,
100, 150, and 300 MU. The dose in Fastcat was adjusted such that the average
CNR over all inserts in the Catphan phantom agreed with that of the
experimental CBCT with the lowest dose averaged. This Fastcat dose was then
multiplied by factors of 1.5, 2, and 4. Agreement between average CNR was then
compared between these Fastcat simulations and the experimental CBCTs.

An MC validation of the mean phantom dose was performed. Mean phantom dose was
calculated in Topas 28 using an MC model of the Catphan phantom with
2$\times$108 initial photons. This dose was compared to the dose calculated in
Fastcat for the same number of photons. These simulations used the Geant4
Penelope physics list and a particle range cutoff of 5$\mu$m. Simulations were
run on a linux desktop computer with 8 Intel Skylake CPUs. No variance
reduction techniques were used. Two simulations were run in total, one with
the 6 MV Varian phasespace file and one with an analytical 100 kVp x-ray
spectrum filtered by 2.7 mm of aluminum, 0.89 mm titanium, and an aluminum
bowtie filter. Both simulation measured the mean phantom dose to virtual
Catphan 504 phantom.

###  II.F. Validation Metrics

####  II.F.1. Detector MTF

Detector modulation transfer function (MTF) was measured using a slanted
virtual slit of 0.3 mm in Fastcat. The method of measuring the detector MTF is
discussed in previous work 20. Detector MTF was compared to experimental
results from previous works of Howansky et al. and Shi et al. 29, 22.

####  II.F.2. CBCT Contrast and CNR

To validate Fastcat, image quality was compared between Fastcat and
experimental images. Both images were first converted to Hounsfield Units (HU)
by subtracting the CT value of water in the image and scaling by the
difference between water and air regions in the image.

$HU=1000\frac{im-\mu_{water}}{(\mu_{water}-\mu_{air})}$ (1)

Where $im$ is the reconstructed image and $\mu_{water}$ and $\mu_{air}$ are
the linear attenuation coefficients of water and air, respectively. For
Fastcat, water values were taken from a separate scan of a water phantom.
Contrast was compared using regions of interest (ROIs) in each insert of the
CTP404 sensiometry module. Regions of interest were made slightly smaller than
the inserts to avoid partial volume effects. The mean value over sixteen
slices was used as the contrast value while the standard deviation of the
contrast in the sixteen slices was used as an estimate of the standard
deviation. Contrast to noise ratio (CNR) was measured using the same ROIs. The
standard deviation of each ROI was used as an estimate of the noise. Contrast,
again, was measured against water, the CNR was then

$CNR=\frac{\mu_{ROI}-\mu_{water}}{(\sigma_{ROI}^{2}+\sigma_{water}^{2})^{1/2}}$
(2)

Where $\mu_{ROI}$ and $\sigma_{ROI}$ are the mean and standard deviation of
each ROI. CNR was measured in each slice over sixteen slices, with the mean
value used as a final estimate of the CNR. The standard deviation of the
sixteen slices was used as an estimate of the standard deviation of the CNR.

##  III. Results

###  III.A. CBCT Image Comparison

Figure 2: A qualitative comparison between experimental CBCTs and Fastcat
simulations averaged over 16 slices and for one CBCT slice, respectively. The
relative electron densities of the inserts are shown in black. The mean of the
ROI shown in red are plotted in Figure 3. An arrow indicates a crescent
artifact from mechanical misalignment. Window (W) and level (L) of 1000/300.

Experimental and Fastcat simulated CBCTs were compared. Figure 2 shows a
comparison of 100 kVp and 6 MV experimental and Fastcat images, respectively.
Qualitatively, the experimental and Fastcat-simulated CBCT are similar. The
effect of the bowtie filter can be seen in the 100 kVp CBCTs which are
qualitatively uniform. The effect of the flattening filter can be seen in the
6 MV images where beam hardening artifacts can be seen.

Figure 3: Profiles over the ROI marked in red in Figure 2.

Different features can be seen in the averaged CT images in Figure 2. The 100
kVp image shows a radial uniformity thanks to the presence of the bowtie
filter. However, the image is not completely uniform. The bowtie filter ends
just inside the lower density outer layer of the Catphan. This causes non-
uniformity in the phantom in this lower density region. A difference between
the experimental and Fastcat kV images is a large crescent artifact denoted by
a white arrow in Figure 2. The effect of this artifact can also be seen at the
edges of the kV profile in Figure 3. In the Varian documentation such
artifacts are said to be related to mechanical misalignment between the
detector, x-ray tube and bowtie filter. Since Fastcat does not account for
this sort of artifact it is not present in the Fastcat image.

In the 6 MV image, the flattening filter causes the beam to be harder in the
center of the phantom than the periphery. This leads to cupping in the center
of the phantom as seen in both images. Some difference is seen between MV
profiles in the center of the phantom in Figure 3. This difference may be from
ring artifacts or due to the estimation of the flattening filter as Gaussian.

Overall, the contrast is visibly similar in between both sets of images in
Figure 2. The 100 kVp image has high contrast, clearly differentiating all of
the inserts while the 6 MV image has low contrast reflecting the lower number
of photo-electric interactions at this energy.

Figure 4: (L) A comparison is demonstrated between the experimental kV and MV
CBCTs and the Fastcat simulations in terms of HU values in an experimental and
virtual CTP404 module, respectively. (R) Comparison of detector MTFs
calculated by Fastcat and based on experimental data presented by Howansky et
al. and Shi et al. 22 for the kV and MV detector, respectively.

###  III.B. Contrast and MTF

Table 1: kV and MV HU values in CTP404 module, values are in HU while standard deviations are in parentheses.  | PMP | LDPE | Polystyrene | Acrylic | Delrin | Teflon
---|---|---|---|---|---|---
kV Exp. | -204 (91) | -102 (102) | -48 (92) | 135 (91) | 357 (105) | 995 (87)
kV Fastcat | -192 (103) | -104 (91) | -62 (93) | 133 (97) | 356 (102) | 1005 (102)
Abs. Diff. | 12 | 2 | 14 | 2 | 1 | 10
MV Exp. | -70 (124) | 14 (136) | 74 (131) | 219 (129) | 397 (121) | 847 (115)
MV Fastcat | -73 (129) | 17 (126) | 65 (135) | 222 (124) | 398 (128) | 850 (144)
Abs. Diff. | 3 | 3 | 1 | 3 | 1 | 3

Looking quantitatively at these images, there is also close agreement. In
Figure 4, HU values are compared between experimental and Fastcat CBCT images.
The HU values in the inserts are shown in Table 1. HU values for the kV and MV
images agreed within 14 and 9 HU values, respectively. The kV HU values were
all within the range defined in the Catphan documentation. kV and MV detector
MTF was also compared to measurements made by Shi et al. and Howansky et al.,
respectively. The measurements for the CsI detector were within 4.2% of
measurements by Howansky et al. with an average RMSE of 1.7%. MTF deviated
mostly at frequencies between 0.5-1.3 mm-1 while agreeing closely at larger
spatial frequencies. The MTF measurements for the GOS detector were within
2.5% of measurements by Shi et al. agreeing well over all spatial frequencies
with an average root mean squared error (RMSE) of 0.5% and a slightly higher
MTF at spatial frequencies between 0.5-1.0 mm-1.

###  III.C. CNR and Dose

CNR was examined in each insert for each of the images seen in Figure 2. The
CNR results are shown in Figure 5. CNR values are summarized in Table 2. CNR
agreed within 0.4 and 0.2 for the kV and MV images, respectively. CNR as a
function of dose was examined as seen in Figure 5. Fastcat CNR values were
seen to have average RMSE of 2.6% and 1.4% for kV and MV images, respectively.
RMSE between experimental and Fastcat CNRs as a function of dose were seen to
be 1.19% and 1.22%, respectively.

Table 2: kV and MV CNRs in CTP404 module, standard deviations are in parentheses.  | PMP | LDPE | Polystyrene | Acrylic | Delrin | Teflon
---|---|---|---|---|---|---
kV Exp. | 6.1 (.55) | 6.7 (.62) | 6.9 (.72) | 7.9 (.50) | 8.9 (.56) | 12.0 (1.11)
kV Fastcat | 6.5 (.57) | 6.9 (.54) | 7.1 (.74) | 7.9 (.46) | 9.1 (.64) | 12.0 (.88)
Abs. Diff. | 0.4 | 0.2 | 0.2 | 0.0 | 0.2 | 0.0
MV Exp. | 6.2 (.34) | 6.7 (.35) | 7.0 (.70) | 7.8 (.56) | 8.7 (.59) | 11.0 (.74)
MV Fastcat | 6.4 (.60) | 6.9 (.56) | 7.1 (.76) | 7.8 (.65) | 8.9 (.66) | 10.8 (.55)
Abs. Diff. | 0.2 | 0.2 | 0.1 | 0.0 | 0.2 | 0.2
Figure 5: (L) A comparison is shown between the Varian Truebeam kV and MV
CBCTs and the Fastcat simulated CBCTs in terms of CNR in an experimental and
virtual CTP404 module, respectively. (R) CNR is compared between Truebeam
CBCTs and Fastcat CBCTs as a function of dose.

###  III.D. Dose and Speed

Dose was compared between MC simulations and Fastcat. Dose was calculated for
both the kV and the MV CBCT aquisitions. The mean doses to the entire Catphan
for the kV beam were 0.416 and 0.426 $\mu$Gy for the MC and Fastcat
calculation, respectively. The mean doses to the Catphan for the MV beam were
7.31 and 7.19 $\mu$Gy for the MC and Fastcat calculation, respectively. Thus,
there were differences of 2.4% and 1.6% between Fastcat and MC dose
calculations for kV and MV, respectively.

Simulation times were recorded for four Fastcat simulations to estimate the
relationship between computation time and the number of projections. Results
are shown in Figure 6. Computation time scaled linearly with the number of
projections for the MV and kV Fastcat simulations with both linear fits having
R2 values above 0.999. kV simulations are faster, scaling at 0.37 s/projection
while MV simulations scaled at 0.55 s/projection. kV simulations are seen to
be more computationally efficient since only ten discrete energies are
simulated between 10 and 100 keV while the MV simulation used 18 energies
between 10 keV and 6 MeV.

Figure 6: A time comparison as a function of the number of projections in a
Fastcat CBCT simulation.

##  IV. Discussion

We present and perform a two energy validation of Fastcat, a free open-source
CBCT simulator against CBCTs acquired on a Varian Truebeam linac. This
simulator performed simulations of high-resolution voxel based phantoms on a
desktop computer with 16 GB of memory and an Nvidia RTX 2070 GPU. Simulation
times were on the order of a minute to simulate the images used in the
validation. Simulated CBCT images demonstrate unprecedented accuracy for their
speed, enabling novel application of CBCT simulation in research and clinical
workflows. A comparison of Fastcat to four other simulation tools is
discussed. DukeSim, CatSim, VOXSI, and gDDR were selected for comparison since
they accept voxelized phantoms, use analytical methods to reduce simulation
times, and were validated with experimental data.

Compared to the validation of other CT and CBCT simulators, the experimental
validation of Fastcat is exceptional in its comprehensiveness and accuracy.
The two energy validation presented, using very different imaging systems
mitigates possible overfitting resulting from a validation on a single scanner
as was conducted for DukeSim, CatSim, and gDRR. VOXSI provides an experimental
validation at three tube potentials, however, it uses the same source and
detector in each simulation. This might lead to uncertainty in extrapolating
the simulator to different detectors and sources, as simulation parameters may
be unintentionally specific to the one setup.

Fastcat was experimentally validated with the ubiquitous Catphan 504 phantom
and standard imaging metrics such as HU values and CNR. Of the simulation
tools discussed, only VOXSI is experimentally validated using a common phantom
while other tools were validated using only a single material. VOXSI
simulations were within 42 HU of a Gammex RMI 467 tissue characterization
phantom (Gammex, Middleton, WI), while Fastcat which was within 14 HU of a
comparable Catphan CTP404 module. DukeSim displayed simulated CT images of a
Mercury phantom 30, however neither contrast nor CNR comparisons were shown.
This Fastcat validation presents CNR agreement to experimental data with an
average RMSE of 2.6% and 1.4% in the CTP404 module inserts for kV and MV
images, respectively.

Unique among the simulation tools discussed, Fastcat demonstrates agreement
with detector MTF. Values were within 4.2% and 2.5% of measurements for CsI
and GOS detectors, respectively. CatSim and DukeSim use a simple estimation of
detector spatial resolution based on neighbouring crosstalk. In reality,
scintillating detectors have crosstalk that extends further than their nearest
neighbors. Fastcat accounts for this based on the fastEPID framework 31 which
found an 81 $\times$ 81 pixel OSF was necessary to simulate an accurate
detector response. VOXSI and gDRR do not include simunelations of the spatial
resolution.

Computational efficiency was paramount in Fastcat’s methodology. As such,
Fastcat’s computational efficiency compares favorably to other CT and CBCT
simulation tools. Direct comparisons currently remain unfeasible as simulation
times depend on phantom resolution, processors, noise, and the number of
projections, variables which are not always presented in the literature.
Nevertheless, qualitative comparisons can be made by examining reported
simulation times: gDDR and Fastcat are estimated to have similar simulation
speeds, 360 projections of a gDRR high resolution anthropomorphic head phantom
took 95.3 s on an Nvidia GTX 590 GPU. Analogously, Fastcat took 28 seconds to
simulate 360 views of a similar resolution Catphan 504 phantom with a more
powerful Nvidia RTX 2070 GPU. VOXSI is less computationally efficient than
Fastcat, taking 45 seconds to simulate a CT with 512 $\times$ 512 $\times$ 1
voxels, 1024 detectors, and 780 projections, on two Intel Xeon 2.67 GHz
processors. Simulations in DukeSim invoke a GPU MC tool to calculate scatter,
the scatter calculation speed and total calculation speed thus depend on the
desired noise. A sample DukeSim CT simulation with a phantom of 1900 $\times$
1900 $\times$ 1000 voxels, 6912 projection images, and 47104 detectors took
about 10 minutes on four Nvidia Titan Xp GPUs. In this case the simulation of
such a large phantom and the added aspect of the MC noise is considered too
different from Fastcat to compare computational efficiencies.

Fastcat introduces a new CBCT simulation method that could be a useful
research tool. While MC simulations can agree with experimental results to the
level shown in this work, they are computationally expensive. Fastcat is
computationally efficient such that imaging settings can be quickly explored.
Fastcat also contains a complex enough simulation method to agree closely with
experimental results in terms of contrast and CNR, as seen in this work.
Fastcat could be used as a quick and accurate exploration tool to examine the
benefit of many CBCT imaging protocols. Further confirmation of promising
protocols could then be performed using established MC and demonstrated with
experimental methods.

Fastcat could also be used in a clinical setting: A Fastcat model of a given
linac can be constructed such that it agrees with clinical CBCT quality
assurance (QA) through the use of the virtual Catphan phantom. This can be
done by adjusting the beam energy, focal spot size, detector material, and
other parameters. Unlike with MC models, due to its fast simulation times
Fastcat could build a scanner specific, virtual clinical datasets of hundreds
of CBCT volumes of an anthropomorphic phantom can be constructed overnight.
These datasets could be used to estimate the correct imaging settings for a
given clinical situation. Such as estimating the CTDI necessary to achieve a
given soft tissue contrast for a patient of a specific size. Likewise, Fastcat
could be used in conventional workflows. Fastcat, with the addition of a
mapping between experimental DICOM voxel values into material compositions
like that of Schneider et al. 32, could be used to simulate an MV CBCT from a
kV CBCT, this would allow generation of MV beam’s eye view images for the
validation of a given treatment quickly from a given patient-specific CBCT
acquired by the OBI.

Fastcat is unique in its simulation speed and accuracy. No other analytical
CBCT simulator has shown such close agreement with experimental results, while
no MC simulator has similar computational speed. Fastcat is limited, however,
due to its dependence on pre-calculated data. Detector optical spread
functions (OSFs), MV beams, and scatter kernels for different phantoms are
stored in Fastcat. A simulation that uses a different detector, MV beam, or
phantom than those available in Fastcat may need the additional step of
simulating the necessary pre-calculated data in an MC application and be
subject to new experimental validation. As an open-source simulator, ideally
this new pre-calculated data could be uploaded by groups who use Fastcat,
making a variety of commercial setups available and improving the simulator as
a whole. Currently however, the type of accuracy shown in this work is only
available for the Varian Truebeam EPID and Varian Truebeam OBI systems.

##  V. Conclusion

In this work we demonstrate experimental agreement between the Fastcat CBCT
simulator and a Varian Truebeam linac. Fastcat 100 kVp and 6 MV CBCT images
agreed closely with experimental kV and MV CBCT images, respectively. Close
agreement was seen between experimental and Fastcat’s HU values, detector MTF,
dose, and CNR. CBCT simulation took 71 and 72 seconds for the kV and MV CBCT,
respectively. We demonstrate the accuracy of a fast, free, open-source CBCT
simulator that is well suited for the evaluation of new CBCT imaging
protocols.

##  VI. Acknowledgments

The authors would like to acknowledge Chelsea Dunning for the contribution to
the name Fastcat. We would also like to thank Marios Myronakis for sharing the
GOS MTF data with us. This research was enabled in part by support provided by
WestGrid (www.westgrid.ca) and Compute Canada Calcul Canada
(www.computecanada.ca). The work was partly funded by an NSERC Discovery Grant
and the Canada Research Chairs program.

## References

##

  * 1 D. A. Jaffray and J. H. Siewerdsen,  Cone-beam computed tomography with a flat-panel imager: Initial performance characterization,  Medical Physics 27 (2000).
  * 2 D. A. Jaffray, J. H. Siewerdsen, J. W. Wong, and A. A. Martinez,  Flat-panel cone-beam computed tomography for image-guided radiation therapy,  International Journal of Radiation Oncology Biology Physics 53 (2002).
  * 3 T. Xu, J. L. Ducote, J. T. Wong, and S. Molloi,  Dynamic dual-energy chest radiography: a potential tool for lung tissue motion monitoring and kinetic study,  Physics in Medicine and Biology 56, 1191–1205 (2011).
  * 4 C. Lee, J. L. Williams, C. Lee, and W. E. Bolch,  The UF series of tomographic computational phantoms of pediatric patients,  Medical Physics 32 (2005).
  * 5 S. X. Zhang, P. A. Heng, and Z. J. Liu,  Chinese visible human project,  Clinical Anatomy 19 (2006).
  * 6 R. Kramer, V. F. Cassola, H. J. Khoury, J. W. Vieira, V. J. De Melo Lima, and K. Robson Brown,  FASH and MASH: Female and male adult human phantoms based on polygon mesh surfaces: II. Dosimetric calculations,  Physics in Medicine and Biology 55 (2010).
  * 7 W. P. Segars, G. Sturgeon, S. Mendonca, J. Grimes, and B. M. Tsui,  4D XCAT phantom for multimodality imaging research,  Medical Physics 37, 4902–4915 (2010).
  * 8 A. Badal and A. Badano,  Accelerating Monte Carlo simulations of photon transport in a voxelized geometry using a massively parallel graphics processing unit,  Medical Physics 36, 4878–4880 (2009).
  * 9 X. Li, E. Samei, W. P. Segars, G. M. Sturgeon, J. G. Colsher, G. Toncheva, T. T. Yoshizumi, and D. P. Frush,  Patient-specific radiation dose and cancer risk estimation in CT: Part II. Application to patients,  Medical Physics 38 (2011).
  * 10 H. Jiang and H. Paganetti,  Adaptation of GEANT4 to Monte Carlo dose calculations based on CT data,  Medical Physics 31 (2004).
  * 11 S. J. Blake, P. Vial, L. Holloway, P. B. Greer, A. L. McNamara, and Z. Kuncic,  Characterization of optical transport effects on EPID dosimetry using Geant4,  Medical Physics 40, 041708 (2013).
  * 12 Y. S. Yeom, M. C. Han, C. Choi, H. Han, B. Shin, T. Furuta, and C. H. Kim,  Computation Speeds and Memory Requirements of Mesh-Type ICRP Reference Computational Phantoms in Geant4, MCNP6, and PHITS,  Health Physics 116 (2019).
  * 13 J. Bert, H. Perez-Ponce, Z. E. Bitar, S. Jan, Y. Boursier, D. Vintache, A. Bonissent, C. Morel, D. Brasse, and D. Visvikis,  Geant4-based Monte Carlo simulations on GPU for medical applications,  Physics in Medicine and Biology 58, 5593–5611 (2013).
  * 14 B. van der Heyden, L. E. Schyns, M. Podesta, A. Vaniqui, I. P. Almeida, G. Landry, and F. Verhaegen,  VOXSI: A voxelized single- and dual-energy CT scenario generator for quantitative imaging,  Physics and Imaging in Radiation Oncology 6, 47–52 (2018).
  * 15 G. Landry, F. deBlois, and F. Verhaegen,  ImaSim, a software tool for basic education of medical x-ray imaging in radiotherapy and radiology,  Frontiers in Physics 1 (2013).
  * 16 X. Jia, H. Yan, L. Cerviño, M. Folkerts, and S. B. Jiang,  A GPU tool for efficient, accurate, and realistic simulation of cone beam CT projections,  Medical Physics 39 (2012).
  * 17 B. De Man, S. Basu, N. Chandra, B. Dunham, P. Edic, M. Iatrou, S. McOlash, P. Sainath, C. Shaughnessy, B. Tower, and E. Williams,  CatSim: a new computer assisted tomography simulation environment,  in Medical Imaging 2007: Physics of Medical Imaging, volume 6510, 2007.
  * 18 N. Li, H. X. Zhao, S. H. Cho, J. G. Choi, and M. H. Kim,  A fast algorithm for voxel-based deterministic simulation of X-ray imaging,  Computer Physics Communications 178 (2008).
  * 19 E. Abadi, B. Harrawood, S. Sharma, A. Kapadia, W. P. Segars, and E. Samei,  DukeSim: A realistic, rapid, and scanner-specific simulation framework in computed tomography,  IEEE Transactions on Medical Imaging 38, 1457–1465 (2019).
  * 20 J. O’Connell and M. Bazalova-Carter,  FastCAT: Fast Cone Beam CT (CBCT) Simulation,  arXiv 2011.04736 (2020).
  * 21 L. A. Feldkamp, L. C. Davis, and J. W. Kress,  Practical cone-beam algorithm,  Journal of the Optical Society of America A 1, 612 (1984).
  * 22 M. Shi, M. Myronakis, Y. H. Hu, D. Morf, J. Rottmann, and R. Berbeco,  A Monte Carlo study of the impact of phosphor optical properties on EPID imaging performance,  Physics in Medicine and Biology 63, 165013 (2018).
  * 23 B. Münch, P. Trtik, F. Marone, and M. Stampanoni,  Stripe and ring artifact removal with combined wavelet-Fourier filtering,  EMPA Activities (2009).
  * 24 G. Hernández and F. Fernández,  xpecgen: A program to calculate x-ray spectra generated in tungsten anodes,  The Journal of Open Source Software 1, 62 (2016).
  * 25 M. Freed, S. Miller, K. Tang, and A. Badano,  Experimental validation of Monte Carlo (MANTIS) simulated x-ray response of columnar CsI scintillator screens,  Medical Physics 36, 4944–4956 (2009).
  * 26 J. Wiegert, M. Bertram, D. Schaefer, N. Conrads, J. Timmer, T. Aach, and G. Rose,  Performance of standard fluoroscopy antiscatter grids in flat-detector-based cone-beam CT,  in Medical Imaging 2004: Physics of Medical Imaging, volume 5368, 2004.
  * 27 J. Star-Lack, D. Shedlock, D. Swahn, D. Humber, A. Wang, H. Hirsh, G. Zentai, D. Sawkey, I. Kruger, M. Sun, E. Abel, G. Virshup, M. Shin, and R. Fahrig,  A piecewise-focused high DQE detector for MV imaging,  Medical Physics 42, 5084–5099 (2015).
  * 28 J. Perl, J. Shin, J. Schumann, B. Faddegon, and H. Paganetti,  TOPAS: An innovative proton Monte Carlo platform for research and clinical applications,  Medical Physics 39, 6818 (2012).
  * 29 A. Howansky, A. R. Lubinsky, S. K. Ghose, K. Suzuki, and W. Zhao,  Direct measurement of Lubberts effect in CsI:Tl scintillators using single x-ray photon imaging,  in Medical Imaging 2017: Physics of Medical Imaging, volume 10132, 2017.
  * 30 J. M. Wilson, O. I. Christianson, S. Richard, and E. Samei,  A methodology for image quality evaluation of advanced CT systems,  Medical Physics 40 (2013).
  * 31 M. Shi, M. Myronakis, Y.-H. Hu, M. Jacobson, M. Lehmann, R. Fueglistaller, P. Huber, P. Baturin, A. Wang, D. Ferguson, T. Harris, D. Morf, and R. Berbeco,  A novel method for fast image simulation of flat panel detectors.,  Physics in medicine and biology 64, 095019 (2019).
  * 32 W. Schneider, T. Bortfeld, and W. Schlegel,  Correlation between CT numbers and tissue parameters needed for Monte Carlo simulations of clinical dose distributions,  Physics in Medicine and Biology 45 (2000).