# Configuration Validation with Large Language Models

Xinyu Lian  University of IllinoisUrbana-ChampaignILUSA<EMAIL_ADDRESS>,
Yinfang Chen  University of IllinoisUrbana-ChampaignILUSA
<EMAIL_ADDRESS>,  Runxiang Cheng  University of IllinoisUrbana-
ChampaignILUSA<EMAIL_ADDRESS>,  Jie Huang  University of
IllinoisUrbana-ChampaignILUSA<EMAIL_ADDRESS>,  Parth Thakkar  Meta
Platforms, Inc.Menlo ParkCAUSA<EMAIL_ADDRESS>and  Tianyin Xu  University of
IllinoisUrbana-ChampaignILUSA<EMAIL_ADDRESS>
###### Abstract.

Misconfigurations are the major causes of software failures. Existing
configuration validation techniques rely on manually written rules or test
cases, which are expensive to implement and maintain, and are hard to be
comprehensive. Leveraging machine learning (ML) and natural language
processing (NLP) for configuration validation is considered a promising
direction, but has been facing challenges such as the need of not only large-
scale configuration data, but also system-specific features and models which
are hard to generalize. Recent advances in Large Language Models (LLMs) show
the promises to address some of the long-lasting limitations of ML/NLP-based
configuration validation techniques. In this paper, we present an exploratory
analysis on the feasibility and effectiveness of using LLMs like GPT and Codex
for configuration validation. Specifically, we take a first step to
empirically evaluate LLMs as configuration validators without additional fine-
tuning or code generation. We develop a generic LLM-based validation
framework, named Ciri, which integrates different LLMs. Ciri devises effective
prompt engineering with few-shot learning based on both valid configuration
and misconfiguration data. Ciri also validates and aggregates the outputs of
LLMs to generate validation results, coping with known hallucination and
nondeterminism of LLMs. We evaluate the validation effectiveness of Ciri on
five popular LLMs using configuration data of six mature, widely deployed
open-source systems. Our analysis (1) confirms the potential of using LLMs for
configuration validation, (2) understands the design space of LLM-based
validators like Ciri, especially in terms of prompt engineering with few-shot
learning, and (3) reveals open challenges such as ineffectiveness in detecting
certain types of misconfigurations and biases to popular configuration
parameters.

∗Co-primary authors

††copyright: none

##  1\. Introduction

Modern cloud and web services evolve rapidly and deploy hundreds to thousands
of configuration changes to production systems on a daily basis (tang:sosp:15,
; maurer:15, ; barroso:2018, ; sherman:05, ; mehta:2020, ; beyer:2018, ;
sayagh18, ). For example, at Meta/Facebook, thousands of configuration changes
are committed daily, outpacing the frequency of source-code changes
(tang:sosp:15, ; maurer:15, ). Other cloud services such as Google and
Microsoft also frequently deploy configuration changes (mehta:2020, ;
barroso:2018, ; beyer:2018, ). Such rapid configuration changes inevitably
lead to misconfigurations, resulting in system failures. Today,
misconfigurations are among the dominating causes of production incidents
(maurer:15, ; gunawi:16, ; yin:sosp:11, ; xu:15, ; oppenheimer:03, ;
kendrick:12, ; rabkin:13, ; sayagh18, ). For example, misconfiguration is the
second largest root-cause category of service disruptions at a main Google
production service (barroso:2018, ).

To detect misconfigurations before deployment, today’s configuration
management systems typically employ the “configuration-as-code” paradigm and
enforce continuous configuration validation, ranging from static validation,
to configuration testing, and to manual review and approval. The configuration
is first checked by validation code (aka validators) based on predefined
correctness rules (tang:sosp:15, ; raab:17, ; baset:middleware:2017, ;
huang:15, ; Leuschner:2017, ; liao:18, ; NadiTSE2015, ; zhang:oopsla:21, ); in
practice, validators are written by engineers when introducing configuration
parameters. After passing validators, configuration changes are then tested
together with the code to ensure expected program behavior under the changed
configuration (sun:osdi:20, ; xu:19, ). Lastly, the configuration changes go
through the same process as source-code review, where the change, commonly in
the form of a configuration file “diff”, is reviewed before production
deployment.

The aforementioned configuration validation pipeline either relies on manual
inspection to spot misconfigurations in the configuration file diffs, or
requires significant engineering efforts to implement and maintain validators
or test cases. However, these efforts are known to be costly and
incomprehensive. For example, despite the fact that mature projects all
include extensive configuration checks, recent work (xu:sosp:13, ; xu:osdi:16,
; li:18, ; keller:dsn:08, ; li:issta:21, ; wang:ase:23, ; wangt:icse:23, )
repeatedly shows that existing checks are far from sufficient to catch
misconfigurations. The reasons are twofold. First, with large-scale systems
exposing hundreds to thousands of configuration parameters (xu:fse:15, ),
implementing validators (or test cases) for every parameter becomes a
significant overhead. Recent studies (xu:sosp:13, ; tang:sosp:15, ) report
that many parameters are not covered by existing checks, even in mature
software projects with many years of development history. Second, it is
nontrivial to validate each individual parameter, which could have many
different correctness properties, such as type, range, semantic meaning,
dependencies with other parameters, etc.; encoding each of them as validators
could be laborious and error-prone, not to mention the high maintenance cost
due to configuration-related software evolution (zhang:icse:14, ;
zhang:icse:21, ). These limitations also apply to the configuration tests
(sun:osdi:20, ). Compared with static validation, configuration testing is
also more time-consuming to run and more expensive in terms of computing
resources (cheng:issta:21, ).

Using machine learning (ML) and natural language processing (NLP) to detect
misconfigurations has been considered a promising approach to addressing the
above challenges of configuration validation. Compared with manually written
static validators and test cases, ML or NLP-based approaches are automatic,
easy to scale to a large number of parameters, and applicable to different
systems and environments. A number of ML/NLP-based misconfiguration detection
techniques are proposed in the literature (Bhagwan:21, ; zhang:asplos:14, ;
wang:osdi:04, ; le:06, ; xiang:atc:20, ; palatin:kdd:06, ; potharaju:vldb:15,
; wang:lisa:03, ). The key idea is to first learn correctness rules from field
configuration data (kiciman:04, ; zhang:asplos:14, ; wang:osdi:04, ; le:06, ;
palatin:kdd:06, ; wang:lisa:03, ; Bhagwan:21, ; santo:16, ; santo:17, ) or
from documents (xiang:atc:20, ; potharaju:vldb:15, ), and then use the learned
rules to detect misconfigurations in new configuration files. ML/NLP-based
approaches have achieved good success. For example, Microsoft adopted
PeerPressure (wang:osdi:04, ; huang:ndss:05, ) as a part of Microsoft Product
Support Services (PSS) toolkits. It collects configuration data in Windows
Registry from a large number of Windows users to learn statistical golden
states of system configurations.

However, ML/NLP-based misconfiguration detection has also revealed significant
limitations. First, the need of large volumes of system-specific configuration
data makes it hard to apply those techniques to outside corporations that
collect user configurations (e.g., Windows Registry (wang:lisa:03, )) or
maintain knowledge base or customer tickets (potharaju:vldb:15, ). For
example, in cloud/datacenter systems where the same set of configurations is
maintained by a small DevOps team, there is no enough information for learning
(xu:sosp:13, ). Moreover, prior ML/NLP-based detection techniques all target
specific systems/projects, and rely on predefined learning features
(palatin:kdd:06, ), templates (zhang:asplos:14, ), or models
(potharaju:vldb:15, ). As a result, it is hard to generalize them to different
systems and their configurations.

Recent advances on Large Language Models (LLMs), such as GPT (chatgpt, ) and
Codex (codex, ), show promises to address some of the long-lasting limitations
of traditional ML/NLP-based misconfiguration detection techniques.
Specifically, LLMs are trained on massive amounts of internet data, including
configuration data—configuration files in software repositories, configuration
documents, knowledge-based articles, Q&A websites for resolving configuration
issues, etc. Consequently, LLMs encode the extensive knowledge of both common
and even project-specific configuration for well-known projects. Such
knowledge can be utilized for configuration validation without the need for
manual rule engineering. Furthermore, LLMs show the capability of
generalization and reasoning (wei:23, ; huang:acl:23, ) unlike the traditional
ML approaches, and can potentially “understand” the configuration semantics.
For example, they can not only generalize that values of a port must be in the
range of [0, 65535], but also reason that a specific configuration value
represents a port (e.g., based on the parameter name and description) and thus
has to be within the range.

Certainly, LLMs have limitations. Notably, they are known to hallucinate
responses and can be nondeterministic (bang:arxiv:23, ; zhang:arxiv:23, ).
Additionally, LLMs have limited input context, which can pose challenges when
encoding extensive contexts like configuration-related code and execution
environments. Moreover, they are reported to be biased to popular content in
the training dataset. However, there are active efforts (anthropic100k, ;
nakano2022webgpt, ; wang:arxiv:23, ; liu2023trustworthy, ;
manakul2023selfcheckgpt, ) addressing these limitations, making them promising
tools.

In this paper, we present an exploratory analysis on the feasibility and
effectiveness of using LLMs like GPT and Codex for configuration validation.
Our goal is to empirically evaluate the promises of leveraging LLMs to develop
effective configuration validators and to understand the challenges. As a
first step, we empirically evaluate LLMs as configuration validators, without
additional fine-tuning or code generation. We focus on basic misconfigurations
(those violating explicit correctness constraints) which can potentially be
detected by LLMs directly. We do not target misconfigurations specific to the
execution environments or correct configuration changes triggering bugs in the
code. We discuss how to further build on this work to detect those in §7.

To this end, we develop Ciri, an LLM-empowered configuration validation
framework. Ciri takes a configuration file or a file diff as the input; it
outputs a list of detected misconfigurations along with the reasons that
explain the misconfigurations. Ciri integrates different LLMs such as Code-
Davinci-002, GPT-3.5-turbo, and GPT-4. Ciri devises effective prompt
engineering with few-shot learning based on existing configuration data.
Additionally, Ciri validates and aggregates the outputs of LLMs to generate
validation results, coping with known hallucination and nondeterminism of
LLMs. A key design principle of Ciri is separation of policy and mechanism—it
implements different mechanisms to support various policies. Ciri can serve as
an open framework for experimenting with different prompt engineering
approaches, training datasets, and validation and aggregation methods.

We study the validation effectiveness of Ciri backed by five popular LLMs
including advanced models (Code-davinci-002, GPT-3.5 Turbo, and GPT-4), and
basic models (Babbage-002, and Davinci-002). We use misconfiguration datasets
of six mature, widely deployed open-source systems (HCommon, HBase, Alluxio,
HDFS, YARN, and ZooKeeper). Our study confirms the potential of using LLMs for
configuration validation, e.g., Ciri with GPT-4 shows promising results at
both file- and parameter-level, achieving up to 0.75 and 0.56 F1-scores,
respectively. Our study also helps understand the design space of LLM-based
validators like Ciri, especially in terms of prompt engineering with few-shot
learning and voting. We find that few-shot learning using both valid
configuration and misconfiguration data as shots (examples) can significantly
enhance validation effectiveness. Specifically, labeled misconfigurations are
pivotal to validation effectiveness. Our results also reveal open challenges
such as ineffectiveness in detecting certain types of misconfigurations and
biases to popular configuration parameters. While Ciri excels in identifying
misconfigurations, it struggles with specific misconfiguration types such as
dependency violations and version-specific misconfigurations. We also observe
that the popularity of configuration parameters creates biases in validation
results, causing both false positives and false negatives. In summary, this
paper makes the following main contributions:

  * •

A new direction of configuration validation using pre-trained large language
models.

  * •

Ciri, an LLM-empowered configuration validation framework and an open platform
for followup research. Ciri is released at https://github.com/ciri4conf/ciri.

  * •

An empirical study on the effectiveness of LLM-based configuration validation
which confirms the potential of the direction, explores the design space, and
reveals open challenges.

  * •

A discussion on future endeavours such as fine-tuning, test generation to
detect more types of misconfigurations, and detecting configuration-related
bugs.

  * •

All code and data for results presented in this paper will be released with
detailed steps for reproducing our experiments.

Figure 1. Example 1 and 2 show the LLM correctly catches and reasons the
misconfigurations. Example 3 and 4 show the LLM misses a misconfiguration or
reports a valid configuration as erroneous.

##  2\. Exploratory Examples

We explore the capability of utilizing LLMs to validate configuration files
off-the-shelf. We argue that vanilla LLMs are capable of detecting
sophisticated misconfigurations. However, they are prone to both false
negatives and false positives that require further attention and handling.
Figure 1 presents four examples, two of which the LLM correctly detects the
misconfiguration, and two of which the LLM misses a misconfiguration or
reports a false alarm. These examples were generated using the Codex LLM
(Code-Davinci-002) developed by OpenAI (codex, ; chen:arxiv:21, ).

Detecting violation in configuration dependency. Extracting and validating
against dependency relationships between configuration parameters has been a
challenging task in highly-configurable systems (chen:fse:20, ; xu:15, ). LLM
has the ability to infer relations between entities from text at the level of
human experts (brown:arxiv:20, ). This ability allows LLM to infer
dependencies between parameters in a configuration file based on their
corresponding names and descriptions. By simply asking if any violation exists
in the configuration, off-the-shelf LLM can check if configured values satisfy
the extracted relations at runtime. This allows better applicability of LLMs
for validating configuration dependency when compared to prior techniques that
require manually codified rules (tang:sosp:15, ; baset:middleware:2017, ),
program analysis (chen:fse:20, ; xu:sosp:13, ), or specialized learning
(xiang:atc:20, ; potharaju:vldb:15, ).

Figure 1 (Example 1) presents a case where values of two dependent parameters
were changed (i.e., buffer.size and bytes.per.checksum). After understanding
the value relationship dependency between these two parameters, the model
determines that the change in bytes.per.checksum has violated the enforced
dependency, and provides the correct reason for the misconfiguration.

Detecting violation with domain knowledge. Written configuration validation
rules often require significant manual efforts to produce and maintain. They
are difficult to scale, due to the diverse types and functionalities of
configuration parameters. A state-of-the-art LLM is trained on a massive
amount of textual data collected from the Internet, and possesses basic
knowledge across a wide range of professional domains. An LLM thus could be
capable of understanding the definition of a configuration parameter and
reasoning with its semantics. When the LLM encounters a configuration
parameter such as IP Address, permissions, masks, it invokes the domain
knowledge specific to the properties of those parameters for user’s further
instructions, such as validation. Figure 1 (Example 2) presents a case where
an HTTP address has been misconfigured to a semantically invalid value. The
model identifies the misconfiguration, reasons that its configured value has
been out of range, and further suggests potential direction for fixing.

Missed misconfiguration and false alarm. Despite that LLMs have demonstrated
impressive performance across many tasks since its recent emergence, at the
current stage of development, however, LLMs as configuration validators are
not without errors. Examples 3 and 4 in Figure 1 show two cases where the LLM
makes mistakes in the configuration validation.

In Example 3, the configuration file has provided a description of the changed
parameter hostname.verifier, and explicitly listed the valid value options for
the parameter. However, the model is unable to identify that the parameter has
been misconfigured to an invalid, non-existent option (STRICT_I8). Example 4
is interesting — the description suggests that the parameter bloom.error.rate
ranges from 0 to 100 (percentage), whereas the actual scale is 0 to 1
(fraction). This inconsistency supposedly confuses the model making it mark
0.01 as invalid even though it is valid (1%) — a fairly reasonable mistake for
a human to make as well.

Both examples demonstrate that employing off-the-shelf LLMs as configuration
validators can result in false negatives and false positives, thereby making
the predictions less trustworthy. Incorrect validation outcomes could be
attributed towards a phenomenon in LLMs termed hallucination, which is being
actively investigated (gpt4, ). A simple explanation is that LLMs are exposed
to potentially contradictory data during training, which causes confusion to
the model at inference time.

To account for these factors, our study applies and evaluates several
mechanisms that can mitigate the impact of wrongful predictions made by LLMs
in the context of configuration validation, including few-shot learning, and
reaching validation consensus through majority voting (§3).

##  3\. Ciri: A LLM-empowered Configura-
tion Validation Framework

We develop Ciri, a LLM-empowered configuration validation framework. Ciri
takes a configuration file or a file diff as the input. It outputs a list of
detected misconfigurations along with the reasons to explain the
misconfigurations. If no misconfiguration is detected, Ciri outputs an empty
list.

Ciri now supports five LLMs (Code-Davinci-002, GPT-3.5-turbo, GPT-4,
Babbage-002 and Davinci-002). Adding a new LLM in Ciri takes a few lines of
code to adopt the LLM’s query APIs. Figure 2 gives an overview of Ciri. Ciri
turns a configuration validation request into a prompt to the LLMs (§3.1). The
prompt includes 1) the input configuration file or diff, 2) a few examples
(referred to as shots) to demonstrate the task of configuration validation,
and 3) directive question and metadata. To generate shots, Ciri uses its
database that contains labeled configuration data, including both valid
configurations and misconfigurations. To validate a configuration file, Ciri
sends the same query to the LLMs multiple times and aggregates the responses
into the final validation result (§3.2).

Ciri can be applied to any software project, even if it has no configuration
data of that project. Ciri can directly query advanced LLMs like GPT-4 with
zero shot, and achieves considerable effectiveness (Finding 2). Ciri exhibits
the ability to transfer configuration-related knowledge across projects when
using configurations from different projects as shots (Finding 4). Ciri’s
configuration validation effectiveness can also be further improved with high-
quality generated shots (Finding 3).

Figure 2. System overview of Ciri.

###  3.1. Prompt Engineering

Prompt Structure. Ciri generates a prompt that includes three elements: 1) the
content of input configuration file or file diff, 2) the shots as ValidConfig
or misconfiguration files with example questions and ground truth responses
for few-shot learning, and 3) a directive question for LLM to respond in
formatted output. Figure 3 shows an illustrative example of the prompt
generated by Ciri. It contains $N$ shots, content of the validating
configuration file, followed by the directive question.

Ciri phrases the prompting question as “Are there any mistakes in the above
configuration file for [PROJECT] version [VERSION]? Respond in a JSON format
similar to the following: …”. The [PROJECT] and [VERSION] are required inputs
of Ciri because validity of a configuration file can change by project and
project version. This prompt format enforces the LLM to respond in a unified
JSON format for automated result aggregation (§3.2). However, responses from
LLMs sometimes may still deviate from the anticipated format (bang:arxiv:23, ;
zhang:arxiv:23, ). In such cases, Ciri retries a new query to the LLM.

Few-Shot Learning. Ciri leverages the LLM’s ability to learn from examples at
inference time (referred to as few-shot learning) to improve configuration
validation effectiveness. To do so, Ciri simply inserts shots at the beginning
of each prompt. Each shot contains a configuration file, the prompting
question, and its corresponding ground truth. Figure 3 provides an example. In
Figure 3, there are $N$ shots. “Configuration File Shot #1” is the first shot,
in which the parameter file.bytes-per-checksum in the configuration file is
misconfigured. This shot also contains the prompting question (orange box) and
the corresponding ground truth (blue box).

Shot Generation and Selection. Ciri maintains a database of labeled valid
configurations and misconfigurations. It is used for generating valid
configuration shots (ValidConfig) and misconfiguration shots (Misconfig). A
ValidConfig shot specifies a set of configuration parameters and their valid
values. A valid value of a parameter can be its default value, or other valid
values used in practice. A Misconfig shot specifies a set of parameters and
their values, where only one of the parameters’ value is invalid. We provide
more details on how to generate valid/invalid configuration values in this
paper in §4.

For a configuration file or diff of a specific project, Ciri by default
generates shots using the configuration data of the same project. If Ciri’s
database does not contain configuration data for the target project, Ciri
would use available data (from other projects) to generate shots. As we will
show (§5.2), LLMs possess transferrable knowledge in configuration across
different projects.

Ciri supports various strategies to select the shot data, including randomized
selection, selecting from different configuration/misconfiguration types, and
selecting from configuration parameters similar to the validating
configuration (using cosine similarity). We did not observe major differences
in these selection strategies so Ciri uses randomized selection by default.

Addressing Token Limits. LLMs limit input size per query by the number of
input tokens. For example, the token limits for GPT-4 and GPT-3.5-turbo (the
16K variant) are 4,097 and 16,385 respectively. To navigate these constraints,
Ciri compresses the prompt if its size cannot fit the token limit. Ciri
prioritizes putting the validating configuration file and the directive
question in the prompt, then applies a number of strategies to maximize the
number of shots that can fit into the remaining token limit. If the validating
configuration file itself cannot fit into the token limit, Ciri transforms the
original file into a more compact format, e.g., transforming an XML file into
INI format. If the compressed input still cannot fit, Ciri aborts and returns
errors. In practice, real-world configuration file are small (openctest, ),
which lefts enough space to include shots. For example, prior study inspects
configuration files collected from Docker, where each file contains 1 to 18
parameters, with 8 parameters on average (openctest, ). For the extremely
large configuration file, Ciri can split it into smaller snippets, which can
be validated separately and reasoned together.

Figure 3. An example prompt generated by Ciri.

###  3.2. Result Generation

The JSON response from LLMs contains three primary fields: 1) hasError: a
boolean value indicating if a misconfiguration is detected, 2) errParameter:
an array of strings specifying the misconfigured parameter, and 3) reason: an
array of strings explaining the detected misconfiguration.

Validation against Hallucination. We use a few rules to counter the
hallucination of LLMs. For example, if hasError is False, both errParameter
and reason must be empty. Conversely, if hasError returns True, errParameter
and reason should not be empty and have the same array size. The answer to
errParameter also should not contain repeated values. If a response fails
these rules, Ciri discards it and retries until the LLM returns a valid
response.

Voting against Inconsistency. LLMs can produce inconsistent outputs in
conversation (adiwardana:arxiv:20, ), explanation (camburu:arxiv:19, ), and
knowledge extraction tasks (elazar:tacl:21, ). To mitigate inconsistency, Ciri
uses a multi-query strategy—querying the LLM multiple times using the same
prompt and aggregating the responses. When aggregated with a voting mechanism,
these responses converge towards a solution that is both representative of the
model’s understanding and more consistent than a single query. Ciri uses a
frequency-based voting strategy: the output that recurs most often among the
responses is selected as the final output (wang:arxiv:23, ).

Note that the reason field is not considered during voting due to the diverse
nature of the response. After voting, Ciri collects the reason from all
responses that are associated with the selected errParameter. The reason field
is important as it provides users with insights into the misconfiguration,
which is different from the traditional ML approaches that only provide a
binary answer with a confidence score. However, the content of reason may not
always be useful due to hallucination. Ciri clusters reasons based on TF-IDF
similarity, and picks a reason from the dominant cluster. We found that
hallucinated reasons were often avoided this way as they tended to be very
different from each other.

###  3.3. Ciri Configuration

Ciri is highly customizable, with a basic principle of separating policy and
mechanism. Users can customize Ciri via its own configurations. Table 1 shows
several important Ciri configurations and default values. The values are
chosen by pilot studies using a subset of our dataset (§4).

Table 1. System config of Ciri and their default values. Parameter | Description | Default Value
---|---|---
Model | Backend LLM. Also allows users to add other LLMs. | GPT-4
Temperature | Tradeoff between creativity and determinism. | 0.2
# Shots | The number of shots included in a prompt. | Dynamic
# Queries | The number of queries with the same prompt. | 10

##  4\. Benchmarks and Metrics

Table 2. Misconfiguration generation rules (extended from prior work (li:issta:21, )). “(Sub-)Category” list different sets of violations that can be applied to a configuration parameter to generate its misconfigured values. Category | Sub-Category | Specification | Generation Rules
---|---|---|---
Syntax | Data type | Value set = {Integer, Float, Long…} | Use a value that doesn’t belong to set
Numbers with units | Use a non-existent unit (e.g., ”nounit”)
Path | `^(\/[^\/ ]*)+\/?$` |  Use a value that violates the pattern (e.g., `/hello//world`)
URL | `[a-z]+://.*` |  Use a value that violates the pattern (e.g., `file///`)
IP Address | `[\d]{1,3}(.[\d]{1,3}){3}` |  Use a value that violates the pattern (e.g., `127.x0.0.1`)
Port | Data type, value set = {Octet} | Use a value that doesn’t belong to set
Permission | Data type, value set = {Octet} | Use a value that doesn’t belong to set
Range | Basic numeric | Valid Range constrainted by data type |  Use the values beyond the valid range (e.g., `Integer.MAX_VALUE+1`)
Bool | Options, value set = {true, false} | Use a value that doesn’t belong to set
Enum | Options, value set = {“enum1”, “enum2”, …} | Use a value that doesn’t belong to set
IP Address | Range for each octet = [0, 255] |  Use a value beyond the valid range (e.g., `256.123.45.6`)
Port | Range = [0, 65535] | Use a value beyond the valid range
Permission | Range = [000, 777] | Use a value beyond the valid range
Dependency | Control |  $(P_{1},V,\Diamond)\mapsto P_{2}$, $\Diamond\in\\{>,\geq,=,\neq,<,\leq\\}$ |  Use invalid control condition $(P_{1},V,\lnot\Diamond)$
Value Relationship |  $(P_{1},P_{2},\Diamond)$, $\Diamond\in\\{>,\geq,=,\neq,<,\leq\\}$ |  Use invalid value relationship $(P_{1},P_{2},\lnot\Diamond)$
Version | Parameter change |  $(V_{1},Pset_{1})\mapsto(V_{2},Pset_{2})$, $Pset_{1}\neq Pset_{2}$ |  Use a removed parameter in $V_{2}$ or use an added paraemter in $V_{1}$

Software Systems We evaluate six popular, open-source systems: Hadoop Common,
HBase, Alluxio, HDFS, YARN, and ZooKeeper. They all are mature, widely
deployed systems. These systems are highly configurable with a large number of
configuration parameters. Table 3 lists the version (SHA), and the total
number of parameters at that version for each system.

Configuration Dataset To evaluate the effectiveness of configuration
validation, we create new datasets for each system. First, we collect valid
configuration values based on the default configuration file from each system,
as well as configuration files from the Ctest dataset (openctest, ). The
configuration files in the Ctest dataset was collected from public Docker
images that deploy the target systems (wang:icse:23, ; xu:18, ; sun:osdi:20,
). We then generate misconfigurations of different types. The generation is
based on prior studies on misconfigurations (xu:sosp:13, ; li:18, ;
keller:dsn:08, ; li:issta:21, ), which violates the constraints of
configuration parameters as shown in Table 2.

Table 3. Software systems, and configuration datasets (including both ValidConfig and Misconfig datasets). Software | Version (SHA) | #Params | ValidConfig Dataset | Misconfig Dataset
---|---|---|---|---
Shot Pool | Eval. Set | Shot Pool | Eval. Set
HCommon | aa96f18 | 395 | 16 | 64 | 16 | 64
HBase | 0fc18a9 | 221 | 12 | 50 | 12 | 50
Alluxio | 76569bc | 494 | 13 | 54 | 13 | 54
HDFS | aa96f18 | 566 | 16 | 64 | 16 | 64
YARN | aa96f18 | 525 | 10 | 40 | 10 | 40
ZooKeeper | e3704b3 | 32 | 8 | 32 | 8 | 32

For each project, we build two distinct sets of configuration files. First, we
build a dataset of configuration files with no misconfiguration (denoted as
ValidConfig) to measure true negatives and false positives (Table 4).
Concurrently, we also build a dataset of configuration files in which each
file has one misconfiguration (denoted as Misconfig) to measure true positives
and false negatives (Table 4). A misconfiguration can be a dependency
violation between values of multiple parameters.

To build Misconfig for a project, we first check if a configuration parameter
fits the specification of any sub-category in Table 2, and assign it to all
sub-categories that fit. For example, an IP-address parameter can be assigned
to “Syntax: IP Address” and “Range: IP Address”. And we do so for all
parameters in the project. Then, we randomly sample at most 5 parameters in
each sub-category that has a non-empty set of assigned parameters, and
generate invalid value(s) per sampled parameter using the corresponding
generation rules. For each non-empty sub-category, we further randomly select
one parameter and its generated invalid value(s) from the 5 previously-sampled
parameters. We use that one parameter to create a faulty configuration file as
a Misconfig shot (§3) for that sub-category, and add this shot to the
project’s shot pool. For the remaining 4 parameters, we use them to create 4
faulty configuration files for that sub-category, and add them to the
evaluation set. If a sub-category does not have enough parameters for the
aforementioned samplings, we use all its assigned parameters to create files
for the evaluation set. We separate the evaluation set and shot pool to follow
the practice that the learning shot data does not overlap with the testing
data (brown:arxiv:20, ).

We build ValidConfig for a project following the same methodology we used to
build the Misconfig mentioned above, except that we generate valid values for
the sampled parameters. Table 3 shows the size for both the ValidConfig and
Misconfig datasets per project. It’s worth noting that our datasets cover
72%–100% of the total number of parameters in each evaluated system.

Models We evaluate Ciri with five state-of-the-art LLMs: GPT-4, GPT-3.5-turbo
(16k), Code-Davinci-002, Babbage-002, Davinci-002. These models are the most
widely used LLMs, each of which differs in training procedures and/or training
data. They are trained on a large amount of code data, and show promising
capability in handling a number of software engineering tasks (xia:icse:23, ;
thakur:arxiv:23, ; chen:arxiv:23, ).

  * •

Code-Davinci-002 is optimized for code completion tasks based on GPT-3.5, and
is capable at translating natural language to code. Code-Davinci-002 has 175
billion parameters, and was trained on data collected until June 2021. Its
token limit per query is 8,001.

  * •

GPT-3.5-turbo (16k) is a successor to Code-Davinci-002. Compared with Code-
Davinci-002, GPT-3.5-turbo further uses an effective optimization technique
called RLHF to follow instructions (ziegler:arxiv:19, ). GPT-3.5-turbo has
unknown number of parameters and was trained on data available up to September
2021. Since the base turbo model has a token limit of only 4,097 per query, we
use its variant that extends the token limit to 16,385.

  * •

GPT-4 is claimed to be the most advanced and widely-used LLM up to date (gpt4,
). Compared with GPT-3.5-turbo, GPT-4 is larger in size. GPT-4 is trained on
data prior to September 2021, its token limit per query is 8,192.

  * •

Babbage-002 and Davinci-002, successors to the legacy GPT-3, are two base
models (openaimodels, ). They were not fine-tuned with instruction-following
technique (long:arxiv:22, ), which aligns models with specific prompts and
desired outputs. Both models have a 16,384 token limit per query.

Metrics We evaluate the effectiveness of LLMs on configuration validation at
both configuration file and configuration parameter levels. At file level, we
want to know whether the model can determine if a configuration file is fully
valid or misconfigured. At parameter level, we want to know whether the model
can determine if each parameter in the configuration file is valid or
erroneous. We describe the definitions of the terms used in confusion matrix
in Table 4. We then compute the Precision, Recall, and F1-score at both levels
to assess the LLM’s effectiveness. If not specified, we default to Macro
averaging since each project is regarded equally. We prioritize studying
parameter-level effectiveness because it provides more fine-grained
measurements. We default to discussing parameter-level metrics in §5 unless
noted otherwise.

Table 4. Definitions for confusion matrix (CM) terms. Level | CM | Definition
---|---|---
File | TP | A misconfigured configuration file correctly identified
FP | A correct configuration file wrongly flagged as misconfigured
TN | A correct configuration file rightly identified as valid
FN | A misconfigured configuration file overlooked or deemed correct
Param. | TP | A misconfigured parameter correctly identified
FP | A correct parameter wrongly flagged as misconfigured
TN | A correct parameter rightly identified as valid
FN | A misconfigured parameter overlooked or deemed correct

##  5\. Evaluation and Findings

In this section, we first present results on evaluating the effectiveness of
LLMs as configuration validators with Ciri (§5.1). We then analyze how
validation effectiveness changes with regard to shots in few-shot learning
(§5.2). We also present our understanding on when Ciri produces wrongful
validation results (§5.3) and observed biases from LLMs’ training (§5.4).

###  5.1. Validation Effectiveness

######  Finding 1.

Ciri demonstrates the effectiveness of using state-of-the-art LLMs as
configuration validators. It achieves file-level and parameter-level F1-scores
of up to 0.75 and 0.56, respectively.

Ciri exhibits remarkable capability in configuration validation. Table 5 shows
the F1-score, precision, and recall for each project and LLM under four-shot
setting (§3)—the most effective few-shot learning setting obtained from our
later experiments (§5.2). Table 5 shows that: beyond merely identifying
misconfiguration files, with an average F1-score ranging from 0.62 to 0.75,
LLMs can also adept at pinpoint erroneous parameters and discern the causes of
the misconfigurations. Among the top-three LLMs, the parameter-level F1-scores
are approximately 25% lower than their file-level counterparts, this shows
that identifying misconfigured parameters is currently a more challenging task
for LLMs than classifying if a configuration change is erroneous.

When using legacy models, we observe they lack the ability of effective
configuration validation. Specifically, for Babbage-002, its F1-score has a
sharp drop from file-level (0.62) to parameter-level (0.09), indicating that
it is not able to localize the actual misconfigured parameter accurately. One
reason is that Babbage-002 lacks optimization for instruction-following
(long:arxiv:22, ), leading it to produce inappropriate results. Furthermore,
we also evaluate the Davinci-002, and it cannot detect any misconfiguration
within our dataset (omitted from Table 5).

######  Finding 2.

Providing configuration file examples (shots) for the validation query can
effectively improve LLMs’ configuration validation effectiveness. Without
shots, LLMs often report false alarms or miss misconfigurations, e.g., Code-
Davinci-002’s F1-score at parameter-level is as low as 0.08—with shots, its
file-level and parameter-level F1-scores can be improved by 0.56 and 0.48
respectively.

Validation examples (shots) play an important role in improving the
effectiveness of LLMs for configuration validation. Table 6 shows the
performance of LLMs when the configuration validation query does not include
shots from Ciri. In particular, comparing Table 6 to Table 5, the average
F1-score of the top-three LLMs has decreased by 0.08-0.56 at the file-level,
and decreased by 0.09-0.48 at the parameter-level. Without any shots, both
Davinci-002 and Babbage-002 cannot detect any misconfiguration in our dataset;
we thus omitted them in Table 6.

Implication. Our result suggests that state-of-the-art LLMs (e.g., GPT-4,
GPT-3.5-turbo) can be applied to configuration validation in a properly
designed framework like Ciri to achieve promising effectiveness. Specifically,
generating and providing configuration validation examples along with the
validation query can improve off-the-shelf LLMs’ misconfiguration detection
effectiveness. However, legacy LLMs (e.g., Davinci-002, Babbage-002) are often
incapable of configuration validation due to insufficient training data and/or
outdated training mechanisms (long:arxiv:22, ; chatgpt, ).

Table 5. Effectiveness of LLMs under Ciri. Models | F1-score |  |  | Precision |  |  | Recall
---|---|---|---|---|---|---|---
File-Level (F.L.) |  | Parameter-Level (P.L.) |  |  | F.L. | P.L. |  |  | F.L. | P.L.
HC. | HB. | AL. | HD. | YA. | ZK. | Avg |  | HC. | HB. | AL. | HD. | YA. | ZK. | Avg |  |  | Avg | Avg |  |  | Avg | Avg
GPT-4 | 0.80 | 0.74 | 0.74 | 0.77 | 0.72 | 0.72 | 0.75 |  | 0.64 | 0.56 | 0.57 | 0.60 | 0.53 | 0.47 | 0.56 |  |  | 0.63 | 0.43 |  |  | 0.93 | 0.83
GPT-3.5-turbo | 0.74 | 0.72 | 0.76 | 0.74 | 0.71 | 0.72 | 0.73 |  | 0.52 | 0.60 | 0.47 | 0.56 | 0.46 | 0.55 | 0.52 |  |  | 0.67 | 0.42 |  |  | 0.82 | 0.73
Code-Davinci-002 | 0.74 | 0.75 | 0.79 | 0.68 | 0.70 | 0.71 | 0.73 |  | 0.59 | 0.62 | 0.53 | 0.51 | 0.54 | 0.59 | 0.56 |  |  | 0.63 | 0.48 |  |  | 0.87 | 0.71
Babbage-002 | 0.66 | 0.65 | 0.66 | 0.64 | 0.58 | 0.54 | 0.62 |  | 0.08 | 0.11 | 0.11 | 0.02 | 0.08 | 0.14 | 0.09 |  |  | 0.51 | 0.08 |  |  | 0.81 | 0.11
Table 6. Effectiveness of LLMs without using shots from Ciri. Models | F1-score |  |  | Precision |  |  | Recall
---|---|---|---|---|---|---|---
File-Level (F.L.) |  | Parameter-Level (P.L.) |  |  | F.L. | P.L. |  |  | F.L. | P.L.
HC. | HB. | AL. | HD. | YA. | ZK. | Avg |  | HC. | HB. | AL. | HD. | YA. | ZK. | Avg |  |  | Avg | Avg |  |  | Avg | Avg
GPT-4 | 0.66 | 0.68 | 0.64 | 0.69 | 0.62 | 0.73 | 0.67 |  | 0.53 | 0.62 | 0.46 | 0.48 | 0.39 | 0.33 | 0.47 |  |  | 0.76 | 0.44 |  |  | 0.62 | 0.55
GPT-3.5-turbo | 0.61 | 0.61 | 0.75 | 0.66 | 0.68 | 0.61 | 0.65 |  | 0.20 | 0.37 | 0.19 | 0.25 | 0.24 | 0.21 | 0.24 |  |  | 0.66 | 0.16 |  |  | 0.67 | 0.55
Code-Davinci-002 | 0.24 | 0.25 | 0.00 | 0.36 | 0.14 | 0.06 | 0.17 |  | 0.11 | 0.10 | 0.00 | 0.17 | 0.10 | 0.00 | 0.08 |  |  | 0.56 | 0.08 |  |  | 0.11 | 0.09

###  5.2. Impacts of Few-shot Learning

Following the implication in Section 5.1, we conduct a series of experiments
to study how the configuration validation effectiveness of LLMs can be
improved over different shot combinations.

We evaluate six $N$-shot learning settings, where $N$ ranges from 0 to 5. For
each of these settings, we use Ciri to generate different combinations of
shots drawn from the ValidConfig and Misconfig datasets (§4). For example, to
evaluate GPT-3.5-turbo on HCommon with a two-shot setting, three experiments
will be performed: (1) two ValidConfig shots; (2) one ValidConfig shot plus
one Misconfig shot; (3) two Misconfig shots, drawn from the HCommon’s shot
database. In total, we experiment with 21 shot combinations for each project
and LLM. To control cost, we limit the experiment to two LLMs (GPT-3.5-turbo
and Code-Davinci-002) on three systems (HCommon, HBase, Alluxio).

######  Finding 3.

Including both ValidConfig and Misconfig shots for LLMs delivers the optimal
configuration validation effectiveness. Meanwhile, Misconfig shots are more
crucial to validation effectiveness than ValidConfig shots. For example, both
GPT-3.5-turbo and Code-Davinci-002 achieve their highest F1-score with three
Misconfig shots and one ValidConfig shot.

Figure 4 shows how the average F1-score, precision and recall across projects
for GPT-3.5-turbo and Code-Davinci-002 under different shot combinations.
Without Misconfig, LLMs’ performance is easily limited. For example, when only
using ValidConfig shots in the prompt, Code-Davinci-002 only gets an F1-score
around 0.2 (i.e., the first column of the heat map in Figure 4(b)). Compared
with ValidConfig shots, Misconfig shots allow Ciri to more effectively
identify patterns and attributes of misconfigurations at inference time. In
both Figure 4(a)-4(b), F1-score increases at more Misconfig shots are used in
the prompt.

On the other hand, providing only Misconfig shots can introduce bias to the
LLM and lead to a performance decrease. This is because the text distribution
in the input query can significantly impact the performance of LLMs
(min:arxiv:22, ). In our evaluation, after providing a sufficient number of
Misconfig shots, we indeed see that providing more ValidConfig shots can
sometimes reduce the number of false positives for Code-Davinci-002 and false
negatives for GPT-3.5-turbo. Overall, using both Misconfig and ValidConfig in
few-shot learning settings mitigates the biases of LLMs and delivers the
optimal configuration validation performance.

######  Finding 4.

Using configuration files from the same system as shots for LLMs delivers the
optimal configuration validation effectiveness. When same-system shots are
unavailable, using configuration files from a different system could also
improve validation effectiveness over zero-shot. For example, on HCommon, the
parameter-level F1-score improved by 0.39 averaged across the top three LLMs.

(a) Heatmaps of F1-score, Precision and Recall of GPT-3.5-turbo

(b) Heatmaps of F1-score, Precision and Recall of Code-Davinci-002

Figure 4.  Evaluation results under different shot combinations. Table 7. Results on HCommon using shots from other systems, e.g., the HB. columns show results of using HBase shots for HCommon. The HC. columns show results of using HCommon shots for HCommon. Models | F1-score |  |  | Precision |  |  | Recall
---|---|---|---|---|---|---|---
File-Level (F.L.) |  | Parameter-Level (P.L.) |  |  | F.L. | P.L. |  |  | F.L. | P.L.
HC. | HB. | AL. | HD. | YA. | ZK. | Avg |  | HC. | HB. | AL. | HD. | YA. | ZK. | Avg |  |  | HC. | Avg | HC. | Avg |  |  | HC. | Avg | HC. | Avg
GPT-4 | 0.80 | 0.72 | 0.77 | 0.73 | 0.74 | 0.71 | 0.73 |  | 0.64 | 0.42 | 0.44 | 0.47 | 0.46 | 0.33 | 0.43 |  |  | 0.67 | 0.58 | 0.50 | 0.30 |  |  | 0.98 | 0.99 | 0.87 | 0.77
GPT-3.5-turbo | 0.74 | 0.72 | 0.66 | 0.71 | 0.64 | 0.68 | 0.68 |  | 0.52 | 0.30 | 0.22 | 0.47 | 0.34 | 0.21 | 0.31 |  |  | 0.71 | 0.62 | 0.42 | 0.21 |  |  | 0.77 | 0.78 | 0.67 | 0.61
Code-Davinci-002 | 0.74 | 0.73 | 0.70 | 0.71 | 0.71 | 0.62 | 0.69 |  | 0.59 | 0.47 | 0.41 | 0.42 | 0.41 | 0.48 | 0.44 |  |  | 0.65 | 0.64 | 0.51 | 0.38 |  |  | 0.85 | 0.79 | 0.69 | 0.53

In situations where configuration data of target systems is unavailable, we
evaluate the possibility of using configuration files from other systems as
shots for LLMs can improve the configuration validation effectiveness on the
target system. Table 7 shows our evaluation results of using shots from other
systems for LLMs to do configuration validation on HCommon. By comparing the
HC. columns with other columns in Table 7, we can see that using shots from
other systems is not as effective as using shots from the target system.
However, by comparing the average F1-score in Table 7 with the HC. columns in
Table 6, we find that using shots from other systems is generally more
effective than not using any shots. Our observations highlight that Ciri with
the underlying LLMs can transfer configuration-related knowledge across
different systems for effective configuration validation compared to
traditional validation approaches (§1).

In contrast to the other LLMs, using cross-system shots actually decreases the
F1-score compared to the zero-shot setting on GPT-4. One possible explanation
is that examples from other systems also introduce noise that is contradictory
to the target system’s configuration data. LLMs can pick up such noise and
produce an inaccurate validation outcome.

######  Finding 5.

The validation effectiveness is higher if the misconfiguration in the
validating configuration file belongs to the same violation (sub-)categories
as the misconfigurations in the shots, compared to when it does not.

Table 8 shows that when the validating configuration file contains
misconfiguration belonging to the same (sub-)categories of violation as the
misconfigurations in the shots, the LLMs’ F1-score improves significantly.
When the misconfigurations between the shot and the evaluated file are caused
by the same category (e.g., syntax or range error from Table 2), parameter-
level F1-score improves by up to 30% compared to when they are not caused by
the same category of violations. When they are caused by the same sub-category
of violation (i.e., they are violated similarly and have the same parameter
type), F1-score improves by up to 69.5%.

Table 8. Improvement in parameter-level F1-score when the misconfiguration in the evaluated file belongs to the same violation (sub-)category as the misconfiguration in the shot, compared to when they do not. Models | Category | Sub-Category
---|---|---
Diff. | Same | %Improv. | Diff. | Same | %Improv.
GPT-4 | 0.57 | 0.65 | +12.9% | 0.60 | 0.57 | -5.3%
GPT-3.5-turbo | 0.41 | 0.53 | +28.2% | 0.45 | 0.51 | +12.8%
Code-Davinci-002 | 0.44 | 0.61 | +37.0% | 0.48 | 0.81 | +69.5%

Implication. To improve LLM’s performance as a configuration validator with
few-shot learning, developers can leverage frameworks like Ciri to collect and
generate high-quality, comprehensive configuration validation examples as
shots. The provided shots in the prompt should be composed of misconfiguration
files in which the misconfigured parameter(s) have been identified and
reasoned, as well as configuration files that are entirely valid. Our
experience suggests that prioritizing the provision of misconfiguration shots
is more crucial than supplying valid configuration shots.

When configuration data of target systems is unavailable for few-shot
learning, using configuration data from other systems as shots could improve
configuration validation effectiveness of LLMs, compared with zero shot.
Moreover, misconfigurations that may fall into the same possible
(sub-)category of violation are particularly useful as shots. However, shots
from other systems may introduce bias, and affect the validation
effectiveness.

###  5.3. Ineffectiveness and Difficulties

######  Finding 6.

Under Ciri, LLMs excel at pinpointing misconfigurations caused by syntax and
range violations (i.e., 12 out of all 15 sub-categories of violations), with
an average F1-score of 0.8 across corresponding sub-categories. However, LLMs
show limited effectiveness in pinpointing misconfigurations caused by
dependency and version violations (i.e., 3 out of all 15 sub-categories), with
an average F1-score of 0.2 across corresponding sub-categories.

Table 9 shows the validation effectiveness of Ciri broken down by the types of
misconfigurations. The average F1-score across systems on detecting
misconfigurations due to Syntax and Range violations is consistently above 0.5
and often reaches 0.8 for all three state-of-the-art LLMs, with one exception
in Code-Davinci-002 on “Range: Permission” misconfigurations (an F1-score of
0.44). Meanwhile, however, F1-score rarely exceeds 0.3 when detecting
misconfigurations due to violations in Dependency and Version. Only GPT-4
achieves a slightly better F1-score of 0.46 in the Value Relationship sub-
category, which is still much lower than its F1-scores in other sub-
categories.

Table 9. Parameter-level F1-score by misconfiguration types from Table 2. N.A. means no evaluation samples. Category | Sub-category | GPT-4 |  | GPT-3.5-turbo |  | Code-Davinci-002
---|---|---|---|---|---|---
HC. | HB. | AL. | HD. | YA. | ZK. | avg |  | HC. | HB. | AL. | HD. | YA. | ZK. | avg |  | HC. | HB. | AL. | HD. | YA. | ZK. | avg
Syntax | Data type | 1.00 | 0.89 | 1.00 | 0.89 | 1.00 | 0.73 | 0.92 |  | 0.61 | 0.89 | 0.70 | 0.94 | 0.89 | 0.80 | 0.77 |  | 0.94 | 0.86 | 0.67 | 0.80 | 1.00 | 1.00 | 0.86
Path | 1.00 | 0.80 | 0.80 | 1.00 | 0.89 | 0.67 | 0.85 |  | 1.00 | 0.86 | 0.43 | 0.55 | 1.00 | 0.89 | 0.74 |  | 0.50 | 1.00 | 0.75 | 0.75 | 1.00 | 0.75 | 0.79
URL | 1.00 | N.A. | 0.00 | 0.80 | N.A. | N.A. | 0.82 |  | 1.00 | N.A. | 0.00 | 1.00 | N.A. | N.A. | 0.94 |  | 1.00 | N.A. | 0.00 | 1.00 | N.A. | N.A. | 0.89
IP Address | 1.00 | 0.89 | 1.00 | 1.00 | 1.00 | 0.73 | 0.92 |  | 0.86 | 0.80 | 0.89 | 1.00 | 0.89 | 0.89 | 0.88 |  | 1.00 | 0.80 | 0.50 | 0.89 | 0.75 | 0.89 | 0.81
Port | 0.89 | 0.89 | 0.89 | 1.00 | N.A. | 0.67 | 0.85 |  | 0.80 | 0.89 | 0.80 | 1.00 | N.A. | 0.89 | 0.86 |  | 1.00 | 1.00 | 0.89 | 0.89 | N.A. | 1.00 | 0.95
Permission | 1.00 | 1.00 | 0.57 | 1.00 | N.A. | N.A. | 0.88 |  | 0.86 | 1.00 | 1.00 | 1.00 | N.A. | N.A. | 0.95 |  | 1.00 | 1.00 | 0.67 | 0.86 | N.A. | N.A. | 0.90
Range | Basic numeric | 0.67 | 1.00 | 0.80 | 0.57 | 0.75 | 0.73 | 0.75 |  | 0.67 | 0.55 | 0.67 | 0.75 | 0.67 | 0.53 | 0.62 |  | 0.57 | 0.57 | 0.60 | 0.67 | 0.86 | 1.00 | 0.70
Bool | 0.89 | 0.55 | 1.00 | 0.80 | 0.80 | 0.62 | 0.75 |  | 0.57 | 0.57 | 0.47 | 0.00 | 0.44 | 0.80 | 0.50 |  | 0.57 | 0.75 | 1.00 | 0.67 | 0.75 | 0.80 | 0.77
Enum | 0.67 | 0.80 | 0.57 | 1.00 | 0.89 | N.A. | 0.78 |  | 0.75 | 0.67 | 0.40 | 0.86 | 0.67 | N.A. | 0.63 |  | 0.75 | 0.86 | 0.86 | 0.57 | 1.00 | N.A. | 0.81
IP Address | 0.89 | 0.89 | 0.89 | 1.00 | 0.89 | 0.57 | 0.83 |  | 0.89 | 1.00 | 0.57 | 0.86 | 0.86 | 0.67 | 0.77 |  | 0.89 | 0.89 | 0.50 | 0.75 | 0.89 | 0.57 | 0.74
Port | 1.00 | 0.86 | 0.86 | 0.67 | N.A. | 0.80 | 0.83 |  | 0.75 | 0.73 | 0.67 | 0.67 | N.A. | 1.00 | 0.76 |  | 0.86 | 0.86 | 0.86 | 1.00 | N.A. | 1.00 | 0.91
Permission | 0.75 | 1.00 | 0.50 | 1.00 | N.A. | N.A. | 0.82 |  | 0.50 | 0.00 | 0.57 | 0.86 | N.A. | N.A. | 0.63 |  | 0.57 | 1.00 | 0.50 | 0.00 | N.A. | N.A. | 0.44
Dependency | Control | 0.00 | 0.57 | 0.00 | 0.00 | 0.00 | N.A. | 0.13 |  | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | N.A. | 0.00 |  | 0.00 | 0.57 | 0.00 | 0.00 | 0.25 | N.A. | 0.18
Value Relationship | 0.57 | 0.57 | 0.67 | 0.00 | 0.57 | N.A. | 0.46 |  | 0.00 | 0.40 | 0.44 | 0.00 | 0.29 | N.A. | 0.25 |  | 0.00 | 0.29 | 0.00 | 0.40 | 0.25 | N.A. | 0.21
Version | Parameter Change | 0.75 | 0.00 | 0.00 | 0.57 | 0.00 | N.A. | 0.27 |  | 0.20 | 0.00 | 0.00 | 0.00 | 0.00 | N.A. | 0.06 |  | 0.50 | 0.00 | 0.00 | 0.00 | 0.40 | N.A. | 0.17

The performance difference can be attributed to the inherent nature of the
misconfigurations. Misconfigurations due to violations in the Syntax and Range
categories are more common in practice (yin:sosp:11, ), from which LLMs have
learned extensive knowledge. In such a case, domain-specific knowledge from
LLM is sufficient to spot Syntax or Range violations. On the other hand,
misconfiguration data from the Dependency and Version categories is often
project-specific, e.g., the example shown in Figure 5. They are tied to
detailed history and features of individual projects, thus harder to be
captured or memorized by LLMs if the LLMs have not been heavily re-trained or
fine-tuned on project-specific data. This performance discrepancy across
different misconfiguration types exposes existing LLMs’s limitation on
detecting misconfigurations that require highly project-specific knowledge.

Figure 5. Misconfiguration of Control Dependency that LLMs cannot detect. The
update interval for authentication is set but the secure authentication is
disabled. Figure 6. Effectiveness of GPT-3.5-turbo under different number of
parameters in the configuration file.

######  Finding 7.

LLMs show a trend of decrease in configuration validation effectiveness as the
number of parameters increases in the to-be-validated configuration file.

We evaluate how the F1-score from GPT-3.5-turbo relates to the number of
configuration parameters in the validating configuration file. We present our
results in Figure 6, which indicates that configuration validation becomes
more challenging as the number of parameters in the validating configuration
file grows. For instance, when the number of parameters jumps from 8 to 16,
the performance of Ciri begins to deteriorate severely. This performance
decline can be attributed to a potential information overload for LLMs at
inference time. As the complexity of the configuration file grows, the
difficulty of validating it also increases. The model may struggle to process
all the validating configuration parameters and their relationships as more
parameters are included.

######  Finding 8.

Among the correctly identified misconfigurations, 93.6% of the reasons from
LLMs directly address the root causes of the misconfigurations. Meanwhile,
6.4% of the reasons are misleading.

When an LLM identifies a misconfiguration from the validating configuration
file, Ciri also requests the LLM to provide explanations to its judgment to
aid developers in debugging the root cause and fixing the misconfiguration
(§3.1). To assess the clarity and accuracy of these explanations, we randomly
select one answer in which the misconfiguration is correctly identified per
(sub-category, system, LLM) tuple, and collect a total of 204 answers
(resulted from 2,040 queries). Upon careful manual review, we determined that
93.6% of the reasons given by the LLMs are clear and directly address the root
cause of the misconfigurations. This indicates that LLMs can both detect
misconfigurations and provide meaningful explanations for them. 1.9% of the
answers contain a mix of correct and incorrect explanations across their
queries. However, these incorrect reasons were filtered out by the text
clustering method outlined in §3.1 because the correct reasons are dominating.
Figure 7 presents an example of mixed reasons, with the second reason being an
instance of hallucination.

Figure 7. Correct and incorrect misconfig. reasons by LLMs.

Implication. With frameworks like Ciri, state-of-the-art LLMs can effectively
validate configurations for syntax or range violations. However, they are less
effective for the configurations that involve dependencies between parameters
and software versions, showing the challenges for LLMs to reason the
interactions between configuration and between configuration and code
(meinicke:16, ). To improve the effectiveness for those misconfigurations, one
can re-train or fine-tune LLMs with data related to dependency and versions.
Our results also show that small configuration snippets are much easier for
LLMs to validate, supporting incremental, continuous configuration change
practice which is already widely adoped in practice (tang:sosp:15, ;
beyer:2018, ). Lastly, while LLMs often provide correct explanations on
misconfigurations that can aid debugging, it is crucial for developers to use
these explanations with discretion, as they may not be consistently accurate.

###  5.4. Biases in Validation Results

######  Finding 9.

During configuration validation, LLMs more frequently pinpoint parameters that
are more popular on the Internet. When a configuration file is entirely valid,
LLMs more frequently report false alarms on the more popular parameters. When
a parameter is misconfigured, LLMs demonstrate higher accuracy in identifying
misconfigurations on the more popular parameters.

To quantify the popularity of a configuration parameter on the Internet, we
measure the number of exact-match search results returned by Google for a
keyword and term it as G-hits.

We first study whether there is a correlation between a parameter’s popularity
and the frequency it is reported as a false alarm by LLMs in valid
configuration files. For the configuration files in ValidConfig dataset, we
obtain the G-hits of each parameter in each file. We then track the frequency
of LLMs pinpointing the parameter with the $i^{th}$ highest G-hits in each
file, where $i=1...8$. Figure 8 shows the overall frequencies of the parameter
with $i^{th}$ highest G-hits in the file being pinpointed. The smaller figures
show the overall frequencies under different shot combinations (§5.2), the
larger figure simply sums up the frequencies from the smaller figures.
Overall, the frequency distributions of being pinpointed reveal a clear
skewness towards parameters with higher G-hits.

We then study the correlation between a parameter’s popularity and its
validation accuracy. We perform similar calculations for parameters in
configuration files from Misconfig dataset, and further separate the cases
when the misconfigured parameter is identified versus when it is missed. We
observe that the median G-hits of the misconfigured parameters being correctly
identified is higher than the median G-hits of the misconfigured parameters
being missed.

The correlations between the parameter popularity and the parameter G-hits
across both datasets can be attributed to the nature of the training data of
LLMs. Training data of LLMs is often sourced from publicly accessible domains
(e.g., Common Crawl (commoncrawl, )), which are easily accessible by search
engines like Google. Topics or parameters that are popularly discussed are
more likely to be memorized by the LLMs than the less popular ones, due to
more frequent presence in the training data.

Implication. LLMs are predisposed to prioritize configuration parameters that
are more frequently discussed on the internet during the configuration
validation. As a result, when employed as configuration validators, LLMs can
effectively detect misconfigurations in parameters that are commonly
referenced online, while posing limited capacity in validating those that are
not.

Figure 8. Frequency of the identified parameter with $i^{th}$ highest G-hits
in a configuration file. In different shot settings (sub-figures), VC stands
for a ValidConfig shot, and MC stands for a Misconfig shot. Figure 9. The
G-hits distribution of the correctly detected misconfigurations (orange), and
the G-hits distribution of the missed misconfigurations (blue). The bars in
box plots indicate medians.

##  6\. Threats To Validity

External. The external threats come from our evaluated LLMs and dataset. We
evaluate Ciri with five state-of-the-art LLMs that are widely used to mitigate
threats on evaluated models. To mitigate threats from the evaluated projects,
we select six mature, widely used software systems with different types. These
systems are commonly used in prior studies (wang:icse:23, ; sun:osdi:20, ;
cheng:issta:21, ; zhang:icse:21, ; rabkin:11, ; rabkin2:11, ; chen:fse:20, ).
To account for bias in the evaluated configuration data, we include many types
of configuration parameters and their generation rules based on prior studies
(xu:sosp:13, ; li:18, ; keller:dsn:08, ; li:issta:21, ) to synthesize our
evaluation dataset. Our results cannot generalize to other types of
misconfigurations (discussed in §7). We believe that the overall trend is
general, but the precise numbers could vary with other LLMs, software systems,
and configuration files in the field.

Internal. The internal threats lie in potential bugs in the implementation of
Ciri, and experimental scripts for evaluation. To mitigate such threats, we
have rigorously reviewed our code and carefully analyzed the experiment
results.

Construct. The threats to construct validity mainly lie in the metrics used in
the study. We use the popular F1-score, Precision, and Recall for our
evaluations, and carefully define our confusion matrices at both configuration
file granularity and configuration parameter granularity for misconfiguration
detection.

##  7\. Discussion and Future Work

Improving effectiveness of LLMs as validators. Despite the promising results,
using LLMs directly as configuration validators like Ciri is only a starting
point to fully leverage LLMs’ capabilities for configuration validation.
Specifically, there are circumstances where Ciri exhibits limitations and
biases (§5.3, §5.4). One of the intricate aspects of configuration validation
is understanding and validating configuration dependencies. To address this,
an initial step can be introduced where an LLM first tries to analyze the
dependencies of the target configuration before validation. Moreover,
integrating LLMs with automated configuration dependency discovery tools
(chen:fse:20, ) can be helpful. For example, tools that use static taint
analysis (chen:fse:20, ; wang:ase:23, ) or dynamic information flow analysis
(attariyan:osdi:10, ; attariyan2012x, ) can be utilized to extract and analyze
configuration dependencies, which can then be fed as input to LLMs. This
ensures that dependencies are accurately captured for LLMs.

To further enhance the understanding of LLMs, more information related to
configurations can be incorporated, such as code comments, descriptions,
change logs, and specifications. Such information can provide valuable
context, which has been proven by the prior work (chen:arxiv:23, ;
zhang2023multimodal, ; nedelkoski2019anomaly, ).

Moreover, we plan to investigate advanced prompting techniques, such as Chain-
of-Thoughts (CoT) prompting (wei2022chain, ; wang:arxiv:23, ;
zhang2022automatic, ). In configuration validation context, CoT prompting can
mimic the reasoning process a system expert might follow during the validation
process. By eliciting LLMs to generate intermediate reasoning steps leading to
the validation answer, it not only makes the validation process more
transparent but also potentially more accurate. This step-by-step reasoning
may also help in identifying and rectifying biases in the model’s validation
process.

Lastly, integrating user feedback loops can be valuable. With user feedback on
validation results, the iterative procedure can refine LLMs over time, leading
to more accurate results.

Detecting environment-related misconfigurations. While our study primarily
targets basic misconfigurations, such as syntax and semantic violations, the
validity of a configuration file can vary across environments. For instance, a
system’s configuration might specify a file location, but the file’s
existence, readability, and format can determine its actual validity. To
address these, LLMs can generate environment-specific scripts that can be run
in the context of the environment. For example, given the configuration file
as input, the LLM can generate a Python script like the following to validate
the specified file path.

Such an approach can help identify issues like misconfigured paths,
unreachable network addresses, missing packages, or invalid permissions.
Notably, these scripts offer a lightweight alternative to more intensive
configuration tests (xu:19, ; cheng:issta:21, ).

For security concern, running those checks generated by LLMs need to be
sandboxed. Moreover, the scripts can be reviewed by humans and transformed
into lightweight validators. Given the recent success of code generation tools
such as GitHub Copilot, we believe this is a promising direction to explore.
In fact, our preliminary experiments show that, given appropriate examples,
LLMs can generate such scripts quite well.

Detecting source-code related misconfigurations. In addition to the deployment
environment, the system’s source code can also affect the validity of a
configuration. Implicit assumptions or latent software bugs can create
ambiguities in understanding the true requirements of a configuration. To
further illustrate this point, continuing the above example, if the
documentation does not mention that the file needs to be in JSON format, but
the code expects such a format, neither an LLM nor a human could infer this
constraint based solely on the documentation.

To detect such issues, we can leverage LLMs’ ability to reason about code. The
strategy involves presenting both the configuration file and the relevant
source code that exercises this configuration to the LLM. Techniques like
static or dynamic program slicing (staticslicing, ; dynamicslicing, ) can help
pinpoint the relevant code blocks. The LLM can then be tasked with distilling
this code into a validator script. While this poses a challenge, the code
reasoning capability of LLMs (ahmad2020transformer, ; li2022automating, )
suggests that this is promising and worth further exploration.

Fine-tuning LLMs for configuration validation. Apart from these, there is also
the problem of tackling very system-specific parameters, which cannot be
reasoned about based on common-sense knowledge. This problem is further
exacerbated by the fact that software evolves constantly (zhang:icse:14, ;
zhang:icse:21, ) by introducing new parameters and changing existing
parameters to take different meanings and constraints. This is an important
problem to tackle to unleash the full potential of LLMs for configuration
validation. The most obvious approach for tackling this is to fine-tune models
on new data to keep LLMs updated, but this is non-trivial, especially due to
lack of data on the newly introduced parameters. The LLM community has found
promising results in using synthetic data (xu2023wizardlm, ; huang2022large, ;
haluptzok2023language, ; wang2023selfinstruct, ) for fine-tuning these models,
reducing the need for large amounts of real data. We believe that this is a
promising direction to explore for configuration validation as well.

##  8\. Related Work

Configuration Validation. Prior studies developed frameworks for developers to
implement validators (tang:sosp:15, ; raab:17, ; baset:middleware:2017, ;
huang:15, ) and test cases (sun:osdi:20, ; xu:19, ), as well as techniques to
extract configuration constraints (liao:18, ; NadiTSE2015, ; xu:sosp:13, ;
zhang:oopsla:21, ). However, manually writing validators and tests requires
extensive engineering efforts, and is hard to cover various properties of
different configurations (xu:sosp:13, ; xu:osdi:16, ; li:18, ; keller:dsn:08,
; li:issta:21, ). ML/NLP-based configuration validation techniques have been
investigated. Traditional ML/NLP-based approaches learn correctness rules from
configuration data (Bhagwan:21, ; zhang:asplos:14, ; wang:osdi:04, ; le:06, ;
palatin:kdd:06, ; wang:lisa:03, ; santo:16, ; santo:17, ) and documents
(xiang:atc:20, ; potharaju:vldb:15, ) and then use the learned rules for
validation. These techniques face data challenges and rely on predefined
learning features and models, making them hard to generalize to different
projects and deployment scenarios. Complementary to prior work, we explore
using LLMs for configuration validation, which can potentially address the
limitations of traditional ML/NLP techniques towards automatic, effective
validation solutions.

Large Language Models for Software Engineering. LLMs have become an exciting
utility in the past few years, achieving impressive performance across various
tasks such as text classification, text summarization, and logical reasoning
(wei:arvix:22, ; kojima:nips:22, ; devlin:arxiv:18, ; brown:arxiv:20, ;
touvron:arxiv:23, ). Recently, they are being actively adopted to the software
engineering domain, where they have demonstrated abilities in generating,
summarizing, and translating code (chen:arxiv:21, ; li:22, ; lyer:arxiv:18, ;
ahmed:ase:22, ; lu:arxiv:21, ; roziere:nips:20, ; roziere:arxiv:21, ), failure
diagnosis (chen:arxiv:23, ; ahmed:icse:23, ), fault localization and program
repair (mirsky:ss:21, ; fan:icse:23, ; xia:icse:23, ; xia2023universal, ).
Large pre-trained models of coding data (LLMs for code) are also increasingly
prominent (feng:arxiv:20, ; chen:arxiv:21, ; nijkamp:arxiv:20, ;
fried:arxiv:23, ; xu:arxiv:22, ; nedelkoski2019anomaly, ), and have been used
for the aforementioned code-specific tasks. We take the first step to
comprehensively evaluate LLMs for configuration validation. Our proposed
framework for adopting LLMs as configuration validators, Ciri, is general to
different LLMs.

##  9\. Conclusion

As a first step to harvest recent advances of LLMs such as GPT and Codex for
configuration validation, we developed Ciri as an open platform to experiment
with LLMs as configuration validators and to analyze the promises and
challenges of LLM-based validators. In this paper, we presented our analysis
of Ciri’s validation effectiveness on five popular LLMs using configuration
data of six mature, widely deployed open-source systems. Our findings showed
the potential of using LLMs for configuration validation—Ciri demonstrates the
effectiveness of state-of-the-art LLMs as configuration validators, achieving
file-level and parameter-level F1-scores of up to 0.75 and 0.56, respectively.
We also explored the design space of LLM-based validators in terms of prompt
engineering with few-shot learning. Despite the encouraging results, our study
revealed that directly using LLMs as configuration validators is ineffective
in detecting certain types of misconfigurations such as dependency violations
and version-related misconfigurations and induces biases to popular
parameters. We discuss the open challenges which shedding light on new,
exciting research directions of the LLM-empowered validation techniques.

## References

  * (1) Openctest.  https://github.com/xlab-uiuc/openctest, 2020.
  * (2) ChatGPT.  https://openai.com/blog/chatgpt, 2022.
  * (3) Codex.  https://openai.com/blog/openai-codex, 2022.
  * (4) Common Crawl.  https://commoncrawl.org/, 2023.
  * (5) GPT Base Models.  https://platform.openai.com/docs/models/gpt-base, 2023.
  * (6) Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. Towards a human-like open-domain chatbot.  arXiv:2001.09977 (2020).
  * (7) Ahmad, W. U., Chakraborty, S., Ray, B., and Chang, K.-W. A transformer-based approach for source code summarization.  arXiv:2005.00653 (2020).
  * (8) Ahmed, T., and Devanbu, P. Few-shot training LLMs for project-specific code-summarization.  In ASE (2022).
  * (9) Ahmed, T., Ghosh, S., Bansal, C., Zimmermann, T., Zhang, X., and Rajmohan, S. Recommending Root-Cause and Mitigation Steps for Cloud Incidents Using Large Language Models.  In ICSE (2023).
  * (10) Anthropic.  Introducing 100k context windows.  https://www.anthropic.com/index/100k-context-windows, 2023.
  * (11) Attariyan, M., and Flinn, J. Automating configuration troubleshooting with dynamic information flow analysis.  In OSDI (2010).
  * (12) Attariyan, Mona and Chow, Michael and Flinn, Jason.  X-ray: automating $\\{$Root-Cause$\\}$ diagnosis of performance anomalies in production software.  In OSDI (2012).
  * (13) Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.  arXiv:2302.04023 (2023).
  * (14) Barroso, L. A., Hölzle, U., and Ranganathan, P. The Datacenter as a Computer: Designing Warehouse-Scale Machines, 3 ed.  Morgan and Claypool Publishers, 2018.
  * (15) Baset, S., Suneja, S., Bila, N., Tuncer, O., and Isci, C. Usable Declarative Configuration Specification and Validation for Applications, Systems, and Cloud.  In Middleware (2017).
  * (16) Beyer, B., Murphy, N. R., Rensin, D. K., Kawahara, K., and Thorne, S. Site Reliability Workbook: Practical Ways to Implement SRE.  O’Reilly Media Inc., 2018.
  * (17) Bhagwan, R., Mehta, S., Radhakrishna, A., and Garg, S. Learning Patterns in Configuration.  In ASE (2021).
  * (18) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners.  arXiv:2005.14165 (2020).
  * (19) Camburu, O.-M., Shillingford, B., Minervini, P., Lukasiewicz, T., and Blunsom, P. Make up your mind! adversarial generation of inconsistent natural language explanations.  arXiv:1910.03065 (2019).
  * (20) Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code.  arXiv:2107.03374 (2021).
  * (21) Chen, Q., Wang, T., Legunsen, O., Li, S., and Xu, T. Understanding and Discovering Software Configuration Dependencies in Cloud and Datacenter Systems.  In ESEC/FSE (2020).
  * (22) Chen, Y., Xie, H., Ma, M., Kang, Y., Gao, X., Shi, L., Cao, Y., Gao, X., Fan, H., Wen, M., et al. Empowering Practical Root Cause Analysis by Large Language Models for Cloud Incidents.  arXiv:2305.15778 (2023).
  * (23) Cheng, R., Zhang, L., Marinov, D., and Xu, T. Test-Case Prioritization for Configuration Testing.  In ISSTA (2021).
  * (24) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv:1810.04805 (2018).
  * (25) Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., and Goldberg, Y. Measuring and improving consistency in pretrained language models.  arXiv:2102.01017 (2021).
  * (26) Fan, Z., Gao, X., Mirchev, M., Roychoudhury, A., and Tan, S. H. Automated repair of programs from large language models.  In ICSE (2023).
  * (27) Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. Codebert: A pre-trained model for programming and natural languages.  arXiv:2002.08155 (2020).
  * (28) Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., tau Yih, W., Zettlemoyer, L., and Lewis, M. InCoder: A Generative Model for Code Infilling and Synthesis.  arXiv:2204.05999 (2023).
  * (29) Gunawi, H. S., Hao, M., Suminto, R. O., Laksono, A., Satria, A. D., Adityatama, J., and Eliazar, K. J. Why Does the Cloud Stop Computing? Lessons from Hundreds of Service Outages.  In SOCC (2016).
  * (30) Haluptzok, P., Bowers, M., and Kalai, A. T. Language models can teach themselves to program better.  arXiv:2207.14502 (2023).
  * (31) Huang, J., and Chang, K. C.-C. Towards Reasoning in Large Language Models: A Survey.  In Findings of the Association for Computational Linguistics: ACL 2023 (2023).
  * (32) Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve.  arXiv:2210:11610 (2022).
  * (33) Huang, P., Bolosky, W. J., Sigh, A., and Zhou, Y. ConfValley: A Systematic Configuration Validation Framework for Cloud Services.  In EuroSys (2015).
  * (34) Huang, Q., Wang, H. J., and Borisov, N. Privacy-Preserving Friends Troubleshooting Network.  In NDSS (2005).
  * (35) Iyer, S., Konstas, I., Cheung, A., and Zettlemoyer, L. Mapping language to code in programmatic context.  arXiv:1808.09588 (2018).
  * (36) Keller, L., Upadhyaya, P., and Candea, G. ConfErr: A Tool for Assessing Resilience to Human Configuration Errors.  In DSN (2008).
  * (37) Kendrick, S. What Takes Us Down?  USENIX ;login: 37, 5 (2012), 37–45.
  * (38) Kiciman, E., and Wang, Y.-M. Discovering Correctness Constraints for Self-Management of System Configuration.  In ICAC (2004).
  * (39) Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners.  Advances in neural information processing systems 35, 22199.
  * (40) Korel, B., and Laski, J. Dynamic program slicing.  Information Processing Letters 29, 3 (1988), 155–163.
  * (41) Le, F., Lee, S., Wong, T., Kim, H. S., and Newcomb, D. Minerals: Using Data Mining to Detect Router Misconfigurations.  Tech. Rep. CMU-CyLab-06-008, Carnegie Mellon University, 2006.
  * (42) Leuschner, L., Küttler, M., Stumpf, T., Baier, C., Härtig, H., and Klüppelholz, S. Towards Automated Configuration of Systems with Non-Functional Constraints.  In HotOS-XVI (2017).
  * (43) Li, S., Li, W., Liao, X., Peng, S., Zhou, S., Jia, Z., and Wang, T. ConfVD: System Reactions Analysis and Evaluation Through Misconfiguration Injection.  IEEE Transactions on Reliability 67, 4 (2018), 1393–1405.
  * (44) Li, W., Jia, Z., Li, S., Zhang, Y., Wang, T., Xu, E., Wang, J., and Liao, X. Challenges and Opportunities: An In-Depth Empirical Study on Configuration Error Injection Testing.  In ISSTA (2021).
  * (45) Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. Competition-level code generation with alphacode.  Science 378, 6624 (2022), 1092–1097.
  * (46) Li, Z., Lu, S., Guo, D., Duan, N., Jannu, S., Jenks, G., Majumder, D., Green, J., Svyatkovskiy, A., Fu, S., et al. Automating code review activities by large-scale pre-training.  In ESEC/FSE (2022).
  * (47) Liao, X., Zhou, S., Li, S., Jia, Z., Liu, X., and He, H. Do You Really Know How to Configure Your Software? Configuration Constraints in Source Code May Help.  IEEE Transactions on Reliability 67, 3 (2018), 832–846.
  * (48) Liu, Y., Yao, Y., Ton, J.-F., Zhang, X., Guo, R., Cheng, H., Klochkov, Y., Taufiq, M. F., and Li, H. Trustworthy llms: a survey and guideline for evaluating large language models’ alignment.  arXiv:2308.05374 (2023).
  * (49) Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. Codexglue: A machine learning benchmark dataset for code understanding and generation.  arXiv:2102.04664 (2021).
  * (50) Manakul, P., Liusie, A., and Gales, M. J. F. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.  arXiv:2303.08896 (2023).
  * (51) Maurer, B. Fail at Scale: Reliability in the Face of Rapid Change.  Communications of the ACM 58, 11 (2015), 44–49.
  * (52) Mehta, S., Bhagwan, R., Kumar, R., Ashok, B., Bansal, C., Maddila, C., Bird, C., Asthana, S., and Kumar, A. Rex: Preventing Bugs and Misconfiguration in Large Services using Correlated Change Analysis.  In NSDI (2020).
  * (53) Meinicke, J., Wong, C.-P., Kästner, C., Thüm, T., and Saake, G. On Essential Configuration Complexity: Measuring Interations in Highly-Configurable Systems.  In ASE (2016).
  * (54) Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work?  arXiv:2202.12837 (2022).
  * (55) Mirsky, Y., Macon, G., Brown, M., Yagemann, C., Pruett, M., Downing, E., Mertoguno, S., and Lee, W. Vulchecker: Graph-based vulnerability localization in source code.  In USENIX Security (2023).
  * (56) Nadi, S., Berger, T., Kästner, C., and Czarnecki, K. Where do configuration constraints stem from? an extraction approach and an empirical study.  IEEE Transactions on Software Engineering (TSE) 99 (2015), 820–841.
  * (57) Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. Webgpt: Browser-assisted question-answering with human feedback.  arXiv:2112.09332 (2022).
  * (58) Nedelkoski, S., Cardoso, J., and Kao, O. Anomaly detection from system tracing data using multimodal deep learning.  In CLOUD (2019).
  * (59) Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi-turn program synthesis.  arXiv:2203.13474 (2022).
  * (60) OpenAI.  Gpt-4 technical report.  arXiv:2303.08774 (2023).
  * (61) Oppenheimer, D., Ganapathi, A., and Patterson, D. A. Why Do Internet Services Fail, and What Can Be Done About It?  In USITS (2003).
  * (62) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback.  arXiv:2203.02155 (2022).
  * (63) Palatin, N., Leizarowitz, A., Schuster, A., and Wolff, R. Mining for Misconfigured Machines in Grid Systems.  In KDD (2006).
  * (64) Potharaju, R., Chan, J., Hu, L., Nita-Rotaru, C., Wang, M., Zhang, L., and Jain, N. ConfSeer: Leveraging Customer Support Knowledge Bases for Automated Misconfiguration Detection.  In VLDB (2015).
  * (65) Raab, M., and Barany, G. Challenges in Validating FLOSS Configuration.  In OSS (2017).
  * (66) Rabkin, A., and Katz, R. Precomputing Possible Configuration Error Diagnosis.  In ASE (2011).
  * (67) Rabkin, A., and Katz, R. Static Extraction of Program Configuration Options.  In ICSE (2011).
  * (68) Rabkin, A., and Katz, R. How Hadoop Clusters Break.  IEEE Software Magazine 30, 4 (2013), 88–94.
  * (69) Roziere, B., Lachaux, M.-A., Chanussot, L., and Lample, G. Unsupervised translation of programming languages.  Advances in Neural Information Processing Systems 33, 20601.
  * (70) Roziere, B., Zhang, J. M., Charton, F., Harman, M., Synnaeve, G., and Lample, G. Leveraging automated unit tests for unsupervised code translation.  arXiv:2110.06773 (2021).
  * (71) Santolucito, M., Zhai, E., Dhodapkar, R., Shim, A., and Piskac, R. Synthesizing Configuration File Specifications with Association Rule Learning.  In OOPSLA (2017).
  * (72) Santolucito, M., Zhai, E., and Piskac, R. Probabilistic Automated Language Learning for Configuration Files.  In CAV (2016).
  * (73) Sayagh, M., Kerzazi, N., Adams, B., and Petrillo, F. Software Configuration Engineering in Practice: Interviews, Surveys, and Systematic Literature Review.  TSE (2018).
  * (74) Sherman, A., Lisiecki, P., Berkheimer, A., and Wein, J. ACMS: Akamai Configuration Management System.  In NSDI (2005).
  * (75) Sun, X., Cheng, R., Chen, J., Ang, E., Legunsen, O., and Xu, T. Testing Configuration Changes in Context to Prevent Production Failures.  In OSDI (2020).
  * (76) Tang, C., Kooburat, T., Venkatachalam, P., Chander, A., Wen, Z., Narayanan, A., Dowell, P., and Karl, R. Holistic Configuration Management at Facebook.  In SOSP (2015).
  * (77) Teng Wang, Haochen He, X. L. S. L. Z. J. Y. J. Q. L. W. L. ConfTainter: Static Taint Analysis For Configuration Options.  In ASE (2023).
  * (78) Thakur, S., Ahmad, B., Pearce, H., Tan, B., Dolan-Gavitt, B., Karri, R., and Garg, S. VeriGen: A Large Language Model for Verilog Code Generation.  arXiv:2308.00708 (2023).
  * (79) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models.  arXiv:2302.13971 (2023).
  * (80) Wang, H. J., Platt, J. C., Chen, Y., Zhang, R., and Wang, Y.-M. Automatic Misconfiguration Troubleshooting with PeerPressure.  In OSDI (2004).
  * (81) Wang, S., Lian, X., Marinov, D., and Xu, T. Test Selection for Unified Regression Testing.  In ICSE (2023).
  * (82) Wang, T., Jia, Z., Li, S., Zheng, S., Yu, Y., Xu, E., Peng, S., and Liao, X. Understanding and Detecting On-the-Fly Configuration Bugs.  In ICSE (2023).
  * (83) Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-Consistency Improves Chain of Thought Reasoning in Language Models.  arXiv:2203.11171 (2023).
  * (84) Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions.  arXiv:2212.10560 (2023).
  * (85) Wang, Y.-M., Verbowski, C., Dunagan, J., Chen, Y., Wang, H. J., Yuan, C., and Zhang, Z. STRIDER: A Black-box, State-based Approach to Change and Configuration Management and Support.  In LISA (2003).
  * (86) Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. Emergent abilities of large language models.  arXiv:2206.07682 (2022).
  * (87) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  arXiv:2201.11903 (2023).
  * (88) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models.  Advances in Neural Information Processing Systems 35 (2022), 24824–24837.
  * (89) Weiser, M. D. Program Slices: Formal, Psychological, and Practical Investigations of an Automatic Program Abstraction Method.  PhD thesis, USA, 1979.  AAI8007856.
  * (90) Xia, C. S., Paltenghi, M., Tian, J. L., Pradel, M., and Zhang, L. Universal fuzzing via large language models.  arXiv:2308.04738 (2023).
  * (91) Xia, C. S., Wei, Y., and Zhang, L. Automated Program Repair in the Era of Large Pre-Trained Language Models.  In ICSE (2023).
  * (92) Xiang, C., Huang, H., Yoo, A., Zhou, Y., and Pasupathy, S. PracExtractor: Extracting Configuration Good Practices from Manuals to Detect Server Misconfigurations.  In ATC (2020).
  * (93) Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions.  arXiv:2304.12244 (2023).
  * (94) Xu, F. F., Alon, U., Neubig, G., and J.Hellendoorn, V. A Systematic Evaluation of Large Language Models of Code.  arXiv:2202.13169 (2022).
  * (95) Xu, T., Jin, L., Fan, X., Zhou, Y., Pasupathy, S., and Talwadker, R. Hey, You Have Given Me Too Many Knobs! Understanding and Dealing with Over-Designed Configuration in System Software.  In ESEC/FSE (2015).
  * (96) Xu, T., Jin, X., Huang, P., Zhou, Y., Lu, S., Jin, L., and Pasupathy, S. Early Detection of Configuration Errors to Reduce Failure Damage.  In OSDI (2016).
  * (97) Xu, T., and Legunsen, O. Configuration Testing: Testing Configuration Values as Code and with Code.  arXiv:1905.12195 (2019).
  * (98) Xu, T., and Marinov, D. Mining Container Image Repositories for Software Configurations and Beyond.  In ICSE-NIER (2018).
  * (99) Xu, T., Zhang, J., Huang, P., Zheng, J., Sheng, T., Yuan, D., Zhou, Y., and Pasupathy, S. Do Not Blame Users for Misconfigurations.  In SOSP (2013).
  * (100) Xu, T., and Zhou, Y. Systems Approaches to Tackling Configuration Errors: A Survey.  ACM Computing Surveys 47, 4 (2015).
  * (101) Yin, Z., Ma, X., Zheng, J., Zhou, Y., Bairavasundaram, L. N., and Pasupathy, S. An Empirical Study on Configuration Errors in Commercial and Open Source Systems.  In SOSP (2011).
  * (102) Zhang, J., Piskac, R., Zhai, E., and Xu, T. Static Detection of Silent Misconfigurations with Deep Interaction Analysis.  In OOPSLA (2021).
  * (103) Zhang, J., Renganarayana, L., Zhang, X., Ge, N., Bala, V., Xu, T., and Zhou, Y. EnCore: Exploiting System Environment and Correlation Information for Misconfiguration Detection.  In ASPLOS (2014).
  * (104) Zhang, S., and Ernst, M. D. Which Configuration Option Should I Change?  In ICSE (2014).
  * (105) Zhang, Y., He, H., Legunsen, O., Li, S., Dong, W., and Xu, T. An Evolutionary Study of Configuration Design and Implementation in Cloud Systems.  In ICSE (2021).
  * (106) Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models.  arXiv:2309.01219 (2023).
  * (107) Zhang, Z., Zhang, A., Li, M., and Smola, A. Automatic chain of thought prompting in large language models.  arXiv:2210.03493 (2022).
  * (108) Zhang, Z., Zhang, A., Li, M., Zhao, H., Karypis, G., and Smola, A. Multimodal chain-of-thought reasoning in language models.  arXiv:2302.00923 (2023).
  * (109) Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences.  arXiv:1909.08593 (2019).