# An LLM-based Readability Measurement for Unit Tests’ Context-aware Inputs

Zhichao Zhou, Yutian Tang, Yun Lin, and Jingzhu He Z. Zhou and J. He are with
School of Information Science and Technology, ShanghaiTech University, ChinaY.
Lin is with the Shanghai Jiao Tong University, ChinaY. Tang is with University
of Glasgow, United KingdomJ. He<EMAIL_ADDRESS>is the
corresponding author.

###### Abstract

Automated test techniques usually generate unit tests with higher code
coverage than manual tests. However, the readability of automated tests is
crucial for code comprehension and maintenance. The readability of unit tests
involves many aspects. In this paper, we focus on test inputs. The central
limitation of existing studies on input readability is that they focus on test
codes alone without taking the tested source codes into consideration, making
them either ignore different source codes’ different readability requirements
or require manual efforts to write readable inputs. However, we observe that
the source codes specify the contexts that test inputs must satisfy. Based on
such observation, we introduce the Context Consistency Criterion (a.k.a, C3),
which is a readability measurement tool that leverages Large Language Models
to extract primitive-type (including string-type) parameters’ readability
contexts from the source codes and checks whether test inputs are consistent
with those contexts. We have also proposed EvoSuiteC3. It leverages C3’s
extracted contexts to help EvoSuite generate readable test inputs. We have
evaluated C3’s performance on $409$ JAVA classes and compared manual and
automated tests’ readability under C3 measurement. The results are two-fold.
First, The Precision, Recall, and F1-Score of C3’s mined readability contexts
are $84.4\%$, $83\%$, and $83.7\%$, respectively. Second, under C3’s
measurement, the string-type input readability scores of EvoSuiteC3,
ChatUniTest (an LLM-based test generation tool), manual tests, and two
traditional tools (EvoSuite and Randoop) are $90\%$, $83\%$, $68\%$, $8\%$,
and $8\%$, showing the traditional tools’ inability in generating readable
string-type inputs. We have conducted a survey based on the questionnaires
collected from 30 programmers with varied backgrounds. The results reveal that
when C3 identifies readable differences between tests, programmers tend to
give similar opinions of the test’s readability of C3.

###### Index Terms:

readability, test generation, large language models.

##  1 Introduction

Figure 1: Overview of test inputs from manual and automated tests for $30$
JAVA classes (see Sec. 5.2): Manual, EvoSuiteC3 (see Sec. 4), and ChatUniTest
[1] tests contain many real-world entities, such as person names, emails,
URLs, and locations; EvoSuite [2] and Randoop [3] tests contain many random
inputs, as well as JAVA class names, JSON strings, and UUIDs [4] since these
two tools tend to use pre-defined seeds and static or dynamic information of
classes under test as test inputs

Unit testing is a popular testing technique among developers. A unit test
checks whether a method inside the source code behaves functionally correctly.
Existing unit test generation tools (e.g., search-based tool EvoSuite [2, 5]
and feedback-directed random tool Randoop [3]) can generate unit tests that
are close to and even outperform the manually written ones in many coverage
criteria [6], such as branch coverage.

We observe that automated tests’ readability is crucial for programmers. For
example, in a scenario where programmers use tools to obtain regression tests
for two code versions [7, 8], they must review a test failing at an assertion
to locate and fix new bugs. It is time-consuming and takes lots of manual
effort [9, 10, 11].

The readability of unit tests involves many aspects, including but not limited
to test method names [12], comments [13, 14], documents [15], identifiers, and
assertions [16, 17]. In this paper, we focus on test inputs. Many studies [10,
18, 19, 20] and our survey (see Sec. 5.3) show that test inputs are crucial in
readability. However, the random inputs generated by existing tools (EvoSuite
and Randoop) cost understanding time. A former survey [10] showed that
compared to strings occurring in natural languages (e.g., "mail"), random
strings (e.g., "Qu5ua") require people at most $1.6$ times more comprehension
time.

⬇

1// method under test

2public User(String username, String password, String email) {

3 this.username = username; this.password = password;

4 this.email = email;

5}

6public void testCreateUser() { // manual test

7 User user = new User("Simon", "secret", "simon@example.org");

8}

9public void testConstructor() { // automated test by ChatUniTest

10 User user = new User("testUser", "testPassword", "test@example.com");

11}

12public void test05() throws Throwable { // automated test by EvoSuite

13 User user0 = new User("|x45e*3q4+", " [stream]", "");

14}

15public void test001() throws Throwable { // automated test by Randoop

16 User user7 = new User("hi!", "", "");

17}

18public void test25() throws Throwable { // automated test by EvoSuiteC3

19 User user0 = new User("Enrico Fermi", "guest", "careers@jobs.com");

20}

Code 1: A constructor from Airsonic [21] with three tests differing in inputs

To deal with this issue, Afshan et al. [10] used a statistical model to
predict the likelihood of the test input occurring in the real world. They
formulated the likelihood as a secondary objective to guide test generation.
The main issue with this approach is that its model ignores the source code
when judging the input readability. However, different source codes have
different requirements for input readability. Deljouyi et al. [22] proposed to
extract understandable inputs from end-to-end (E2E) tests’ execution trace
logs since an E2E test aims to simulate a real user to execute an
application’s entire workflow [23, 24] and may contain understandable inputs.
However, this approach is semi-automated since it requires human effort to
write E2E tests.

###  1.1 A Motivating Example

We leverage an example to demonstrate the lack of readability of automated
tests and the challenges of existing studies. This example is a simplified
constructor method under test (MUT) with a manual and two automated tests on
Code 1. The MUT comes from an open-source project, Airsonic [21]. It has three
string-type parameters, which are the user’s name, password, and email,
respectively. The manual test assigns the values "Simon", "secret", and
<EMAIL_ADDRESS>to them, matching the contexts of the three parameters.
The input values generated by an LLM-based test generation tool (ChatUniTest
[1]) are "testUser", "testPassword", and<EMAIL_ADDRESS>They also match
the contexts. Conversely, the input values from EvoSuite [2] are "|x45e*3q4+",
" [stream]", and "". These values make the automated test less readable than
the manual one. In our survey (see Sec. 5.3), $22$ out of $30$ respondents
prefer the manual test to EvoSuite’s test regarding readability. The
corresponding values from Randoop [3] are "hi!", "", and "", posing the same
issue. Such deficiency prevails in tests generated by traditional tools. Fig.
1 aggregates the inputs from manual tests and tests generated by various
testing tools for 30 classes (see Sec. 5.2). It shows that manual tests
contain diversified real-world objects (e.g., person names, emails, and
locations), while EvoSuite and Randoop [3] generate a large number of random
inputs. It could be worse for methods with multiple parameters. One
interviewee gave us the following feedback:

> “When there are many parameters, readers can not easily match parameters
> with values if a test uses a bunch of random inputs.”

The secondary objective complementing branch objectives proposed by Afshan et
al. [10] guides genetic algorithms [5, 25] to generate inputs that are more
likely to occur in the real language. However, this objective ignores the
source code, thus ignoring the different input readability requirements of
username and email. Hence, the optimized inputs may also not satisfy these
readability requirements hidden in the source code. The approach of Deljouyi
et al. [22] to extract readable test inputs from manual-written E2E tests can
not applied in this JAVA class under test without extra manual efforts since
the Airsonic project does not have E2E tests.

To cope with this readability issue, we have to solve two challenges:

$\bullet$Challenge 1 is how to extract the readability requirements for the
parameters from the source code. Due to their varied occurrence (e.g.,
comments, variable names, and their operations), it is challenging to identify
such requirements using traditional techniques like data-flow analysis [26].

$\bullet$Challenge 2 is how to improve existing traditional tools (e.g.,
EvoSuite) to fulfill such requirements after they are extracted.

###  1.2 Contributions

To solve the challenges shown in the motivating example, this paper proposes a
novel concept called readability context, which refers to the requirements for
test readability described in the source code (e.g., valid person names for
the username parameter in Code 1). Based on the readability context, this
paper proposes the Context Consistency Criterion (C3). First, C3 passes
detailed information (including the source code and the comments) to a Large
Language Model (LLM) and requests it to mine the readability contexts for the
primitive-type (including string-type) parameters. Besides, we also pass few-
shots [27] (i.e., pre-defined similar questions and their answers for LLM to
quickly adapt to this mine-context task) to LLM. Second, C3 checks how well
the inputs of the tests are consistent with readability contexts based on
LLMs, NLP tools, and regex matching.

To sum up, this paper’s contributions include:

$\bullet$ We observe that test inputs highly depend on the source codes. Based
on the observation, we propose a novel concept, readability context, to
represent the test readability requirements for the primitive-type variables
of source codes. We design and implement an automatic tool, i.e., C3, to mine
the readability contexts and leverage these contexts to measure the
consistency of the tests’ inputs with readability contexts (Sec. 3).

$\bullet$ We propose a variant of EvoSuite [2] (a search-based test generation
tool), called EvoSuiteC3. EvoSuiteC3 leverages the readability contexts mined
by C3 to help EvoSuite generate test inputs better in readability (Sec. 4).
The EvoSuiteC3 test in Code 1 exemplifies readability improvements brought by
EvoSuiteC3.

$\bullet$ We evaluate C3 on $107$ JAVA projects, including $409$ classes. C3
detects $1024$ ones with readability contexts from $4485$ primitive-type
parameters. After a human review process, we confirm that C3’s Precision,
Recall, and F1-Score are $84.4\%$, $83\%$, and $83.7\%$, respectively (Sec.
5.1).

$\bullet$ We compare manual tests, two traditional test generation tools
(EvoSuite [2] and Randoop [3]), EvoSuiteC3, and an LLM-based tool (ChatUniTest
[1]) on $30$ JAVA classes with $115$ parameters that have readability
contexts. Fig. 1 summarizes test inputs for these parameters and reveals that
EvoSuiteC3 and ChatUniTest’s tests perform closely to manual ones in input
readability. Meanwhile, EvoSuite and Randoop largely fail to generate readable
string-type inputs (Sec. 5.2). Furthermore, we invite $30$ programmers with
varied backgrounds to quantify the readability of manual and automated tests.
The results show that programmers reach a consensus with C3 (Sec. 5.3).

The rest of this paper is organized as follows. Sec. 2 discusses the related
work. Sec. 3 and 4 describe C3 and EvoSuiteC3’s system design, respectively.
Sec. 5 illustrates our evaluation. Sec. 6 presents the threats to validity.
Sec. 7 concludes the whole paper. Sec. 8 provides the data availability.

##  2 Related Work

In this section, we discuss related studies.

Traditional test generation. This paper uses two traditional test generation
tools, Randoop [3] and EvoSuite [2], to represent two methods. The first,
random testing [3, 28, 29, 30], generates random test inputs to satisfy
coverage criteria or expose faults in the software under test. Its lightweight
nature allows for easy testing of large-scale systems, providing a solution to
the scalability issues of static methods like model checking and symbolic
execution [31, 32]. However, random testing has limitations. While it can
cover source lines, the random inputs may lack readability. Our evaluation
(Sec. 5) reveals that most string-type inputs generated by Randoop are
meaningless (e.g., "hi!"), which can confuse developers. Additionally, random
testing without explicit guidance is inefficient in generating high-coverage
tests. The second method, search-based testing [33, 25, 34, 5, 2, 35],
addresses this by using coverage criteria as fitness functions to guide
genetic algorithms in generating test inputs. This approach may produce
readable inputs if the optimization of a fitness function requires it (e.g.,
name == "Peter" in a condition statement). However, most readability
requirements are not explicitly defined. Consequently, our experiments
indicate that EvoSuite’s readability is close to Randoop’s.

Readability of traditional test generation. Existing studies on the
readability of traditional test generation can be divided into two categories.
The first focuses on generating test summaries. Daka et al. [12] proposed
summarizing a test’s covered goals as its name. Panichella et al. [13]
suggested a more thorough summary incorporating covered goals, the JAVA class
being tested, and additional details to create comments. Roy et al. [14]
employed deep learning to name tests and internal variables and create
comments. Li et al. [15] proposed an approach to generate test documentation.
These summary-based approaches ignore the concrete inputs, which are also
regarded as necessary for readability (see Sec. 5.3). The second category
applies code readability studies to test readability. Based on Buse and
Weimer’s research on code readability [36], Daka et al. [17] gathered a
dataset of tests along with their human-rated readability, used common code
metrics as features (including identifier length, parentheses, test length,
and casts), and added extra features for tests (e.g., assertions). They
trained a linear regression model to improve the readability of automated unit
tests. Grano et al. [37] leveraged the existing code readability model [38] to
compare the readability of manual and automated tests. Gregory Gay [16]
proposed improving tests’ readability by leveraging LLMs to refactor tests,
such as dividing a test into small ones and formatting test code to satisfy
common code style standards. This paper closely aligns with the studies of
Afshan et al. [10] and Deljouyi et al. [22]. Afshan et al. [10] incorporated a
statistical language model into search-based test generation to predict the
likelihood of the test input occurring in the real world and guide test
generation as a secondary fitness function. Deljouyi et al. [22] proposed to
leverage end-to-end (E2E) tests to generate understandable unit test inputs
since an E2E test simulates a real user to execute an application’s entire
workflow and may contain understandable inputs. However, this approach
requires human effort to write E2E tests. All of the above-mentioned studies
ignore the source code, which plays a crucial role in determining the
readability of the test code. C3 complements these studies. It mines the
readability contexts of the primitive-type parameter from the source code and
judges whether a test input is consistent with the context as a readability
indicator. Our experiments show that C3 identifies the shortcomings of
EvoSuite and Randoop in readability, and EvoSuiteC3 improves EvoSuite’s
readability.

Figure 2: Overview of Context Consistency Criterion (C3) TABLE I: Context Profile Type | Context Category | Contexts | Judging Method
---|---|---|---
| Cyberspace | EMAIL, FILE, PATH, PROGRAM, WEBSITE, URL, NETADDR | LLM + Regex matching
| Geo-Abstract |  CITY, ORGANIZATION, COUNTRY, GPE (e.g., "UN"), FAC ("Route 66"), LOCATION |  LLM + CoreNLP [39]
| Temporal |  DATE ("Sat 26 April"), DURATION ("15 years"), TIMESET ("Every Monday"), TIME | LLM + CoreNLP
| Number |  PERCENT ("20%"), ORDINAL ("first"), CARDINAL ("two") | LLM + CoreNLP
| Personal Identifier |  PERSON ("Simon"), NORP ("American") | LLM + CoreNLP
String | Finance |  Money ("2 euros") | LLM + CoreNLP
Number | Base Formats |  BINARY (0b100), HEXADECIMAL (0xfe), OCTAL (007) | Regex matching
Other Formats |  FIXEDLENGTH, LONGNUMBER, SCIENTIFIC (1.2e20f) | Rule + Regex matching

LLM-based test generation. Large language models (e.g., the GPT series [27])
have demonstrated capabilities across diverse domains, including unit test
generation. Tang et al. [40] compared test suites generated by both LLM and
EvoSuite in correctness, code coverage, complexity metrics [41], and bug
detection. Lemieux et al. [42] suggested using LLMs to overcome plateaus in
fitness functions during search-based test generation. AthenaTest [43] employs
a pre-trained BART model [44] for test generation, while ChatUniTest [1] is an
automated tool that guides LLMs in generating and repairing tests. More
studies can be found in the survey paper [45]. Given its ready-to-use nature
and ability to continuously guide LLM in generating syntactically correct
tests, this paper uses ChatUniTest as a baseline for comparison with EvoSuite
and Randoop. Many LLM studies [43, 1, 46] assert that tests generated by LLMs
are more readable than those produced by traditional tools, a conclusion
reached through manual review or developer interviews. Our approach, C3,
offers a quantitative metric for measuring test readability automatically. Our
experimental results validate their findings that LLMs surpass EvoSuite and
Randoop in generating readable tests. We also find that LLMs outperform
developers as their tests are readable and achieve higher code coverage than
manual tests.

##  3 Context Consistency Criterion (C3)

This section describes the system design of Context Consistency Criterion
(C3). Fig. 2 shows its overview. Without losing any generality, we use JAVA as
the target language to introduce C3, while C3 is not tied to any specific
language. C3 takes the tested source code (e.g., a JAVA class or a method) and
the test code as the input. C3’s framework comprises two components, i.e., the
Mining Component and the Judging Component. The Mining Component mines the
readability contexts from the tested source code (Sec. 3.1). C3 extracts the
primitive-type parameters from the source code. Then, for each parameter, C3
passes its text information (i.e., the source code and comments) to a Large
Language Model (LLM) and lets LLM choose a context from a prepared context
collection (shown in Table I). The Judging Component measures the consistency
of test inputs with the readability contexts (Sec. 3.2). C3 extracts the
primitive-type inputs from the test. Then, it uses various tools (i.e., LLM,
NLP tools, and regex matching) to check whether an input satisfies its context
as the readability result.

###  3.1 Mining Component

The first step is to mine the contexts from the code under test. C3 extracts
the primitive-type parameters; then, C3 uses each parameter’s text information
and the prepared contexts to ask an LLM which context it matches.

Primitive-type parameter extractor. We leverage an open-source JAVA parser
[47] to analyze the code and extract its parameters of primitive types,
including string, byte, short, int, long, float, and double. Note that (1) the
primitive types in this paper include string although JAVA regards it as an
object type since it is common in nearly all real-world projects and can be
initialized by literal values like other primitive types, and (2) we omit char
and boolean since their range of values is small and finite, posing no issues
with readability. Besides, for a parameter, we extract its method’s source and
comments used to construct LLM’s prompt. Note that we only use the method
source code instead of the whole class code, thus significantly reducing the
prompt size and decreasing the model’s financial costs.

Prepared contexts. We divide the concrete primitive types into two abstract
types, i.e., String (including string) and Number (others), and introduce
their contexts, respectively.

(1) String. We use the named entity (a concept from NLP) [48, 49, 39, 50] to
prepare contexts for String. Named entity refers to real-world context-aware
objects, such as person names and emails. Referencing previous studies [39,
50], we prepare $23$ contexts shown in Table I, which can be grouped into six
context categories: Cyberspace ($7$), Geographic and Abstract Location ($6$),
Temporal ($4$), Number (not Number type, but Number as String contexts) ($3$),
Personal Identifier ($2$), and Finance ($1$). Note that “LOCATION” and “TIME”
are two generic contexts. We use them to capture potential contexts in the
source code that belong to Geo-Abstract Location and Temporal but do not match
any other contexts in these two categories. For example, “restaurant” does not
match any specific contexts (e.g., “CITY” and “GPE”) of Geo-Abstract Location
but can be captured by “LOCATION”.

(2) Number. While named entities serve as String’s contexts, we employ literal
formats for Number’s contexts, including base formats (e.g., 0b100, 0xfe) and
Scientific Notation (e.g., 1.2e20f). We add two more contexts, “FIXEDLENGTH”
and “LONGNUMBER”. The former targets numbers of fixed lengths, such as credit
card numbers, while the latter is designed for potentially overly long
numbers. Code 3 from our survey questions (Sec. 5.3) provides an example.

Figure 3: An example of context mining prompts for LLM

Primitive parameter context miner. Based on a primitive parameter’s text
information (the source code and comments) and the prepared contexts, we
construct a prompt to ask an LLM which context the parameter matches. The
prompt is created by following the few-shot technique [27]. Few-shot means
that before asking a question, pass some similar questions and the prepared
answers to an LLM in advance. This technique is proven to work well in many
fields [27, 51, 52, 53]. Fig. 3 shows our prompt example with one shot for one
context category. The first part of Fig. 3 is to set the model role and
control its behaviors. The second part is one shot, i.e., a question example
with an answer. The third part is the question of interest. The first three
placeholders store the parameter name, the method name, and the source code
with comments. The following placeholders store each context and its instance
(e.g., “PERSON” and "Simon").

We start illustrating the prompt-constructing process by introducing two parameters for controlling the prompt size. In LLM, a token is a basic unit of text [27], and an LLM has a token limit for a conversation. We use MaxToken to represent the token size limit. Then, we use Remain to keep the token space for the target question (i.e., the third part of Fig. 3) and LLM’s response. First, we calculate the RestToken by $(\textit{MaxToken}-(\textit{Remain}+strToken(\textit{targetQuestion})))$, where strToken computes the number of tokens within a string. Then, we add the role setting message to Messages and update RestToken. Next, we iterate to add a shot and update RestToken if there is enough token space. Finally, we add the targeting question. For the Number type, we have an extra operation. We observe that an important factor determining a Number parameter’s context is the operators it uses. The or method on Code 2 presents an example. The | operator denotes that the values of a and b should be in binary format, i.e., their contexts are “BINARY”. Therefore, we emphasize this factor by appending a sentence to the end of the targeting question, i.e., “The operators involved by this parameter are [op1, ...]”. Meanwhile, two tests identical for compilers, test1 and test2, vary in readability: The second is more readable since its inputs match the contexts.

⬇

1// method under test

2int or(int a, int b) { return a | b; }

3// a test satisfying traditional criteria like line coverage

4void test1() { assert 7 == or(4, 3); }

5// a more readable test

6void test2() { assert 0b111 == or(0b100, 0b011); }

Code 2: An or method with two tests differing in number formats

After generating messages, we pass them to LLM to get the result, which could
be a concrete context or MISC. MISC means that LLM regards that no context
matches this parameter.

###  3.2 Judging Component

The Judging Component accepts parameters with contexts and a test as the
inputs. For each parameter, it judges whether the test has an input passed to
it and matches its context. We utilize various tools to judge the context
matching, including LLM, arbitrary NLP tool (this paper uses CoreNLP [39]),
and regex matching. We first introduce each of them. Then, we describe
combining them to complete the judging process.

Tools.

(1) LLM. Fig. 4 shows an example of asking LLM to check whether an input
matches the context. We prepare a shot for each context category like in the
Mining Component (see Sec. 3.1). The prompt-constructing process is similar to
the process of constructing mining prompts, too.

(2) CoreNLP. Traditional NLP tools (e.g., CoreNLP) can verify the consistency
of a character string to the context. CoreNLP supports all context categories
of String [39], except Cyberspace (see Table I). Its advantage over LLM is
that it is financially cost-free.

(3) Regex matching. Regex matching works for String’s Cyberspace and Number’s
all categories. For example, we can use "^0[bB][01]+$" to check whether a
number is binary.

Figure 4: An example of context matching prompts for LLM

Primitive-type input extractor. We use a JAVA parser [47] to extract the
test’s inputs whose parameter has a context. Then, we run the judger on each
input. Note that we only take three input types into account: the literal
values, the primitive-type variables defined with a literal in the test
method, and the test’s final static fields. This choice ignores many input
sources, such as another class’s field. However, according to our survey (see
Sec. 5.3), programmers prefer straightforward test statements. For example,
pass-by-value is preferable to other ways when calling the method under test.
Hence, this choice has a tiny impact on readability judgment.

Primitive-type input judger. We use different tool combinations for context
categories (shown in Table I’s last column).

(1) LLM + Regex matching. We use this combination to check String’s Cyberspace
category (see Table I). Regex matching is enough to check whether an input is
valid for these contexts (e.g., “EMAIL”). However, in realistic development,
programmers may use an invalid but readable input, such as "testEmail" for
“EMAIL”. Hence, we complement LLM to make the judge results more elastic than
they are judged by regex matching only. We regard an input as readable as long
as one tool gives a positive result.

(2) LLM + CoreNLP. This combination checks all other String’s categories. Like
the previous combination, an input is readable if at least one tool says yes,
reducing the risk of ignored readable inputs.

(3) Regex matching. We use regex matching to check Number’s Base Formats, such
as "^0[bB][01]+$" for “BINARY”.

(4) Rule + Regex matching. This combination checks Number’s Other Formats.
Before leveraging regex matching, we add a parameter called BearableLength
(this paper set it to $9$). For this category, if an input’s length is shorter
than BearableLength, we regard it as readable. Otherwise, we use regex
matching to check the inputs: the requirement for “LONGNUMBER” is that an
underscore must appear after at most every three consecutive digits (e.g.,
11_321_22_745); for “FIXEDLENGTH”, the requirement is that an underscore must
exist and appear after a fixed number of consecutive digits (e.g.,
1111_1111_1111_1111); for “SCIENTIFIC”, we checks whether the input contains
an e or E. We assume the test code is grammatically correct, so as long as a
number contains such symbols, it satisfies “SCIENTIFIC”.

##  4 EvoSuite Enhanced by C3 (EvoSuiteC3)

The readability contexts mined by the Mining Component of C3 (Sec. 3.1) can
not only used by C3’s Judging Component (Sec. 3.2) to judge tests but also can
help automated tools generate tests better in readability. In this paper, we
propose EvoSuiteC3, i.e., EvoSuite [2] enhanced by C3. Fig. 5 shows its
overview. EvoSuite is a search-based test generation tool. It formulates
coverage metrics (e.g., Branch Coverage and Weak Mutation [54, 55]) as fitness
functions and uses them to guide genetic algorithms [25, 34, 5] to evolve
tests until all fitness functions reach optimum or the search budget is out.
To enhance EvoSuite, we take different strategies for string and number
contexts: For each string context, we provide a group of string seeds (e.g.,
"Beijing" and "London" for “CITY”) for EvoSuite. These seeds can be randomly
selected by EvoSuite to be directly inserted into tests. Secondly, we provide
extra fitness functions to guide EvoSuite’s genetic algorithm so that it can
generate tests with string values satisfying contexts. Unlike string contexts,
number contexts focus on the numbers’ formats instead of values. Hence, we
modify the formats of numbers according to contexts after generating tests
without any operations in the test generation process.

Figure 5: Overview of EvoSuiteC3

To demonstrate the fitness function for string contexts, we first introduce
several auxiliary functions: $getSeeds(context)$ gets prepared string seeds by
a specific context. $isSatisfy(value,context)$ checks whether a string value
satisfies a specific context by invoking C3’s Judging Component (Sec. 3.2). It
returns $1$ if the value satisfies the context; otherwise, it returns $0$.
Note that EvoSuiteC3 forces C3 to only use the regex matcher and CoreNLP
without LLM for judging (see Sec. 3.2) so that EvoSuiteC3 does not waste
EvoSuite’s time budget due to network calls and avoids model costs.
$stringDistance(value1,value2)$ measures the distance (ranging from $0$ to
$1$) between two string values. It returns $0$ if they are identical. We use
Jaro-Winkler Distance [56] to implement this function.

Based on these auxiliary functions, we define the readability fitness function
$f_{c3}(value,context)$ as follows:

$\left\\{\begin{array}[]{lcl}0&&{isSatisfy(value,}\\\ &&{context)=1,}\\\
minDistance(value,context)&&{\textnormal{otherwise,}}\\\ \end{array}\right.$

where $minDistance(value,context)$ is a function defined as
$\min{\\{stringDistance(value,v)|v\in getSeeds(context)\\}}$.

Besides, a method may have multiple parameters with a context (e.g., the
constructor in Code 1). To guide Evosuite in generating a method invocation
satisfying multiple contexts, we define an extra fitness function
$f_{c3invo}(invocation)$ as:

$\max{\\{f_{c3}(v,c)|(v,c)\in getVCPairs(invocation)\\}},$

where $getVCPairs$ returns an invocation’s all value-context pairs. This
function returns $0$ only if all contexts are satisfied.

Based on the above definitions, EvoSuiteC3 extracts all $f_{c3}$ and
$f_{c3invo}$ functions for parameters with contexts and methods containing
such parameters from the JAVA class under test. It combines them with the
existing fitness functions of EvoSuite to guide test generation.

##  5 Evaluation

Figure 6: Readability contexts’ proportion

The evaluation mainly assesses C3’s performance, i.e., false positive and
negative rates, in mining readability contexts from real projects.
Additionally, we aim to explore the differences between manual and automated
tests under C3 measurement. Moreover, we investigate whether programmers agree
with C3. The research questions are as follows:

$\bullet$RQ1: What are the false positive and negative rates of C3?

$\bullet$RQ2: What differentiates manual from automated tests in C3
measurement?

$\bullet$RQ3: Whether programmers agree with C3?

###  5.1 RQ1: C3’s Performance

This RQ aims to answer what the false positive and negative rates of C3 are.

####  5.1.1 Methodology

(1) Subjects. Table II presents the real projects used in this RQ. These
projects come from two sources: (1) We select $254$ classes from $100$
projects within the DynaMOSA benchmark [5, 57]; (2) To enrich the project
diversity and obtain more classes with manual tests for subsequent RQs, we add
$155$ classes from seven open-source projects, namely, Airsonic [58],
BroadleafCommerce [59], GoogleMapService [60], OpenMRS [61], OpenRefine [62],
PetClinic [63], and Shopizer [64]. In summary, these classes contain $4485$
primitive-type parameters.

(2) Procedure. We employ GPT-4-Turbo [65] to run with C3’s mining component
(Sec. 3.1). Specifically, C3 uses GPT-4-Turbo to analyze every primitive-type
parameter and further extract readability contexts. The default GPT-4-Turbo
settings are used when invoking its APIs. Two parameters’ concrete values of
C3’s mining component (see Sec. 3.1) are $4096$ (MaxToken) and $20$ (Remain).
Although multiple executions of this procedure could mitigate the effects of
the LLM’s randomness, C3 only uses the result of one run due to model
financial costs. Then, we conduct a manual review of C3’s results. Following
the previous study [66], two authors independently examined each parameter,
its associated JAVA source code, and comments to check whether the readability
context is correct (True Positive), wrong (False Positive), or missed (False
Negative). They then compared the label results and found a few differences.
In the end, they discussed and resolved the differences. Note that a context
may be both False Positive and False Negative, as C3 might incorrectly assign
a context to a parameter with a correct readability context. In such
instances, they only label these cases as False Positive.

(3) Data analysis. Firstly, we summarize the results as a Confusion Matrix
[67]. Secondly, we calculate the Precision, Recall, and F1-Score [67].
Thirdly, we group C3’s results by project categories to show the distribution
of readability contexts. Finally, we compute the proportion of each context
type.

####  5.1.2 Results

TABLE II: JAVA projects used in our study group by category. Category | Projects | Classes
---|---|---
Frameworks and Libraries | 48 | 173
Utilities and Tools | 40 | 103
Content Management Systems | 13 | 125
Games and Entertainment | 6 | 8
Total | 107 | 409
TABLE III: C3 accuracy summary: 240 parameters | C3 Positive | C3 Negative
---|---|---
Actual Positive | 864 | 177
Actual Negative | 160 | 3284
TABLE IV: Readability contexts group by project category Category | Primitive-Type Parameters
---|---
| With contexts | Total
Frameworks and Libraries |  604 ($21.3\%$) | 2832
Utilities and Tools |  212 ($25.9\%$) | 820
Content Management Systems |  226 ($28.1\%$) | 804
Games and Entertainment |  6 ($20.7\%$) | 29
Total |  1048 ($23.4\%$) | 4485
TABLE V: Projects used in RQ2-4 Project | Category | Classes | Contexts | True Positive Contexts
---|---|---|---|---
|  |  | Total | String | Number | Total | String | Number
Airsonic [58] | Utilities and Tools | 9 | 42 | 26 | 16 | 37 | 21 | 16
GoogleMapService [60] | Frameworks and Libraries | 6 | 20 | 20 | 0 | 19 | 19 | 0
OpenMRS [61] | Content Management Systems | 3 | 31 | 31 | 0 | 26 | 26 | 0
Shopizer [64] | Content Management Systems | 8 | 20 | 19 | 1 | 19 | 18 | 1
Twitter4J [5] | Frameworks and Libraries | 4 | 15 | 8 | 7 | 14 | 7 | 7
Total | - | 30 | 128 | 104 | 24 | 115 | 91 | 24

(1) Confusion Matrix. Table III shows the True Positives (TPs), False
Positives (FPs), True Negatives (TNs), and False Negatives (FNs).

(2) Metrics.

$Precision=\frac{TP}{TP+FP}=\frac{864}{864+160}=84.4\%.$ (1)
$Recall=\frac{TP}{TP+FN}=\frac{864}{864+177}=83\%.$ (2)
$F1=2\times\frac{Precision\times
Recall}{Precision+Recall}=2\times\frac{84.4\%\times
83\%}{84.4\%+83\%}=83.7\%.$ (3)

(3) Context distribution. After manually checking C3’s results, we confirmed
that $1048$ ($23.4\%$) out of $4485$ primitive-type parameters have
readability contexts. Table IV presents the distribution of readability
contexts across each group.

(4) Context proportion. Fig. 6 shows the proportions of $1048$ parameters’
readability contexts. Overall, string-type readability contexts constitute
$57.44\%$, while number-type contexts represent $42.56\%$. The leading two
context categories are “Other Formats” ($37.35\%$) and “Cyberspace”
($24.52\%$). The top six contexts are “LONGNUMBER” ($35.88\%$), “PERSON”
($12.88\%$), “PATH” ($6.39\%$), “LOCATION” ($6.11\%$), “FILE” ($5.92\%$) and
“CARDINAL” ($4.96\%$).

####  5.1.3 Discussion

Table IV shows that all project categories have a similar proportion of
readability contexts ($20\%$ to $30\%$). Second, “LONGNUMBER” ($35.88\%$) is
the most common context. After a manual review, we find that GPT-4-Turbo tends
to associate a parameter with “LONGNUMBER” if its name ends with “id” (e.g.,
“id” and “userId”) and concrete type is long. GPT-4-Turbo might do this
because such parameters are typically used for large-value database keys.
Meanwhile, such parameters are common in data-manipulating projects, leading
to a high proportion of this context.

The statistical data shows C3’s ability to correctly identify positive
instances while minimizing false positive cases. There are still $177$ FNs and
$160$ FPs. We classify the analyzed reasons as follows.

(1) LLM’s randomness. We find $154$ ($87\%$) FNs and $49$ ($31\%$) FPs whose
projects contain many parameters with similar names, comments, and meanings
that are accurately annotated with context, except for them. We assume this is
due to LLM’s randomness, which could be mitigated by calling LLM multiple
times. We randomly select $30$ cases as a sample. After calling LLM three
times, $20$’s readability contexts can be correctly extracted at least one
time. After calling LLM five times, the correct cases increase to $24$.

(2) LLM’s insufficient understanding. We find that $28$ ($13\%$) FNs and $102$
($64\%$) FPs are due to LLM’s insufficient understanding. The first example is
a class Playlist from Airsonic [58], which stores music playlists. Its one
parameter, name (the playlist’s name), is wrongly recognized as “PERSON”. The
second instance is a class Billing from Shopizer [64]. Its parameter, state
that could be tagged as “COUNTRY”, is missed by GPT-4-Turbo. We assume that
these cases are due to LLM’s insufficient understanding, which could be solved
by the LLM with better understanding in the future. On the other hand, our
mine-context task can be used as a benchmark to compare different LLMs.

Another issue undermining C3 is that our contexts are incomplete. One example
is a class Location from OpenMRS [61]. Its one string parameter, postalCode is
not recognized since we do not define “POSTCODE” context (see Sec. 3.1). We do
not count these instances as FN, as they fall outside C3’s contexts and exceed
its capabilities. To cope with this issue, we allow C3’s users to integrate
new contexts into C3 by providing their names, descriptions, and some examples
for LLMs (see Sec. 3.1).

Answer to RQ1 In a sample of $4485$ primitive-type parameters, C3 detects 1024
ones with readability contexts. After a manual review process, we confirm that
its Precision, Recall, and F1-Score are $84.4\%$, $83\%$, and $83.7\%$,
respectively.

TABLE VI: Judge summary group by project with true positive contexts (Text
marked with blue background when Read.% >70%

Project Manual ChatUniTest EvoSuite Randoop EvoSuiteC3 Cover. (%) Read. (%)
Cover. (%) Read. (%) Cover. (%) Read. (%) Cover. (%) Read. (%) Cover. (%)
Read. (%) Airsonic string 15 (71%) 8 (53%) 21 (100%) 17 (81%) 19 (90%) 1 (5%)
19 (90%) 4 (21%) 21 (100%) 20 (95%) number 10 (62%) 9 (90%) 15 (94%) 15 (100%)
11 (69%) 11 (100%) 8 (50%) 8 (100%) 10 (62%) 10 (100%) GoogleMap string 12
(63%) 10 (83%) 17 (89%) 9 (53%) 16 (84%) 2 (12%) 16 (84%) 0 (0%) 17 (89%) 13
(76%) Service number - - - - - - - - - - OpenMRS string 0 (0%) - 26 (100%) 25
(96%) 26 (100%) 2 (8%) 20 (77%) 1 (5%) 26 (100%) 22 (85%) number - - - - - - -
- - - Shopizer string 13 (72%) 9 (69%) 18 (100%) 17 (94%) 18 (100%) 0 (0%) 18
(100%) 1 (6%) 18 (100%) 18 (100%) number 0 (0%) - 0 (0%) - 0 (0%) - 0 (0%) - 0
(0%) - Twitter4J string 4 (57%) 3 (75%) 7 (100%) 6 (86%) 5 (71%) 2 (40%) 5
(71%) 0 (0%) 5 (71%) 5 (100%) number 2 (29%) 2 (100%) 6 (86%) 6 (100%) 6 (86%)
6 (100%) 7 (100%) 7 (100%) 7 (100%) 7 (100%) Total string 44 (48%) 30 (68%) 89
(98%) 74 (83%) 84 (92%) 7 (8%) 78 (86%) 6 (8%) 87 (96%) 78 (90%) number 12
(50%) 11 (92%) 21 (88%) 21 (100%) 17 (71%) 17 (100%) 15 (62%) 15 (100%) 17
(71%) 17 (100%)

TABLE VII: Judge summary group by project with all string contexts Project | Manual | ChatUniTest | EvoSuite | Randoop | EvoSuiteC3
---|---|---|---|---|---
Cover. (%) | Read. (%) | Cover. (%) | Read. (%) | Cover. (%) | Read. (%) | Cover. (%) | Read. (%) | Cover. (%) | Read. (%)
Total | string | 50 (48%) | 31 (62%) | 102 (98%) | 76 (75%) | 96 (92%) | 7 (7%) | 90 (87%) | 6 (7%) | 100 (96%) | 91 (91%)
TABLE VIII: Average judge summary of all tests from EvoSuite, Randoop, and EvoSuiteC3 with true positive contexts Project | EvoSuite | Randoop | EvoSuiteC3
---|---|---|---
Cover. (%) | Read. (%) | Cover. (%) | Read. (%) | Cover. (%) | Read. (%)
Airsonic | string | 20.25 (96%) | 3.54 (17%) | 20.9 (100%) | 5.67 (27%) | 21.0 (100%) | 20.15 (96%)
number | 9.7 (61%) | 9.65 (99%) | 8.43 (53%) | 8.4 (100%) | 9.92 (62%) | 9.92 (100%)
GoogleMapService | string | 15.36 (81%) | 0.69 (4%) | 13.47 (71%) | 0.23 (2%) | 16.28 (86%) | 10.77 (66%)
number | - | - | - | - | - | -
OpenMRS | string | 25.5 (98%) | 1.83 (7%) | 22.37 (86%) | 1.4 (6%) | 26.0 (100%) | 23.33 (90%)
number | - | - | - | - | - | -
Shopizer | string | 17.94 (100%) | 1.84 (10%) | 18.0 (100%) | 1.5 (8%) | 18.0 (100%) | 17.8 (99%)
number | 0.0 (0%) | - | 0.0 (0%) | - | 0.0 (0%) | -
Twitter4J | string | 5.0 (71%) | 3.13 (63%) | 5.97 (85%) | 3.53 (59%) | 5.0 (71%) | 4.97 (99%)
number | 6.43 (92%) | 6.43 (100%) | 6.53 (93%) | 6.53 (100%) | 6.43 (92%) | 6.43 (100%)
Total | string | 84.05 (92%) | 11.03 (13%) | 80.71 (88%) | 12.33 (15%) | 86.28 (94%) | 77.02 (89%)
number | 16.13 (67%) | 16.08 (99%) | 14.96 (62%) | 14.93 (99%) | 16.35 (68%) | 16.35 (100%)

###  5.2 RQ2: Readability of Manual and Automated Tests

This RQ aims to study the differences between manual and automated tests under
C3 measurement.

####  5.2.1 Methodology

(1) Subjects. Table V shows this RQ’s subjects, i.e., $30$ classes with manual
tests whose false positive cases have been identified in RQ1 (see Sec. 5.1).
We chose these classes because they have manual tests, and we aim to compare
manual and automated tests in this RQ.

(2) Baselines. We have five baselines: manual tests, EvoSuite [2], Randoop
[3], ChatUniTest [1], and EvoSuiteC3 (Sec. 4). We use EvoSuite and Randoop to
represent two traditional test generation techniques, i.e., search-based
software testing [25, 2, 34, 5, 35] and random software testing [3, 28, 29,
30]. We use ChatUniTest to represent LLM-based test generation [43, 1, 45]
since it is ready-to-use and can continuously guide LLM to generate
syntactically correct tests. EvoSuiteC3 is EvoSuite enhanced by C3 (see Sec.
4).

(3) Test generation procedure. For EvoSuite, Randoop, and EvoSuiteC3, we use
their default settings to run on each class for producing tests. Due to their
randomness, we repeat the procedure for $30$ rounds, and each round’s time
budget is two minutes. Besides, we pass all contexts, including false positive
ones mined by C3 (Table V), to EvoSuiteC3. For ChatUniTest, we adopt the
authors’ settings [1]. For example, the test number for each method under test
(MUT) is six, and the LLM is GPT-3.5-Turbo [65]. The only difference is that
we use at most ten rounds of calling GPT-3.5-Turbo to generate and repair a
test instead of six rounds to obtain more syntactically correct tests.
Although GPT-3.5-Turbo also has randomness, we only execute the routine once
due to financial costs.

(4) C3 measurement procedure. After the test generation procedure, we employ
C3’s judging component (see Sec. 3.2) to evaluate all tests. We execute C3
with GPT-4-Turbo on both manual and automated tests produced by ChatUniTest.
Likewise, we utilize GPT-4-Turbo for the first round of tests generated by
EvoSuite, Randoop, and EvoSuiteC3. However, considering the substantial model
financial costs brought by the tests of 30 rounds, we use GPT-3.5-Turbo to run
with C3 for them.

(5) Data analysis. We calculate two metrics for both manual and automated
tests generated by ChatUniTest, as well as the first round of EvoSuite and
Randoop tests: the number of covered parameters (i.e., the existence of a test
that calls the MUT with an input) and parameters with readable inputs (i.e.,
the existence of a test that calls the MUT with a readable input).
Additionally, we show the metric data by true positive contexts and all
contexts (see Table V), respectively, to show the effects of false positive
cases from C3 on measuring test generation tools. Note that we only show
string-type data for all contexts since there are no false positive cases in
number-type contexts (see Table V). Furthermore, we compute the average values
of these two metrics across all rounds of tests generated by EvoSuite,
Randoop, and EvoSuiteC3 to check whether the randomness affects their
performance in generating readable inputs.

####  5.2.2 Results

(1) String coverage and readability. Table VI presents the performance of five
baselines in covered parameters and parameters with readable inputs on $91$
true positive string-type contexts. Overall, four automated tools surpass
manual tests in parameter coverage ($86\%$ to $98\%$ versus $48\%$). The
readability rates (i.e., the ratio of parameters with readable input to
covered parameters) indicate that EvoSuiteC3, ChatUniTest, and manual tests
significantly outperform EvoSuite and Randoop in generating readable inputs
($90\%$, $83\%$, and $68\%$ versus $8\%$ and $8\%$). Fig. 1 summarizes test
inputs of five baselines.

(2) Number coverage and readability. Table VI also presents baselines’
performance on $24$ number-type contexts. Similarly, four test generation
tools (ChatUniTest, EvoSuite, EvoSuiteC3, and Randoop) surpass manual tests in
parameter coverage ($88\%$, $71\%$, $71\%$, and $62\%$ versus $50\%$). Their
readability rates are close: $92\%$ (manual tests) and $100\%$ (four tools).

(3) Effects of false positive contexts. Table VII shows the results of all
string-type contexts (including false positive cases). The parameter coverage
delta between all and true positive contexts only exists in Randoop ($86\%$
and $87\%$). The readability deltas range from $1\%$ (EvoSuite and EvoSuiteC3)
to $8\%$ (ChatUniTest).

(4) Thirty rounds. We run EvoSuite, Randoop, and EvoSuiteC3 $30$ times to
eliminate the randomness. Table VIII shows their average performance. In
total, the parameter coverage deltas between one and $30$ rounds range from
$0\%$ (EvoSuite’s string) to $4\%$ (EvoSuite’s number). The readability deltas
range from $0\%$ (EvoSuiteC3’s number) to $7\%$ (Randoop’s string).

####  5.2.3 Discussion

(1) Parameter coverage. The parameter coverage figures show that all test
generation tools can generate higher code coverage than manual tests,
confirming many studies’ claims and findings [54, 6].

(2) String readability. EvoSuiteC3 and ChatUniTest’s tests surpass manual ones
regarding string-input readability, while EvoSuite and Randoop almost fail to
generate readable string-type inputs.

(3) Number readability. All baselines achieve scores near $100\%$ in number
readability. This is primarily because all these contexts are “LONGNUMBER”,
and C3 regards that a number shorter than $9$ meets this context’s readability
(see Sec. 3.2). Meanwhile, such numbers are prevalent in all baseline tests.

(4) Effects of false positive contexts. False positive contexts lead to
underestimating the string-type readability performance of ChatUniTest and
manual tests. The primary reason is that these two baselines perform well in
string-type readability, thus making them sensitive to incorrect contexts.
However, such contexts have tiny impact on EvoSuite and Randoop, as these
tools fail to generate readable string inputs. Therefore, C3, even with false
positive contexts, can be utilized to evaluate traditional tools. Besides,
since we pass all contexts to EvoSuiteC3, its performance is nearly identical
with or without false positive contexts.

(5) Thirty rounds. The thirty rounds of data confirm that EvoSuite and Randoop
stably lag far behind manual tests, ChatUniTest, and EvoSuiteC3 in generating
readable string inputs.

Answer to RQ2 ChatUniTest and EvoSuiteC3’s tests surpass manual ones regarding
parameter coverage and C3 measurement, while EvoSuite and Randoop almost fail
to generate readable string-type inputs.

###  5.3 RQ3: Programmers’ Readability Measurements

Figure 7: The question framework of the questionnaire

In this RQ, we aim to determine whether programmers agree with C3’s
measurement on tests’ readability.

####  5.3.1 Methodology

(1) Subjects. From manual and automated tests used in Sec. 5.2, we randomly
select ten pairs, each testing the same method and covering identical
readability contexts. Each pair consists of a manual and an automated test. Of
these ten pairs, C3 identifies manual tests of five pairs as more readable
than automated ones (referred to as C3-Diff pairs). In comparison, the other
five pairs are determined to have equivalent readability (C3-Equal pairs).

(2) Questionnaire. We arrange these ten pairs into a questionnaire, forming
ten questions by applying the question framework depicted in Fig. 7 to each
pair of tests. We shuffle a pair’s tests to ensure that a manual or an
automated test is not always shown in the Test 1 or Test 2 of the framework.

(3) Participants. Following the previous study [13], we invite $30$
participants to answer the questionnaire. Table IX shows their jobs and
programming experiences.

TABLE IX: Participants’ jobs and programming experiences

Job | Number
---|---
Developer | 17
Graduate Student | 11
Professor | 1
Test Engineer | 1

Experience | Number
---|---
Over 3 years | 27
1 to 3 years | 2
Within 1 year | 1

(4) Data analysis. After participants finish the questionnaire, firstly, we
group the data by C3-Diff and C3-Equal, then decode and count their choices,
which include “manual is better”, “automated is better”, and “no difference”.
Secondly, for each group, we calculate the Cohen’s Kappa coefficient [68, 69,
70] for every two participants to determine their agreement level. The Cohen’s
Kappa coefficient, ranging from $-1$ to $1$, measures the agreement between
two raters, where $1$ means perfect agreement, $0.61$ means substantial
agreement, and $0$ means random agreement [69, 70].

####  5.3.2 Results

TABLE X: Participants’ choice summary Group | Manual | Automated | No Diff. | Total
---|---|---|---|---
C3-Equal | 52 | 88 | 10 | 150
C3-Diff | 88 | 47 | 15 | 150
Figure 8: Participants’ Cohen’s Kappa coefficients

(1) Choice summary. Table X summarizes participants’ choices. When C3 asserts
the manual test is more readable, 88 out of 150 instances ($58.7\%$) show
agreement, while 47 instances ($31.3\%$) display explicit disagreement. For
the C3-Equal cases, explicit preferences are expressed in $52+88=140$
instances ($93.3\%$).

(2) Cohen’s Kappa coefficients. Fig. 8 displays the levels of agreement among
participants, grouped by C3-Diff and C3-Equal. The agreement levels for
C3-Diff range from $-0.563$ to $1$, whereas for C3-Equal, they range from
$-0.923$ to $1$. The median values for C3-Diff and C3-Equal are $0.063$ and
$0$, respectively, while the mean values are $0.118$ and $0.067$. For C3-Diff,
there are $37$ instances reaching substantial agreement (values $\geq 0.61$),
compared to $23$ such instances for C3-Equal.

####  5.3.3 Discussion

We observe that for C3-Diff pairs, where C3 deems manual tests more readable
than automated ones, $103$ instances ($68.7\%$) of participants’ choices
express at least no objection. Moreover, several metrics of Cohen’s Kappa
coefficients from C3-Diff pairs surpass those from C3-Equal pairs, including
the minimum, median, and mean values and substantial agreement count. These
findings suggest that when C3 distinguishes readable differences between two
tests, programmers are more inclined to reach a consensus and concur with C3.
However, $31.3\%$ of choices still disagree with C3 in C3-Diff pairs.
Concurrently, the choice summary and Cohen’s Kappa coefficients of C3-Equal
pairs indicate that programmers’ preferences are not uniform. This is because
apart from test inputs matching contexts, different programmers emphasize
various other factors in readability, which are shown in the programmers’
feedback.

Feedback from programmers. After completing the questionnaire, we specifically
asked participants about the importance of matching test inputs to contexts
for readability and which other factors they emphasize. We categorize their
feedback as follows:

(1) Programmers approve readability context matching. Most programmers concur
that matching test inputs to contexts enhances readability. Twelve
participants expressed their dissatisfaction with random and meaningless
inputs (generated by EvoSuite and Randoop). One participant stated that the
situation deteriorates when a test calls a method with multiple random inputs,
as the mapping from values to parameters becomes unidentifiable. The EvoSuite
test of Code 1 provides an example.

(2) Short and explicit tests are preferred. Beyond input contexts, two hot
keywords from programmers discussing test readability are shortness and
explicitness. Shortness is a requirement for all test elements, such as
variable names, statements, and literals. Explicitness implies that (1) a test
should be self-contained, i.e., the overuse of methods other than the MUT is
discouraged. This prevents readers from referring to different methods for
understanding the test; (2) statements should be straightforward. For
instance, pass-by-value is preferable to pass-by-variable when calling the
MUT. Code 3 exemplifies this: the manual test violates both principles (too
long literals and calling two methods), whereas the automated one adheres to
them. Hence, $21$ out of $30$ programmers preferred the latter, although the
manual one satisfies the URL context. However, for the MUT in Code 1 with two
parameters requiring readability contexts, $22$ out of $30$ preferred the
manual test to the EvoSuite test, indicating dominating factors vary per
scenario.

⬇

1// object creation of manual test

2OEmbed oembed = twitter1.getOEmbed(new OEmbedRequest(240192632003911681L,
"http://somesite.com/"));

3// object creation of automated test

4OEmbedRequest oEmbedRequest0 = new OEmbedRequest(1784L, "kP");

Code 3: Two object creation statements

(3) Unreadable inputs are necessary. Two participants mentioned that
unreadable inputs are necessary when testing boundary conditions since many
defects are triggered by unexpected inputs. This posed a challenge for C3 to
measure tests since C3 cannot identify unreadable inputs that are on purpose.
Still, our experimental setup avoided this issue: For a context, as long as
one input meets the requirement, we assert that context is to be satisfied
(see Sec. 5.2).

Answer to RQ3 When C3 identifies readable differences between tests,
programmers tend to reach a consensus and agree with it, although C3 alone
does not include all readability factors.

##  6 Threats to Validity

Construct Validity. In Sec. 3.1, we use NLP’s named entities [48, 49, 39, 50]
to construct string type’ contexts and number formats to construct number
types’ contexts, which brings a threat that such named entities are not
complete to represent all contexts of real-world projects. We provide an
example in Sec. 5.1. Firstly, We plan to extend C3 to mine contexts beyond
prepared contexts in the source code automatically. Secondly, in practice, we
allow C3’s users to integrate new contexts into C3 by providing their names,
descriptions, and some examples for LLMs.

Internal Validity. The first threat to internal validity comes from two
parameters (MaxToken and Remain) of C3’s Mining Component (Sec. 3.1). We use
MaxToken to represent the token size limit and Remain to keep the token space
for LLM’s response. With these two parameters, our prompt token size will not
exceed the LLM’s token limit. In our experiment, we used GPT-3.5/4-turbo, and
their token limit was $4096$ when we ran our experiment. C3’ users need to
reset MaxToken when their adopted LLM has another token limit. Remain is used
to remain the token space for LLM’s response. C3’s expected response should be
a small string, i.e., a context name (e.g., “EMAIL” and “PERSON”). Hence, we
set Remain to a small number ($20$ in our experiment). However, C3’s users
also need to adjust this parameter’s concrete value when using an LLM with a
different string-tokenization algorithm from GPT-3.5/4 since a different
tokenization algorithm may lead to a significantly different token size, even
for a small string. The second threat to internal validity comes from a
parameter (BearableLength) of the C3’s Judging Component (Sec. 3.2). This
parameter is used to control the readable threshold for Number’s three
contexts (FIXEDLENGTH, LONGNUMBER, and SCIENTIFIC). For these three contexts,
when the inputs are not too long (e.g., $12$ and $12345$), we regard them as
readable. Hence, we add the bearable length to filter out such numbers when we
apply our judger on the tests. Since many meaningful numbers (e.g., American
phone numbers and credit card numbers) usually have at least ten digits, we
set the bearable length at $9$. The third threat is the randomness of the test
generation tools (ChatUnitTest, EvoSuiteC3, EvoSuite, and Randoop) when
comparing their test input readability in Sec. 5.2. To reduce this risk, We
repeated running EvoSuiteC3, EvoSuite, and Randoop $30$ times on the same JAVA
class. However, we only ran ChatUnitTest once because it leveraged
GPT-3.5-turbo to generate tests and could bring many financial costs if we
repeated running it $30$ times. The last threat is due to potential bugs
introduced in implementing C3 and EvoSuiteC3. For C3, we manually checked its
extracted contexts and confirmed that its Precision, Recall, and F1-Score are
$84.4\%$, $83\%$, and $83.7\%$, respectively (see Sec. 5.1). For EvoSuiteC3,
we extracted the inputs from its generated tests and manually checked whether
these inputs match the contexts. The result confirmed EvoSuite’s readability
increase brought by EvoSuiteC3. Fig. 1 demonstrates the comparison between
EvoSuiteC3 and EvoSuite.

External Validity. Experimental subjects form a threat. Many classes from the
DynaMOSA benchmark [5] are obsolete [35]. Hence, we complement it by adding
$155$ classes from seven open-source projects (see Sec. 5.1). The choice of
LLMs in our evaluation forms another threat since C3’s performance depends on
LLMs. Moreover, we implemented C3 in JAVA and evaluated it using JAVA
subjects. Hence, our experimental findings are not verified in other
programming languages. Nonetheless, the fundamental concepts of C3 are not
confined to JAVA.

##  7 Conclusion

This paper proposes the Context Consistency Criterion (C3), an automatic
readability measurement tool for primitive-type test input. C3 can mine the
readability contexts for primitive-type parameters in the code under test and
use these contexts to judge tests’ consistency as a readability metric.
Besides, this paper proposes EvoSuiteC3 (i.e., EvoSuite enhanced by C3). Our
experimental results show the distribution of readability contexts and confirm
C3’s accuracy. Furthermore, while LLM-based test generation and EvoSuiteC3
surpass manual tests under C3 measurement, two traditional tools (Randoop and
the original EvoSuite) almost fail to generate readable string-type inputs.

##  8 Data Availability

C3’s implementation and the experimental data are at
https://doi.org/10.5281/zenodo.13028768.

## References

  * [1] Y. Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test generation,” in _Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering_ , 2024, pp. 572–576.
  * [2] G. Fraser and A. Arcuri, “Evosuite: Automatic test suite generation for object-oriented software,” in _Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering_ , 2011, p. 416–419.
  * [3] C. Pacheco and M. D. Ernst, “Randoop: feedback-directed random testing for java,” in _Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion_ , 2007, pp. 815–816.
  * [4] Java Platform, “Uuid introduction,” 2024. [Online]. Available: https://docs.oracle.com/javase/8/docs/api/java/util/UUID.html
  * [5] A. Panichella, F. M. Kifetew, and P. Tonella, “Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets,” _IEEE Transactions on Software Engineering_ , vol. 44, pp. 122–158, 2018.
  * [6] S. Wang, N. Shrestha, A. K. Subburaman, J. Wang, M. Wei, and N. Nagappan, “Automatic unit test generation for machine learning libraries: How far are we?” in _Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering_ , 2021, pp. 1548–1560.
  * [7] J. Zhang, Y. Lou, L. Zhang, D. Hao, L. Zhang, and H. Mei, “Isomorphic regression testing: executing uncovered branches without test augmentation,” in _Proceedings of the 24th ACM SIGSOFT international symposium on foundations of software engineering_ , 2016, pp. 883–894.
  * [8] A. Salahirad, H. Almulla, and G. Gay, “Choosing the fitness function for the job: Automated generation of test suites that detect real faults,” _Software Testing, Verification and Reliability_ , vol. 29, no. 4-5, p. e1701, 2019.
  * [9] P. McMinn, M. Stevenson, and M. Harman, “Reducing qualitative human oracle costs associated with automatically generated test data,” in _Proceedings of the First International Workshop on Software Test Output Validation_ , 2010, pp. 1–4.
  * [10] S. Afshan, P. McMinn, and M. Stevenson, “Evolving readable string test inputs using a natural language model to reduce human oracle cost,” in _Proceedings of the Sixth IEEE International Conference on Software Testing, Verification and Validation_ , 2013, pp. 352–361.
  * [11] E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” _IEEE Transactions on Software Engineering_ , vol. 41, no. 5, pp. 507–525, 2014.
  * [12] E. Daka, J. M. Rojas, and G. Fraser, “Generating unit tests with descriptive names or: Would you name your children thing1 and thing2?” in _Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , 2017, pp. 57–67.
  * [13] S. Panichella, A. Panichella, M. Beller, A. Zaidman, and H. C. Gall, “The impact of test case summaries on bug fixing performance: An empirical investigation,” in _Proceedings of the 38th international conference on software engineering_ , 2016, pp. 547–558.
  * [14] D. Roy, Z. Zhang, M. Ma, V. Arnaoudova, A. Panichella, S. Panichella, D. Gonzalez, and M. Mirakhorli, “Deeptc-enhancer: Improving the readability of automatically generated tests,” in _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering_ , 2020, pp. 287–298.
  * [15] B. Li, C. Vendome, M. Linares-Vásquez, D. Poshyvanyk, and N. A. Kraft, “Automatically documenting unit test cases,” in _2016 IEEE international conference on software testing, verification and validation_ , 2016, pp. 341–352.
  * [16] G. Gay, “Improving the readability of generated tests using gpt-4 and chatgpt code interpreter,” in _International Symposium on Search Based Software Engineering_. Springer, 2023, pp. 140–146.
  * [17] E. Daka, J. Campos, G. Fraser, J. Dorn, and W. Weimer, “Modeling readability to improve unit tests,” in _Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering_ , 2015, pp. 107–118.
  * [18] A. Alsharif, G. M. Kapfhammer, and P. McMinn, “What factors make sql test cases understandable for testers? a human study of automated test data generation techniques,” in _Proceedings of the 35th IEEE International Conference on Software Maintenance and Evolution_ , 2019, pp. 437–448.
  * [19] M. M. Almasi, H. Hemmati, G. Fraser, A. Arcuri, and J. Benefelds, “An industrial evaluation of unit test generation: Finding real faults in a financial application,” in _Proceedings of the 39th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice Track_ , 2017, pp. 263–272.
  * [20] D. Winkler, P. Urbanke, and R. Ramler, “Investigating the readability of test code,” _Empirical Software Engineering_ , vol. 29, no. 2, p. 53, 2024.
  * [21] Online, “A constructor method from airsonic: A free, web-based media streamer,” 2024. [Online]. Available: https://tinyurl.com/5d2fnc4w
  * [22] A. Deljouyi and A. Zaidman, “Generating understandable unit tests through end-to-end test scenario carving,” in _2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation_. IEEE, 2023, pp. 107–118.
  * [23] M. Leotta, H. Z. Yousaf, F. Ricca, and B. Garcia, “Ai-generated test scripts for web e2e testing with chatgpt and copilot: A preliminary study,” in _Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering_ , 2024, pp. 339–344.
  * [24] C. Augusto, “Efficient test execution in end to end testing: Resource optimization in end to end testing through a smart resource characterization and orchestration,” in _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings_ , 2020, pp. 152–154.
  * [25] G. Fraser and A. Arcuri, “Whole test suite generation,” _IEEE Transactions on Software Engineering_ , vol. 39, no. 2, pp. 276–291, 2013.
  * [26] F. E. Allen and J. Cocke, “A program data flow analysis procedure,” _Communications of the ACM_ , vol. 19, no. 3, p. 137, 1976.
  * [27] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [28] C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball, “Feedback-directed random test generation,” in _Proceedings of the IEEE/ACM 29th International Conference on Software Engineering_ , 2007, pp. 75–84.
  * [29] C. Pacheco and M. D. Ernst, “Eclat: Automatic generation and classification of test inputs,” in _Proceedings of the 19th European Conference on Object-Oriented Programming_. Springer, 2005, pp. 504–527.
  * [30] J. H. Andrews, T. Menzies, and F. C. Li, “Genetic algorithms for randomized unit testing,” _IEEE Transactions on software engineering_ , vol. 37, no. 1, pp. 80–94, 2011.
  * [31] K. Havelund and T. Pressburger, “Model checking java programs using java pathfinder,” _International Journal on Software Tools for Technology Transfer_ , vol. 2, pp. 366–381, 2000.
  * [32] P. Braione, G. Denaro, and M. Pezzè, “Jbse: A symbolic executor for java programs with complex heap inputs,” in _Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering_ , 2016, pp. 1018–1022.
  * [33] P. Tonella, “Evolutionary testing of classes,” in _Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis_ , 2004, p. 119–128.
  * [34] A. Panichella, F. M. Kifetew, and P. Tonella, “Reformulating branch coverage as a many-objective optimization problem,” in _Proceedings of the 8th IEEE international conference on software testing, verification and validation_ , 2015, pp. 1–10.
  * [35] Y. Lin, Y. S. Ong, J. Sun, G. Fraser, and J. S. Dong, “Graph-based seed object synthesis for search-based unit testing,” in _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2021, p. 1068–1080.
  * [36] R. P. Buse and W. R. Weimer, “Learning a metric for code readability,” _IEEE Transactions on software engineering_ , vol. 36, no. 4, pp. 546–558, 2009.
  * [37] G. Grano, S. Scalabrino, H. C. Gall, and R. Oliveto, “An empirical investigation on the readability of manual and generated test cases,” in _Proceedings of the 26th Conference on Program Comprehension_ , no. 4, 2018, p. 348–351.
  * [38] S. Scalabrino, M. Linares-Vásquez, D. Poshyvanyk, and R. Oliveto, “Improving code readability models with textual features,” in _2016 IEEE 24th International Conference on Program Comprehension_ , 2016, pp. 1–10.
  * [39] C. D. Manning, M. Surdeanu, J. Bauer, J. R. Finkel, S. Bethard, and D. McClosky, “The stanford corenlp natural language processing toolkit,” in _Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations_ , 2014, pp. 55–60.
  * [40] Y. Tang, Z. Liu, Z. Zhou, and X. Luo, “Chatgpt vs sbst: A comparative assessment of unit test suite generation,” _IEEE Transactions on Software Engineering_ , 2024.
  * [41] C. E. C. Dantas and M. A. Maia, “Readability and understandability scores for snippet assessment: an exploratory study,” _arXiv preprint arXiv:2108.09181_ , 2021.
  * [42] C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,” in _Proceedings of the 45th IEEE/ACM International Conference on Software Engineering_ , 2023.
  * [43] M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transformers and focal context,” _arXiv preprint arXiv:2009.05617_ , 2020.
  * [44] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Online: ACL, 2020, pp. 7871–7880.
  * [45] J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,” _IEEE Transactions on Software Engineering_ , 2024.
  * [46] Z. Yuan, M. Liu, S. Ding, K. Wang, Y. Chen, X. Peng, and Y. Lou, “Evaluating and improving chatgpt for unit test generation,” _Proceedings of the ACM on Software Engineering_ , vol. 1, no. FSE, pp. 1703–1726, 2024.
  * [47] Online, “Javaparser: A java 1.0 - java 18 parser,” 2024. [Online]. Available: http://tinyurl.com/5n7h2pnb
  * [48] E. T. K. Sang and F. De Meulder, “Introduction to the conll-2003 shared task: Language-independent named entity recognition,” in _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , 2003, pp. 142–147.
  * [49] A. Ritter, S. Clark, O. Etzioni _et al._ , “Named entity recognition in tweets: an experimental study,” in _Proceedings of the 2011 conference on empirical methods in natural language processing_ , 2011, pp. 1524–1534.
  * [50] S. Wang, X. Sun, X. Li, R. Ouyang, F. Wu, T. Zhang, J. Li, and G. Wang, “Gpt-ner: Named entity recognition via large language models,” _arXiv preprint arXiv:2304.10428_ , 2023.
  * [51] T. Ahmed and P. Devanbu, “Few-shot training llms for project-specific code-summarization,” in _Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering_ , 2022, pp. 1–5.
  * [52] ——, “Better patching using llm prompting, via self-consistency,” in _Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering_ , 2023, pp. 1742–1746.
  * [53] X. Liu, D. McDuff, G. Kovacs, I. Galatzer-Levy, J. Sunshine, J. Zhan, M.-Z. Poh, S. Liao, P. Di Achille, and S. Patel, “Large language models are few-shot health learners,” _arXiv preprint arXiv:2305.15525_ , 2023.
  * [54] J. M. Rojas, J. Campos, M. Vivanti, G. Fraser, and A. Arcuri, “Combining multiple coverage criteria in search-based unit test generation,” in _Search-Based Software Engineering_ , M. Barros and Y. Labiche, Eds., 2015, pp. 93–108.
  * [55] G. Fraser and A. Arcuri, “Achieving scalable mutation-based generation of whole test suites,” _Empirical Software Engineering_ , vol. 20, no. 3, p. 783–812, 2015.
  * [56] W. E. Winkler, “String comparator metrics and enhanced decision rules in the fellegi-sunter model of record linkage.” 1990.
  * [57] J. Campos, Y. Ge, N. Albunian, G. Fraser, M. Eler, and A. Arcuri, “An empirical evaluation of evolutionary algorithms for unit test suite generation,” _Information and Software Technology_ , vol. 104, pp. 207–235, 2018.
  * [58] Online, “Airsonic: A free, web-based media streamer,” 2024. [Online]. Available: https://tinyurl.com/4ye4xsmn
  * [59] ——, “Broadleafcommerce: An e-commerce framework,” 2024. [Online]. Available: https://tinyurl.com/3j2345sb
  * [60] ——, “Googlemapservice: Client for google maps,” 2024. [Online]. Available: http://tinyurl.com/y99bzypc
  * [61] ——, “Openmrs: A patient-based medical record system,” 2024. [Online]. Available: https://tinyurl.com/mrxr69s2
  * [62] ——, “Openrefine: A data manipulator,” 2024. [Online]. Available: https://tinyurl.com/4xev45tz
  * [63] ——, “Petclinic: A spring sample application,” 2024. [Online]. Available: https://tinyurl.com/5d9amebv
  * [64] ——, “Shopizer: An e-commerce software,” 2024. [Online]. Available: https://tinyurl.com/33z3sdvs
  * [65] OpenAI, “Gpt series,” 2024. [Online]. Available: https://tinyurl.com/vcfzar2f
  * [66] D. Wang, S. Li, G. Xiao, Y. Liu, and Y. Sui, “An exploratory study of autopilot software bugs in unmanned aerial vehicles,” in _Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering_ , 2021, pp. 20–31.
  * [67] D. M. W. Powers, “Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation,” _arXiv preprint arXiv:2010.16061_ , 2020.
  * [68] J. Cohen, “Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit.” _Psychological bulletin_ , vol. 70, no. 4, p. 213, 1968.
  * [69] G. Landis JRKoch, “The measurement of observer agreement for categorical data,” _Biometrics_ , vol. 33, no. 1, p. 159174, 1977.
  * [70] M. L. McHugh, “Interrater reliability: the kappa statistic,” _Biochemia medica_ , vol. 22, no. 3, pp. 276–282, 2012.