# An Empirical Study on Challenging Math Problem Solving with GPT-4

Yiran Wu
Pennsylvania State University
<EMAIL_ADDRESS>
Feiran Jia
Pennsylvania State University
<EMAIL_ADDRESS>Shaokun Zhang
Pennsylvania State University
<EMAIL_ADDRESS>
Hangyu Li
Imperial College London
<EMAIL_ADDRESS>
Erkang Zhu
Microsoft Research Redmond
<EMAIL_ADDRESS>
Yue Wang
Microsoft Research Redmond
<EMAIL_ADDRESS>
Yin Tat Lee
University of Washington
<EMAIL_ADDRESS>
Richard Peng
University of Waterloo
<EMAIL_ADDRESS>Qingyun Wu
Pennsylvania State University
<EMAIL_ADDRESS>Chi Wang
Microsoft Research Redmond
<EMAIL_ADDRESS>

###### Abstract

Employing Large Language Models (LLMs) to address mathematical problems is an
intriguing research endeavor, considering the abundance of math problems
expressed in natural language across numerous science and engineering fields.
While several prior works have investigated solving elementary mathematics
using LLMs, this work explores the frontier of using GPT-4 for solving more
complex and challenging math problems. We evaluate various ways of using
GPT-4. Some of them are adapted from existing work, and one is MathChat, a
conversational problem-solving framework newly proposed in this work. We
perform the evaluation on difficult high school competition problems from the
MATH dataset, which shows the advantage of the proposed conversational
approach.

##  1 Introduction

With Large Language Models (LLMs) demonstrating remarkable proficiency in
various tasks spanning diverse domains [1], it is compelling to explore the
potential of LLMs in tackling mathematical problems considering the crucial
role of mathematics [34] and the prevalence of mathematical problems expressed
in natural language throughout numerous scientific and engineering
disciplines. Prior works on this endeavor mostly concern relatively simple
math problems such as elementary school math [4, 18, 24], or concerns a
specific aspect of math problem solving, for instance, arithmetic operations
and reasoning [12], and quantitative reasoning [16]. In this work, we
investigate the frontier of GPT-4 [22], arguably the most advanced LLM to
date, on advanced math problem-solving.

Creative ways of using LLMs to solve math problems have emerged lately [3, 6,
40, 31, 36]; however, they are rarely evaluated on GPT-4. [9] and [1]
performed studies on math problem solving with GPT-3.5 and an early version of
GPT-4 respectively, reporting interesting observations on the capability and
limitations of recent LLMs on this endeavor. In this work, we adapt and
evaluate several existing methods of using LLMs to the usage of GPT-4,
including a vanilla prompt, Program of Thoughts (PoT) prompt [3], and Program
Synthesis prompt [6]. We further propose MathChat, a conversational framework
tailored to chat-based LLMs, where the math problem is solved with a mock
conversation between the model and a user proxy agent (See Figure 1 for an
example and Figure 2 for the workflow). We perform the evaluation on the MATH
dataset [11], a comprehensive collection of mathematical problems derived from
various competitions and educational levels. Our study targets the level-5
difficulty problems within this dataset, which primarily consist of
challenging high school competition problems that even college students find
difficult. The evaluation shows that MathChat can further improve previous
prompting methods by 6%, and it can reach 60% accuracy on half of the
categories while having competitive performance across all categories. We also
demonstrate the extensibility of MathChat with different prompts and different
tools.

Figure 1: Example of a math problem-solving process with MathChat. The user
proxy agent initiates a conversation by sending the math problem to be solved,
along with a preset prompt to GPT-4 (the LLM assistant). From GPT-4’s
response, the user proxy agent extracts all code and executes them
sequentially. Valid code from previous runs is recorded and will be executed
together with the new code to reflect the step-by-step reasoning progress of
the model. The results will be returned to GPT-4 and GPT-4 will continue its
problem-solving process. While GPT-4 solves this problem with only one turn of
user message in this example, our framework allows multi-turn conversations
and additional query handling, shown in Figure 3. The user proxy agent will do
pattern-matching (in our case, the appearance of \boxed{} containing a final
answer) in the assistant’s response to determine whether to end the
conversation.

##  2 Existing methods of using LLMs on math problem solving

In this section, we introduce the various methods of using LLMs from existing
literature that can be applied to solve math problems. We first introduce
techniques specifically designed for mathematical problem-solving and then
summarize the general approaches. From our investigation, we only find one
method [6] that targets challenging math problems, while most other methods
are evaluated on elementary school math [4, 18, 24].

Methods tailored for math solving. Despite creative methods to use LLMs for
different tasks, there are only a few methods proposed that target math
problem-solving. One particular endeavor is using LLMs to offload arithmetic
calculations and other basic operations involved in math problem-solving to
programs. Specifically, [10] uses Codex [2] to decompose math problems into
runnable steps by interleaving CoT prompting with code, resulting in an
executable Python code to solve math problems. [6, 3] also use Codex to
generate code-based answers to problems. While [3, 10] prompt the model to
tackle math word problems, [6] investigates the ability of program-aided
coding on university-level problems, including randomly sampled problems from
the MATH [11] dataset. Other than using Python, [33] proposes a method for
basic arithmetic and logical problems, that concatenates the generated answer
back to the problem and lets the model predict the original conditions to
verify the correctness of this answer. Among these methods, [6, 3] can be used
with zero-shot prompts to solve math problems. MathPrompter is another method
tailored for math solving but it specifically targets improving arithmetic
operations and reasoning performance of LLMs [12].

General approaches. General approaches that can be used to improve math
solving include (1) chain-of-thought prompting [32, 15], (2) multi-stage
prompting [8, 14, 26, 5, 13, 39, 23, 37, 35, 30], (3) tool-using [10, 3, 6,
28, 23, 37]; and (4) self-consistency [31, 20]. Among them, here we review
some of the latest ones. [28] trains a Toolformer that can decide when to send
API calls. [23] use few-shot prompting to interleave step-by-step reasoning
and tool-using. ART [23] maintains a library of demonstrations, such as using
calculators, and retrieves the most similar examples for prompting a new
problem. [20] first generates an output using GPT-3.5 and uses the same model
to give feedback on the output, then the model is asked to refine the output
based on the feedback. Although these general approaches can be applied to
solve math word problems, these methods are not tailored for solving math
problems and are mostly evaluated on easier math tasks.

Due to the page limit, we include a comprehensive discussion on existing
methods in Appendix C.

Figure 2: MathChat workflow: After a math problem is fed into MathChat, the
user proxy agent will initiate a conversation with the LLM Assistant to solve
the problem. In each turn of interaction, the user proxy agent processes the
message from the assistant (Assistant Message) and responds with a User
Message. This process continues until the user proxy agent detects a certain
pattern to end the conversation. The process in the rectangular on the right-
hand side of the figure shows the inner workflow of the user proxy agent once
an Assistant Message is received. It shows the functionality of executing any
tool-using queries, such as Python code. It is also responsible for giving
different instructions corresponding to different types of messages from the
LLM Assistant (More in Appendix A). To illustrate this feature of the proxy
agent, we give a concrete example in Figure 3.

##  3 MathChat: A conversational framework for math problem solving

In this section, we introduce MathChat, a conversational framework for math
problem-solving.

Figure 3: An example demonstrating how the user proxy agent handles different
types of messages received from GPT-4 in MathChat. Specifically, the user
proxy agent may respond in the following ways: (1) asking the assistant to
continue because no code block (i.e., query) is detected; (2) returning the
valid results from code execution; and (3) returning the error message from
Python execution. Note that GPT-4 may change the query if the old code is
undesired based on the messaged from the user proxy agent. In the last step,
GPT-4 corrects the query, and the final result is returned.

A conversational framework with user proxy agent. MathChat is a framework that
simulates a mock conversation between an LLM assistant (GPT-4 in our case) and
a user proxy agent. Here a user proxy agent is an agent playing the user’s
role in conversations with the LLM assistant. In MathChat, the assistant and
the user proxy agent work together to solve the math problem. The workflow of
this framework is presented in Figure 2. The user proxy agent takes a math
problem to be solved as input and would initiate a conversation with the LLM
assistant. The initial message from the user proxy agent consists of an
initial prompt and the problem to be solved. The initial prompt is used to
instruct the LLM assistant to solve the problem collaboratively with the user
(effectively the user proxy agent in the MathChat system) in a certain desired
manner. This framework is designed in this conversational manner in order to
leverage the chat-optimized feature of state-of-the-art LLMs, e.g., GPT-4.
Another distinct benefit of this framework is that it enables multi-turn
dialogues, which can be particularly useful in addressing complex issues that
require multi-step reasoning and tool using.

Figure 4: The prompt used in the initial message of the user proxy agent in
MathChat. It instructs the LLM assistant to solve a problem collaboratively
with the user proxy agent in a certain way.

Prompting and tool-using in MathChat. With proper modifications, effective
prompting methods from existing research, such as CoT and tool-using, can be
integrated into the MathChat framework. Specifically, for the prompt in the
initial message, we aggregate multiple effective prompting techniques to
instruct the LLM assistant. We present the designed prompt in Figure 4, which
consists of three main components.

  * •

Tool-using Prompt: This component prompts the LLM to use Python programming in
the correct format to tackle the problem. We use the ‘query requirement’
subsection to specify the coding format so that the user proxy agent can parse
the code and return the corresponding results.

  * •

Problem-Solving Strategy Selection Prompt: This component instructs the
assistant to select from three possible problem-solving strategies and to
perform multi-stage reasoning and tool-using in the last strategy. The
problem-solving strategies include the following three cases, which cover the
most effective strategies from existing literature on math problem-solving.
_(Case 1) Write a Python program to solve the problem directly._ This
corresponds to single-stage tool-using methods similar to [10, 6, 3]. _(Case
2) Solve the problem directly without Python._ This strategy allows GPT-4 to
exercise its inherent reasoning capacity to solve the problem at hand. _(Case
3) Solve the problem step by step and use Python to help with math
operations._ If the first two ways are not suitable, we ask the model to
choose this way to solve the problem. We craft a zero-shot version of the
multi-step tool-using prompt that allows the model to flexibly interleave
between multi-step reasoning and Python code, similar to [37, 23, 28]. In this
case, we also ask the model to handle errors and unexpected results from the
run of programs [21].

  * •

Final Answer Encapsulation Prompt: This component of the prompt instructs the
LLM assistant to enclose the final answer in \boxed{}, which will be used as
an indicator to end the conversation. This interaction between the LLM
assistant (GPT-4) and the user proxy agent will not be ended until \boxed{} is
detected or max rounds of conversations are reached.

We acknowledge that there could be alternative ways to design the prompt.
Fortunately, it is fairly easy to refine the prompt, for example, further
enabling the usage of Wolfram Alpha in addition to Python, in our framework.
We perform an empirical evaluation accordingly to test two alternative
versions of the prompt in Section 5.2.

##  4 Evaluation

Dataset. We perform evaluations on all the level-5 (the highest difficulty
level) problems from the test set of MATH dataset [11]. Compared to other
datasets for mathematical problems such as GSM8k [4], the level-5 problems are
much more challenging and include the application of theorems and complex
equation derivation. The MATH dataset has 7 categories of problems:
Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry,
Intermediate Algebra, and Precalculus. In our evaluation, we remove Geometry
from the evaluation to make it consistent with previous work [7]. Most
geometry problems from this dataset contain an Asymptote code to plot the
figure. But the currently available version of GPT-4 cannot accept image
input. If the raw code is included, it can leak information to the model
through exact numbers for the coordinates. Taking these issues into
consideration, we skip the evaluation on Geometry problems and remove ASY code
from all the other categories (though this could result in a lack of enough
information for some problems). The correct answer to each problem is
deterministic and is enclosed in \boxed{} in the dataset as ground truth (but
not disclosed to the methods solving the problem).

Evaluated Methods. Most previous work uses few-shot examples to elicit the
reasoning of LLMs and tool-using so that the model will imitate the examples
to follow a desired format or pattern. It is important to select similar
examples to the unanswered problem, and then annotate the examples to cover
all the cases that the LLMs might encounter. A considerable amount of effort
and careful consideration are required in this process. For example, [14, 39]
relies on elaborate examples to showcase the patterns, [23] maintains an
example library to choose examples, and [37] finetunes a smaller model for
annotation. Note that these methods use elementary math problems and it
requires even more effort to prepare and choose the examples needed for
challenging math problems. On the other hand, multiple existing studies [22,
1] reveal GPT-4’s remarkable capacity to follow instructions. Thus, we are
interested in zero-shot prompting techniques that could enhance math-solving
of GPT-4, without any example selection and annotations. Following this
criterion, we evaluate our MathChat framework with the introduced prompt and
the following methods which are all zero-shot methods: vanilla prompt, Program
of Thoughts [3], and the Program Synthesis prompt from [6].

  1. 1.

Vanilla prompting: GPT-4 can perform CoT reasoning without few-shot examples.
To evaluate GPT-4’s performance on solving the problem directly, we use a
default prompt adapted from the few-shot prompt in MATH dataset: "Solve the
problem carefully. Put the final answer in \boxed{}. {Problem}".

  2. 2.

Program of Thoughts (PoT): We use the zero-shot PoT prompting from [3], which
asks a model to write a Solver function to solve a problem and return the
final answer directly.

  3. 3.

Program Synthesis (PS) prompting: Similar to PoT, the Program Synthesis (PS)
prompting method [6] uses a prompt to ask the model to write a program to
solve a problem: "Write a program that answers the following question:
{Problem}"

Evaluation Details. We compare different methods on GPT-4 with the default
configurations from the OpenAI API. In MathChat, we allow a max round of 15
messages between GPT-4 and the user proxy agent. The agent will explicitly ask
GPT-4 to solve each step by itself if it detects errors from 3 consecutive
executions. To avoid extremely long responses from the user proxy agent, the
agent will replace any result that exceeds 600 tokens with a warning text in
the user message to ask the GPT-4 to revise the previous code. We manually go
through the answer of all the methods to count all the correct answers. For
vanilla prompt, Program Synthesis, and MathChat, we ask GPT-4 to enclose the
final answer in \boxed{}, so only the answers in the box will be extracted.
For PoT, we follow the original paper to take the return of the Solver
function as the final answer.

##  5 Results

###  5.1 Main Results

We perform an evaluation on six categories of level-5 problems from the MATH
dataset. We report the problem-solving accuracy of different methods in each
category in Table 1. Compared to vanilla prompting, which shows the native
capability of GPT-4, using Python with PoT or PS improves the overall accuracy
by around 10%. We can see this improvement mostly in the categories that
involve more number manipulations (Counting & Probability and Number Theory)
and more challenging categories (Intermediate Algebra and Precalculus). For
Algebra and Prealgebra, however, PoT and PS have little improvement or even
lead to lower accuracy. Compared with PoT and PS, MathChat can further improve
the total accuracy by around 6%, and have competitive performance across all
the categories. It is worth highlighting that MathChat improves the accuracy
in the Algebra category over other methods by around 15%. Considering all the
methods, Intermediate Algebra and Precalculus can only be solved with a low
accuracy rate of around 20%. More than half of the problems from the other
categories can be solved correctly by MathChat.

| Algebra | C.Prob | I.Alg | N.Theory | Prealg | Precalc | Total
---|---|---|---|---|---|---|---
Problem Count | 307 | 123 | 280 | 154 | 193 | 135 | 1192
MathChat | 59.93% | 52.03% | 17.85% | 60.39% | 60.10% |  19.26% | 44.71%
PoT | 42.67% | 50.41% | 17.50% | 54.55% | 52.33% | 16.30% | 37.67%
PS | 43.32% | 44.71% | 20.36% |  61.03% | 55.96% | 18.52% | 39.60%
Vanilla | 46.58% | 25.20% | 2.86% | 28.57% | 54.92% | 7.41% | 28.69%
Table 1: Accuracy on all the problems with difficulty level-5 from different
categories of the MATH dataset with different methods.

###  5.2 Additional evaluation on MathChat with alternative prompts

| Algebra | C.Prob | I.Alg | N.Theory | Prealg | Precalc | Total
---|---|---|---|---|---|---|---
Problem Count | 50 | 50 | 50 | 50 | 50 | 50 | 300
MathChat w/ Two-tools | 33 | 22 | 6 | 27 | 29 | 10 | 127
MathChat w/ Python | 26 | 19 | 7 | 22 | 31 | 13 | 118
MathChat | 30 | 24 | 8 | 34 | 28 | 10 | 134
PoT | 20 | 19 | 9 | 24 | 24 | 7 | 103
PS | 17 | 19 | 12 | 31 | 26 | 5 | 110
Vanilla | 26 | 13 | 1 | 17 | 21 | 1 | 79
Table 2: Additional evaluation of MathChat with two alternative prompts. 50
problems are sampled from each problem category for this evaluation. MathChat
w/Two-tools and MathChat w/ Python are two alternative prompts.

MathChat allows easy incorporation of different prompts and tools. We perform
an additional evaluation to test two alternative initial prompts with MathChat
to demonstrate its extensibility. (1) A simplified prompt with Python: In this
alternative, we only keep the ‘query requirements’ subsection for python
coding format and the step-by-step tool-using (i.e., case 3) from the default
prompt. (2) A simplified prompt with Python and Wolfram Alpha: In this
alternative, on top of alternative (1), we add Wolfram Alpha, a computational
engine, as an additional tool for the LLM assistant to choose from. Details of
these two alternative prompts are in Appendix B. We perform an evaluation on
randomly sampled 50 examples from each of the six problem categories. We also
include results from other methods on the sample problems for comparison in
Table 2. MathChat still performs better than other methods with the two newly
crafted prompts. With MathChat, the step-by-step prompt that allows both
Python and Wolfram performs the best on Algebra, while the new prompt with
only Python solves the most problems on Prealgebra and Precalculus, but has a
worse performance on Number Theories. Overall, MathChat with the default
prompt still performs the best.

##  6 Failure Analysis

Figure 5: One example is selected for each of the first two failures. Type 1
failure: in the first problem, the assistant fails to give a plausible plan.
It chooses to enumerate all sequences, and it does not use tools to help with
it. Type 2 failure: the second problem shows that the model fails to give the
correct code to solve the problem, while it follows the problem requirements
and the overall direction is correct. With minor changes to the code, the
final answer can be correct.

In this section, we analyze failure cases to gain insight to the reason of
failure and compare different methods in depth.

###  6.1 Failure reasons

We first summarize the failure cases according to the reasons for failure,
based on the systematic math problem-solving process established by George
Pólya [25]. The process consists of (1) understanding the problem; (2)
devising a plan; (3) executing the plan; (4) reviewing and extending. We
observe failures of the following three main types.

Type 1. Failure to devise or select an appropriate plan or path to solve the
problem. This type encompasses cases where GPT-4 fails to provide a proper way
to approach the problem. In these instances, the answer is consistently
incorrect, even though each individual step of the calculation is accurate.
Failure cases of this type are typically tricky problems that even math
experts find challenging.

Type 2. Failure to flawlessly execute the devised plan. Math problem-solving
requires rigorous and precise reasoning steps. A minor error in calculation or
symbol manipulation could be fatal and lead to a wrong answer. This type of
error is considered ‘minor’ because they are easier to be fixed. This type of
error contributes to a fair amount of failures, where the overall direction of
the problem-solving is correct, but one mistake in a basic derivation leads to
the wrong answer. Note that an error for Python execution is also included in
this type, where GPT-4 fails to write a runnable code leading to the error.

Type 3. Other technical errors. There are other technical errors causing the
failure. One example of such failure is lack of information due to the removal
of ASY code.

In Figure 5, we give one example for each of the first two types of failures.
More details for these examples and additional examples are in Appendix E.

###  6.2 Failure cases using different methods on GPT-4

Figure 6: An example where MathChat is correct and others fail. All other
methods fail due to Type 2 failure. 1. Vanilla prompt: when calculating $b$,
didn’t include $-4^{2}$. 2. PoT: it first calculates vieta_product wrong, even
is this is corrected, another TyperError will occur. 3. PS: it solves for $b$
correctly, but gets an ValueError when using the program to extract $m$ and
$n$.

In Table 3, we present the frequency of successful outcomes for each method
(represented in each row), while all other methods fail, categorized according
to different problem instances. This table serves to highlight the distinct
advantage that a particular method exhibits across various problem categories.
Similarly, in Table 4, we summarize the frequency of instances where one
method fails while all other methods succeed. A high number in this table
signifies the unique disadvantage of the method in question.

These statistics demonstrate the robustness of MathChat in comparison to other
methods. MathChat leverages conversation to enhance error correction when
utilizing external tools, thereby hypothesizing a reduction in failures within
the third type.

| Algebra | C.Prob | I.Alg | N.Theory | Prealg | Precalc | Total
---|---|---|---|---|---|---|---
MathChat | 27 | 8 | 21 | 13 | 6 | 9 | 84
PoT | 11 | 9 | 19 | 6 | 3 | 5 | 53
PS | 12 | 6 | 22 | 11 | 10 | 8 | 69
Vanilla | 12 | 4 | 5 | 3 | 10 | 3 | 37
Table 3: The number of problems where one math succeeds, and all the other methods fail (the higher the better for the concerned method in each row). | Algebra | C.Prob | I.Alg | N.Theory | Prealg | Precalc | Total
---|---|---|---|---|---|---|---
MathChat | 6 | 2 | 0 | 5 | 4 | 1 | 18
PoT | 22 | 5 | 0 | 6 | 18 | 2 | 53
PS | 17 | 5 | 1 | 5 | 14 | 0 | 42
Vanilla | 16 | 19 | 11 | 28 | 19 | 5 | 98
Table 4: The number of problems where one method fails and all the other
methods succeed (the lower, the better for the concerned method in each row).

We take one example from each table to analyze the failures of these methods.
We first take an Algebra problem that MathChat succeeds but others fail to
analyze the failures of other methods (Figure 6) For this problem, other
methods fail to execute this plan without any mistakes, causing the second
type of failure. While vanilla prompting has a calculation error, the other
two methods get execution errors from running the code. We run these methods
three more times and they still fail to solve the problem. From Table 4, we
take the only Precalculus instance that MathChat is wrong while all the other
methods are correct. Through investigating, we find that MathChat gives longer
solutions to the problem than all the other methods, and also contains Type 2
failures. This problem might indicate a potential correlation between the
accuracy and length of responses. We present more details in investigating
this possible correlation and also the Precalculus example in Appendix E.

##  7 Summary and future work

In this paper, we evaluate different ways of using GPT-4 in advanced math
problem-solving. We introduce MathChat, a conversational framework based on
GPT-4, that allows the model to interact with a proxy user agent to solve math
problems. MathChat is designed for chat-optimized models like GPT-4, and it is
extensible to be used with different prompts and different tools with minimal
effort. Based on the framework, we also derive a prompt that aggregates
previous prompting techniques to be used on MathChat.

Our evaluation on level-5 problems from the MATH dataset demonstrates the
effectiveness of MathChat to solve more complex and challenging problems.
Despite its improvements over previous methods, the results show that complex
math problem is still challenging for recent powerful LLMs, like GPT-4, even
with help from external tools. For one, GPT-4 fails to solve a fair amount of
problems due to error in execution. While MathChat tries to improve this,
further work can be done to enhance this framework, for example, training an
assistant model on top of the user proxy agent to give more specific
instructions [40, 38], or to derive new prompts and incorporate new tools.

Human-in-the-loop extension. MathChat with GPT-4 has great potential to be
adapted into a copilot system to assist people in solving mathematics
problems. MathChat is natural in its design to assist people in communicating
with GPT-4. With a preset prompt in MathChat, the user can directly input a
problem and get a step-by-step response. Moreover, the MathChat system could
be designed to stop at each step to allow the user to override the message of
the proxy agent. We envision the system could allow users to engage in the
math-solving process and facilitate learning.

## References

  * Bubeck et al. [2023] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_ , 2023.
  * Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ , 2021.
  * Chen et al. [2022] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen.  Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.  _arXiv preprint arXiv:2211.12588_ , 2022.
  * Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Creswell et al. [2022] Antonia Creswell, Murray Shanahan, and Irina Higgins.  Selection-inference: Exploiting large language models for interpretable logical reasoning.  _arXiv preprint arXiv:2205.09712_ , 2022.
  * Drori et al. [2022a] Iddo Drori, Sarah Zhang, Reece Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng, et al.  A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level.  _Proceedings of the National Academy of Sciences_ , 119(32):e2123433119, 2022a.
  * Drori et al. [2022b] Iddo Drori, Sarah Zhang, Reece Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng, et al.  A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level.  _Proceedings of the National Academy of Sciences_ , 119(32):e2123433119, 2022b.
  * Dua et al. [2022] Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner.  Successive prompting for decomposing complex questions.  _arXiv preprint arXiv:2212.04092_ , 2022.
  * Frieder et al. [2023] Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner.  Mathematical capabilities of chatgpt.  _arXiv preprint arXiv:2301.13867_ , 2023.
  * Gao et al. [2022] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.  Pal: Program-aided language models.  _arXiv preprint arXiv:2211.10435_ , 2022.
  * Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.  Measuring mathematical problem solving with the math dataset.  _arXiv preprint arXiv:2103.03874_ , 2021.
  * Imani et al. [2023] Shima Imani, Liang Du, and Harsh Shrivastava.  Mathprompter: Mathematical reasoning using large language models.  _arXiv preprint arXiv:2303.05398_ , 2023.
  * Jung et al. [2022] Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi.  Maieutic prompting: Logically consistent reasoning with recursive explanations.  _arXiv preprint arXiv:2205.11822_ , 2022.
  * Khot et al. [2022] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal.  Decomposed prompting: A modular approach for solving complex tasks.  _arXiv preprint arXiv:2210.02406_ , 2022.
  * Kojima et al. [2022] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  _arXiv preprint arXiv:2205.11916_ , 2022.
  * Lewkowycz et al. [2022] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al.  Solving quantitative reasoning problems with language models.  _arXiv preprint arXiv:2206.14858_ , 2022.
  * Li et al. [2022] Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen.  On the advance of making language models better reasoners.  _arXiv preprint arXiv:2206.02336_ , 2022.
  * Ling et al. [2017] Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom.  Program induction by rationale generation: Learning to solve and explain algebraic word problems.  _arXiv preprint arXiv:1705.04146_ , 2017.
  * Liu et al. [2021] Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi.  Generated knowledge prompting for commonsense reasoning.  _arXiv preprint arXiv:2110.08387_ , 2021.
  * Madaan et al. [2023] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al.  Self-refine: Iterative refinement with self-feedback.  _arXiv preprint arXiv:2303.17651_ , 2023.
  * Ni et al. [2023] Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I Wang, and Xi Victoria Lin.  Lever: Learning to verify language-to-code generation with execution.  _arXiv preprint arXiv:2302.08468_ , 2023.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Paranjape et al. [2023] Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro.  Art: Automatic multi-step reasoning and tool-use for large language models.  _arXiv preprint arXiv:2303.09014_ , 2023.
  * Patel et al. [2021] Arkil Patel, Satwik Bhattamishra, and Navin Goyal.  Are nlp models really able to solve simple math word problems?  _arXiv preprint arXiv:2103.07191_ , 2021.
  * Polya [2004] George Polya.  _How to solve it: A new aspect of mathematical method_.  Number 246. Princeton university press, 2004.
  * Press et al. [2022] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis.  Measuring and narrowing the compositionality gap in language models.  _arXiv preprint arXiv:2210.03350_ , 2022.
  * Roberts et al. [2019] Adam Roberts, Colin Raffel, Katherine Lee, Michael Matena, Noam Shazeer, Peter J Liu, Sharan Narang, Wei Li, and Yanqi Zhou.  Exploring the limits of transfer learning with a unified text-to-text transformer.  2019\.
  * Schick et al. [2023] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_ , 2023.
  * Taylor et al. [2022] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic.  Galactica: A large language model for science.  _arXiv preprint arXiv:2211.09085_ , 2022.
  * Wang et al. [2022a] Boshi Wang, Xiang Deng, and Huan Sun.  Shepherd pre-trained language models to develop a train of thought: An iterative prompting approach.  _arXiv preprint arXiv:2203.08383_ , 2022a.
  * Wang et al. [2022b] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_ , 2022b.
  * Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ , 2022.
  * Weng et al. [2022] Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu, and Jun Zhao.  Large language models are reasoners with self-verification.  _arXiv preprint arXiv:2212.09561_ , 2022.
  * Wigner [1990] Eugene P Wigner.  The unreasonable effectiveness of mathematics in the natural sciences.  In _Mathematics and science_ , pages 291–306. World Scientific, 1990\.
  * Yang et al. [2022a] Jingfeng Yang, Haoming Jiang, Qingyu Yin, Danqing Zhang, Bing Yin, and Diyi Yang.  Seqzero: Few-shot compositional semantic parsing with sequential prompts and zero-shot models.  _arXiv preprint arXiv:2205.07381_ , 2022a.
  * Yang et al. [2022b] Zhicheng Yang, Jinghui Qin, Jiaqi Chen, Liang Lin, and Xiaodan Liang.  Logicsolver: Towards interpretable math word problem solving with logical prompt-enhanced learning.  _arXiv preprint arXiv:2205.08232_ , 2022b.
  * Yao et al. [2022] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.  React: Synergizing reasoning and acting in language models.  _arXiv preprint arXiv:2210.03629_ , 2022.
  * Zelikman et al. [2022] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman.  Star: Bootstrapping reasoning with reasoning.  _Advances in Neural Information Processing Systems_ , 35:15476–15488, 2022.
  * Zhou et al. [2022] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi.  Least-to-most prompting enables complex reasoning in large language models.  _arXiv preprint arXiv:2205.10625_ , 2022.
  * Zhu et al. [2022] Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang.  Solving math word problem via cooperative reasoning induced language models.  _arXiv preprint arXiv:2210.16257_ , 2022.

##  Appendix A Supplementary Details on the User Proxy Agent

The user proxy agent in MathChat takes a problem and put it in a message with
an initial prompt, and sends the message to the LLM assistant. Then the agent
is responsible for extracting and executing queries and also providing
additional guidance. Here are all functionalities of the user proxy agent (the
workflow is shown in Figure 2):

  1. 1.

Extract Queries: The user proxy agent needs to match the pattern specified in
the initial message to extract all tool-using queries. With our designed
prompt, the agent matches all code blocks in the message and extracts the
code.

  2. 2.

"Continue": If no query is detected in the message, the agent will send this
message to the LLM assistant: "Continue. Please keep solving the problem until
you need to query. (If you get to the answer, put it in \boxed{}.". This asks
the assistant to keep solving the problem and reminds it to end the
conversation by putting the answer in the box.

  3. 3.

Query Execution: Any tool-using queries extracted will be executed
sequentially. For Python, we set the time limit to be 5 seconds for execution.
As shown in Figure 1, the previous valid code is recorded. All the execution
results will be concatenated sequentially (including errors).

  4. 4.

Recurrent Error detection: If LLM assistant sends 3 consecutive errors, the
user proxy agent will replace the third error message with this message:
"Please revisit the problem statement and your reasoning. If you think this
step is correct, solve it yourself and continue the next step. Otherwise,
correct this step.". To avoid sticking to this error, the LLM assistant is
asked to solve this step without tools and move on.

  5. 5.

Repetitive results: This is not shown in the workflow, but the agent also
detects another situation where the LLM assistant gives the same tool-using
query from the last one or the result is the same as the last query. Then the
message is appended to the execution result to remind the assistant to avoid
giving the same queries: "Your query or result is same from the last, please
try a new approach.".

  6. 6.

Long query results: It is possible that LLM assistant requires a query result
that is too long to be passed back (such as long results from the print
function in a for loop in Python). The proxy agent will replace any query
result that is longer than 2000 chars (approximately 600 tokens) with this
message: "Your requested query response is too long. You might have made a
mistake. Please revise your reasoning and query.".

In MathChat, if the tool-using query and the end indicator are detected in the
same message, the result from the query will be returned, and the conversation
will continue. This is to prevent early stops where the LLM agent predicts the
execution result and puts it in a box other than waiting for the result.

##  Appendix B Supplementary Details on Experiment Settings

The code is in this
GitHub111https://github.com/kevin666aa/FLAML/tree/gpt_math_solver/flaml/autogen/math
repository. In our experiment, we use the default configuration from OpenAI,
specifically temperature=1, and max_token=inf (See OpenAI API Reference for
more details). We use the system message "You are a helpful assistant" for
vanilla prompt, PS, and MathChat. For PoT, we do not add this system message,
since our evaluation shows that PoT without system message has a better
performance. We discuss the effect of system message below in Section D.

Here is the prompts for PoT [3], PS [6], and the additional two prompts we
designed:

  * •

Program of Thoughts (PoT). See Figure 7. The whole prompt uses the Python code
format, where information such as the problem or instructions is in the
comments.

  * •

Program Synthesis (PS). The prompt for PS is "Write a program that answers the
following question: {Problem}". Since the definition of "program" is unclear
and sometimes the LLM assistant won’t write code to solve the problem, we add
the keyword ’Python’ in front of ’program’. After the message is returned, we
used the proxy agent to return the result (by default, GPT-4 would return the
code in the code block). Then we send another message to the model with the
Python execution result and ask it to enclose the final answer: {Return from
Python}. Please put the final answer in \boxed{}. See Figure 10 for an example
of the whole process.

  * •

Python prompt (w/ MathChat). See Figure 8.

  * •

Two-tools prompt (w/ MathChat). See Figure 9.

Figure 7: PoT prompt. Comparing to the original prompt from [3], we add
"import sympy as sp" that gives the LLM assistant hint to use the sympy
library. The placeholder "{problem}" will be replaced with the actual problem.
Figure 8: The Python prompt used on MathChat from Section 5.2. Figure 9: The
Two-tools prompt used on MathChat from Section 5.2. The added requirement
compared to the Python prompt is highlighted in yellow. This prompt allows the
LLM assistant to choose from Python or Wolfram Alpha. Figure 10: An example of
the process of PS. The query result will be returned to the LLM assistant and
ask it to put the answer in box. The process of PS is exactly the same as
MathChat when the assistant in MathChat chooses to solve the problem with one
Python program.

##  Appendix C Additional Discussion on Related Work

While there are only a few methods targeting math problems, more prompt
techniques are proposed to improve reasoning tasks. We summarize the three
main ways that can be used to improve math solving: Chain of Thought
prompting, multi-stage prompting, and tool-using:

Chain-of-thought prompting. Chain-of-thought (CoT) prompting [32, 15] is a
type of prompting technique found useful for improving LLMs’ performance in
math problem-solving. [32] propose the initial CoT technique with step-by-step
answer examples and show that this technique could lead to better performance
on math word problems. [15] further demonstrates that CoT can be done in a
zero-shot manner by simply attaching "Let’s think step by step" to the
questions fed to LLMs. This simple CoT technique also leads to better
performance on math problem-solving tasks when used in conjunction with GPT-3.

Multi-stage prompting Another effective way is to prompt LLMs to solve math
problems in a multi-stage manner [8, 14, 26, 5, 13, 39, 23, 37, 35, 30]. The
most common strategy for multi-stage prompting is divide-and-conquer, which
decomposes complex problems into several subproblems. For example, least-to-
most prompting [39] and Decomposed prompting [14] propose to break down a
complex problem into smaller subproblems, and the subproblems are solved
sequentially to reach the final solution. [19] first uses GPT-3 to generate
knowledge statements for a problem and then presents both the question and
generated knowledge to another model (T5-11b [27], etc.) to answer the
question.

Tool-using. As we discussed in Section 2, a particular effort for math solving
is to use Python code [10, 3, 6]. [28, 23, 37] further combine multi-stage
prompting with external tools in the usage of LLMs on math problem-solving.
Specifically, [28] trains a Toolformer that can decide when to send API calls.
[37, 23] both use few-shot prompting to interleave step-by-step reasoning and
tool-using. ReAct [37] finetunes a smaller model in a bootstrapping manner to
help annotation, while ART [23] maintains a library of demonstrations, such as
using calculators and retrieves the most similar examples for prompting a new
problem.

#### Other methods.

There exist other methods that are potentially useful to improve math problem-
solving with LLMs. Self-consistency [31], built on top of CoT, samples several
different reasoning paths for a problem and selects the answer with the
majority vote. [17] extends self-consistency by training a verifier to verify
the correctness of each step. [20] first generates an output using recent LLMs
like GPT-3.5 and uses the same model to give feedback on the output, then the
model is asked to refine the output based on the feedback. We also refer to
[16, 29] for training LLM on Science problems.

##  Appendix D Supplementary Experiments and Results

We further evaluate a vanilla few-shot prompt, PoT with and without system
message, and Vanilla prompt with and without system message on the randomly
selected 50 problems from each category and present the results in Figure 5.

In the few-shot prompt, we randomly select 3 level-5 problem-solution pairs
from the train set. These examples are selected from each category and are
used for all the problems from that category. The vanilla few-shot prompt
starts with "Solve the problem carefully. Put the final answer in \boxed{}
just like the vanilla prompt, and then three "Problem: ... Solution: ..."
pairs are attached. Compared with the vanilla prompt, adding three additional
examples does not make any obvious difference, and the overall performance is
slightly worse.

From our experiment, we also notice that the system message affects the
performance of the LLM assistant. However, the impact significantly differs
between methods. As shown in Table 5, using a system message is crucial for
the Vanilla prompt: adding the system message doubles the success rate
compared to the one without a system message. However, for PoT, adding the
system message only slightly increases the performance. We add a further
evaluation on all the level-5 problems and find the PoT with the system
message has an overall accuracy of 35.82%, which is lower than the accuracy of
PoT without the system message (37.67% as shown in the main results in Table
1). We hypothesize the difference in the prompt format across different
methods is the reason for this behavior. The method with the Vanilla prompt
imitates a conversation between the LLM assistant and humans via natural
language, but PoT prompt is in Python code format, which explicitly directs
the model for code completion. Thus, the system message "you are a helpful
assistant" is more suitable for Vanilla prompt but doesn’t align with PoT.
More investigation is needed to understand the effect of system messages.

| Algebra | C.Prob | I.Alg | N.Theory | Prealg | Precalc | Total
---|---|---|---|---|---|---|---
Problem Count | 50 | 50 | 50 | 50 | 50 | 50 | 300
MathChat | 30 | 24 | 8 | 34 | 28 | 10 | 134
PS | 17 | 19 | 12 | 31 | 26 | 5 | 110
PoT w/o sys | 20 | 19 | 9 | 24 | 24 | 7 | 103
PoT w/ sys | 18 | 23 | 9 | 23 | 29 | 7 | 109
Vanilla w/o sys | 14 | 4 | 0 | 4 | 13 | 1 | 35
Vanilla w/ sys | 26 | 13 | 1 | 17 | 21 | 1 | 79
Few-shot (k=3) | 21 | 6 | 2 | 18 | 24 | 1 | 72
Table 5: Results for few-shot prompt, PoT w/ and w/o system message, Vanilla
prompt w/ and w/o system message.

##  Appendix E Supplementary Failure Analysis

###  E.1 Failure under different forms of problem-solving processes in
MathChat

The default prompt in MathChat allows the LLM assistant to choose from
different forms of problem-solving processes to solve the problem, and we
investigate how choosing different forms could affect the performance. We plot
the correct rate when a problem from each category is solved with three forms
of problem-solving approaches depending on the existence and validity of the
query in the generated solution in Figure 11: 1. The LLM Assistant doesn’t
make any tool-using queries (Python) when solving the problem. 2. The
assistant makes one or more queries, but at least one query is invalid. 3. The
assistant makes all valid queries. It is shown in the plot that using Python
correctly could significantly increase the correct rate, while doesn’t use
Python is worse than using Python but having invalid queries. The results in
Figure 11 show that especially for intermediate algebra and prealgebra, the
gap in accuracy between "using no query" and "have invalid query" is large,
indicating that using Python is very helpful to solve problems from the two
categories.

Figure 11: Success rate of MathChat under different forms of problem-solving
processes: 1. the LLM assistant solves the problem without making any tool-
using queries. 2. The assistant makes queries and has at least one invalid
query 3. All queries made are valid.  Figure 12: Additional example of Type 1
failure (Fail to devise a proper plan) and Type 2 failure (Fail to execute the
plan flawlessly). Figure 13: An example of Type 3 failure where the ASY code
is removed.

###  E.2 Examples of 3 types of failures

In Section 6.1, we summarize 3 main type of failures: type 1: failure to
devise an appropriate plan. type 2: failure to flawlessly execute the plan.
type 3: other technical errors. We give one additional example for type 1
error and type 2 error, and an example where the removal of ASY code leads to
a leak of information (Figure 12, Figure 13). We note that among all the
problems, the ASY code from 72 problems are removed, but 12 problems can still
be solved correctly.

###  E.3 Failure under different methods

We present the precalculus example where MathChat fails but all other methods
success (Figure 15, Figure 16, Figure 17). The results from PS and PoT show
that it is easy to get this problem correct with Python (using the
sympy.simplify function). However, in MathChat, the LLM assistant chooses to
solve the problem via direct reasoning. Both MathChat and vanilla prompt solve
this problem by writing extremely long derivations. MathChat solves the
problem with an even longer step-by-step response and makes a calculation
error during the process.

Additionally, we also provide an overview of the number of problems where all
methods either fail or succeed in Table 6.

| Algebra | C.Prob | I.Alg | N.Theory | Prealg | Precalc | Total
---|---|---|---|---|---|---|---
All Success | 46 | 13 | 0 | 18 | 45 | 1 | 176
All Fail | 57 | 32 | 171 | 20 | 36 | 86 | 402
Table 6: The number of problems where all methods fail, and all methods
succeed.

###  E.4 The Relationship Between Failure Rate and Generated Solution Length

Figure 14: Distribution of solution length of both correctly and incorrectly
solved problems in MathChat. The distribution of length of the given solution
(ground truth) is also shown. The left figure represents the distribution of
the less challenging categories and the right figure represents problems from
Intermediate Algebra and Precalculus. We cut off outliers that the split
string length is longer than 1500.

Chain of Thought (CoT) prompting shows that extra reasoning steps for a
problem can improve the ability of LLMs [32]. With GPT-4, explicit reasoning
is no longer an issue. Instead, we find that a long and tedious reasoning
process may result in more type 2 failures, such as calculation errors, which
results in a wrong answer even the overall direction is correct. We plot the
distribution of correct and wrong answer lengths and also the answer length of
the given solution (The length of the string list from splitting with a single
space is used here). Since more complex and challenging problems are likely to
have a longer solving process but still a lower success rate, we separate
problems from Intermediate Algebra and Precalculus with other categories
(Figure 14), to distinguish less challenging problems from harder problems. We
note that the success rate of MathChat on the four less challenging categories
goes over 50%, but the rate is lower than 20% for Intermediate Algebra and
Precalculus.

Overall, the solution length of MathChat is longer than the ground truth
solution. The length of the given solution on the two fundamentally
challenging categories is longer than other categories. For MathChat, correct
answers and wrong answers from the less challenging categories have a similar
distribution in solution length, where the majority of problems are solved
with 50 to 500 string length. For harder problems, however, an answer with
more than 600 string lengths is likely to be wrong. From the precalculus
problem shown in Figure 17, the LLM Assistant can choose a plausible strategy
to solve the problem, but that strategy is less efficient and involve more
math operations compared to the given solution, this results in a much longer
response, and it is more likely to make errors during the process.

Figure 15: The precalculus problem where other methods are correct but
MathChat is wrong. This figure shows the ground truth solution and the
response with vanilla prompt. Figure 16: The precalculus problem where other
methods are correct but MathChat is wrong (Continued). This figure shows the
PS and PoT code. Both code returns the correct result: "D". Figure 17: The
precalculus example where all the other methods are correct but MathChat is
wrong (Continued). This figure shows the conversation generated in MathChat.
The LLM assistant in MathChat chooses to solve the problem via direct
reasoning, and it makes a calculation error when expanding the terms in Step
3.


# Questionnaires for Everyone: Streamlining Cross-Cultural Questionnaire
Adaptation with GPT-Based Translation Quality Evaluation

Otso Haavisto  0009-0001-5865-1047<EMAIL_ADDRESS>Aalto
UniversityEspoo02150Finland and  Robin Welsch  0000-0002-7255-7890
<EMAIL_ADDRESS>Aalto UniversityEspoo02150Finland

###### Abstract.

Adapting questionnaires to new languages is a resource-intensive process often
requiring the hiring of multiple independent translators, which limits the
ability of researchers to conduct cross-cultural research and effectively
creates inequalities in research and society. This work presents a prototype
tool that can expedite the questionnaire translation process. The tool
incorporates forward-backward translation using DeepL alongside
GPT-4-generated translation quality evaluations and improvement suggestions.
We conducted two online studies in which participants translated
questionnaires from English to either German (Study 1; $n=10$) or Portuguese
(Study 2; $n=20$) using our prototype. To evaluate the quality of the
translations created using the tool, evaluation scores between conventionally
translated and tool-supported versions were compared. Our results indicate
that integrating LLM-generated translation quality evaluations and suggestions
for improvement can help users independently attain results similar to those
provided by conventional, non-NLP-supported translation methods. This is the
first step towards more equitable questionnaire-based research, powered by AI.

Figure 1. The translation flow of the final prototype application. Users are
able to translate questionnaires forwards and backwards; edit the translations
manually; and to request GPT-4-generated evaluation of their translation.

The flow within the prototype application. First, users enter the
questionnaire to translate. After a forward machine translation to the target
language has been performed, users can manually edit the translation and
backtranslate it to the source language for comparison to the original.
Finally, users can request a GPT-generated numeric and verbal evaluation of
the translated questionnaire.

##  1\. Introduction

Currently, most HCI studies utilizing standardized user experience
questionnaires are conducted in Europe (Díaz-Oreiro et al., 2021). In
psychology, the availability of clinical questionnaires vary greatly between
high- and low-income countries (Ali et al., 2016; Hoffman et al., 2022). This
creates structural inequalities for conducting high-quality questionnaire-
based research in non-English-speaking and low-income parts of the world.

When questionnaires are adapted to new languages, Brislin’s (Brislin, 1970)
translation methods for cross-cultural research are commonly cited. However,
using these is not always straightforward. Cha et al. (Cha et al., 2007) note
that, while Brislin’s backtranslation and committee methods undoubtedly yield
the best results, the number of translators required to reach a consensus is
not always known, which makes the process resource-intensive. For example, in
the backtranslation approach, the source text is translated forward (to the
target language) and backward (to the original language) by separate and
independent translators until the two versions are identical.

In practice, this can be seen in the adoption of mixed-methods translation
processes. Two recent studies adapting questionnaires to Finnish by Heilala et
al. (Heilala et al., 2023) and Westerlund et al. (Westerlund et al., 2023)
used combined approaches, with the authors taking part in the conflict
resolution process in both cases and even doing a part of the translation in
the latter. Notably, while both studies employed the backtranslation method,
neither used it iteratively as intended (Brislin, 1970). Partly automating the
translation process would lower the threshold of iterative questionnaire
translation, potentially resulting in higher-quality translations overall.

We set out to develop a prototype of a questionnaire translation tool that
would exploit the versatility of large language models (LLMs) in natural
language processing tasks (e.g. (Gillioz et al., 2020; Floridi and Chiriatti,
2020; Brin et al., 2023; Kasneci et al., 2023)) to the benefit of researchers
conducting cross-cultural studies. The prototype allows users to translate
questionnaires, edit translations, backtranslate to the source language for
comparisons against the original and receive evaluations of the quality of the
translation generated by GPT-4 (OpenAI et al., 2023).

To evaluate the assistive potential of LLMs in questionnaire translation, we
conducted two online studies in which participants used our prototype to
translate the ATI scale (Franke et al., 2019) from English to German (Study 1)
and BFI-10 (Rammstedt and John, 2007) from English to Portuguese (Study 2). In
both studies, participants reached equivalent translation quality to
conventionally translated (i.e. not translated using NLP tools) versions of
the questionnaires using machine translation and GPT-4-generated quality
evaluations and suggestions for improvement. Participants found the
evaluations helpful and made use of the suggestions provided by the prototype:
In the latter study, over half of the participants reported having implemented
at least some of the suggestions.

##  2\. Background

Cha et al. (Cha et al., 2007) note that there is “no gold standard” for
questionnaire adaptation due to varying resources and research contexts.
Another literature review of guidelines for cross-cultural adaptation of
questionnaires by Epstein et al. (Epstein et al., 2015) could not find a
consensus regarding recommendations either. However, while the translation
methods themselves have not been standardized, the questionnaire adaptation
process should address a few key issues.

Translation, in general, can be approached from multiple different angles,
which in turn determine the criteria for a translation being good (House,
2014). In questionnaire translation, the semantic similarity between the
translated version and the original is often more important than word-by-word
match (Su and Parham, 2002); both versions should equally measure the same
phenomena.

The construction of the original questionnaire also affects how well it
translates over to other languages: Using colloquial or culture-specific
terminology makes it difficult for a translated questionnaire to be accurate
(Brislin, 1970; Su and Parham, 2002). Finally, questionnaires should undergo
an intensive validation process with real participants before they are
deployed in research (Brislin, 1970; Epstein et al., 2015; Taherdoost, 2016).
Kocmi and Federmann (Kocmi and Federmann, 2023) evaluated a number of LLMs as
translation quality assessors. In their work, they developed a series of
prompts that would return quantifiable scores for a translation, collectively
named GEMBA. The authors argue that LLMs — especially GPT-4 — are “the state-
of-the-art” in translation quality assessment. Promising results were also
achieved by Lu et al. (Lu et al., 2023), who attained a similar level of
quality when prompting a verbal analysis of errors in a given translation when
compared to a human reference.

The cross-cultural questionnaire adaptation process could, then, be considered
to consist of two major steps: translation and validation. As all
questionnaire translations should undergo a rigorous validation process before
they are applied in practice regardless of the method used to translate them,
incorporating LLM-prompted heuristic assessments of semantic similarity and
suggestions for improvements to translation software has a higher potential to
be helpful to researchers during the translation process.

##  3\. Prototype

###  3.1. Task Analysis

To begin the prototyping process, hierarchical task analyses (HTA; (Shepherd,
1998)) were performed on existing methods for cross-cultural questionnaire
adaptation. The methods selected were Brislin’s (Brislin, 1970)
backtranslation and committee approaches and the recent approaches taken by
Heilala et al. (Heilala et al., 2023) and Westerlund et al. (Westerlund et
al., 2023) when translating the ATI (Franke et al., 2019) and CLEFT-Q
(Tsangaris et al., 2017) questionnaires into Finnish. The four HTA graphs are
presented in Appendix A alongside snapshots of the prototyping process.
Questionnaire validation was not broken down following the recommendations by
both Cha et al. (Cha et al., 2007) and Epstein et al. (Epstein et al., 2015)
to keep the validation process separate.

The HTA identified three common features in the translation processes. First,
all used independent translators at some point in the process. Second, all
groups used discussions as a tool for assessing translation quality and
resolving potential conflicts of opinion or ambiguities in meaning. Finally,
three out of four of the processes described the use of forward-backward
translation, in which a translated version of the questionnaire is translated
back to its original language and the two original-language versions compared.
The aim of using forward-backward translation is to assert a high level of
interchangeability between the original and final translation.

###  3.2. Prototype Construction

An initial interactive wireframe based on the task analyses and sketches was
constructed for usability testing. Two participants, both acquainted with HCI
research (a doctoral researcher and a research assistant), were asked to
translate a four-item questionnaire from English to Finnish while explaining
their thought process. The issues uncovered by the usability tests were
addressed during the development of the functional prototype.

Based on the results of the usability tests, a functional prototype was
developed. The result was a web application with a front-end developed with
React.JS111https://react.dev/; Version 18.2.0 and the ChakraUI component
library222https://chakra-ui.com/; Version 2.8.2 and a server developed with
Python333https://www.python.org/; Version 3.12.0 using
Flask444https://flask.palletsprojects.com/en/3.0.x/; Version 3.0.0. As a final
measure, the Aalto Interface Metrics (Oulasvirta et al., 2018) evaluation
tool’s UMSI (Fosco et al., 2020) functionality was used to confirm that the
interface maintained a coherent visual hierarchy. The user flow in the
prototype can be seen in Figure 1, with an overview of the editing and
evaluating interface in Figure 2. The initial wireframe, the resulting
prototype, the saliency heatmap and an example sequence diagram of interacting
with the application are included in Appendix B.

Figure 2. The view for editing and evaluating translations in the final
prototype application.

The server employed the DeepL and OpenAI Python
libraries555https://github.com/DeepLcom/deepl-python; Version
1.16.1666https://github.com/openai/openai-python; Version 1.3.8 to generate
neural machine translations and GPT-4-generated translation evaluations
respectively. DeepL was chosen over GPT-4 as the translation provider due to
its API providing reliable output and promising results in translating
scientific text (Takakusagi et al., 2021).

Two evaluation methods were offered: GEMBA-DA (Kocmi and Federmann, 2023)
(using GPT-4 and the [noref] version of the prompt) and a custom semantic
similarity assessment (SSA; using GPT-4-1106-preview for its JSON mode)
intended to provide concrete suggestions for improving the semantic similarity
of the translation compared to the original (Appendix C). As the results by
Kocmi and Federmann (Kocmi and Federmann, 2023) and Lu et al. (Lu et al.,
2023) indicated that GPT-based evaluations would not achieve a desired level
of quality in segment-level evaluation, the prompt included the original and
translated contents in full.

The code for the prototype is publicly available on GitHub777Front-end:
https://github.com/otsha/questionnaires-for-everyone888Back-end:
https://github.com/otsha/questionnaires-for-everyone-server.

##  4\. Study 1: Prototype-assisted Translation of the ATI Scale from English
to German

###  4.1. Method

As discussed, large language models have shown promise in delivering human-
readable translation quality evaluations (Kocmi and Federmann, 2023; Lu et
al., 2023). While delivering a good user experience was one of the goals of
developing the prototype, it was only a means to answer the present study’s
research question:

RQ: Do LLM-generated evaluations help users reach similar questionnaire
translation quality to that achieved with conventional, non-NLP-supported,
methods?

The study was conducted online with the prototype hosted on
Netlify999https://www.netlify.com/. To compare conventional and LLM-assisted
translation methods, participants were instructed to utilize the tool to
translate the ATI questionnaire (Franke et al., 2019) from English to German
—, a language a validated version of the questionnaire is available in. Each
participant was asked to use the tool until they thought they had achieved a
satisfying result, after which they recorded their GEMBA-DA and semantic
similarity scores. The participant scores were then compared with scores
received by the conventionally translated and validated German version.

Participant experiences with the system as a whole were gathered with the
system usability scale (SUS; (Brooke, 1996)) and their opinion of the quality
of the evaluations with the short version of the user experience questionnaire
(UEQ-S; (Schrepp et al., 2017)). Additional measures gathered participants’
opinions of the correspondence of the scores to the translation quality and
thoughts on the evaluations. Demographic information such as the participants’
age, gender, occupation, fluency in Finnish and German and prior experiences
with questionnaire translation and using LLMs for translation were also
requested.

###  4.2. Participants

10 German-speaking participants (5 M; 5 F) between ages 24 and 59 ($M=32.9$;
$SD=12.1$) were recruited via Prolific. Participants rated their fluency in
English and German on scales of 1 (No knowledge at all) to 7 (Native). Median
German fluency was 4.0 ($IQR=3.75$; $Min=2.0$; $Max=7.0$) with 1 participant
estimating their fluency to be 1–2 and 4 participants giving their fluency a
rating of 6–7. Median English fluency was 6.0 ($IQR=0.75$; $Min=4.0$;
$Max=6.0$).

4 participants reported having participated in questionnaire translation
before, while 4 had used LLMs for translation in general.

Informed consent was collected from each participant at the beginning of the
study. The participants were also informed of their right to stop
participating at any time. Participants were compensated £2.00 upon completion
of the study, which took them approximately 9 minutes.

###  4.3. Analysis

1 participant was excluded due to not translating the questionnaire as
instructed. 2 participants were further excluded due to them not using the
designated method for importing their results to the questionnaire. This left
7 participants for the analysis.

In order to be able to compare the translations achieved by the participants
to the versions translated using conventional, non-NLP-supported methods, the
conventionally translated German language version of the ATI (Franke et al.,
2019) was evaluated using the two metrics — GEMBA-DA and SSA — available in
the tool. Following the number of participants left after exclusions and to
account for potential variation in scores given by GPT-4, each metric was run
on the original questionnaire 7 times.

###  4.4. Results

####  4.4.1. Evaluation Scores

The original German translation of the ATI scale received a mean GEMBA-DA
score of 100 ($SD=0$) and a mean SSA score of 97.86 ($SD=2.47$).

Four participants reported implementing at least some of the suggestions
generated by the evaluation. However, only four participants returned their
evaluation scores. Of those, all achieved scores of 100 out of 100 for both
GEMBA-DA and SSA evaluations. We used the evaluation prompts to separately
evaluate the remaining three translations, all of which also scored 100 on
both measures.

####  4.4.2. Participant Experience

5 participants reported having implemented at least some of the changes
suggested by the semantic similarity evaluation (in response to ”How many of
the suggestions given in the evaluation did you implement?”; 4-step scale:
”None”, ”Some”, ”Most”, ”All”). When asked to rate how well the numeric
evaluation scores matched the translation quality (on a scale from 1–7; Very
badly–Very well), participants gave the evaluations a median score of 7.0
($IQR=0.5$).

The median SUS score given to the prototype was 92.50 ($IQR=13.75$;
$Mean=86.43$; $SD=16.47$), indicating a relatively high degree of usability.
The UEQ-S scores indicate that participants perceived the prototype to have a
higher pragmatic ($Median=1.50$; $IQR=0.38$) than hedonic ($Median=0.50$;
$IQR=1.63$) quality.

One participant was particularly impressed by the evaluation quality: ”I’m
surprised at how accurate the evaluation is. Although not a professional
translator, I have done some translation work in the past, and I think this
tool can further enhance the quality of translations in the context of
questionnaires.” (P10). Another might have been slightly confused by GPT-4’s
tendency to suggest changes even if none were needed: ”I think the evaluations
made were pretty spot on. Some just said it were not matching exactly the
english version but from what I could tell it seemed exactly the same to me”
(P8).

##  5\. Study 2: Prototype-assisted Translation of the BFI-10 Scale from
English to Portuguese

In Study 1, Participants managed to achieve evaluation scores similar to those
received by the conventionally translated questionnaire. Outlook on the user
experience of the tool was generally positive.

The most significant limitations of this study were the low number of
participants and that participants did not follow the instructions correctly.
Due to these issues, we decided to run a follow-up study with a larger sample
that would clarify the instructions.

###  5.1. Method

The follow-up study was shorter and focused more on the evaluations. As the
user experience of the tool seemed to be quite good based on the first study,
we decided not to include the SUS and UEQ-S questionnaires.

In the second study, participants were tasked to translate the BFI-10
(Rammstedt and John, 2007) personality questionnaire from English to
Portuguese; a language the questionnaire has not yet been translated to using
conventional methods and one that features a large participant pool on
Prolific.

###  5.2. Participants

20 Portuguese-speaking participants (9 M; 11 F) between ages 21 and 44
($M=27.65$; $SD=6.05$) and no self-reported difficulty with reading were
recruited via Prolific. Median self-reported Portuguese fluency was 7.0
($IQR=0.0$; $Min=5.0$; $Max=7.0$), and English fluency 6.0 ($IQR=1.0$;
$Min=4.0$; $Max=6.0$). 6 of the participants had been involved in translating
a questionnaire. 12 participants had used LLMs for translation.

Informed consent was collected from the participants at the beginning of the
study. Participants were compensated £1.32 upon completion of the study. Based
on the first study, we estimated the shortened survey to take the participants
approximately 8 minutes, but participants ended up spending more time on
completion (around 11 minutes on average).

###  5.3. Analysis

We scored 4 conventionally translated versions of the BFI-10 (Danish (Palsson
et al., 2023), French (Courtois et al., 2020), German (Rammstedt and John,
2007) and Chinese (Carciofo et al., 2016)) individually using both GEMBA-DA
and SSA to obtain a baseline for comparison. Matching the number of
participants, each version was scored 20 times on both metrics. Then, the
scores achieved by the participants using our prototype were compared against
scores received by the conventionally translated versions.

###  5.4. Results

####  5.4.1. Evaluation Scores

The conventionally translated and validated versions received a mean GEMBA-DA
score of 97.56 ($SD=2.66$) and a mean SSA score of 94.88 ($SD=0.78$). A one-
way ANOVA revealed that there was statistically significant language-dependent
variation in the mean GEMBA scores received by these versions
($F\textsubscript{Trad-GEMBA-DA}(3,76)=17.39;p<0.001$). Post-hoc pairwise
t-tests showed that the Chinese-language translation (Carciofo et al., 2016)
was an outlier, receiving lower GEMBA-DA scores ($M\textsubscript{Trad-ZH-
GEMBA-DA}=94.75;SD=2.86$) compared to the Danish ($t=-4.43;p<0.001$), French
($t=-6.46;p<0.001$) and German ($t=-6.01;p<0.001$) versions. However, no
significant language-dependent variation was found in the SSA scores achieved
by the conventionally translated versions ($F\textsubscript{Trad-
SSA}(3,76)=6.0;p>0.1$). This indicates that even if the translation quality of
the Chinese-language version was not as high as the other versions, it
delivered the semantics of the original questionnaire equally well.

Translations generated by the participants had a mean GEMBA-DA score of 99.75
($SD=1.09$; $Min=95$; $Max=100$) and a mean SSA score of 95.75 ($SD=2.75$;
$Min=90$; $Max=100$). The tool-assisted GEMBA-DA scores did significantly
differ from scores achieved by conventional methods
($F\textsubscript{GEMBA}(4,95)=20.27;p<0.001$). Pairwise t-tests showed that
participants achieved significantly higher scores than both the Danish
($t=3.33;p<0.01$) and Chinese ($t=8.12;p<0.001$) versions. The SSA scores
achieved by participants did not significantly differ from those received by
the conventionally translated versions
($F\textsubscript{SSA}(4,95)=1.93;p>0.1$).

The mean scores received by each conventional translation, as well as the mean
scores participants achieved, can be found in Table 1.

Translation method | Conventional | Prototype-assisted
---|---|---
Target language | Chinese | Danish | French | German | Portuguese
Mean GEMBA-DA[noref] score | 94.75* | 97.70 | 99.05 | 98.75 | 99.75*
Mean SSA score | 94.50 | 95.00 | 95.00 | 95.00 | 95.75
Table 1. The mean scores achieved by each of the conventionally translated
versions, as well as the prototype-assisted translations of the BFI-10
(Rammstedt and John, 2007). The Chinese-language version received
significantly lower GEMBA scores than all other versions, while prototype-
assisted Portuguese translations received significantly higher scores than the
Chinese and Danish versions.

####  5.4.2. Participant Experience

Participants found the suggestion quality high ($Median=6.0$; $IQR=1.0$).
While 12 participants reported having implemented at least some of the
suggestions, only 1 reported having implemented most, with no participants
having implemented all of them. Nearly all (18) participants reported having
manually edited the translations.

One participant was not quite satisfied with the quality of the evaluation:
”[…] I had to manually edit the second item of the questionnaire to preserve
it’s original meaning […] The tool suggested to revert to the original
translation […] which has a different meaning” (P19). Another noted that as
the initial translation had received a perfect score, they did not receive any
suggestions to implement: ”Since no suggestions for correction were made, I
can not tell how useful this feature is.” (P18).

##  6\. General Discussion and Limitations

The results from our two studies show promising results regarding LLM adoption
in the questionnaire translation process. We found that machine translation,
accompanied by GPT-4 generated quality scores, helped participants reach at
least a similar level of translation quality and semantic similarity to
questionnaire translations generated using conventional, non-NLP-supported
methods. Participants found the evaluations to accurately represent
translation quality.

Participants found the suggestions provided by our experimental semantic
similarity evaluation to be moderately helpful. In the second study, while
nearly all participants had opted to edit the translations manually, only just
over half reported having implemented suggestions. A factor limiting
meaningful evaluation of how practical the suggestions provided by the tool
was the high performance of the DeepL translator. As the initial translations
were scored highly, the suggestions generated by GPT-4 were of limited value.

One factor for the high scores received by the translations generated by
participants could be the high initial quality of the questionnaires. As noted
by (Brislin, 1970; Su and Parham, 2002), the wording of the original
questionnaire highly affects how well its semantics translate to other
languages. Future research should investigate LLM translation quality
evaluations with questionnaires that have been found more difficult to
translate due to culture-specific expressions.

Another factor contributing to high translation quality could have been the
close relationship of the languages. English, German and Portuguese are all
Indo-European languages, with English and German especially close: both are
Germanic languages (Kapović, 2017). Another topic for future research is
investigating LLM assessments when questionnaires are translated between
language families.

Beyond making the translation of questionnaires more accessible, LLMs could be
used for their preliminary validation as well. Schmidt et al. (Schmidt et al.,
2024) provide an overview of the potential use cases of LLMs in complementing
human participants in human-centric design. Studies by Tavast et al. (Tavast
et al., 2022) and Hämäläinen et al. (Hämäläinen et al., 2023) have found that
LLMs can be used to simulate participant responses in both Likert-style
questionnaires and open-ended questions. Future research should, therefore,
investigate whether LLMs can be prompted to generate questionnaire validation
results similar to those obtained through user testing with real people.

We provide two design recommendations regarding the integration of LLM-
generated evaluations into translation software. First, we noticed that
evaluation generation with LLMs can take time, especially when dealing with
longer questionnaires. To avoid user frustration, it is important to
communicate longer wait times to the user in some manner (Nielsen, 1994); we
simply used a spinner accompanied by a short message. Second, to avoid
evaluations or suggestions that might be perceived as redundant, prompt
engineering could discourage the system from providing feedback after a
certain score threshold.

##  7\. Conclusion

LLM-generated translation quality evaluations can help researchers identify
and resolve issues in their translations based on context-specific
requirements. This work presents promising results regarding the integration
of LLM-generated feedback to translation software to expedite the
questionnaire translation process for cross-cultural research. We demonstrate
that GPT-4-generated translation quality assessments, alongside machine
translation, can help users independently reach translation results similar to
those attained with conventional, non-NLP-supported methods.

It should be kept in mind that validation is as much an integral part of the
questionnaire adaptation process as translation. Questionnaires, whether
translated using automated tools or conventional methods, should be validated
with participants before deploying them in research.

## References

  * (1)
  * Ali et al. (2016) Gemma-Claire Ali, Grace Ryan, and Mary J. De Silva. 2016.  Validated Screening Tools for Common Mental Disorders in Low and Middle Income Countries: A Systematic Review.  _PLOS ONE_ 11, 6 (June 2016), e0156939.  https://doi.org/10.1371/journal.pone.0156939
  * Brin et al. (2023) Dana Brin, Vera Sorin, Akhil Vaid, Ali Soroush, Benjamin S. Glicksberg, Alexander W. Charney, Girish Nadkarni, and Eyal Klang. 2023.  Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments.  _Scientific Reports_ 13, 1 (Oct. 2023), 16492.  https://doi.org/10.1038/s41598-023-43436-9
  * Brislin (1970) Richard W. Brislin. 1970.  Back-Translation for Cross-Cultural Research.  _Journal of Cross-Cultural Psychology_ 1, 3 (Sept. 1970), 185–216.  https://doi.org/10.1177/135910457000100301 Number: 3.
  * Brooke (1996) John Brooke. 1996.  SUS: A ’Quick and Dirty’ Usability Scale.  In _Usability Evaluation In Industry_ (1 ed.), Patrick W. Jordan, B. Thomas, Ian Lyall McClelland, and Bernard Weerdmeester (Eds.). CRC Press.  https://doi.org/10.1201/9781498710411
  * Carciofo et al. (2016) Richard Carciofo, Jiaoyan Yang, Nan Song, Feng Du, and Kan Zhang. 2016.  Psychometric Evaluation of Chinese-Language 44-Item and 10-Item Big Five Personality Inventories, Including Correlations with Chronotype, Mindfulness and Mind Wandering.  _PLOS ONE_ 11, 2 (Feb. 2016), e0149963.  https://doi.org/10.1371/journal.pone.0149963
  * Cha et al. (2007) Eun‐Seok Cha, Kevin H. Kim, and Judith A. Erlen. 2007.  Translation of scales in cross‐cultural research: issues and techniques.  _Journal of Advanced Nursing_ 58, 4 (May 2007), 386–395.  https://doi.org/10.1111/j.1365-2648.2007.04242.x Number: 4.
  * Courtois et al. (2020) R. Courtois, J.-M. Petot, O. Plaisant, B. Allibe, B. Lignier, C. Réveillère, G. Lecocq, and O. John. 2020.  Validation française du Big Five Inventory à 10 items (BFI-10).  _L’Encéphale_ 46, 6 (Dec. 2020), 455–462.  https://doi.org/10.1016/j.encep.2020.02.006
  * Díaz-Oreiro et al. (2021) Ignacio Díaz-Oreiro, Gustavo López, Luis Quesada, and Luis A. Guerrero. 2021.  UX Evaluation with Standardized Questionnaires in Ubiquitous Computing and Ambient Intelligence: A Systematic Literature Review.  _Advances in Human-Computer Interaction_ 2021 (May 2021), 1–22.  https://doi.org/10.1155/2021/5518722
  * Epstein et al. (2015) Jonathan Epstein, Ruth Miyuki Santo, and Francis Guillemin. 2015.  A review of guidelines for cross-cultural adaptation of questionnaires could not bring out a consensus.  _Journal of Clinical Epidemiology_ 68, 4 (April 2015), 435–441.  https://doi.org/10.1016/j.jclinepi.2014.11.021 Number: 4.
  * Floridi and Chiriatti (2020) Luciano Floridi and Massimo Chiriatti. 2020.  GPT-3: Its Nature, Scope, Limits, and Consequences.  _Minds and Machines_ 30, 4 (Dec. 2020), 681–694.  https://doi.org/10.1007/s11023-020-09548-1
  * Fosco et al. (2020) Camilo Fosco, Vincent Casser, Amish Kumar Bedi, Peter O’Donovan, Aaron Hertzmann, and Zoya Bylinskii. 2020.  Predicting Visual Importance Across Graphic Design Types. In _Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology_. ACM, Virtual Event USA, 249–260.  https://doi.org/10.1145/3379337.3415825
  * Franke et al. (2019) Thomas Franke, Christiane Attig, and Daniel Wessel. 2019.  A Personal Resource for Technology Interaction: Development and Validation of the Affinity for Technology Interaction (ATI) Scale.  _International Journal of Human–Computer Interaction_ 35, 6 (April 2019), 456–467.  https://doi.org/10.1080/10447318.2018.1456150
  * Gillioz et al. (2020) Anthony Gillioz, Jacky Casas, Elena Mugellini, and Omar Abou Khaled. 2020.  Overview of the Transformer-based Models for NLP Tasks. 179–183.  https://doi.org/10.15439/2020F20
  * Heilala et al. (2023) Ville Heilala, Riitta Kelly, Mirka Saarela, Päivikki Jääskelä, and Tommi Kärkkäinen. 2023.  The Finnish Version of the Affinity for Technology Interaction (ATI) Scale: Psychometric Properties and an Examination of Gender Differences.  _International Journal of Human–Computer Interaction_ 39, 4 (Feb. 2023), 874–892.  https://doi.org/10.1080/10447318.2022.2049142 Number: 4.
  * Hoffman et al. (2022) Joel Hoffman, Ziv Ben-Zion, Adrián Arévalo, Or Duek, Talya Greene, Brian J. Hall, Ilan Harpaz-Rotem, Belinda Liddell, Cosima Locher, Naser Morina, Angela Nickerson, Monique C. Pfaltz, Matthis Schick, Ulrich Schnyder, Soraya Seedat, Fatlinda Shatri, Hao Fong Sit, Roland Von Känel, and Tobias R. Spiller. 2022.  Mapping the availability of translated versions of posttraumatic stress disorder screening questionnaires for adults: A scoping review.  _European Journal of Psychotraumatology_ 13, 2 (Dec. 2022), 2143019.  https://doi.org/10.1080/20008066.2022.2143019
  * House (2014) Juliane House. 2014.  Translation Quality Assessment: Past and Present.  In _Translation: A Multidisciplinary Approach_ , Juliane House (Ed.). Palgrave Macmillan UK, London, 241–264.  https://doi.org/10.1057/9781137025487_13
  * Hämäläinen et al. (2023) Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. 2023.  Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. ACM, Hamburg Germany, 1–19.  https://doi.org/10.1145/3544548.3580688
  * Kapović (2017) Mate Kapović. 2017.  Indo-European languages – introduction.  In _The Indo-European Languages_ (2 ed.), Mate Kapović, Anna Giancalone Ramat, and Paolo Ramat (Eds.). Routledge, 1–9.  https://doi.org/10.4324/9781315678559
  * Kasneci et al. (2023) Enkelejda Kasneci, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stephan Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia Nerdel, Jürgen Pfeffer, Oleksandra Poquet, Michael Sailer, Albrecht Schmidt, Tina Seidel, Matthias Stadler, Jochen Weller, Jochen Kuhn, and Gjergji Kasneci. 2023.  ChatGPT for good? On opportunities and challenges of large language models for education.  _Learning and Individual Differences_ 103 (April 2023), 102274.  https://doi.org/10.1016/j.lindif.2023.102274
  * Kocmi and Federmann (2023) Tom Kocmi and Christian Federmann. 2023.  Large Language Models Are State-of-the-Art Evaluators of Translation Quality.  (2023).  https://doi.org/10.48550/ARXIV.2302.14520 Publisher: arXiv Version Number: 2.
  * Lu et al. (2023) Qingyu Lu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, and Dacheng Tao. 2023.  Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models: A Case Study on ChatGPT.  (2023).  https://doi.org/10.48550/ARXIV.2303.13809 Publisher: arXiv Version Number: 2.
  * Nielsen (1994) Jakob Nielsen. 1994.  Enhancing the explanatory power of usability heuristics. In _Proceedings of the SIGCHI conference on Human factors in computing systems celebrating interdependence - CHI ’94_. ACM Press, Boston, Massachusetts, United States, 152–158.  https://doi.org/10.1145/191666.191729
  * OpenAI et al. (2023) OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2023.  GPT-4 Technical Report.  (2023).  https://doi.org/10.48550/ARXIV.2303.08774 Publisher: arXiv Version Number: 4.
  * Oulasvirta et al. (2018) Antti Oulasvirta, Samuli De Pascale, Janin Koch, Thomas Langerak, Jussi Jokinen, Kashyap Todi, Markku Laine, Manoj Kristhombuge, Yuxi Zhu, Aliaksei Miniukovich, Gregorio Palmas, and Tino Weinkauf. 2018.  Aalto Interface Metrics (AIM): A Service and Codebase for Computational GUI Evaluation. In _Adjunct Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology_. ACM, Berlin Germany, 16–19.  https://doi.org/10.1145/3266037.3266087
  * Palsson et al. (2023) Thorvaldur Skuli Palsson, Søren Thorgaard Skou, Morten Haugaard Pape, Rogerio Pessoto Hirata, Trine Rafn, Pablo Bellosta‐López, and Steffan Wittrup McPhee Christensen. 2023.  Cross‐cultural translation and adaptation of the Danish version of the brief version of the 10‐item Big Five Inventory.  _Physiotherapy Research International_ 28, 3 (July 2023), e2004.  https://doi.org/10.1002/pri.2004
  * Rammstedt and John (2007) Beatrice Rammstedt and Oliver P. John. 2007.  Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German.  _Journal of Research in Personality_ 41, 1 (Feb. 2007), 203–212.  https://doi.org/10.1016/j.jrp.2006.02.001
  * Schmidt et al. (2024) Albrecht Schmidt, Passant Elagroudy, Fiona Draxler, Frauke Kreuter, and Robin Welsch. 2024.  Simulating the Human in HCD with ChatGPT: Redesigning Interaction Design with AI.  _Interactions_ 31, 1 (Jan. 2024), 24–31.  https://doi.org/10.1145/3637436
  * Schrepp et al. (2017) Martin Schrepp, Andreas Hinderks, and Jörg Thomaschewski. 2017.  Design and Evaluation of a Short Version of the User Experience Questionnaire (UEQ-S).  _International Journal of Interactive Multimedia and Artificial Intelligence_ 4, 6 (2017), 103.  https://doi.org/10.9781/ijimai.2017.09.001 Number: 6.
  * Shepherd (1998) Andrew Shepherd. 1998.  HTA as a framework for task analysis.  _Ergonomics_ 41, 11 (Nov. 1998), 1537–1552.  https://doi.org/10.1080/001401398186063 Number: 11.
  * Su and Parham (2002) Chia-Ting Su and L. Diane Parham. 2002.  Generating a Valid Questionnaire Translation for Cross-Cultural Use.  _The American Journal of Occupational Therapy_ 56, 5 (Sept. 2002), 581–585.  https://doi.org/10.5014/ajot.56.5.581 Number: 5.
  * Taherdoost (2016) Hamed Taherdoost. 2016.  Validity and Reliability of the Research Instrument; How to Test the Validation of a Questionnaire/Survey in a Research.  _SSRN Electronic Journal_ (2016).  https://doi.org/10.2139/ssrn.3205040
  * Takakusagi et al. (2021) Yosuke Takakusagi, Takahiro Oike, Katsuyuki Shirai, Hiro Sato, Kio Kano, Satoshi Shima, Keisuke Tsuchida, Nobutaka Mizoguchi, Itsuko Serizawa, Daisaku Yoshida, Tadashi Kamada, and Hiroyuki Katoh. 2021.  Validation of the Reliability of Machine Translation for a Medical Article From Japanese to English Using DeepL Translator.  _Cureus_ (Sept. 2021).  https://doi.org/10.7759/cureus.17778
  * Tavast et al. (2022) Mikke Tavast, Anton Kunnari, and Perttu Hämäläinen. 2022.  Language Models Can Generate Human-Like Self-Reports of Emotion. In _27th International Conference on Intelligent User Interfaces_. ACM, Helsinki Finland, 69–72.  https://doi.org/10.1145/3490100.3516464
  * Tsangaris et al. (2017) Elena Tsangaris, Karen W.Y. Wong Riff, Tim Goodacre, Christopher R. Forrest, Marieke Dreise, Jonathan Sykes, Tristan De Chalain, Karen Harman, Aisling O’Mahony, Andrea L. Pusic, Lehana Thabane, Achilleas Thoma, and Anne F. Klassen. 2017.  Establishing Content Validity of the CLEFT-Q: A New Patient-reported Outcome Instrument for Cleft Lip/Palate.  _Plastic and Reconstructive Surgery - Global Open_ 5, 4 (April 2017), e1305.  https://doi.org/10.1097/GOX.0000000000001305
  * Westerlund et al. (2023) Mikael Westerlund, Arja Heliövaara, Junnu Leikola, and Pauliina Homsy. 2023.  Finnish Translation and Linguistic Validation of the CLEFT-Q Questionnaire.  _The Cleft Palate Craniofacial Journal_ (March 2023), 105566562311624.  https://doi.org/10.1177/10556656231162454

##  Appendix A Task Analysis Diagrams

Figure 3. Brislin’s (Brislin, 1970) backtranslation approach Figure 4.
Brislin’s (Brislin, 1970) committee approach Figure 5. Finnish ATI translation
process (Heilala et al., 2023) Figure 6. Finnish CLEFT-Q translation process
(Westerlund et al., 2023)

##  Appendix B Prototyping

###  B.1. Initial High-fidelity Prototype

Figure 7. The initial high-fidelity non-functional interactive prototype that
was used for usability testing. The actions taken included changing item
labeling and addition of tooltips, reordering of the layout to match user
expectations of left-to-right logical flow, moving the buttons away from in
between the sides to bring the two sides closer together and delivering the
evaluations on the same screen for less reliance on recall.

###  B.2. Final Prototype

Figure 8. The initial view of the prototype translation application features
language selection and a textarea for inputting the questionnaire items.
Figure 9. The view after performing the initial translation. On the left: The
initial item translations, which can be edited manually by the user. On the
right, the original items. Figure 10. The view after conducting a
backtranslation. The right side allows the users to switch between the
original and the backtranslated version of each non-matching item. Figure 11.
The evaluations provided by the application. The semantic similarity
assessment gives users a verbal justification of the score given, coupled with
suggestions for improvement. Figure 12. UMSI saliency evaluation of the
interface using the Aalto Interface Metrics service (Oulasvirta et al., 2018).
The instructional texts attracted the most emphasis, followed by the core
actions of backtranslation and evaluation. The top-right toolbar was the least
salient. Figure 13. An example sequence diagram of the process of using the
application.

##  Appendix C Prompt for Semantic Similarity Assessment

Assess the semantic similarity of the following texts in {source_lang} and
{target_lang} on a scale from 0 (no semantic similarity at all) to 100
(perfect semantic similarity). Justify the score. Provide a single paragraph
suggesting changes to the {target_lang} version (i.e. word or expression
replacements) to improve the score.

{source_lang}: ”{source_text}”
{target_lang}: ”{target_text}”

Respond with JSON only in the following format:

score
reasoning,
suggestion