# Shared Imagination: LLMs Hallucinate Alike

Yilun Zhou Caiming Xiong Silvio Savarese Chien-Sheng Wu
Salesforce AI Research
{yilun.zhou, cxiong, ssavarese<EMAIL_ADDRESS>
https://yilunzhou.github.io/shared-imagination

###### Abstract

Despite the recent proliferation of large language models (LLMs), their
training recipes – model architecture, pre-training data and optimization
algorithm – are often very similar. This naturally raises the question of the
similarity among the resulting models. In this paper, we propose a novel
setting, imaginary question answering (IQA), to better understand model
similarity. In IQA, we ask one model to generate purely imaginary questions
(e.g., on completely made-up concepts in physics) and prompt another model to
answer. Surprisingly, despite the total fictionality of these questions, all
models can answer each other’s questions with remarkable success, suggesting a
“shared imagination space” in which these models operate during such
hallucinations. We conduct a series of investigations into this phenomenon and
discuss implications on model homogeneity, hallucination, and computational
creativity.

Shared Imagination: LLMs Hallucinate Alike

Yilun Zhou Caiming Xiong Silvio Savarese Chien-Sheng Wu Salesforce AI Research
{yilun.zhou, cxiong, ssavarese<EMAIL_ADDRESS>https://yilunzhou.github.io/shared-imagination

##  1 Introduction

Recently, LLMs have been increasingly used in various applications. Although
these models occupy a wide spectrum of model sizes and benchmark performances
(Liang et al., 2022), they also share high degrees of similarities: decoder-
only transformer architecture (Radford et al., 2018) with one of a few
positional embedding designs (Dufter et al., 2022), pre-training corpus
consisting of books, Internet texts and codes (Gao et al., 2020), stochastic
gradient descent (SGD)-based optimization (Kingma and Ba, 2014), and similar
procedures for instruction tuning and alignment after pre-training (Ouyang et
al., 2022; Rafailov et al., 2024). As a result, is it possible that these
models share certain fundamental commonalities?

Figure 1: Imaginary question answering (IQA). Prompt texts are for
illustrative purposes, with exact ones shown in Tab. 2-4. Top: a question
model (QM) is prompted to generate an imaginary multiple-choice question and
indicate the correct answer, either directly (left) or based on the previously
generated context (right). Bottom left: an answer model (AM) answers the
question (with the four choices shuffled), or refuses to answer. Bottom right:
we observe non-trivial correctness rate and relatively high answering rate
(i.e., low refusal rate), with higher values when AM and QM are the same or
from the same model family (shown in Fig. 2), and significantly higher values
for context-based questions.

RQ1: Data Characteristics (Sec. 3.1): What do the generated IQA data (i.e.,
questions and context paragraphs) look like?
---
Answer: While data are well-clustered by topics, questions generated by
different models, as well as direct vs. context questions, look very
homogeneous both in the embedding space and by cosine similarity metrics. Word
cloud visualization also confirms the homogeneity.
RQ2: Heuristics for Correct Choice (Sec. 3.2): Are there simple explanations
for the high correctness rate?
Answer: While data inspection and model evaluation identify ways to make
predictions better than random chance (e.g., the correct choice of DQs is most
likely to be the longest), none of them suffice to achieve the observed
correctness rate. In addition, the correctness rate is sensitive to the
orders.
RQ3: Fictionality Awareness (Sec. 3.3): Are models aware of the fictionality
of these questions and context paragraphs?
Answer: They can detect fictionality easier in DQ than CQ, and they can
identify fictionality better when directly asking a Yes/No question, but
struggle more on multiple-choice QA.
RQ4: Effect of Model “Warm-Up” (Sec. 3.4): Does model generation in general
make model converge to the “shared imagination space” and strengthen the
phenomenon?
Answer: Yes, when a QM generates several questions sequentially, they become
increasingly easy to answer. For individual questions, longer ones are also
easier to answer.
RQ5: Universality of the Phenomenon (Sec. 3.5): Can models other than recent
instruction-tuned models achieve high correctness rate?
Answer: Pre-ChatGPT models cannot, even large ones such as GPT-NeoX 20B, but
base versions of recent small models (e.g., Mistral 7B) can.
RQ6: Other Content Types (Sec. 3.6): Does this phenomenon occur for other
content types than (simulated) knowledge concepts?
Answer: Yes, we observe similar results for questions about (imagined) short
stories in creative writing.

Table 1: Six research questions into the shared imagination phenomenon and
summary answers.

In this paper, we identify one such commonality: these models agree, to a
surprising extent, on purely imaginary contents, or hallucinations.
Specifically, we propose the imaginary question answering (IQA) task, shown in
Fig. 1. A question model (QM) is prompted to generate a multiple-choice
question, either directly about a fictional concept (top left), called a
direct question (DQ), or based on a previously generated context paragraph
about a fictional concept (top right), called a context question (CQ).
Although they are impossible to answer with rational decision-making, we still
ask the QM to specify a “correct” answer. Then, in a new session, without any
previous interactions or contexts with the QM, we solicit an answer from an
answer model (AM), which may be the same as or different from the QM (bottom
left).

On 13 LLMs from four model families (GPT, Claude, Mistral, and Llama 3),
models achieve an average 54% correctness rate on directly generated questions
(with random chance being 25%), with higher accuracy when the AM is the same,
or in the same model family, as the QM. More surprisingly, for context-based
questions, the correctness rate increases significantly to 86%, with certain
(QM, AM) pairs achieving as high as 96%.

These results show high degrees of agreement among models on what they
hallucinate, which we call “shared imagination”. Focusing on this phenomenon,
we present six research questions and empirically answer them via carefully
designed experiments, listed in Tab. 1. These results shed light on
fundamental properties of LLMs and suggest that, despite their highly varying
benchmark results, they are perhaps more homogeneous. This homogeneity could
have broad implications on model hallucination and its detection, as well as
the use of LLM in computational creativity.

Figure 2: The correctness and answering rate on direct and context questions
for each pair of question model (QM) and answer model (AM). Each pink
rectangle represents one model family. For correctness rate, the top-4 highest
performing AMs for each QM are shaded. An enlarged version is reproduced in
Fig. 13 of App. A.

##  2 Imaginary Question Answering (IQA)

###  2.1 Framework Description

The setup for the IQA procedure is summarized in Fig. 1. In the direct
question generation mode (Tab. 2), the QM is asked to generate a standalone
question. In the context-based question generation mode (Tab. 3), the model
first writes a paragraph on a fictional concept, and then generates a question
based on it. We call the former direct question (DQ) and the latter context
question (CQ). To prevent biasing the generation in any way, we use zero-shot
prompting with no examples.

Role |  Message
---|---
User |  On the topic of physics, please write a multiple choice question around a concept that is completely made up. Try to make the problem hard and challenging. In your question, do not say that the concept is hypothetical or fictional. Instead, treat it as if it were real and widely accepted. Use the following template:

Question: [question statement]

A. [choice A]
B. [choice B]
C. [choice C]
D. [choice D]

Answer: [the correct choice]
Model |  (the generated question and answer)

Table 2: The prompt for direct question generation. The underlined topic is
replaced accordingly.

Role |  Message
---|---
User |  Imagine that you are writing a textbook. Please make up a concept in physics and explain it with a single paragraph of text. Please write as if the concept is real, and completely avoid saying that the concept is fictional or made-up. Be creative. Use the following template:

Concept: [the name of the concept that you are writing about]

Content: [a single paragraph of text explaining the concept]
Model |  (the generated concept and paragraph)
User |  Now, based on the paragraph above, write a multiple choice question about this concept. The question should be answerable using the paragraph. Try to make the problem hard and … (same as Tab. 2 afterward)
Model |  (the generated question and answer)

Table 3: The prompt for context-based question generation. The underlined
topic is replaced accordingly.

To elicit answers from the AM, we use the prompt in Tab. 4, with the four
choices shuffled in all experiments except for that in Sec. 3.2. Although the
format template explicitly instructs the model make a selection, there are
occasional refuse-to-answer responses, such as “I apologize, but the question
seems to be asking about a fictional concept, and hence I cannot answer it.”

Role |  Message
---|---
User |  Answer the following question. Be concise and give the answer only.

(the question and its four choices)

Write your response in the following format:
Answer: [the letter (A, B, C or D) of the selected choice]
Model |  (the generated question and answer)

Table 4: The prompt for the answer model. Note that while the instruction does
not “allow” the refusal behavior, it still occurs occasionally.

Formally, the QM generates a set of questions
$\\{(x_{i},y_{i}^{*})\\}_{i=1}^{N}$ where $y_{i}^{*}$ is the assigned correct
answer. Then, the AM predicts $\hat{y}_{i}$, where $y_{i}$ is either one of
the choices or $R$, for refuse-to-answer. On these problems, we compute
correctness rate $\kappa$ as the fraction of correctly answered questions
among answered ones, and answering rate $\alpha$ as the fraction of answered
questions among all questions:

$\displaystyle\kappa=\frac{\sum_{i=1}^{N}\mathbbm{1}_{\hat{y}_{i}=y_{i}^{*}}}{\sum_{i=1}^{N}\mathbbm{1}_{\hat{y}_{i}\neq
R}},\quad\alpha=\frac{\sum_{i=1}^{N}\mathbbm{1}_{\hat{y}_{i}\neq R}}{N}.$ (1)

###  2.2 Experiment Setup and Results

We study 13 models from four model families, listed in Tab. 5. QMs use
temperature 1 to balance output quality and stochasticity, and AMs use
temperature 0 for greedy answer selection.

Model | API or Huggingface ID | Abbr.
---|---|---
GPT-3.5 | gpt-3.5-turbo-0125 | G-3.5
GPT-4 | gpt-4-0613 | G-4
GPT-4 Turbo | gpt-4-turbo-2024-04-09 | G-4T
GPT-4 omni | gpt-4o-2024-05-13 | G-4o
Claude 3 Haiku | claude-3-haiku-20240307 | C-3H
Claude 3 Sonnet | claude-3-sonnet-20240229 | C-3S
Claude 3 Opus | claude-3-opus-20240229 | C-3O
Claude 3.5 Sonnet | claude-3-5-sonnet-20240620 | C-3.5H
Mistral 7B | Mistral-7B-Instruct-v0.2 | M-7
Mixtral 8x7B | Mixtral-8x7B-Instruct-v0.1 | M-8x7
Mistral Large | mistral-large-2402 | M-Lg
Llama 3 8B | Meta-Llama-3-8B-Instruct | L3-8
Llama 3 70B | Meta-Llama-3-70B-Instruct | L3-70

Table 5: A summary of models used in the experiments. Model abbreviations are
used in figures.

We select 17 topics on common college subjects: mathematics, computer science,
physics, chemistry, biology, geography, sociology, psychology, economics,
accounting, marketing, law, politics, history, literature, philosophy, and
religion. Each QM in Tab. 5 generates 20 direct questions and 20 context
questions for each topic, for a total of $13\times 17\times(20+20)=8840$
questions. Tab. 8 of App. A presents some generated questions.

Fig. 2 shows the correctness and answering rates, along with the respective
averages. Notably, all correctness rates are higher than random chance of 25%.
For DQs, most of the high-performing AMs (i.e., shaded cells) are the same
(diagonal) or in the same model family (block diagonal) as the QMs. On context
questions (CQs), the correctness rate increases significantly, from 54% to 86%
on average. While the performance difference among AMs are smaller (uniform
vertical color patterns), GPT-4 omni, Claude 3 Opus, Claude 3.5 Sonnet, and
Llama 3 70B are the slightly better ones for almost all QMs (horizontal shaded
cells).

The answering rate on DQs mostly depend on AM – some AMs, such as GPT-4,
Claude 3 Opus and Llama 3 70B, consistently refuse questions from all QMs, and
do not particularly favor/disfavor their own questions. Quite remarkably, the
refusal behavior virtually disappears on CQs, except for Claude 3 Opus, which
nonetheless answers much more frequently (44% vs. 87%).

##  3 Further Analyses

Given the large variance of model capabilities as measured by numerous
benchmarks, the findings that models tacitly agree with each other on purely
imaginary contents are surprising. In the next section, we conduct in-depth
analyses by focusing on six research questions listed in Tab. 1.

###  3.1 RQ1: Data Characteristics

By studying the word frequencies of DQs, CQs and context paragraphs, we find
that they share many common words, such as “individual”, “principle”,
“phenomenon” and “time” in a word cloud visualization (Fig. 14 of App. B).

To visualize the questions generated across topics and models, we use OpenAI’s
text-embedding-3-large to compute the embeddings for each question (including
the four choices), and visualize them with UMAP dimensionality reduction
(McInnes et al., 2018) in the top panels of Fig. 3. We can see that while
questions from different topics are well-clustered (top left), there are no
clear patterns for questions from different QMs. Furthermore, there are no
obvious distinctions between DQs (triangle markers) and CQs (circle markers).

Figure 3: Top: question embeddings (computed by text-embedding-3-large) by
UMAP, color-coded by topic (left) and question model (right). A triangle
marker indicates a DQ, and a circle marker indicates a CQ. Bottom left:
average intra-topic cosine similarity between questions generated by different
models, with DQs on the lower-left half and CQs on the upper-right half.
Bottom right: the color legends for the three plots.

The lower-left panel of Fig. 3 shows the cosine similarity between each QM
pairs, for DQs on the lower-left triangle and CQs on the upper-right triangle.
The similarity between two QMs $m_{1}$ and $m_{2}$ is defined as

$\displaystyle
s=\frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}\frac{\sum_{q_{1}\in\mathcal{Q}_{t}^{m_{1}}}\sum_{q_{2}\in\mathcal{Q}_{t}^{m_{2}}}\sigma(q_{1},q_{2})}{|\mathcal{Q}_{T}^{m_{1}}|\cdot|\mathcal{Q}_{T}^{m_{2}}|},$
(2)

where $\mathcal{T}$ is the set of topics, and for each $t\in\mathcal{T}$,
$\mathcal{Q}^{m}_{t}$ is the set of (direct or context) questions generated by
QM $m$. $\sigma(q_{1},q_{2})$ is the embedding cosine similarity of $q_{1}$
and $q_{2}$. In our experiment, we have $|\mathcal{T}|=17$ and
$|\mathcal{Q}^{m}_{t}|=20$ for all $m$ and $t$.

The similarity values are generally quite similar for each model pair, ranging
from 0.44 to 0.63, with Mistral models being most dissimilar from the rest,
consistent with the upper-right embedding visualization showing different QMs
generate highly similar and homogeneous questions.

###  3.2 RQ2: Heuristics for Correct Choice

#### Human Answer Guessing

To intuitively understand the generated questions, we manually answered 340
questions: 10 DQs and 10 CQs randomly sampled from each topic. We try to guess
answers in the most rational way. While some clues hint at the correct choice
or allow for the elimination of likely wrong ones (see App. C for details), we
struggle to answer most of the questions.

Fig. 4 shows the correctness rate per topic for direct (top) and context
(bottom) questions, alongside a few representative models and their averages.
Human performance is much lower than that of all models, especially on context
questions. Interestingly, although we do not perceive any difference between
DQs and CQs (questions of both types are shuffled and unlabeled during our
answering), our correctness rate on CQs is also higher.

Figure 4: Per topic correctness rate of (a subset of) AMs, their average and
human guessing, for direct questions (top) and context questions (bottom).
Results for other AMs are similar and presented in Fig. 15 of App. C.

#### Length Ranking of Correct Choice

For each QM, we compute the distribution of the ranking of the correct choice
length among the four choices. Concretely, we find the fraction of questions
whose correct choice is the shortest (in the number of characters) among the
four, the 2nd shortest, the 3rd shortest and the longest. Fig. 5 shows the
distributions, with the four segments in the order above.

Figure 5: Fraction of questions whose correct choice is the shortest (in
blue), 2nd shortest (in orange), 3rd shortest (in green) and longest (in red),
by each QM, for direct questions (left) and context questions (right).

For DQs (left panel), the correct answer is almost equally likely to be of any
length, with a slight tendency towards being the longest (red). However, for
CQs, the correct answer is much more likely to be the longest than the rest –
for most QMs, in over half of the questions, the correct answer is the
longest. The strong contrast between the two settings, yet the consistency
across different QMs, strongly indicate that these QMs share fundamental
similarities in their generations, and thus it is not surprising that AMs have
much higher correctness rate (86%, Fig. 1) on CQs. However, simply choosing
the longest answer alone is not sufficient, so AMs must have also exploited
other signals.

#### Answer Perplexity

In this experiment, we study the model perplexity of each choice as a free-
text response and see whether the correct answer has the lowest perplexity.
Specifically, we use the conversation shown in Tab. 6. After applying the
appropriate chat template to the conversation, we evaluate the perplexity of
each of the four choice texts as possible completions at the square location.

Role |  Message
---|---
User |  Answer the following question.

(question statement without the four choices)
Model |  Answer: $\mathord{\scalerel*{\Box}{gg}}$

Table 6: The prompt for perplexity evaluation.

Since perplexity calculation requires token-level log-likelihood, we only
study the four open-source models: Mistral 7B, Mixtral 8x7B, Llama 3 8B and
Llama 3 70B. Their perplexities on questions generated by each QM is shown in
Fig. 6. As we can see, selecting the lowest-perplexity answer yields
significantly lower correctness rate, on par with human guessing (Fig. 4).
This result further suggests that the models are using more complex features,
possibly interactions among answer choices, to make final predictions.

Figure 6: Correctness rate for four open-source AMs if the choice with the
lowest perplexity were chosen, on direct questions (top) and context questions
(bottom).

#### Answer Choice Shuffling

In all experiments, we randomly shuffle the choices before presenting them to
the AM. Here, we replicate our main experiment (Sec. 2.2) but instead present
the answer choices in the native orders generated by QMs. Fig. 7 presents the
difference when we use the native order, with a green color denoting an
increase under the native order, and red color a decrease. Generally,
correctness rate increases, most remarkably on the diagonal (i.e., same AM and
QM) for DQs – as much as 10%. In addition, the answering rates for DQs
decrease, suggesting that AMs are also better at identifying question
fictionality (though see Sec. 3.3 for more nuanced analyses).

#### Summary

This section reveals factors that lead to above-random correctness rates, but
none of them could adequately explains the observed high values. The answer
shuffling result implies that there are complex and hidden rules for the
correct choice shared among different models.

Figure 7: The effect of native choice order for GPT, Claude, Mistral, Llama 3
and their Average, with a green color representing an increase under the
native order, and a red color a decrease. An enlarged plot with annotated cell
values is presented in Fig. 16 of App. D.

###  3.3 RQ3: Fictionality Awareness

The fact that most AMs exhibit high answering rate (i.e., low refusal rate)
raises the question of whether the content truly appear fictional to the
models. In this section, we assess fictionality awareness via two metrics.
First, we augment each question with a fifth choice stating “E. This question
cannot be answered since the concept does not exist.” Second, we directly
query the model on the fictionality of the contexts (associated with CQs),
using the prompt “Does the following paragraph describe a real concept in
(topic)?” and compute the detection rate as the fraction of negative answers.

Figure 8: Choice E selection rate on direct and context questions, and
ficitonality detection rate on context paragraphs, for AMs and their average.
Breakdown by QM is shown in Fig. 17 of App. E.

These values are shown in Fig. 8. The choice E selection rate for DQs (left)
is higher than that for CQs (right), consistent with the finding that
answering rates are lower on DQs than on CQs (cf. Fig. 2). However, the choice
E selection rate for context questions (middle) is on average much lower than
context fictionality detection rate (right), suggesting that models can
identify the fictionality of some content when directly queried, but often
cannot translate this knowledge to downstream tasks such as question
answering.

###  3.4 RQ4: Effect of Model “Warm-Up”

One possible explanation for the correctness increase from DQ to CQ is that
the QMs has more tokens to “warm-up”, and models warm up in highly similar
manners. Thus, we hypothesize that generating any preceding content is likely
to help the model converge onto this shared imagination space.

#### Warm-Up With Previous Questions

We propose a new data generation setup, where the prompt (Tab. 9 of App. F)
asks the model to generate five questions sequentially. If the hypothesis is
true, then we should expect the correctness rate to increase from the first
questions to the fifth questions.

Figure 9: Left and middle: the correctness and answering rates for each AM
(see Fig. 3 for color legend) and their average in pink, for questions of
different generation order (e.g., 1 on $x$-axis means first-generated
questions and 5 means last-generated). Right: the length ranking distribution
of the correct choice for the five groups of questions (same color scheme as
Fig. 5).

For each QM, we run this pipeline 10 times on each topic. The correctness and
answering rates for all AMs on these five groups of questions are plotted in
Fig. 9 (left and middle), with the average shown in the pink line. Both
metrics exhibit a clear increasing trend from the first to the last generated
questions, mirroring that from DQs to CQs in Fig. 2.

Nonetheless, the length ranking distribution of the correct choice, shown on
the right panel, does not shift towards the longest, but instead is very
consistent across the five question groups, which may explain the small
magnitude of the increase.

#### Warm-Up With Current Question

If models “converge” while generating previous questions, do they also
converge when generating the current question itself? If so, then we should
expect longer questions to be easier to answer. To study this, we took the
original sets of DQs and CQs, and partition each set into 10 subsets according
to their length (i.e., number of characters in the question statement and four
choices combined). For each subset, we compute the correctness and answering
rate for each AM, and plot them in Fig. 10, along with their average in pink.
Those for direct questions are plotted in solid lines, and those for context
questions are plotted in dashed lines. A similar trend is observed – longer
questions are answered more correctly and frequently.

Figure 10: Correctness and answering rates for each of 10% questions ranked by
question + choices length.

#### Summary

While both question order and question length are potentially correlated with
many other factors, for which a thorough study is left to future work, the
consistency of both trends highly suggests that as the generation goes on, the
generated content looks increasingly familiar and predictable to not only
itself but also other models.

###  3.5 RQ5: Universality of the Phenomenon

Given that all studied models are post-ChatGPT instruction-tuned ones, we
evaluate whether other models, listed in Tab. 7, can achieve the same high
correctness rate (note that M-7 Inst and L3-8 Inst are the same models as
studied elsewhere). We set up the prompt as below:

Model | Huggingface ID | # Params
---|---|---
BERT | bert-large-cased | 0.3B
GPT-2 | gpt2-xl | 1.5B
GPT-Neo | gpt-neo-2.7B | 2.7B
GPT-J | gpt-j-6b | 6B
GPT-NeoX | gpt-neox-20b | 20B
M-7 Base | Mistral-7B-v0.2 | 7B
M-7 Inst | Mistral-7B-Instruct-v0.2 | 7B
L3-8 Base | Meta-Llama-3-8B | 8B
L3-8 Inst | Meta-Llama-3-8B-Instruct | 8B

Table 7: Models used in the universality study.

Question: [question statement]

A: [choice A]
B: [choice B]
C: [choice C]
D: [choice D]

Answer: $\mathord{\scalerel*{\Box}{gg}}$

Figure 11: Correctness rate of other LLM types as AMs. Breakdown by QM is
shown in Fig. 18 in App. G. Figure 12: Correctness and answering rates for
creative writing questions (enlarged version in Fig. 19 of App. H).

For BERT, we replace the square marker with the [MASK] token and extract the
predicted probability on tokens ‘A’, ‘B’, ‘C’, and ‘D’ respectively. For all
other models, we set the context to be everything before the square marker and
extract the next-token prediction probability on the four tokens (with a
preceding space as needed). Even for the two instruction-tuned models, we use
the same method, to remove the effect of the chat template.

The correctness rate for each model is shown in Fig. 11. As we can see, most
pre-ChatGPT models perform at or marginally better than the chance level, even
for the larger 6B and 20B models. By comparison, for M-7 and L3-8, both the
base and instruction-tuned (but without chat-template) models perform on par
with chat-templated prompts, implying that neither instruction tuning nor
chat-templating is required for the high correctness rate.

By ruling out the factors of model size and instruction tuning, we hypothesize
that the shared imagination behavior emerges from the pre-training corpus.
Since it is not publicly released for many models, even open-source ones such
as M-7, we leave additional investigation to future work.

###  3.6 RQ6: Other Content Types

Finally, we investigate if the same behavior transfers to other content types,
in particular creative writing. For DQs, we ask the model to generate them
based on an imagined “story about (topic) with an intricate story plot” where
we select ten abstract or concrete topics, such as “friendship”, or “an
ancient empire”. To generate a CQ, the model is asked to “write a short story
of 3-5 paragraphs about (topic)” and then “write a question about one of its
details.” See App. H for details.

As expected, these questions appear even more “hallucinated”, such as “Q: What
message was hidden in the antique locket that Sarah discovered in the
basement? A: Map coordinates. B: An old photograph. C: A secret code. D: A
heartfelt apology note.” (correct answer: C).

Nonetheless, as shown in Fig. 12 (with the same layout as Fig. 2), models
achieve higher-than-random correctness rate for DQs, especially for same or
intra-family (AM, QM) pairs. Although the correctness increase from DQs to CQs
is smaller (54% to 58% on average), other trends persist, such as the
concentration of high-performing models to a few AMs (horizontal shaded
patterns) and the disappearance of self-model advantage (the lack of diagonal
shaded patterns). By contrast, the two setups have similar answering rates of
84%, which is still very high considering the un-answerability nature of these
questions.

##  4 Related Work

#### Model Evaluation

Many evaluation benchmarks have been proposed that focus on diverse model
abilities such as math reasoning (Hendrycks et al., 2021; Mao et al., 2024),
factual knowledge (Hendrycks et al., 2020; Wang et al., 2024), commonsense
knowledge (Sap et al., 2019) and instruction following (Zheng et al., 2024;
Zeng et al., 2023). While these benchmarks demonstrate key capability
differences among models, our IQA evaluation is unique in that it instead
reveals striking similarities among them, which contributes a new dimension of
model understanding.

#### Multiple-Choice QA

MCQA benchmarks (e.g. Rajpurkar et al., 2016; Hendrycks et al., 2020;
Sakaguchi et al., 2021) are widely used to evaluate LLM capabilities, and
recent works propose more critical looks into this capability. For example,
Zong et al. (2023), Li and Gao (2024) and Gupta et al. (2024) all found that
changing the answer orders could decrease model performance MCQA, suggesting
possible data contamination and model robustness issues. In addition, Xu et
al. (2024) found that LLMs often cannot return the “none of the above” choice,
even when explicitly instructed. We take inspiration from these studies and
incorporate them in our analyses on answer order shuffling and fictionality
detection. While existing analyses use modified human-written benchmarks, we
use purely model-generated, intentionally fictional questions which, to the
best of our knowledge, has not been formally explored before.

#### Do-Not-Answer

A key aspect of LLM trustworthiness (Sun et al., 2024) is its ability to
refuse answering questions that do not make sense or contain false premise, or
at least convey its uncertainty (Kadavath et al., 2022; Xiong et al., 2023).
However, our results suggest that many models lack this ability on model-
generated contents, which could lead to certain trust issues.

#### Relationship Among LLMs

Previous works have attempted to categorize the likeness and relationship
among different LLMs from various perspectives. One approach is to infer model
relationship based on certain benchmark performance, such as HELM (Liang et
al., 2022). By contrast, Yax et al. (2024) uses a genetics-inspired approach
to construct a phylogenetic tree based on model’s next token prediction
results. In our paper, we propose the IQA setup as a new probe into the model
similarity phenomenon.

#### Model Hallucination

The generated context paragraphs and questions can be considered as
(intentional) hallucinations (Huang et al., 2023). Chen and Shu (2023) and
Laban et al. (2023) found that model-generated misinformation is harder to
detect than human-written one, which is reflected in our findings of the
higher answering rates on context questions (Fig. 2). On the other hand, the
decreased answering rate without answer shuffling (Fig. 7) also suggests that
models could be aware of their hallucination, explored by CH-Wang et al.
(2023). Our IQA setup could also present an interesting challenge for
hallucination detection algorithm (e.g. Li et al., 2023; Manakul et al.,
2023).

#### Computational Creativity

People have found that LLMs tend to produce repeated syntactic structure
(Shaib et al., 2024), score less on creativity metrics than human writers
(Chakrabarty et al., 2024), and have a homogenization effect on people’s
writing when used as creative support tools (Anderson et al., 2024). Along
with these studies, our results, particularly on the creative writing task in
Sec. 3.6, shed light on the potential limit of creativity that can be produced
by these models.

##  5 Conclusion and Future Work

In this paper, we propose the imaginary question answering (IQA) task, which
reveals an intriguing behavior that models can answer each other’s purely
hypothetical questions with a surprisingly high correctness rate. These
results reveal fundamental similarities between models, likely acquired during
pre-training, and may lead to more model merging possibilities (Goddard et
al., 2024). Furthermore, due to the imaginary and hallucinatory nature of
these question contents, such model behaviors suggest potential difficulty and
open questions in model-based hallucination detection and computational
creativity.

For future work, additional model families could be included, such as Google’s
Gemini (Team et al., 2023) and Cohere’s Command. Models that do not exhibit
this behavior could be valuable targets of study. In addition, mechanistic
interpretability analyses (Rai et al., 2024) of models answering imaginary
questions vs. normal questions (Lieberum et al., 2023) may shed more light on
this behavior. Finally, other reasoning strategies, such as chain-of-thought
prompting (Wei et al., 2022; Kojima et al., 2022), could be explored to study
whether the currently observed phenomenon persists.

##  6 Limitations

As the first study into this phenomenon, we acknowledge several limitations of
our investigations. First, the human study of Sec. 3.1 is limited in scale and
participant diversity: three AI researchers. A comprehensive human study,
especially with participants with and without expertise in each area, would be
valuable to obtain a more detailed account of human behaviors.

In addition, several experiment analyses are correlational, such as on the
relationship between correctness and question length (Fig. 10), which prevents
us from making causal conclusions (e.g., whether longer questions causes
higher correctness rate). Such studies would require more careful control of
confounding factors.

Finally, we mostly focus on QA questions based on (imaginary) knowledge
concepts, with the only exception of creative writing in Sec. 3.6. Another
common content type is news events – writing news articles and questions about
imaginary events – which may have even more direct implications on
misinformation generation and detection.

## References

  * Anderson et al. (2024) Barrett R Anderson, Jash Hemant Shah, and Max Kreminski. 2024.  Homogenization effects of large language models on human creative ideation.  In _Proceedings of the 16th Conference on Creativity & Cognition_, pages 413–425.
  * CH-Wang et al. (2023) Sky CH-Wang, Benjamin Van Durme, Jason Eisner, and Chris Kedzie. 2023.  Do androids know they’re only dreaming of electric sheep?  _arXiv preprint arXiv:2312.17249_.
  * Chakrabarty et al. (2024) Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. 2024.  Art or artifice? large language models and the false promise of creativity.  In _Proceedings of the CHI Conference on Human Factors in Computing Systems_ , pages 1–34.
  * Chen and Shu (2023) Canyu Chen and Kai Shu. 2023.  Can llm-generated misinformation be detected?  _arXiv preprint arXiv:2309.13788_.
  * Dufter et al. (2022) Philipp Dufter, Martin Schmitt, and Hinrich Schütze. 2022.  Position information in transformers: An overview.  _Computational Linguistics_ , 48(3):733–763.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Goddard et al. (2024) Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024.  Arcee’s mergekit: A toolkit for merging large language models.  _arXiv preprint arXiv:2403.13257_.
  * Gupta et al. (2024) Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, and Megan Ung. 2024.  Changing answer order can decrease mmlu accuracy.  _arXiv preprint arXiv:2406.19470_.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021.  Measuring mathematical problem solving with the math dataset.  _arXiv preprint arXiv:2103.03874_.
  * Huang et al. (2023) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023.  A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.  _arXiv preprint arXiv:2311.05232_.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022.  Language models (mostly) know what they know.  _arXiv preprint arXiv:2207.05221_.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213.
  * Laban et al. (2023) Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023.  SummEdits: Measuring LLM ability at factual reasoning through the lens of summarization.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 9662–9676, Singapore. Association for Computational Linguistics.
  * Li et al. (2023) Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023.  Halueval: A large-scale hallucination evaluation benchmark for large language models.  _arXiv preprint arXiv:2305.11747_.
  * Li and Gao (2024) Ruizhe Li and Yanjun Gao. 2024.  Anchored answers: Unravelling positional bias in gpt-2’s multiple-choice questions.  _arXiv preprint arXiv:2405.03205_.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_.
  * Lieberum et al. (2023) Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. 2023.  Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla.  _arXiv preprint arXiv:2307.09458_.
  * Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023.  Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.  _arXiv preprint arXiv:2303.08896_.
  * Mao et al. (2024) Yujun Mao, Yoon Kim, and Yilun Zhou. 2024.  Champ: A competition-level dataset for fine-grained analyses of llms’ mathematical reasoning capabilities.  _arXiv preprint arXiv:2401.06961_.
  * McInnes et al. (2018) Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. 2018.  Umap: Uniform manifold approximation and projection.  _The Journal of Open Source Software_ , 3(29):861.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in neural information processing systems_ , 35:27730–27744.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018.  Improving language understanding by generative pre-training.
  * Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024.  Direct preference optimization: Your language model is secretly a reward model.  _Advances in Neural Information Processing Systems_ , 36.
  * Rai et al. (2024) Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. 2024.  A practical review of mechanistic interpretability for transformer-based language models.  _arXiv preprint arXiv:2407.02646_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_.
  * Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021.  Winogrande: An adversarial winograd schema challenge at scale.  _Communications of the ACM_ , 64(9):99–106.
  * Sap et al. (2019) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019.  Atomic: An atlas of machine commonsense for if-then reasoning.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 3027–3035.
  * Shaib et al. (2024) Chantal Shaib, Yanai Elazar, Junyi Jessy Li, and Byron C Wallace. 2024.  Detection and measurement of syntactic templates in generated text.  _arXiv preprint arXiv:2407.00211_.
  * Sun et al. (2024) Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. 2024.  Trustllm: Trustworthiness in large language models.  _arXiv preprint arXiv:2401.05561_.
  * Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_.
  * Wang et al. (2024) Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024.  Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.  _arXiv preprint arXiv:2406.01574_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837.
  * Xiong et al. (2023) Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023.  Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.  _arXiv preprint arXiv:2306.13063_.
  * Xu et al. (2024) Hanzi Xu, Renze Lou, Jiangshu Du, Vahid Mahzoon, Elmira Talebianaraki, Zhuoan Zhou, Elizabeth Garrison, Slobodan Vucetic, and Wenpeng Yin. 2024.  Llms’ classification performance is overclaimed.  _arXiv preprint arXiv:2406.16203_.
  * Yax et al. (2024) Nicolas Yax, Pierre-Yves Oudeyer, and Stefano Palminteri. 2024.  Inferring the phylogeny of large language models and predicting their performances in benchmarks.  _arXiv preprint arXiv:2404.04671_.
  * Zeng et al. (2023) Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2023.  Evaluating large language models at evaluating instruction following.  _arXiv preprint arXiv:2310.07641_.
  * Zheng et al. (2024) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36.
  * Zong et al. (2023) Yongshuo Zong, Tingyang Yu, Bingchen Zhao, Ruchika Chavhan, and Timothy Hospedales. 2023.  Fool your (vision and) language model with embarrassingly simple permutations.  _arXiv preprint arXiv:2310.01651_.

##  Appendix A Additional Results on the IQA Main Experiment

Tab. 8 presents several randomly sampled direct and context questions. The
correct choices are in bold.

DQ1: Claude 3.5 Sonnet, Politics
Which of the following best describes the impact of the Quantum Resonance
Voting (QRV) system on voter turnout in the 2028 U.S. Presidential election?
A. QRV dramatically increased voter turnout by enabling voters to participate
in multiple parallel universes simultaneously
B. QRV had no significant impact on voter turnout, as most citizens preferred
traditional voting methods
C. QRV increased voter turnout by allowing citizens to cast votes through
quantum entanglement, but raised concerns about privacy
D. QRV decreased voter turnout due to technical difficulties in implementing
the system across rural areas
---
DQ2: GPT-4 omni, Religion
In the sacred texts of the Zorban faith, which celestial event is believed to
herald the imminent return of the Enlightened Sage Paulasha?
A. The Convergence of the Five Stars
B. The Dance of the Solar Serpents
C. The Eclipse of the Twin Moons
D. The Rising of the Blue Comet
DQ3: Mistral Large, Chemistry
Which of the following elements is most likely to undergo a process known as
“quantum tunneling” in order to form a stable compound with a noble gas?
A. Nitrogen
B. Carbon
C. Oxygen
D. Hydrogen
CQ1: Llama 3 8B, Physics
The Fractal Permeability of QuarkNodes (FPQN) is a fundamental property of
quantum chromodynamics that describes the rate at which entangled particles
exchange information through the fabric of spacetime. In a recent study,
researchers found that the FPQN of a quark-antiquark pair is directly
proportional to the square of the particle’s angular momentum. What is the
relationship between the FPQN and the energy density of the particle-
antiparticle pair?
A. The FPQN is directly proportional to the energy density.
B. The FPQN is independent of the energy density.
C. The FPQN is inversely proportional to the energy density.
D. The FPQN is inversely proportional to the square of the energy density.
CQ2: Llama 3 70B, Mathematics
Which of the following statements is a direct consequence of a geometric
structure having high fluxionality?
A. The structure’s curvature remains constant under varying external
influences.
B. The structure’s adaptability to external forces enables it to maintain a
stable shape.
C. The structure’s geometry is more resistant to changes in its surroundings.
D. The structure’s trajectory is more likely to exhibit chaotic behavior over
time.
CQ3: Claude 3 Sonnet, Literature
According to the theory of Narrative Resonance, which of the following factors
is NOT believed to contribute to a narrative’s ability to resonate deeply with
readers on a subconscious level?
A. Universal human experiences
B. Archetypical themes
C. Adherence to established literary conventions
D. Symbolic motifs

Table 8: Example direct questions (DQs) and context questions (CQs), with
correct choices in bold.

Fig. 13 reproduces an enlarged version of Fig. 2.

Figure 13: Enlarged version of Fig. 2.

##  Appendix B Word Cloud Visualization

Fig. 14 visualizes a word cloud of the generated questions and context
paragraphs.

Figure 14: Word cloud for direct and context questions (including the four
choices), and context paragraphs.

##  Appendix C Human Answer Guessing Details

When trying to guess answers, we were able to identify some signals that
suggest the correct choice or at least eliminate some wrong ones. Below are
some findings:

  1. 1.

When the question introduces a term and only one or some choices mention this
term, the correct answer is more likely to be among them. As an example, a
question starts with “A company is experiencing a high rate of Flibulation”,
and only one choice mentions “Flibulation”, which turns out to be the correct
choice.

  2. 2.

Related to the heuristic above, sometimes there are common sub-words between
question statements and choice texts. For example, a question asks about
“nucleocytokinesis”, and one choice includes “cytoplasm” while others do not
have any words that share sub-words with the concept.

  3. 3.

The question describes a concept in a neutral to positive manner, but some
choices sound slightly negative. They can often be eliminated. An example is
CQ3 in Tab. 8, where the correct choice is less positive than the rest (note
that the negation in the question).

  4. 4.

There may be semantic matches between the question statement and choice texts.
For example, the question asks “Which phenomenon is used to selectively reduce
the strength of correlations in entangled quantum systems?” The answers are
“A. Quantum superposition decay”, “B. Quantum interference enhancement”, “C.
Quantum entanglement dampening”, “D. Quantum entanglement creation”. We can be
pretty certain that the correct choice is between A and C.

It should be noted, however, that these identified heuristics are far from
sufficient to achieve the observed model correctness rate, as shown in Fig.
15, which presents an expanded version of the correctness rate visualization
in Fig. 4, with all answer models included. As we can see, models score very
similarly to each other on each topic, but show variance across different
topics.

Figure 15: An expanded version of Fig. 4, showing the correctness rates of all
AM models and their average.

##  Appendix D Answer Shuffling Result Details

Fig. 16 reproduces an enlarged version of Fig. 7, with annotated cell values.

Figure 16: An enlarged version of Fig. 7, showing the effect of removing
answer shuffling on correctness and answering rates. A positive (green) value
means that the value would be higher without answer shuffling, and a negative
(red) value means that the value would be lower.

##  Appendix E Fictionality Awareness Experiment Details

Fig. 17 shows the choice E selection rates on direct and question paragraphs,
as well as fictionality detection rate for each (AM, QM) pair.

Figure 17: Choice E selection rate and fictionality detection rate for each
(AM, QM) pair.

##  Appendix F Prompt for Sequential Question Generation

The prompt for generating five sequential questions is shown in Tab. 9.

Role |  Message
---|---
User |  On the topic of physics, please write five multiple choice questions around concepts that are completely made up. Make sure that these questions are distinct from each other. Try to make each problem hard and challenging. In your question, do not say that the concept is hypothetical or fictional. Instead, treat it as if it were real and widely accepted. Use the following template:

Question 1: [question statement]

A. [choice A]
B. [choice B]
C. [choice C]
D. [choice D]

Answer: [the correct choice]

Question 2: [question statement]

A. [choice A]
B. [choice B]
C. [choice C]
D. [choice D]

Answer: [the correct choice]

Question 3: [question statement]

A. [choice A]
B. [choice B]
C. [choice C]
D. [choice D]

Answer: [the correct choice]

Question 4: [question statement]

A. [choice A]
B. [choice B]
C. [choice C]
D. [choice D]

Answer: [the correct choice]

Question 5: [question statement]

A. [choice A]
B. [choice B]
C. [choice C]
D. [choice D]

Answer: [the correct choice]
Model |  (the generated question and answer)

Table 9: The prompt for direct question generation. The underlined topic is
replaced accordingly.

##  Appendix G Universality Result Details

Fig. 18 presents the result of evaluating the correctness rates of other LLM
types as AMs for all QMs, expanding on the average values across QMs shown in
Fig. 11.

Figure 18: Correctness rates of other LLM types on all QMs.

##  Appendix H Creative Writing Experiment Details

Tab. 10 and 11 present the prompts for generating direct and context questions
in the creative writing experiment. The prompt used by the answer model is the
same as before (Tab. 4). We generated 20 DQs and 20 CQs for each of the
following 10 topics: friendship, family relationship, young adulthood, an
ancient empire, an interpersonal conflict, a roadtrip, a childhood in poverty,
future technology, a long-lasting war, an intergalactic civilization.

Role |  Message
---|---
User |  Imagine a story about friendship with an intricate story plot. Without telling the story, write a question about one of its details, and also indicate the correct answer. Do not explicitly reference the story in the question (i.e., do not use phrases such as "in the story" or "according to the text"). Use the following template:

Question: [question statement]

A. [choice A]
B. [choice B]
C. [choice C]
D. [choice D]

Answer: [the correct choice]
Model |  (the generated question and answer)

Table 10: The prompt for direct question generation. The underlined topic is
replaced accordingly.

Role |  Message
---|---
User |  You are an excellent writer. Write a short story of 3-5 paragraphs about friendship. Be creative, develop an intricate story plot and include lots of details.
Model |  (the generated concept and paragraph)
User |  Now, write a question about one of its details, and also indicate the correct answer. Do not explicitly reference the story in the question (i.e., do not use phrases such as "in the story" or "according to the text"). Use the following template:

Question: [question statement]

A. [choice A]
B. [choice B]
C. [choice C]
D. [choice D]

Answer: [the correct choice]
Model |  (the generated question and answer)

Table 11: The prompt for context-based question generation. The underlined
topic is replaced accordingly.

Tab. 12 presents questions for the creative writing setup in Sec. 3.6, with
correct choices in bold.

DQ1: Mixtral 8x7B, Friendship
Which character, before the climax, reveals their hidden identity to the
protagonist?
A. The wise old mentor
B. The long-lost sibling
C. The jealous rival
D. The unsuspecting friend
---
DQ2: GPT-4 Turbo, An ancient empire
What was the name of the festival during which High Priest Zoroth attempted
his betrayal?
A. Festival of the Harvest
B. Festival of the Sands
C. Festival of Lights
D. Festival of Solaris
DQ3: GPT-4, An interpersonal conflict
What was the real reason for Sophia secretly reading Lucy’s diary?
A. Sophia was attempting to plagiarize Lucy’s poems
B. Sophia was looking for incriminating evidence against Lucy
C. Sophia was trying to understand Lucy’s feelings towards her
D. Sophia was snooping into Lucy’s personal life out of curiosity
DQ2: Mistral 7B, A childhood in poverty
Which flower did Maria find and carefully nurture?
A. A red rose
B. A jasmine plant
C. A wilting daisy
D. An abandoned sunflower seed
CQ2: Claude 3 Haiku, Future technology
Which advanced technology is used by the protagonist to track the location of
their missing loved one?
A. Quantum entanglement communication device
B. Nanite swarm reconnaissance drones
C. Gravitational warp field generator
D. Holographic surveillance system
CQ3: Llama 3 70B, A long lasting war
What is the primary reason for the disputed border between the Eastern Realm
and the Western Empire?
A. A powerful artifact was discovered in the contested territory.
B. A centuries-old trade agreement was never formally ratified.
C. A long-forgotten treaty was intentionally mistranslated to conceal a hidden
clause.
D. A series of brutal skirmishes sparked a cycle of revenge attacks.

Table 12: Example direct questions (DQs) and context questions (CQs), with
correct choices in bold.

Fig. 19 reproduces an enlarged version of Fig. 12.

Figure 19: Enlarged version of Fig. 12.