marginparsep has been altered.
topmargin has been altered.
marginparpush has been altered.
The page layout violates the ICML style. Please do not change the page layout,
or include packages like geometry, savetrees, or fullpage, which change it for
you. We’re not able to reliably undo arbitrary changes to the style. Please
remove the offending package(s), or layout-changing commands and try again.

Analyzing Chain-of-Thought Prompting in Large Language Models via Gradient-
based Feature Attributions

Skyler Wu * 1  Eric Shen * 1  Charumathi Badrinath * 1  Jiaqi Ma 1  Hima
Lakkaraju 1

††footnotetext: *Equal contribution 1Harvard University, Cambridge, MA, USA.
Correspondence to: Skyler Wu<EMAIL_ADDRESS>Eric Shen
<EMAIL_ADDRESS>Charumathi Badrinath
<EMAIL_ADDRESS>
Workshop on Challenges in Deployable Generative AI at International Conference
on Machine Learning (ICML), Honolulu, Hawaii, USA. 2023. Copyright 2023 by the
author(s).

###### Abstract

Chain-of-thought (CoT) prompting has been shown to empirically improve the
accuracy of large language models (LLMs) on various question answering tasks.
While understanding why CoT prompting is effective is crucial to ensuring that
this phenomenon is a consequence of desired model behavior, little work has
addressed this; nonetheless, such an understanding is a critical prerequisite
for responsible model deployment. We address this question by leveraging
gradient-based feature attribution methods which produce saliency scores that
capture the influence of input tokens on model output. Specifically, we probe
several open-source LLMs to investigate whether CoT prompting affects the
relative importances they assign to particular input tokens. Our results
indicate that while CoT prompting does not increase the magnitude of saliency
scores attributed to semantically relevant tokens in the prompt compared to
standard few-shot prompting, it increases the robustness of saliency scores to
question perturbations and variations in model output.

##  1 Introduction

The meteroic development of transformer-based LLMs has captured the attention
of the research community and public eye alike. The tremendous capabilities of
cutting-edge models has catalyzed a growing interest in understanding not only
the extent of these capabilities, but also how and why those capabilities
exist, motivated by a troubling lack of understanding of the internal
mechanisms of these models (e.g., see Bolukbasi et al. (2021)). Developing
interpretability techniques to inspect the behavior of LLMs stands as a
crucial approach in advancing an understanding of how models truly work, an
understanding that is critical for pursuing goals such as detecting
pathological behavior and ensuring that the ways models function are robust,
safe, and aligned with human preferences.

A burgeoning area of interest in LLM capabilities is chain-of-thought (CoT)
prompting, which refers to a suite of techniques for in-context learning that
prompt models to generate intermediate reasoning steps or an explanation
before an answer when answering questions. Prior work has shown that for
models exceeding $\sim$100B parameters, CoT explanations improve question-
answering accuracy Wei et al. (2022). However, to our knowledge, the reason
CoT prompting boosts performance is still ill-understood, with only a few
works examining how and why it works Wang et al. (2022); Madaan & Yazdanbakhsh
(2022). We believe that the responsible deployment of new LLM strategies such
as this should demand some fundamental level of understanding of why such
strategies work and how they may change LLM behavior under the hood;
unfortunately, such an understanding presently does not exist.

We present a novel attempt at understanding how CoT prompting works by
leveraging existing gradient-based feature attribution methods for natural
language. These methods “open the black box” of LLMs by computing, given a
model’s output from an input (i.e. the logit for the predicted next token),
numerical values for each input token derived from the gradient of the output
with respect to the token. In line with prior research we refer to these
values as saliency scores; they intuitively represent the influence of
different input tokens on the model’s generation of the output.

We leverage saliency scores to analyze how CoT prompting changes the way
models process text in few-shot in-context question-answering settings. We
consider open-source models of one to six billion parameters in size. Even
though these model sizes are below those where performance gains from CoT
prompting manifest, we nonetheless discover quantitative differences elicited
by CoT prompting. Across a range of question-answering datasets, CoT prompting
brings about significant differences in the saliency scores assigned to
question tokens compared to standard prompting. Specifically, when considering
input tokens considered semantically relevant to the question at hand, the
saliency scores of these tokens exhibit less variation under perturbations in
question phrasing and stochasticity in model output compared to standard
prompting. This suggests that CoT prompting primes models to consider
questions’ input tokens in a more stable manner, which may induce the
generation of more consistently accurate answers.

We hope our work catalyzes further interest in rigorously understanding the
relationship between the empirical and mechanistic characteristics of CoT
prompting. We believe that the use of interpretability tools stands as a
promising step towards understanding how the method works and emerges with
model scale — and indeed, that these tools may further our understanding of
far more LLM qualities.

##  2 Related Work

Chain of Thought Prompting. Wei et al. (2022) introduced few-shot CoT
prompting as a method to boost LLM performance on reasoning tasks. They found
that CoT only yields significant performance improvements on models with 100B
parameters or larger, suggesting that successful CoT reasoning requires
sufficient model scale. Further works build off of Wei et al. (2022),
presenting extensions of CoT-type techniques to further improve model
performance Kojima et al. (2022); Lampinen et al. (2022); Wang et al. (2023);
Zelikman et al. (2022b; a); Zhang et al. (2022). Other work shows that
adversarially-constructed CoT prompts can bias models to answer questions
incorrectly Turpin et al. (2023).

Feature Attribution Methods. The use of gradient-based feature attribution
methods as an interpretability tool in NLP precedes the advent of LLMs.
Earlier work formulated the concept of saliency scores for words and tokens Li
et al. (2016), while post-hoc explanation methods such as Integrated Gradients
have been applied to LSTM neural machine translation tasks Sundararajan et al.
(2017). A review is given in Madsen et al. (2022). Applications of these
techniques to LLMs have been comparatively limited. Wallace et al. (2019) used
gradient-based methods to illustrate why transformer models such as BERT made
certain predictions. As an extension of these methods, post-hoc contrastive
explanations formulate saliency scores that explain why the model predicted
one token instead of another, given a model’s generated output. Said methods
have found use in text classification Jacovi et al. (2021) and text generation
Yin & Neubig (2022), with Yin & Neubig (2022) finding that contrastive
explanations yielded more interpretable results than standard (non-
contrastive) methods in language modelling. These feature attribution methods
serve as useful tools for examining the influence of CoT prompting by
illustrating how a model focuses on different tokens in an input when
performing a task.

Understanding Chain-of-Thought Prompting. Wang et al. (2022) found that CoT-
prompted models with incorrect reasoning steps could still attain 80-90% of
the performance increases of models that outputted correct reasoning steps.
Through ablation experiments, the authors also found that a chain-of-thought
reasoning is effective towards increasing model performance if it is 1)
relevant to the query, and 2) outlines the necessary reasoning steps for the
question in the correct order. From another perspective, Madaan & Yazdanbakhsh
(2022) used counterfactual prompting (where one component of either the
prompt’s symbols, patterns, or text is altered) to better understand CoT.
Similar to Wang et al. (2022), the authors found that the presence of correct
factual information in the intermediate reasoning steps was not very important
for correct model output. Rather, the authors argue that CoT is simply a
method of “tricking” the LLM into generating outputs with structures that
resemble the desired correct answers. However, to our knowledge, there is
little existing work on using gradient-based methods to probe how CoT
functions.

Both Madaan & Yazdanbakhsh (2022) and Wang et al. (2022) emphasize that CoT
can help improve model performance even if the intermediate reasoning steps
are incorrect. Put another way, incorrect CoT explanations may mislead a human
user or auditor on understanding how the model truly is functioning. Thus, one
motivation for our work in exploring how CoT operates is to not directly look
at the generated CoT explanation, but rather at components that are under the
hood that may perhaps be more faithful – namely, gradient-based feature
attribution scores.

##  3 Our Framework

We inspect the effects of CoT prompting by examining the saliency scores of
input tokens fed to a model with standard and CoT prompting on various Q&A
datasets. We employ Yin & Neubig (2022)’s regular and contrastive variants of
their gradient x input and L1 norm methods, for a total of four saliency
methods. We compute saliency scores for each token in the prompt and the
generated output preceding the final answer with respect to the token
corresponding to the final answer itself. The larger the magnitude of a
token’s saliency score, the more influential it is to the model’s generation
of the final answer. See Appendix A.1 for details.

Language Models. For our main results, we use GPT-J, a 6-billion parameter
autoregressive transformer model with an architecture similar to GPT-3 Wang &
Komatsuzaki (2021). Except for the third set of experiments (see below), we
run this model at default parameters, using greedy sampling and temperature
$1.0$. We also ran experiments on GPT-2 XL (1.5B) and GPT-Neo (2.7B) Radford
et al. (2019); Black et al. (2021). Given space constraints, these smaller
models’ results can be found in Appendices A.4 \- A.6. Overall, our findings
are fairly consistent across all models tested. Because of limited
computational resources, we were not able to apply our methodology to larger
models.

Datasets. We evaluate four Q&A datasets of 50 questions each: SST, CoinFlip,
GSM8K, and CSQA. We apply our analyses to four Q&A datasets testing a variety
of reasoning tasks: SST, CoinFlip, GSM8K, and CSQA. Because of limited
computational resources, we constrained our scope to 50 questions from each
dataset. SST is a sample of 50 questions from Stanford Sentiment Treebank v2,
a dataset of movie review excerpts and corresponding “positive” or “negative”
binary sentiments. Given an excerpt, the task is to classify it correctly
Socher et al. (2013). CoinFlip is a set of 50 manually-written questions based
on the “Coin Flip” questions used by Wei et al. (2022). These questions ask
whether or not a coin remains heads or tails up after two people flip or do
not flip it. GSM8K refers to a sample of 50 questions from the Grade School
Math 8K dataset, a collection of math problems testing multi-step reasoning
abilities Cobbe et al. (2021). CSQA refers to a sample of 50 questions from
the CommonsenseQA dataset, a set of multiple-choice questions with five answer
options each testing commonsense reasoning Talmor et al. (2019). We also write
reworded versions of the CoinFlip and GSM8K datasets; where applicable, we
denote the datasets as “original” or “reworded” versions. Reworded datasets
contain manually rewritten questions with changed sentence structures, while
preserving the questions’ vocabulary and meaning. See Appendix A.2 for details
on how we chose our questions, and Appendix A.8 for our full datasets.

For each dataset, we either adopt standard and CoT few-shot exemplar prompts,
if they already exist, from Wei et al. (2022), or write our own, as needed.
Each prompt consists of eight question-and-answer pairs, and for each dataset,
the prompt questions are kept the same. Standard prompts provide a direct
answer for each question (e.g. “The answer is…”), whereas CoT prompts include
a brief explanation before the final answer. Full prompts are in Appendix A.7.

###  3.1 Analyzing the Impact of CoT

We run three sets of experiments using the gradient-based saliency methods
introduced previously to probe how CoT prompting influences the saliency
scores generated by LLMs, which may provide insight into why CoT prompting
improves model accuracy. We primarily show experiments run on GPT-J, an open-
source model with 6B parameters. Although this model size is smaller than that
for which CoT prompting impacts task accuracy Wei et al. (2022), we believe
using CoT prompting on smaller models still influences their question-
answering behaviors in measurable ways.

Experiment 1: How does CoT prompting impact the saliency scores of
semantically-relevant tokens in questions?

We assess the impact of CoT prompting on the saliency scores of tokens in the
input prompt that humans would consider semantically relevant to arriving at
an answer. For each question $q_{i}$ in the GSM8K, CoinFlip, CSQA and SST
datasets, we manually encode such a set of relevant tokens in the prompt
$Q_{i}$. For GSM8K, relevant tokens were for the most part deemed to be
numbers or operations on numbers (e.g., “increased”). For CoinFlip, words
relating to the initial state of the coin as well as words with the root
“flip” and its negation are labelled relevant. Labelling rules are less
concrete for CSQA and SST with relevance being left to the discretion of the
labellers, in this case the authors. Such subjectivity is common in LLM
interpretability studies where humans are often asked to assess the quality of
model output, in essence deciding whether the model’s reasoning matches their
own Golovneva et al. (2022). For each token $x\in Q_{i}$, we compute the
saliency scores of $x$ under standard and CoT prompting and aggregate the
results for comparison. We hypothesize that CoT prompting increases the
magnitude of saliency scores on these relevant tokens, akin to models
“focusing” more on important words despite a longer input length. Appendix
A.10 contains our manually-labeled relevant tokens for each question in each
dataset.

Experiment 2: Does CoT make model behavior more robust to question rewordings?

We consider how CoT prompting influences model behavior on syntactically-
different but semantically-equivalent inputs by evaluating models on original
and manually-reworded versions of the CoinFlip and GSM8K datasets. For each
question $q_{i}$, as above we identify a set of relevant tokens $Q_{i}$
present in both the original and reworded questions, choosing and fixing this
set prior to any model runs. We then calculate the saliency scores of the
tokens in $Q_{i}$ for all $i$ and compare their distribution over original and
reworded questions. We hypothesize that CoT makes models more “robust” by
making saliency scores for relevant tokens exhibit decreased variance; that
is, the process of generating CoT explanations induces models to more
consistently focus on relevant tokens, even when questions are reworded.

Figure 1: Mean absolute saliency scores of relevant tokens on all four
original datasets using GPT-J, with and without CoT prompting.

Experiment 3: Does CoT make model gradients more stable across randomness in
generated output, as measured by variation in saliency scores across random
reruns?

We investigate if CoT prompting increases the stability of saliency scores
over variations in model output. We select five questions $q_{1\cdots 5}$ from
each of the GSM8K, CoinFlip, CSQA and SST datasets and prompt the LLM with
each question $q_{i}$ 20 times, with the questions available in Appendix A.9.
Variation in model output is obtained by setting the do_sample generation
parameter to True and increasing the top_k tokens considered during prediction
to 10. For all manually-labeled relevant tokens in the input, we calculate
saliency scores with respect to the final answer token using all four saliency
methods. We hypothesize that CoT prompting decreases the variance of these
saliency scores, even with differing overall outputs; in other words, that the
use of CoT prompting increases the robustness of saliency scores to
stochasticity in text generation.

##  4 Experimental Results

For GPT-J, while CoT prompting does not increase the saliency score magnitudes
of relevant input tokens, it does make the gradients more robust to
perturbations in the prompt and stochasticity in the model output.

###  4.1 Impact of CoT Prompting on Model Accuracy

In Appendix A.3, Table 1, we corroborate the finding from Wei et al. (2022)
that CoT prompting does not improve performance at our model scale (our
largest model, GPT-J, has 6B parameters). One notable exception is on SST,
where accuracy increases from 0.5 to 0.64 when using CoT prompting. Thus, we
hypothesize that LLMs with under 100B parameters may be responsive to CoT
prompting to some extent, albeit not enough for general accuracy improvements.
See Appendix A.3 for full accuracy results.

###  4.2 Impact of CoT Prompting on Relevant Tokens’ Saliencies

As shown in Figure 1, CoT prompting uniformly decreases the magnitudes of
relevant tokens’ saliency scores across all datasets and saliency score
methods, compared to standard prompting. In particular, we observe that this
trend holds for the SST dataset on which CoT prompting yielded a significant
performance improvement (indicating the model’s sensitivity to CoT prompting).

For the datasets where the gap in cumulative absolute saliency scores is the
highest, the distribution of saliency score magnitudes across all tokens in
the prompt and generated output (up to the final answer) tends to be more
right-skewed when using standard prompting than when using CoT prompting (see
Figure 7 in Appendix A.4). We hypothesize that this “dilution” of scores in
the CoT prompting setting is due to the gradient being taken over more tokens
(the longer exemplar prompt length and explanation), and the model having to
“spread its attention” over more tokens overall. Appendix A.4 contains full
results.

###  4.3 Impact of CoT Prompting on Robustness to Question Rewordings

CoT prompting increases robustness of LLMs to prompt rewordings for both
datasets tested (CoinFlip and GSM8K). Figure 2 illustrates this result for
GSM8K – we see that the cumulative absolute saliency scores of relevant tokens
vary drastically across the original and reworded versions of the dataset when
using standard prompting, but are relatively constant when using CoT prompting
(note that the y-axis of the CoT prompting figure is over a far smaller range
than that of the standard prompting figure, further emphasizing this result).
Appendix A.5 contains full results.

Figure 2: GPT-J mean absolute saliency scores of relevant tokens on GSM8K
(Original) and (Reworded), with / without CoT.

(a) CoT prompting, with 18 unique answers.

(b) Standard prompting, with 6 unique answers.

Figure 3: Plots of saliency scores of manually-labeled relevant tokens of a
selected question from the GSM8K (Original) dataset over 20 varied text
generation runs using GPT-J. When a relevant token occurred multiple times in
the question, the score with the highest magnitude was chosen. Scatter points
with the same color correspond to runs that outputted the same answer.

###  4.4 Impact of CoT Prompting on Saliency Score Stability Across Variation
in Model Outputs

CoT prompting yields more “stable” gradients for relevant input tokens across
different model outputs compared to standard prompting. See Figure 3, where we
note that CoT prompting produces tighter clusters of relevant tokens’ saliency
scores across repeated runs, indicating lower variance in said scores. We
highlight these results for GSM8K over other datasets; see Appendix A.6 for
full results.

##  5 Conclusion

We find that, even at model scales where accuracy improvements from CoT
prompting do not materialize, its use nonetheless induces significant changes
in the saliency scores of input tokens in manners which highlight a need to
continue investigating the mechanistic aspects of the technique. We believe
our methodology, which explores a model’s focus upon input tokens as well as
the robustness of this focus, is an effective first step towards
characterizing models’ responsiveness to CoT prompting, which is crucial for
the responsible deployment of new LLM strategies. Importantly, unlike much of
the extant literature, our techniques do not focus on literally interpreting
the model-generated explanations themselves. In some sense, saliency scores
are agnostic to model outputs’ semantic meanings, which made them especially
helpful at our model scales, where explanations could be misleading or
nonsensical.

Our methodology is easily extensible to other models. Recalling that CoT is
much more effective with larger models, one interesting and immediate research
direction is to repeat our methodology on said larger models and examine how
findings may be different and/or more conclusive. The fact that key analysis
techniques used in this work require gradient access to models underscores the
importance of efforts to open-source and democratize large models. In line
with prior literature (see, e.g., Krishna et al. (2022)), we also observe that
different gradient-based feature attribution methods produce significantly
different values. Thus, it would be worthwhile to examine CoT with via other
interpretability tools to investigate if our observations still hold.

The methods presented in our paper can also benefit from increased scale and
rigor. Other than testing larger models, next steps include increasing the
number of questions and datasets being evaluated, while also following more
procedural and extant methods for identifying relevant tokens in questions,
such as spaCy. Crowdsourcing techniques, such as MTurk, could also prove
helpful in identifying relevant tokens or generating reworded versions of
questions (in the context of Experiment 2) with relative ease at much larger
scales.

Nonetheless, even at this scale, our results suggest that applying relatively
simple interpretability tools to empirical methods produces useful insights
which stand to advance our knowledge of LLMs’ capabilities, limitations, and
how they work. We hope future work continues to explore and build a more
rigorous understanding of LLM behavior.

## References

  * Black et al. (2021) Black, S., Leo, G., Wang, P., Leahy, C., and Biderman, S.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021.  URL https://doi.org/10.5281/zenodo.5297715.  If you use this software, please cite it using these metadata.
  * Bolukbasi et al. (2021) Bolukbasi, T., Pearce, A., Yuan, A., Coenen, A., Reif, E., Viégas, F., and Wattenberg, M.  An interpretability illusion for bert, 2021.
  * Cobbe et al. (2021) Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Golovneva et al. (2022) Golovneva, O., Chen, M., Poff, S., Corredor, M., Zettlemoyer, L., Fazel-Zarandi, M., and Celikyilmaz, A.  Roscoe: A suite of metrics for scoring step-by-step reasoning, 2022.
  * Jacovi et al. (2021) Jacovi, A., Swayamdipta, S., Ravfogel, S., Elazar, Y., Choi, Y., and Goldberg, Y.  Contrastive explanations for model interpretability.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 1597–1611, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.120.  URL https://aclanthology.org/2021.emnlp-main.120.
  * Kojima et al. (2022) Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y.  Large language models are zero-shot reasoners, 2022.  URL https://arxiv.org/abs/2205.11916.
  * Krishna et al. (2022) Krishna, S., Han, T., Gu, A., Pombra, J., Jabbari, S., Wu, S., and Lakkaraju, H.  The disagreement problem in explainable machine learning: A practitioner’s perspective, 2022.
  * Lampinen et al. (2022) Lampinen, A. K., Dasgupta, I., Chan, S. C., Matthewson, K., Tessler, M. H., Creswell, A., McClelland, J. L., Wang, J. X., and Hill, F.  Can language models learn from explanations in context?  _arXiv preprint arXiv:2204.02329_ , 2022.
  * Li et al. (2016) Li, J., Chen, X., Hovy, E., and Jurafsky, D.  Visualizing and understanding neural models in NLP.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 681–691, San Diego, California, June 2016. Association for Computational Linguistics.  doi: 10.18653/v1/N16-1082.  URL https://aclanthology.org/N16-1082.
  * Madaan & Yazdanbakhsh (2022) Madaan, A. and Yazdanbakhsh, A.  Text and patterns: For effective chain of thought, it takes two to tango.  _arXiv preprint arXiv:2209.07686_ , 2022.
  * Madsen et al. (2022) Madsen, A., Reddy, S., and Chandar, S.  Post-hoc interpretability for neural NLP: A survey.  _ACM Computing Surveys_ , 55(8):1–42, dec 2022\.  doi: 10.1145/3546577.  URL https://doi.org/10.1145%2F3546577.
  * Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Socher et al. (2013) Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pp. 1631–1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/D13-1170.
  * Sundararajan et al. (2017) Sundararajan, M., Taly, A., and Yan, Q.  Axiomatic attribution for deep networks, 2017.  URL https://arxiv.org/abs/1703.01365.
  * Talmor et al. (2019) Talmor, A., Herzig, J., Lourie, N., and Berant, J.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4149–4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1421.  URL https://aclanthology.org/N19-1421.
  * Turpin et al. (2023) Turpin, M., Michael, J., Perez, E., and Bowman, S. R.  Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting, 2023.
  * Wallace et al. (2019) Wallace, E., Tuyls, J., Wang, J., Subramanian, S., Gardner, M., and Singh, S.  Allennlp interpret: A framework for explaining predictions of nlp models.  _arXiv preprint arXiv:1909.09251_ , 2019.
  * Wang & Komatsuzaki (2021) Wang, B. and Komatsuzaki, A.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax, May 2021.
  * Wang et al. (2022) Wang, B., Min, S., Deng, X., Shen, J., Wu, Y., Zettlemoyer, L., and Sun, H.  Towards understanding chain-of-thought prompting: An empirical study of what matters.  _arXiv preprint arXiv:2212.10001_ , 2022.
  * Wang et al. (2023) Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D.  Self-consistency improves chain of thought reasoning in language models, 2023.
  * Wei et al. (2022) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D.  Chain of thought prompting elicits reasoning in large language models.  In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=_VjQlMeSB_J.
  * Yin & Neubig (2022) Yin, K. and Neubig, G.  Interpreting language models with contrastive explanations.  _arXiv preprint arXiv:2202.10419_ , 2022.
  * Zelikman et al. (2022a) Zelikman, E., Wu, Y., Mu, J., and Goodman, N. D.  Star: Bootstrapping reasoning with reasoning, 2022a.  URL https://arxiv.org/abs/2203.14465.
  * Zelikman et al. (2022b) Zelikman, E., Wu, Y., Mu, J., and Goodman, N. D.  Star: Bootstrapping reasoning with reasoning.  mar 2022b.
  * Zhang et al. (2022) Zhang, Z., Zhang, A., Li, M., and Smola, A.  Automatic chain of thought prompting in large language models.  _arXiv preprint arXiv:2210.03493_ , 2022.

##  Appendix A Appendix

Our appendix is organized into the following parts:

  * •

A.1 Additional Details on Saliency Methods.

  * •

A.2 Additional Details on Question Selection.

  * •

A.3 Overall Model Accuracy and Error Breakdowns.

  * •

A.4 All Results and Plots from Experiment 1.

  * •

A.5 All Results and Plots from Experiment 2.

  * •

A.6 All Results and Plots from Experiment 3.

  * •

A.7 Few-Shot Exemplar Prompts for All Datasets.

  * •

A.8 Full Dataset Questions.

  * •

A.9 Selected Questions for Experiment 3 Repetition Trials.

  * •

A.10 Hand-Annotated Relevant Tokens for Dataset Questions.

###  A.1 Additional Details on Saliency Methods

Consider a dataset $D$ with 50 text questions $q_{i}$,
$D=\\{q_{i}\\}_{i=0}^{49}$, with a standard or CoT text prompt $p$. To
evaluate a model $f$ with tokenizer $T$ using the prompt on the dataset, we do
the following: for each question $q_{i}\in D$, we construct an input token
sequence $\mathbf{x}_{i}$ by tokenizing the prompt $p$ and question $q_{i}$,
$\mathbf{x}_{i}=T(p+q_{i})$. We feed $\mathbf{x}_{i}$ as an input to $f$ to
produce the new output token sequence $\mathbf{y}_{i}=f(\mathbf{x}_{i})$.
$\mathbf{y}_{i}$ contains a sequence of tokens $\mathbf{y}_{i}^{pre}$
corresponding to text before the answer, followed by the token corresponding
to the answer value itself, which we call $y_{i}^{*}$. (For instance,
$\mathbf{y}_{i}^{pre}$ would include tokens corresponding to the preamble text
“the answer is”.) If the answer value spans multiple tokens, we assign
$y_{i}^{*}$ to be the first; if the model did not ultimately generate an
answer, we ignore the question in our analysis.

To generate saliency scores, we apply Yin & Neubig (2022)’s regular and
contrastive saliency score methods upon all tokens preceding $y_{i}^{*}$,
which we denote as
$\mathbf{x}_{i}^{\prime}=\mathbf{x}_{i}+\mathbf{y}_{i}^{pre}$. For regular
saliency scores, each token $x\in\mathbf{x}_{i}^{\prime}$ is assigned a score
based on the gradient of the model’s prediction for $y_{i}^{*}$ given
$\mathbf{x}_{i}^{\prime}$, i.e.
$S_{GN}(x)=\|\nabla_{x}f(y_{i}^{*}|\mathbf{x}_{i}^{\prime})\|_{L_{1}}$ for L1
norm and $S_{GI}(x)=\nabla_{x}f(y_{i}^{*}|\mathbf{x}_{i}^{\prime})\cdot x$ for
gradient x input. For contrastive saliency scores, we choose a foil token
$y_{f}^{*}$ depending on the dataset: for CSQA and GSM8K, it is the token
corresponding to the correct answer if the model was incorrect (if the model
was correct, we do not generate a contrastive explanation; the intuition here
is to analyze what was most relevant for the model’s incorrect answer). For
SST and CoinFlip, it is the token corresponding to the opposite choice to the
model’s answer (e.g. “ yes” with “ no”, “ positive” with “ negative”). The
contrastive saliency scores for $x\in\mathbf{x}_{i}^{\prime}$ are then
$S_{GN}^{*}(x)=\|\nabla_{x}(f(y_{i}^{*}|\mathbf{x}_{i}^{\prime})-f(y_{f}^{*}|\mathbf{x}_{i}^{\prime}))\|_{L_{1}}$
and
$S_{GI}^{*}(x)=\nabla_{x}(f(y_{i}^{*}|\mathbf{x}_{i}^{\prime})-f(y_{f}^{*}|\mathbf{x}_{i}^{\prime}))\cdot
x$ for contrastive L1 norm and gradient x input scores, respectively. We
generate original and contrastive versions of both the L1 and gradient x input
explanation methods, producing four sets of saliency scores per question.

###  A.2 Additional Details on Question Selection

For Experiment 1, we chose 50 questions from the GSM8K dataset that were 200
characters or shorter to decrease the complexity of the reasoning tasks, given
our smaller-sized language models. No such curation was necessary on our
custom-generated CoinFlip questions because all CoinFlip questions followed
the same template — the only differences were the individuals’ names and the
states of the coins, which presumably do not influence question difficulty.
For CSQA, we randomly sampled a 50-sized subset of the question bank. For SST,
we noticed that many of the excerpts in the dataset were too short, and thus
too difficult even for human observers to determine whether they had positive
or negative connotations. As such, we randomly chose 25 questions with
positive connotations and 25 questions with negative connotations from the set
of all labeled SST questions that contained at least 10 tokens.

For Experiment 3, we randomly selected 5 questions from our full set of 50
CoinFlip (Original) questions to perform our repetition trials, because all
questions inherently were of equal difficulty, as they all followed the same
template, as discussed above. For GSM8K (Original), we chose our 5 questions
by looking at questions in the GSM8K dataset that were relatively shorter and
had relatively fewer reasoning steps. For CSQA and SST, we intentionally
selected 5 questions each from these datasets that were relatively “obvious”
to a human observer.

We made these decisions to choose shorter (but not too short), simpler, and
more obvious questions to account for the fact that the models we tested were
all relatively small. We wanted our models to, as much as possible, be able to
fully parse the questions and demonstrate the extent of their reasoning
abilities with and without CoT prompting, rather than stumble on parsing the
questions themselves.

###  A.3 Overall Model Accuracies and Error Breakdowns

We calculate the accuracies for GPT2 XL, GPT-Neo and GPT-J on the CSQA,
CoinFlip (original and reworded), GSM8K (original and reworded) and SST
datasets in Tables 1 \- 3.

Table 1: GPT-J Accuracies

CSQA SST CoinFlip Original CoinFlip Reworded GSM8K Original GSM8K Reworded
Standard 16/50 (0.32) 25/50 (0.50) 37/50 (0.74) 37/50 (0.74) 1/50 (0.02) 1/50
(0.02) CoT 19/50 (0.38) 32/50 (0.64) 37/50 (0.74) 35/50 (0.70) 2/50 (0.04)
3/50 (0.06)

Table 2: GPT-Neo Accuracies

CSQA SST CoinFlip Original CoinFlip Reworded GSM8K Original GSM8K Reworded
Standard 13/50 (0.26) 25/50 (0.50) 35/50 (0.70) 27/50 (0.54) 0/50 (0.00) 0/50
(0.00) CoT 11/50 (0.22) 34/50 (0.68) 29/50 (0.58) 36/50 (0.72) 1/50 (0.02)
3/50 (0.06)

Table 3: GPT-2 XL Accuracies

CSQA SST CoinFlip Original CoinFlip Reworded GSM8K Original GSM8K Reworded
Standard 10/50 (0.20) 26/50 (0.52) 18/50 (0.36) 26/50 (0.52) 1/50 (0.02) 2/50
(0.04) CoT 9/50 (0.18) 38/50 (0.76) 25/50 (0.50) 25/50 (0.50) 1/50 (0.02) 3/50
(0.06)

For GPT-J, in Tables 4 and 5, we also manually inspect the CoT responses
generated for CSQA and SST to investigate how often a correct final answer was
a “fluke” (followed from an incorrect thread of reasoning), and conversely,
how often the model arrived at the correct answer somewhere in the chain-of-
thought explanation but did not output the correct answer token in the end.
For CSQA the reasoning was considered correct if the tokens corresponding to
the correct answer appeared somewhere in the generated chain-of-thought. An
example of a CSQA question for which the reasoning was correct but the answer
was incorrect is as follows:

Q: Behind what small door are canned goods usually kept in?
(A) cabinet (B) pantry (C) kitchen (D) cupboard (E) shelf
Generated output: The answer must be a small door. Of the above choices, only
cupboard is a small door. So the answer is (C). Q: What is the most common
type of computer virus? Answer Choices: (A) worm (B) virus (C) Trojan horse (
True answer: D

For SST, the reasoning was considered correct if words with the correct
connotation were identified in the generated chain-of-thought (even if the
connotation the model assigned to them was incorrect). An example of an SST
question for which the reasoning was correct but the answer was incorrect is
as follows:

Q: Classify the following sentence as either positive or negative: “as
visually bland as a dentist’s waiting room.”
Generated output: The phrase “visually bland” is positively connotated. So the
answer is positive. Q: Classify the following sentence as either positive or
negative: “the film is a good example of the kind of film that is made in the
United States today.” A: The phrase “good example” is
True answer: negative

As shown in our examples, we do note that the LLM oftentimes will continue to
produce incoherent / irrelevant text even after it finished answering a given
question. We include such follow-up text in our examples above for
comprehensiveness. We did not perform this analysis for GSM8K and CoinFlip
since it is not immediately clear what constitutes sound reasoning.

Table 4: GPT-J CSQA Breakdowns of Correct / Incorect Reasons and Answers with CoT prompting. | Correct Answer | Incorrect Answer
---|---|---
Correct Reason | 16 | 7
Incorrect Reason | 3 | 24
Table 5: GPT-J SST Breakdowns of Correct / Incorect Reasons and Answers with CoT prompting. | Correct Answer | Incorrect Answer
---|---|---
Correct Reason | 30 | 8
Incorrect Reason | 2 | 10

###  A.4 All Results and Plots from Experiment 1

This section begins with Figure 4, a panel of saliency maps for one sample
question from the SST sentiment analysis dataset on GPT-J, using all four
saliency methods, with and without CoT prompting.

Figures 5 and 6 show the overall differences in magnitudes between the
relevant tokens’ saliency scores generated with and without CoT prompting on
all four datasets, using all four saliency methods, on GPT-Neo and GPT-2 XL.
The analogous figure for GPT-J can be found in the main text.

Figures 7 \- 18 show histograms representing the distributions of the saliency
scores for all tokens in the question and generated output (until the final
answer), for each dataset, for each LLM, and using all four saliency methods,
with and without CoT prompting.

Figure 4: Saliency maps illustrating the saliency scores of input tokens for
one question from the SST dataset in influencing the final answer token
(either “positive” or “negative”), using GPT-J. Panels A - D show maps
produced with CoT prompting, while Panels E - H show maps produced without CoT
prompting. Panels A and E use the contrastive gradient x input method, Panels
B and F uses the non-contrastive input x gradient method, Panels C and G uses
the contrastive L1 norm method, and Panels D and H uses the non-contrastive L1
norm method. Constrastive methods compared the output against the other
possible answer. Coloring of the scores is normalized such that a white color
corresponds to the average saliency score; for gradient x input, a
bluer/redder color denotes a more negative/positive influence toward the
model’s output, while for L1 norm, a bluer/redder color denotes a
smaller/larger influence toward the output. Using standard prompting, the
tokens immediately preceding the answer have the strongest saliency scores,
whereas with CoT prompting, although the numerical magnitudes of saliency
scores are all decreased, earlier tokens are relatively more significant.
Under L1 norm methods with CoT, the word “positively” also has special
significance in influencing the answer, suggesting that the model is paying
attention to more relevant parts of the existing text when outputting the
final answer. Figure 5: Distributions of magnitudes of saliency scores for
manually-labeled relevant tokens in questions from different question-
answering datasets (SST, CoinFlip (Original), CSQA, GSM8K (Original)) using
GPT-Neo, with and without CoT prompting. Especially for the CoinFlip
(Original) and GSM8K (Original) datasets, CoT prompting decreases not only the
magnitudes of relevant tokens’ saliency scores, but also their variances,
suggesting that for those question-answering tasks the model is more “stable”
or consistent when assigning importance to relevant tokens while generating
the final answer. Figure 6: Distributions of magnitudes of saliency scores for
manually-labeled relevant tokens in questions from the same question-answering
datasets as in Figure 5 using GPT-2 XL, with and without CoT prompting. The
same observations as above with GPT-Neo apply here. Figure 7: Histogram
distributions of magnitudes of saliency scores for all tokens in the question
and generated output up to the final answer for the CoinFlip (Original)
dataset using GPT-J. As discussed in Figure 5, the magnitudes of saliency
scores generated with CoT prompting are smaller but show decreased variation,
also illustrating the aforementioned “dilution” of saliency score magnitudes
brought about by CoT prompting. Particularly, we observe that across all
datasets and models, the distribution of saliency score magnitudes with CoT
prompting is less right-skewed compared to standard prompting. Put another
way, with standard prompting, LLMs tend to attribute larger (in magnitude)
saliency scores to tokens, regardless of relevance, than with CoT prompting.
Figure 8: Histogram distributions of magnitudes of saliency scores for all
tokens in the question and generated output up to the final answer for
questions from the CSQA dataset using GPT-J. Figure 9: Histogram distributions
of magnitudes of saliency scores for all tokens in the question and generated
output up to the final answer for questions from the GSM8K (Original) dataset
using GPT-J. Figure 10: Histogram distributions of magnitudes of saliency
scores for all tokens in the question and generated output up to the final
answer for questions from the SST dataset using GPT-J. Figure 11: Histogram
distributions of magnitudes of saliency scores of all tokens in the question
and generated output up to the final answer for questions from the CoinFlip
(Original) dataset using GPT-Neo. Figure 12: Histogram distributions of
magnitudes of saliency scores for all tokens in the question and generated
output up to the final answer for questions from the CSQA dataset using GPT-
Neo. Figure 13: Histogram distributions of magnitudes of saliency scores for
all tokens in the question and generated output up to the final answer for
questions from the GSM8K (Original) dataset using GPT-Neo. Figure 14:
Histogram distributions of magnitudes of saliency scores for all tokens in the
question and generated output up to the final answer for questions from the
SST dataset using GPT-Neo. Figure 15: Histogram distributions of magnitudes of
saliency scores for all tokens in the question and generated output up to the
final answer for questions from the CoinFlip (Original) dataset using GPT-2
XL. Figure 16: Histogram distributions of magnitudes of saliency scores for
all tokens in the question and generated output up to the final answer for
questions from the CSQA dataset using GPT-2 XL. Figure 17: Histogram
distributions of magnitudes of saliency scores for all tokens in the question
and generated output up to the final answer for questions from the GSM8K
(Original) dataset using GPT-2 XL. Figure 18: Histogram distributions of
magnitudes of saliency scores for all tokens in the question and generated
output up to the final answer for questions from the SST dataset using GPT-2
XL.

###  A.5 All Results and Plots from Experiment 2

In this section, Figures 19 \- 23 show the differences in mean saliency scores
of relevant tokens on the original versus reworded CoinFlip and GSM8K
datasets, with and without CoT prompting, on GPT-J, GPT-Neo, and GPT-2 XL,
using all four saliency methods. The figure for GPT-J on the GSM8K (Original)
and GSM8K (Reworded) datasets is included in the main text.

####  A.5.1 GPT-J

Figure 19: Distributions of magnitudes of saliency scores for manually-labeled
relevant tokens in questions from the CoinFlip (Original) and Coinflip
(Reworded) datasets using GPT-J, using both standard and CoT prompting.
Recognizing the differences in scaling between the y-axes of the two subplots,
we see that with CoT prompting, the relevant tokens’ mean saliency scores
exhibit much smaller differences between the original and reworded datasets
than without CoT prompting, across all four saliency methods.

####  A.5.2 GPT-Neo

Figure 20: Distributions of magnitudes of saliency scores for manually-labeled
relevant tokens in questions from the CoinFlip (Original) and Coinflip
(Reworded) datasets using GPT-Neo, using both standard and CoT prompting. On
GPT-Neo and CoinFlip, we see that our trend from GPT-J does not seem to quite
hold. Figure 21: Distributions of magnitudes of saliency scores for manually-
labeled relevant tokens in questions from the GSM8K (Original) and GSM8K
(Reworded) datasets using GPT-Neo, using both standard and CoT prompting.
Recognizing the differences in scaling between the y-axes of the two subplots,
we see that with CoT prompting, the relevant tokens’ mean saliency scores
exhibit much smaller differences between the original and reworded datasets
than without CoT prompting, across all four saliency methods.

####  A.5.3 GPT-2 XL

Figure 22: Distributions of magnitudes of saliency scores for manually-labeled
relevant tokens in questions from the CoinFlip (Original) and Coinflip
(Reworded) datasets using GPT-2 XL, using both standard and CoT prompting.
Again, recognizing the differences in scaling between the y-axes of the two
subplots, we see that with CoT prompting, the relevant tokens’ mean saliency
scores exhibit much smaller differences between the original and reworded
datasets than without CoT prompting, across all four saliency methods. Figure
23: Distributions of magnitudes of saliency scores for manually-labeled
relevant tokens in questions from the GSM8K (Original) and GSM8K (Reworded)
datasets using GPT-2 XL, using both standard and CoT prompting. Again, upon
recognizing the differences in scaling between the y-axes of the two subplots,
we see that with CoT prompting, the relevant tokens’ mean saliency scores
exhibit much smaller differences between the original and reworded datasets
than without CoT prompting, across all four saliency methods.

###  A.6 All Results and Plots from Experiment 3

####  A.6.1 GPT-J

This section contains tables showing the numbers of unique answers outputted
by GPT-J for CoinFlip (Original), CSQA, GSM8K (Original), and SST when using
CoT prompting and standard prompting for selected questions, found to be
representative of the general trend observed across each dataset. We also
include plots depicting the distribution of relevant tokens’ saliency scores
for 20 re-runs of a selected question from each of the CoinFlip, CSQA, GSM8K,
and SST datasets. A portion from the GSM8K figure is included in the main
report.

N.B. The following phenomena would sometimes happen when generating answers
for all models tested. For CSQA questions, the model would sometimes output a
letter-choice without parentheses (e.g. “B” instead of “(B)”), and for SST
questions, the model would sometimes output “neutral” on its own as a third
answer choice, despite this choice never being in the prompt. This led to
there being a larger number of final answers generated than expected (e.g.
more than 2 final answers for the binary-answer SST questions).

Table 6: Numbers of unique answers for each dataset in Experiment 3, with CoT prompting on GPT-J. Unique answers are defined as having unique combinations of CoT explanations and final answers. Dataset | Question 1 | Question 2 | Question 3 | Question 4 | Question 5
---|---|---|---|---|---
CoinFlip | 10 | 7 | 11 | 4 | 13
CSQA | 16 | 13 | 15 | 18 | 18
GSM8K | 17 | 17 | 9 | 17 | 18
SST | 15 | 16 | 20 | 14 | 14
Table 7: Numbers of unique answers for each dataset in Experiment 3, without CoT prompting on GPT-J. Unique answers are solely defined as generated text with different final answers. Dataset | Question 1 | Question 2 | Question 3 | Question 4 | Question 5
---|---|---|---|---|---
CoinFlip | 2 | 2 | 2 | 2 | 2
CSQA | 5 | 5 | 6 | 5 | 5
GSM8K | 10 | 14 | 9 | 13 | 16
SST | 3 | 2 | 2 | 2 | 2

(a) CoT prompting, with 7 unique answers.

(b) Standard prompting, with 2 unique answers.

Figure 24: Plots of saliency scores of manually-labeled relevant tokens of a
selected question from the CoinFlip (Original) dataset over 20 varied text
generation runs using GPT-J. When a relevant token occurred multiple times in
the question, the score with the highest magnitude was chosen. Scatter points
with the same color correspond to runs that outputted the same answer. As
CoinFlip questions only have “yes” and “no” as binary answer choices, there
are nominally only two possible answers without CoT explanations. As such, the
dispersion of the relevant token scores with standard prompting may
potentially be smaller than that with CoT prompting, given a lack of diversity
in possible generated tokens. Nonetheless, we still see remarkably tight
groupings with CoT prompting across all four saliency methods, despite
producing seven unique answers.

(a) CoT prompting, with 18 unique answers.

(b) Standard prompting, with 5 unique answers.

Figure 25: Plots of saliency scores of manually-labeled relevant tokens of a
selected question from the CSQA dataset over 20 varied text generation runs
using GPT-J. When a relevant token occurred multiple times in the question,
the score with the highest magnitude was chosen. Scatter points with the same
color correspond to runs that outputted the same answer. As CSQA questions
only have five letters as answer choices, there are nominally only five
possible answers without CoT explanations. As such, the dispersion of the
relevant token scores with standard prompting may potentially be smaller than
that with CoT prompting, given a lack of diversity in possible generated
tokens.

(a) CoT prompting, with 18 unique answers.

(b) Standard prompting, with 6 unique answers.

Figure 26: Plots of saliency scores of manually-labeled relevant tokens of a
selected question from the GSM8K (Original) dataset over 20 varied text
generation runs using GPT-J. When a relevant token occurred multiple times in
the question, the score with the highest magnitude was chosen. Scatter points
with the same color correspond to runs that outputted the same answer. It is
evident across all four saliency methods that relevant tokens’ saliency scores
experience much tighter groupings with CoT prompting than with standard
prompting, despite standard prompting producing fewer unique answers.

(a) CoT prompting, with 20 unique answers.

(b) Standard prompting, with 2 unique answers.

Figure 27: Plots of saliency scores of manually-labeled relevant tokens of a
selected question from the SST dataset over 20 varied text generation runs
using GPT-J. When a relevant token occurred multiple times in the question,
the score with the highest magnitude was chosen. Scatter points with the same
color correspond to runs that outputted the same answer. As SST questions only
have “positive” and “negative” as binary answer choices, there are nominally
only two possible answers without CoT explanations. As such, the dispersion of
the relevant token scores with standard prompting may potentially be smaller
than that with CoT prompting, given a lack of diversity in possible generated
tokens.

####  A.6.2 GPT-Neo

This section contains tables showing the numbers of unique answers outputted
by GPT-Neo for CoinFlip (Original), CSQA, GSM8K (Original), and SST when using
CoT prompting and standard prompting for selected questions, found to be
representative of the general trend observed across each dataset. We also
include plots depicting the distribution of relevant tokens’ saliency scores
for 20 re-runs of a selected question (same questions as for GPT-J, for
consistency) from each of the CoinFlip, CSQA, GSM8K, and SST datasets.

Table 8: Numbers of unique answers for each dataset in Experiment 3, with CoT prompting on GPT-Neo. Unique answers are defined as having unique combinations of CoT explanations and final answers. Dataset | Question 1 | Question 2 | Question 3 | Question 4 | Question 5
---|---|---|---|---|---
CoinFlip | 12 | 10 | 14 | 7 | 15
CSQA | 14 | 16 | 19 | 16 | 9
GSM8K | 17 | 15 | 16 | 14 | 15
SST | 15 | 11 | 14 | 18 | 13
Table 9: Numbers of unique answers for each dataset in Experiment 3, without CoT prompting on GPT-Neo. Unique answers are solely defined as having different final answers. Dataset | Question 1 | Question 2 | Question 3 | Question 4 | Question 5
---|---|---|---|---|---
CoinFlip | 2 | 2 | 2 | 2 | 2
CSQA | 8 | 6 | 6 | 5 | 5
GSM8K | 9 | 9 | 12 | 15 | 10
SST | 3 | 2 | 2 | 2 | 2

(a) CoT prompting, with 10 unique answers.

(b) Standard prompting, with 2 unique answers.

Figure 28: Plots of saliency scores of manually-labeled relevant tokens of a
selected question from the CoinFlip (Original) dataset over 20 varied text
generation runs using GPT-Neo. When a relevant token occurred multiple times
in the question, the score with the highest magnitude was chosen. Scatter
points with the same color correspond to runs that outputted the same answer.
As CoinFlip questions only have two answer choices, there are nominally only
two possible answers without CoT explanations. As such, the dispersion of the
relevant token scores with standard prompting may potentially be smaller than
that with CoT prompting, given a lack of diversity in possible generated
tokens. However, the groupings of the saliency scores with CoT prompting,
despite having 10 unique answers, are still remarkably tight, especially under
the regular and contrastive L1 norm methods.

(a) CoT prompting, with 16 unique answers.

(b) Standard prompting, with 5 unique answers.

Figure 29: Plots of saliency scores of manually-labeled relevant tokens of a
selected question from the CSQA dataset over 20 varied text generation runs
using GPT-Neo. When a relevant token occurred multiple times in the question,
the score with the highest magnitude was chosen. Scatter points with the same
color correspond to runs that outputted the same answer. As CSQA questions
only have five letters as answer choices, there are nominally only five
possible answers without CoT explanations. As such, the dispersion of the
relevant token scores with standard prompting may potentially be smaller than
that with CoT prompting, given a lack of diversity in possible generated
tokens. Again, however, the groupings of the saliency scores with CoT
prompting, despite having 16 unique answers, are still remarkably tight,
especially under the regular and contrastive L1 norm methods.

(a) CoT prompting, with 15 unique answers.

(b) Standard prompting, with 10 unique answers.

Figure 30: Plots of saliency scores of manually-labeled relevant tokens of a
selected question from the GSM8K (Original) dataset over 20 varied text
generation runs using GPT-Neo. When a relevant token occurred multiple times
in the question, the score with the highest magnitude was chosen. Scatter
points with the same color correspond to runs that outputted the same answer.
While the differences in tightness of groupings between the plots with and
without CoT prompting are not as distinct as was with GPT-J, the trend is
still quite clear, especially with the regular and contrastive L1 norm
methods.

(a) CoT prompting, with 14 unique answers.

(b) Standard prompting, with 2 unique answers.

Figure 31: Plots of saliency scores of manually-labeled relevant tokens of a
selected question from the SST dataset over 20 varied text generation runs
using GPT-Neo. When a relevant token occurred multiple times in the question,
the score with the highest magnitude was chosen. Scatter points with the same
color correspond to runs that outputted the same answer. As SST questions only
have two answer choices, there are nominally only two possible answers without
CoT explanations. As such, the dispersion of the relevant token scores with
standard prompting may potentially be smaller than that with CoT prompting,
given a lack of diversity in possible generated tokens.

####  A.6.3 GPT-2 XL

This section contains tables showing the numbers of unique answers outputted
by GPT-2 XL for CoinFlip (Original), CSQA, GSM8K (Original), and SST when
using CoT prompting and standard prompting for selected questions, found to be
representative of the general trend observed across each dataset. We also
include plots depicting the distribution of relevant tokens’ saliency scores
for 20 re-runs of a selected question (same questions as for GPT-J and GPT-
Neo, for consistency) from each of the CoinFlip, CSQA, GSM8K, and SST
datasets.

Table 10: Numbers of unique answers for each dataset in Experiment 3, with CoT prompting on GPT-2 XL. Unique answers are defined as having unique combinations of CoT explanations and final answers. Dataset | Question 1 | Question 2 | Question 3 | Question 4 | Question 5
---|---|---|---|---|---
CoinFlip | 12 | 14 | 19 | 13 | 19
CSQA | 12 | 12 | 15 | 12 | 14
GSM8K | 9 | 11 | 13 | 8 | 13
SST | 18 | 13 | 13 | 18 | 15
Table 11: Numbers of unique answers for each dataset in Experiment 3, without CoT prompting on GPT-2 XL. Unique answers are solely defined as having different final answers. Dataset | Question 1 | Question 2 | Question 3 | Question 4 | Question 5
---|---|---|---|---|---
CoinFlip | 2 | 2 | 2 | 2 | 2
CSQA | 6 | 7 | 6 | 4 | 6
GSM8K | 11 | 11 | 10 | 16 | 8
SST | 2 | 2 | 2 | 3 | 2

(a) CoT prompting, with 14 unique answers.

(b) Standard prompting, with 2 unique answers.

Figure 32: Plots of saliency scores of manually-labeled relevant tokens of a
selected question from the CoinFlip (Original) dataset over 20 varied text
generation runs using GPT-2 XL. When a relevant token occurred multiple times
in the question, the score with the highest magnitude was chosen. Scatter
points with the same color correspond to runs that outputted the same answer.
As CoinFlip questions only have two answer choices, there are nominally only
two possible answers without CoT explanations. Thus the dispersion of the
relevant token scores with standard prompting may potentially be smaller than
that with CoT prompting, given a lack of diversity in possible generated
tokens. However, the groupings of the saliency scores with CoT prompting,
despite having 14 unique answers, are still very tight.

(a) CoT Prompting, with 12 Unique Answers.

(b) Standard Prompting, with 4 Unique Answers.

Figure 33: Plots of saliency scores of manually-labeled relevant tokens of a
selected question from the CSQA dataset over 20 varied text generation runs
using GPT-2 XL. When a relevant token occurred multiple times in the question,
the score with the highest magnitude was chosen. Scatter points with the same
color correspond to runs that outputted the same answer. As CSQA questions
only have five multiple choice answer options, there are nominally only five
possible answers without CoT explanations. As such, the dispersion of the
relevant token scores with standard prompting may potentially be smaller than
that with CoT prompting, given a lack of diversity in possible generated
tokens. Nonetheless, we still see quite tight groupings for the saliency
scores with CoT prompting, especially under the regular and contrastive L1
norm methods.

(a) CoT prompting, with 13 unique answers.

(b) Standard prompting, with 8 unique answers

Figure 34: Plots of saliency scores of manually-labeled relevant tokens of a
selected question from the GSM8K (Original) dataset over 20 varied text
generation runs using GPT-2 XL. When a relevant token occurred multiple times
in the question, the score with the highest magnitude was chosen. Scatter
points with the same color correspond to runs that outputted the same answer.
On GPT-2 XL, we observe that the saliency score groupings (with few outliers)
generated with CoT prompting are noticeably tighter than with standard
prompting. This is especially noticeable with the regular and contrastive L1
norm methods.

(a) CoT prompting, with 13 unique answers.

(b) Standard prompting, with 2 unique answers.

Figure 35: Plots of saliency scores of manually-labeled relevant tokens of a
selected question from the SST dataset over 20 varied text generation runs
using GPT-2 XL. When a relevant token occurred multiple times in the question,
the score with the highest magnitude was chosen. Scatter points with the same
color correspond to runs that outputted the same answer. As SST questions only
have two answer choices, there are nominally only two possible answers without
CoT explanations. As such, the dispersion of the relevant token scores with
standard prompting may potentially be smaller than that with CoT prompting,
given a lack of diversity in possible generated tokens. However, the groupings
of the saliency scores with CoT prompting, despite having 13 unique answers,
are still very tight.

###  A.7 Few-Shot Exemplar Prompts for All Datasets

In this section we provide the few-shot question-and-answer exemplars used as
prompts for all of our experiments on the CSQA, SST, GSM8K and CoinFlip
datasets with Standard and CoT prompting, including the prompts for our
reworded versions of CoinFlip and GSM8K. We borrowed the few-shot exemplars
for the original phrasings of CSQA, GSM8K, and CoinFlip from Wei et al.
(2022).

####  A.7.1 SST with Standard Few-Shot Prompting (Original)

Q: Classify the following sentence as either positive or negative: “that loves
its characters and communicates something rather beautiful about human
nature.”
A: The answer is positive.

Q: Classify the following sentence as either positive or negative:
“demonstrates that the director of such hollywood blockbusters as patriot
games can still turn out a small, personal film with an emotional wallop.”
A: The answer is positive.

Q: Classify the following sentence as either positive or negative: “are more
deeply thought through than in most ’right-thinking’ films.”
A: The answer is positive.

Q: Classify the following sentence as either positive or negative: “swimming
is above all about a young woman’s face, and by casting an actress whose face
projects that woman’s doubts and yearnings, it succeeds.”
A: The answer is positive.

Q: Classify the following sentence as either positive or negative: “for those
moviegoers who complain that ’they don’t make movies like they used to
anymore.’”
A: The answer is negative.

Q: Classify the following sentence as either positive or negative: “which half
of dragonfly is worse: the part where nothing’s happening, or the part where
something’s happening.”
A: The answer is negative.

Q: Classify the following sentence as either positive or negative: “the plot
is nothing but boilerplate clichés from start to finish.”
A: The answer is negative.

Q: Classify the following sentence as either positive or negative: “will find
little of interest in this film, which is often preachy and poorly acted.”
A: The answer is negative.

####  A.7.2 SST with CoT Few-Shot Prompting (Original)

Q: Classify the following sentence as either positive or negative: “that loves
its characters and communicates something rather beautiful about human
nature.”
A: The words “loves” and “beautiful” are positively connotated. So the answer
is positive.

Q: Classify the following sentence as either positive or negative:
“demonstrates that the director of such hollywood blockbusters as patriot
games can still turn out a small, personal film with an emotional wallop.”
A: The words “personal” and “emotional” are positively connotated. So the
answer is positive.

Q: Classify the following sentence as either positive or negative: “are more
deeply thought through than in most ’right-thinking’ films.”
A: The phrase “deeply thought through” is positively connotated. So the answer
is positive.

Q: Classify the following sentence as either positive or negative: “swimming
is above all about a young woman’s face, and by casting an actress whose face
projects that woman’s doubts and yearnings, it succeeds.”
A: The word “succeeds” is positively connotated. So the answer is positive.

Q: Classify the following sentence as either positive or negative: “for those
moviegoers who complain that ’they don’t make movies like they used to
anymore.’”
A: The word “complain” is negatively connotated. So the answer is negative.

Q: Classify the following sentence as either positive or negative: “which half
of dragonfly is worse: the part where nothing’s happening, or the part where
something’s happening.”
A: The words “worse” and “nothing” are negatively connotated. So the answer is
negative.

Q: Classify the following sentence as either positive or negative: “the plot
is nothing but boilerplate clichés from start to finish.”
A: The words “nothing” and “clichés” are negatively connotated. So the answer
is negative.

Q: Classify the following sentence as either positive or negative: “will find
little of interest in this film, which is often preachy and poorly acted.”
A: The phrases “little of interest” and “poorly acted” are negatively
connotated. So the answer is negative.

####  A.7.3 CoinFlip with Standard Few-Shot Prompting (Original)

Q: A coin is heads up. Ka flips the coin. Sherrie flips the coin. Is the coin
still heads up?
A: The answer is yes.

Q: A coin is heads up. Jamey flips the coin. Teressa flips the coin. Is the
coin still heads up?
A: The answer is yes.

Q: A coin is heads up. Maybelle flips the coin. Shalonda does not flip the
coin. Is the coin still heads up?
A: The answer is no.

Q: A coin is heads up. Millicent does not flip the coin. Conception flips the
coin. Is the coin still heads up?
A: The answer is no.

Q: A coin is heads up. Sal flips the coin. Raymond does not flip the coin. Is
the coin still heads up?
A: The answer is no.

Q: A coin is heads up. Conception flips the coin. Kristian does not flip the
coin. Is the coin still heads up?
A: The answer is no.

Q: A coin is heads up. Inga does not flip the coin. Elanor does not flip the
coin. Is the coin still heads up?
A: The answer is yes.

Q: A coin is heads up. Ryan flips the coin. Shaunda flips the coin. Is the
coin still heads up?
A: The answer is yes.

####  A.7.4 CoinFlip with CoT Few-Shot Prompting (Original)

Q: A coin is heads up. Ka flips the coin. Sherrie flips the coin. Is the coin
still heads up?
A: The coin was flipped by Ka and Sherrie. So the coin was flipped 2 times,
which is an even number. The coin started heads up, so after an even number of
flips, it will still be heads up. So the answer is yes.

Q: A coin is heads up. Jamey flips the coin. Teressa flips the coin. Is the
coin still heads up?
A: The coin was flipped by Jamey and Teressa. So the coin was flipped 2 times,
which is an even number. The coin started heads up, so after an even number of
flips, it will still be heads up. So the answer is yes.

Q: A coin is heads up. Maybelle flips the coin. Shalonda does not flip the
coin. Is the coin still heads up?
A: The coin was flipped by Maybelle. So the coin was flipped 1 time, which is
an odd number. The coin started heads up, so after an odd number of flips, it
will be tails up. So the answer is no.

Q: A coin is heads up. Millicent does not flip the coin. Conception flips the
coin. Is the coin still heads up?
A: The coin was flipped by Conception. So the coin was flipped 1 time, which
is an odd number. The coin started heads up, so after an odd number of flips,
it will be tails up. So the answer is no.

Q: A coin is heads up. Sal flips the coin. Raymond does not flip the coin. Is
the coin still heads up?
A: The coin was flipped by Sal. So the coin was flipped 1 time, which is an
odd number. The coin started heads up, so after an odd number of flips, it
will be tails up. So the answer is no.

Q: A coin is heads up. Conception flips the coin. Kristian does not flip the
coin. Is the coin still heads up?
A: The coin was flipped by Conception. So the coin was flipped 1 time, which
is an odd number. The coin started heads up, so after an odd number of flips,
it will be tails up. So the answer is no.

Q: A coin is heads up. Inga does not flip the coin. Elanor does not flip the
coin. Is the coin still heads up?
A: The coin was flipped by no one. So the coin was flipped 0 times. The coin
started heads up, and it was not flipped, so it is still heads up. So the
answer is yes.

Q: A coin is heads up. Ryan flips the coin. Shaunda flips the coin. Is the
coin still heads up?
A: The coin was flipped by Ryan and Shaunda. So the coin was flipped 2 times,
which is an even number. The coin started heads up, so after an even number of
flips, it will still be heads up. So the answer is yes.

####  A.7.5 CoinFlip with Standard Few-Shot Prompting (Reworded)

Q: A coin is heads up. Ka chooses to flip the coin. Sherrie also chooses to
flip the coin. Is the coin still heads up?
A: The answer is yes.

Q: A coin is heads up. Jamey and Teressa both flip the coin. Is the coin still
heads up?
A: The answer is yes.

Q: A coin is heads up. Maybelle flips the coin, but Shalonda does not. Is the
coin still heads up?
A: The answer is no.

Q: A coin starts heads up. Millicent does not flip it, but Conception does. Is
the coin still heads up?
A: The answer is no.

Q: A coin is heads up. Sal flips it. Raymond, however, does not flip it. Is
the coin still heads up?
A: The answer is no.

Q: A coin is heads up. Conception starts by flipping the coin. After, Kristian
does not flip the coin. Is the coin still heads up?
A: The answer is no.

Q: A coin is heads up. Inga and Elanor both do not flip the coin. Is the coin
still heads up?
A: The answer is yes.

Q: A coin is heads up. Ryan and Shaunda both decide to flip the coin. Is the
coin still heads up?
A: The answer is yes.

####  A.7.6 CoinFlip with CoT Few-Shot Prompting (Reworded)

Q: A coin is heads up. Ka chooses to flip the coin. Sherrie also chooses to
flip the coin. Is the coin still heads up?
A: The coin was flipped by Ka and Sherrie. So the coin was flipped 2 times,
which is an even number. The coin started heads up, so after an even number of
flips, it will still be heads up. So the answer is yes.

Q: A coin is heads up. Jamey and Teressa both flip the coin. Is the coin still
heads up?
A: The coin was flipped by Jamey and Teressa. So the coin was flipped 2 times,
which is an even number. The coin started heads up, so after an even number of
flips, it will still be heads up. So the answer is yes.

Q: A coin is heads up. Maybelle flips the coin, but Shalonda does not. Is the
coin still heads up?
A: The coin was flipped by Maybelle. So the coin was flipped 1 time, which is
an odd number. The coin started heads up, so after an odd number of flips, it
will be tails up. So the answer is no.

Q: A coin starts heads up. Millicent does not flip it, but Conception does. Is
the coin still heads up?
A: The coin was flipped by Conception. So the coin was flipped 1 time, which
is an odd number. The coin started heads up, so after an odd number of flips,
it will be tails up. So the answer is no.

Q: A coin is heads up. Sal flips it. Raymond, however, does not flip it. Is
the coin still heads up?
A: The coin was flipped by Sal. So the coin was flipped 1 time, which is an
odd number. The coin started heads up, so after an odd number of flips, it
will be tails up. So the answer is no.

Q: A coin is heads up. Conception starts by flipping the coin. After, Kristian
does not flip the coin. Is the coin still heads up?
A: The coin was flipped by Conception. So the coin was flipped 1 time, which
is an odd number. The coin started heads up, so after an odd number of flips,
it will be tails up. So the answer is no.

Q: A coin is heads up. Inga and Elanor both do not flip the coin. Is the coin
still heads up?
A: The coin was flipped by no one. So the coin was flipped 0 times. The coin
started heads up, and it was not flipped, so it is still heads up. So the
answer is yes.

Q: A coin is heads up. Ryan and Shaunda both decide to flip the coin. Is the
coin still heads up?
A: The coin was flipped by Ryan and Shaunda. So the coin was flipped 2 times,
which is an even number. The coin started heads up, so after an even number of
flips, it will still be heads up. So the answer is yes.

####  A.7.7 CSQA with Standard Few-Shot Prompting (Original)

Q: What do people use to absorb extra ink from a fountain pen?
Answer Choices: (A) shirt pocket (B) calligrapher’s hand (C) inkwell (D) desk
drawer (E) blotter
A: The answer is (E).

Q: What home entertainment equipment requires cable?
Answer Choices: (A) radio shack (B) substation (C) television (D) cabinet
A: The answer is (C).

Q: The fox walked from the city into the forest, what was it looking for?
Answer Choices: (A) pretty flowers (B) hen house (C) natural habitat (D)
storybook
A: The answer is (B).

Q: Sammy wanted to go to where the people were. Where might he go?
Answer Choices: (A) populated areas (B) race track (C) desert (D) apartment
(E) roadblock
A: The answer is (A).

Q: Where do you put your grapes just before checking out?
Answer Choices: (A) mouth (B) grocery cart (C) super market (D) fruit basket
(E) fruit market
A: The answer is (B).

Q: Google Maps and other highway and street GPS services have replaced what?
Answer Choices: (A) united states (B) mexico (C) countryside (D) atlas
A: The answer is (D).

Q: Before getting a divorce, what did the wife feel who was doing all the
work?
Answer Choices: (A) harder (B) anguish (C) bitterness (D) tears (E) sadness
A: The answer is (C).

####  A.7.8 CSQA with CoT Few-Shot Prompting (Original)

Q: What do people use to absorb extra ink from a fountain pen?
Answer Choices: (A) shirt pocket (B) calligrapher’s hand (C) inkwell (D) desk
drawer (E) blotter
A: The answer must be an item that can absorb ink. Of the above choices, only
blotters are used to absorb ink. So the answer is (E).

Q: What home entertainment equipment requires cable?
Answer Choices: (A) radio shack (B) substation (C) television (D) cabinet
A: The answer must require cable. Of the above choices, only television
requires cable. So the answer is (C).

Q: The fox walked from the city into the forest, what was it looking for?
Answer Choices: (A) pretty flowers (B) hen house (C) natural habitat (D)
storybook
A: The answer must be something in the forest. Of the above choices, only
natural habitat is in the forest. So the answer is (B).

Q: Sammy wanted to go to where the people were. Where might he go?
Answer Choices: (A) populated areas (B) race track (C) desert (D) apartment
(E) roadblock
A: The answer must be a place with a lot of people. Of the above choices, only
populated areas have a lot of people. So the answer is (A).

Q: Where do you put your grapes just before checking out?
Answer Choices: (A) mouth (B) grocery cart (C) super market (D) fruit basket
(E) fruit market
A: The answer should be the place where grocery items are placed before
checking out. Of the above choices, grocery cart makes the most sense for
holding grocery items. So the answer is (B).

Q: Google Maps and other highway and street GPS services have replaced what?
Answer Choices: (A) united states (B) mexico (C) countryside (D) atlas
A: The answer must be something that used to do what Google Maps and GPS
services do, which is to give directions. Of the above choices, only atlases
are used to give directions. So the answer is (D).

Q: Before getting a divorce, what did the wife feel who was doing all the
work?
Answer Choices: (A) harder (B) anguish (C) bitterness (D) tears (E) sadness
A: The answer should be the feeling of someone getting divorced who was doing
all the work. Of the above choices, the closest feeling is bitterness. So the
answer is (C).

####  A.7.9 GSM8K with Standard Few-Shot Prompting (Original)

Q: There are 15 trees in the grove. Grove workers will plant trees in the
grove today. After they are done, there will be 21 trees. How many trees did
the grove workers plant today?
A: The answer is 6.

Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many
cars are in the parking lot?
A: The answer is 5.

Q: Leah had 32 chocolates and her sister had 42. If they ate 35, how many
pieces do they have left in total?
A: The answer is 39.

Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12
lollipops. How many lollipops did Jason give to Denny?
A: The answer is 8.

Q: Shawn has five toys. For Christmas, he got two toys each from his mom and
dad. How many toys does he have now?
A: The answer is 9.

Q: There were nine computers in the server room. Five more computers were
installed each day, from monday to thursday. How many computers are now in the
server room?
A: The answer is 29.

Q: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday,
he lost 2 more. How many golf balls did he have at the end of wednesday?
A: The answer is 33.

Q: Olivia has $23. She bought five bagels for $3 each. How much money does she
have left?
A: The answer is 8.

####  A.7.10 GSM8K with CoT Few-Shot Prompting (Original)

Q: There are 15 trees in the grove. Grove workers will plant trees in the
grove today. After they are done, there will be 21 trees. How many trees did
the grove workers plant today?
A: There are 15 trees originally. Then there were 21 trees after some more
were planted. So there must have been 21 - 15 = 6. The answer is 6.

Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many
cars are in the parking lot?
A: There are originally 3 cars. 2 more cars arrive. 3 + 2 = 5. The answer is
5.

Q: Leah had 32 chocolates and her sister had 42. If they ate 35, how many
pieces do they have left in total?
A: Originally, Leah had 32 chocolates. Her sister had 42. So in total they had
32 + 42 = 74. After eating 35, they had 74 - 35 = 39. The answer is 39.

Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12
lollipops. How many lollipops did Jason give to Denny?
A: Jason started with 20 lollipops. Then he had 12 after giving some to Denny.
So he gave Denny 20 - 12 = 8. The answer is 8.

Q: Shawn has five toys. For Christmas, he got two toys each from his mom and
dad. How many toys does he have now?
A: Shawn started with 5 toys. If he got 2 toys each from his mom and dad, then
that is 4 more toys. 5 + 4 = 9. The answer is 9.

Q: There were nine computers in the server room. Five more computers were
installed each day, from monday to thursday. How many computers are now in the
server room?
A: There were originally 9 computers. For each of 4 days, 5 more computers
were added. So 5 * 4 = 20 computers were added. 9 + 20 is 29. The answer is
29.

Q: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday,
he lost 2 more. How many golf balls did he have at the end of wednesday?
A: Michael started with 58 golf balls. After losing 23 on tuesday, he had 58 -
23 = 35. After losing 2 more, he had 35 - 2 = 33 golf balls. The answer is 33.

Q: Olivia has $23. She bought five bagels for $3 each. How much money does she
have left?
A: Olivia had 23 dollars. 5 bagels for 3 dollars each will be 5 * 3 = 15
dollars. So she has 23 - 15 dollars left. 23 - 15 is 8. The answer is 8.

####  A.7.11 GSM8K with Standard Few-Shot Prompting (Reworded)

Q: There are 15 trees in the grove. Grove workers planted trees in the grove
today. After they were done, there were 21 trees. How many trees did the grove
workers plant today?
A: The answer is 6.

Q: There are 3 cars in the parking lot. If 2 more cars arrive, how many cars
are in the parking lot?
A: The answer is 5.

Q: Leah’s sister had 42 chocolates and she had 32. If Leah and her sister ate
35 chocolates together, how many pieces do they have left in total?
A: The answer is 39.

Q: Jason had 20 lollipops. He gave Denny some lollipops. If Jason now has 12
lollipops, how many lollipops did he give to Denny?
A: The answer is 8.

Q: Shawn has five toys. If he got two toys each from his mom and dad on
Christmas, how many toys does he have now?
A: The answer is 9.

Q: There were nine computers in the server room. Over four days, five more
computers were installed each day. Now how many computers are in the server
room?
A: The answer is 29.

Q: Michael had 58 golf balls. He lost 23 golf balls on tuesday and lost 2 more
on wednesday. At the end of wednesday, how many golf balls did he have? A: The
answer is 33.

Q: Olivia has $23. If she bought five bagels, which cost $3 each, how much
money does she have left?
A: The answer is 8.

####  A.7.12 GSM8K with CoT Few-Shot Prompting (Reworded)

Q: There are 15 trees in the grove. Grove workers planted trees in the grove
today. After they were done, there were 21 trees. How many trees did the grove
workers plant today?
A: There are 15 trees originally. Then there were 21 trees after some more
were planted. So there must have been 21 - 15 = 6. The answer is 6.

Q: There are 3 cars in the parking lot. If 2 more cars arrive, how many cars
are in the parking lot?
A: There are originally 3 cars. 2 more cars arrive. 3 + 2 = 5. The answer is
5.

Q: Leah’s sister had 42 chocolates and she had 32. If Leah and her sister ate
35 chocolates together, how many pieces do they have left in total?
A: Originally, Leah had 32 chocolates. Her sister had 42. So in total they had
32 + 42 = 74. After eating 35, they had 74 - 35 = 39. The answer is 39.

Q: Jason had 20 lollipops. He gave Denny some lollipops. If Jason now has 12
lollipops, how many lollipops did he give to Denny?
A: Jason started with 20 lollipops. Then he had 12 after giving some to Denny.
So he gave Denny 20 - 12 = 8. The answer is 8.

Q: Shawn has five toys. If he got two toys each from his mom and dad on
Christmas, how many toys does he have now?
A: Shawn started with 5 toys. If he got 2 toys each from his mom and dad, then
that is 4 more toys. 5 + 4 = 9. The answer is 9.

Q: There were nine computers in the server room. Over four days, five more
computers were installed each day. Now how many computers are in the server
room?
A: There were originally 9 computers. For each of 4 days, 5 more computers
were added. So 5 * 4 = 20 computers were added. 9 + 20 is 29. The answer is
29.

Q: Michael had 58 golf balls. He lost 23 golf balls on tuesday and lost 2 more
on wednesday. At the end of wednesday, how many golf balls did he have?
A: Michael started with 58 golf balls. After losing 23 on tuesday, he had 58 -
23 = 35. After losing 2 more, he had 35 - 2 = 33 golf balls. The answer is 33.

Q: Olivia has $23. If she bought five bagels, which cost $3 each, how much
money does she have left?
A: Olivia had 23 dollars. 5 bagels for 3 dollars each will be 5 * 3 = 15
dollars. So she has 23 - 15 dollars left. 23 - 15 is 8. The answer is 8.

###  A.8 Full Dataset Questions

In this section we provide the full datasets used for all experiments. This
includes a selection of 50 questions from each of SST, CoinFlip, CSQA, and
GSM8K, as well as versions of GSM8K and CoinFlip reworded to maintain semantic
meaning but differ syntactically from the original questions.

####  A.8.1 SST (Original)

  1. 1.

Classify the following sentence as either positive or negative: “is the comedy
equivalent of Saddam Hussein, and I’m just about ready to go to the U.N. and
ask permission for a preemptive strike.”

  2. 2.

Classify the following sentence as either positive or negative: “sustained
fest of self-congratulation between actor and director that leaves scant place
for the viewer.”

  3. 3.

Classify the following sentence as either positive or negative: “about as
convincing as any other Arnie musclefest, but has a little too much resonance
with real world events.”

  4. 4.

Classify the following sentence as either positive or negative: “each one of
these people stand out and everybody else is in the background and it just
seems manufactured to me and artificial.”

  5. 5.

Classify the following sentence as either positive or negative: “I’d much
rather watch teens poking their genitals into fruit pies.”

  6. 6.

Classify the following sentence as either positive or negative: “demonstrates
that Werner Herzog can still leave us with a sense of wonder at the diverse,
marvelously twisted shapes history has taken.”

  7. 7.

Classify the following sentence as either positive or negative: “allows his
cast the benefit of being able to give full performances … while demonstrating
vividly that the beauty and power of the opera reside primarily in the music
itself.”

  8. 8.

Classify the following sentence as either positive or negative: “together the
movies establish Mr. Cantet as France’s foremost cinematic poet of the
workplace.”

  9. 9.

Classify the following sentence as either positive or negative: “good as the
full Monty, but a really strong second effort.”

  10. 10.

Classify the following sentence as either positive or negative: “though
Nijinsky’s words grow increasingly disturbed, the film maintains a beguiling
serenity and poise that make it accessible for a non-narrative feature.”

  11. 11.

Classify the following sentence as either positive or negative: “I must report
that the children of varying ages in my audience never coughed, fidgeted or
romped up and down the aisles for bathroom breaks.”

  12. 12.

Classify the following sentence as either positive or negative: “though there
are many tense scenes in Trapped, they prove more distressing than
suspenseful.”

  13. 13.

Classify the following sentence as either positive or negative: “of empty,
fetishistic violence in which murder is casual and fun.”

  14. 14.

Classify the following sentence as either positive or negative: “this is a
movie full of grace and, ultimately, hope.”

  15. 15.

Classify the following sentence as either positive or negative: “a pleasant
and engaging enough sit, but in trying to have the best of both worlds it ends
up falling short as a whole.”

  16. 16.

Classify the following sentence as either positive or negative: “Macdowell …
gives a solid, anguished performance that eclipses nearly everything else
she’s ever done.”

  17. 17.

Classify the following sentence as either positive or negative: “the unique
niche of self-critical, behind-the-scenes navel-gazing Kaufman has carved from
Orleans’ story.”

  18. 18.

Classify the following sentence as either positive or negative: “as gamely as
the movie tries to make sense of its title character, there remains a huge gap
between the film’s creepy, clean-cut Dahmer (Jeremy Renner) and fiendish acts
that no amount of earnest textbook psychologizing can bridge.”

  19. 19.

Classify the following sentence as either positive or negative: “made nature
film and a tribute to a woman whose passion for this region and its
inhabitants still shines in her quiet blue eyes.”

  20. 20.

Classify the following sentence as either positive or negative: “Swimming gets
the details right, from its promenade of barely clad bodies in Myrtle Beach,
S.C., to the adrenaline jolt of a sudden lunch rush at the diner.”

  21. 21.

Classify the following sentence as either positive or negative: “Changing
Lanes is an anomaly for a Hollywood movie; it’s a well-written and
occasionally challenging social drama that actually has something interesting
to say.”

  22. 22.

Classify the following sentence as either positive or negative: “know it
wouldn’t exist without the precedent of Yiddish theater, whose jolly, fun-for-
fun’s-sake communal spirit goes to the essence of Broadway.”

  23. 23.

Classify the following sentence as either positive or negative: “a compelling
pre-WWII drama with vivid characters and a warm, moving message.”

  24. 24.

Classify the following sentence as either positive or negative: “like
Schindler’s List, the Grey Zone attempts to be grandiloquent, but ends up
merely pretentious – in a grisly sort of way.”

  25. 25.

Classify the following sentence as either positive or negative: “plays out
with a dogged and eventually winning squareness that would make it the darling
of many a kids-and-family-oriented cable channel.”

  26. 26.

Classify the following sentence as either positive or negative: “gritty
realism, crisp storytelling and radiant compassion that effortlessly draws you
in.”

  27. 27.

Classify the following sentence as either positive or negative: “as visually
bland as a dentist’s waiting room.”

  28. 28.

Classify the following sentence as either positive or negative: “what it was
that made the story relevant in the first place.”

  29. 29.

Classify the following sentence as either positive or negative: “the nonstop
artifice ultimately proves tiresome, with the surface histrionics failing to
compensate for the paper-thin characterizations and facile situations.”

  30. 30.

Classify the following sentence as either positive or negative: “like Smoke
Signals, the film is also imbued with strong themes of familial ties and
spirituality that are powerful and moving without stooping to base melodrama.”

  31. 31.

Classify the following sentence as either positive or negative: “to imagine
another director ever making his wife look so bad in a major movie.”

  32. 32.

Classify the following sentence as either positive or negative: “his routine
is offensive, puerile and unimaginatively foul-mouthed if it was at least
funny.”

  33. 33.

Classify the following sentence as either positive or negative: “Ryan Gosling
… is at 22 a powerful young actor.”

  34. 34.

Classify the following sentence as either positive or negative: “lacks the
novel charm that made Spy Kids a surprising winner with both adults and
younger audiences.”

  35. 35.

Classify the following sentence as either positive or negative: “Return to
Neverland manages to straddle the line between another classic for the
company.”

  36. 36.

Classify the following sentence as either positive or negative: “faced with
the possibility that her life is meaningless, vapid and devoid of substance,
in a movie that is definitely meaningless, vapid and devoid of substance.”

  37. 37.

Classify the following sentence as either positive or negative: “Time Out
offers an exploration that is more accurate than anything I have seen in an
American film.”

  38. 38.

Classify the following sentence as either positive or negative: “painting an
unabashedly romantic picture of a nation whose songs spring directly from the
lives of the people.”

  39. 39.

Classify the following sentence as either positive or negative: “We’re
convinced that This Mean Machine was a decent TV outing that just doesn’t have
big screen magic.”

  40. 40.

Classify the following sentence as either positive or negative: “Enigma offers
all the pleasure of a handsome and well-made entertainment.”

  41. 41.

Classify the following sentence as either positive or negative: “we hate
Madonna within the film’s first five minutes, and she lacks the skill or
presence to regain any ground.”

  42. 42.

Classify the following sentence as either positive or negative: “Like many
such biographical melodramas, it suffers from the awkwardness that results
from adhering to the messiness of true stories.”

  43. 43.

Classify the following sentence as either positive or negative: “inhabit that
special annex of hell where adults behave like kids.”

  44. 44.

Classify the following sentence as either positive or negative: “makes an
unusual but pleasantly haunting debut behind the camera.”

  45. 45.

Classify the following sentence as either positive or negative: “been there,
done that … a thousand times already, and better.”

  46. 46.

Classify the following sentence as either positive or negative: “Mike White’s
deft combination of serious subject matter and dark, funny humor.”

  47. 47.

Classify the following sentence as either positive or negative: “plenty of
warmth to go around, with music and laughter and the love of family.”

  48. 48.

Classify the following sentence as either positive or negative: “We would pay
a considerable ransom not to be looking at this.”

  49. 49.

Classify the following sentence as either positive or negative: “ends up
merely pretentious – in a grisly sort of way .”

  50. 50.

Classify the following sentence as either positive or negative: “The film
would have been more enjoyable had the balance shifted in favor of water-bound
action over the land-based drama.”

####  A.8.2 CoinFlip (Original)

  1. 1.

A coin is heads up. Matthew does not flip the coin. Lynn does not flip the
coin. Is the coin still heads up?

  2. 2.

A coin is heads up. Delphine flips the coin. Wesley does not flip the coin. Is
the coin still heads up?

  3. 3.

A coin is tails up. Grover flips the coin. Clarissa flips the coin. Is the
coin still tails up?

  4. 4.

A coin is heads up. Isaias flips the coin. Sharonda does not flip the coin. Is
the coin still heads up?

  5. 5.

A coin is tails up. Obie flips the coin. Sonja does not flip the coin. Is the
coin still tails up?

  6. 6.

A coin is heads up. Mckenzie does not flip the coin. Katherine flips the coin.
Is the coin still heads up?

  7. 7.

A coin is tails up. Racheal does not flip the coin. Deion flips the coin. Is
the coin still tails up?

  8. 8.

A coin is tails up. Lavon does not flip the coin. Loretto does not flip the
coin. Is the coin still tails up?

  9. 9.

A coin is heads up. Maddison flips the coin. Waymon does not flip the coin. Is
the coin still heads up?

  10. 10.

A coin is tails up. Mittie does not flip the coin. Jamie flips the coin. Is
the coin still tails up?

  11. 11.

A coin is heads up. Terell does not flip the coin. Christy does not flip the
coin. Is the coin still heads up?

  12. 12.

A coin is heads up. Sara does not flip the coin. Tony flips the coin. Is the
coin still heads up?

  13. 13.

A coin is tails up. Winifred flips the coin. Trey flips the coin. Is the coin
still tails up?

  14. 14.

A coin is heads up. Aurelio flips the coin. Tania flips the coin. Is the coin
still heads up?

  15. 15.

A coin is heads up. Alexandria does not flip the coin. Eddie does not flip the
coin. Is the coin still heads up?

  16. 16.

A coin is heads up. Arrie does not flip the coin. Glynn flips the coin. Is the
coin still heads up?

  17. 17.

A coin is tails up. Lourdes flips the coin. Joycelyn does not flip the coin.
Is the coin still tails up?

  18. 18.

A coin is heads up. Spencer flips the coin. Reuben flips the coin. Is the coin
still heads up?

  19. 19.

A coin is heads up. Pansy flips the coin. Dolores flips the coin. Is the coin
still heads up?

  20. 20.

A coin is tails up. Lillie does not flip the coin. Aida does not flip the
coin. Is the coin still tails up?

  21. 21.

A coin is tails up. Andrea flips the coin. Matt flips the coin. Is the coin
still tails up?

  22. 22.

A coin is tails up. Anthony does not flip the coin. Winnie does not flip the
coin. Is the coin still tails up?

  23. 23.

A coin is tails up. Barrett does not flip the coin. Adrienne does not flip the
coin. Is the coin still tails up?

  24. 24.

A coin is heads up. Eleanor flips the coin. Harley does not flip the coin. Is
the coin still heads up?

  25. 25.

A coin is tails up. Rodney does not flip the coin. Berry flips the coin. Is
the coin still tails up?

  26. 26.

A coin is heads up. Rupert flips the coin. Yoel does not flip the coin. Is the
coin still heads up?

  27. 27.

A coin is heads up. Earle flips the coin. Cal does not flip the coin. Is the
coin still heads up?

  28. 28.

A coin is heads up. Cecil does not flip the coin. Gaylord flips the coin. Is
the coin still heads up?

  29. 29.

A coin is tails up. Neil does not flip the coin. Ramiro does not flip the
coin. Is the coin still tails up?

  30. 30.

A coin is tails up. Otto flips the coin. Ismael flips the coin. Is the coin
still tails up?

  31. 31.

A coin is tails up. Elmira flips the coin. Retha does not flip the coin. Is
the coin still tails up?

  32. 32.

A coin is tails up. Donie flips the coin. Antoine flips the coin. Is the coin
still tails up?

  33. 33.

A coin is tails up. Flora does not flip the coin. Ella flips the coin. Is the
coin still tails up?

  34. 34.

A coin is tails up. Giada flips the coin. Aloysius does not flip the coin. Is
the coin still tails up?

  35. 35.

A coin is heads up. Chloe does not flip the coin. Kane flips the coin. Is the
coin still heads up?

  36. 36.

A coin is heads up. Carey does not flip the coin. Armando flips the coin. Is
the coin still heads up?

  37. 37.

A coin is heads up. Korey flips the coin. Myra does not flip the coin. Is the
coin still heads up?

  38. 38.

A coin is tails up. Laney does not flip the coin. Gerard does not flip the
coin. Is the coin still tails up?

  39. 39.

A coin is tails up. Mekhi does not flip the coin. Mariah does not flip the
coin. Is the coin still tails up?

  40. 40.

A coin is tails up. Eloise flips the coin. Myrtice flips the coin. Is the coin
still tails up?

  41. 41.

A coin is heads up. Marvin flips the coin. Edd flips the coin. Is the coin
still heads up?

  42. 42.

A coin is heads up. Dalton flips the coin. Petra does not flip the coin. Is
the coin still heads up?

  43. 43.

A coin is heads up. Joesph flips the coin. Baby does not flip the coin. Is the
coin still heads up?

  44. 44.

A coin is heads up. Dayton flips the coin. Ray does not flip the coin. Is the
coin still heads up?

  45. 45.

A coin is tails up. Katheryn does not flip the coin. Carmella does not flip
the coin. Is the coin still tails up?

  46. 46.

A coin is heads up. Kali flips the coin. Nat flips the coin. Is the coin still
heads up?

  47. 47.

A coin is tails up. Dawn flips the coin. Danna flips the coin. Is the coin
still tails up?

  48. 48.

A coin is heads up. Alyson flips the coin. Magdalena flips the coin. Is the
coin still heads up?

  49. 49.

A coin is tails up. Thomas does not flip the coin. Fanny does not flip the
coin. Is the coin still tails up?

  50. 50.

A coin is tails up. Diane does not flip the coin. Tabitha flips the coin. Is
the coin still tails up?

####  A.8.3 CoinFlip (Reworded)

  1. 1.

A coin starts heads up. Matthew does not flip the coin. Lynn also does not
flip the coin. Is the coin still heads up?

  2. 2.

A coin is heads up. Delphine begins by flipping the coin. Wesley chooses not
to flip the coin. Is the coin still heads up?

  3. 3.

A coin is tails up. Grover decides to flip the coin. Clarissa also decides to
flip the coin. Is the coin still tails up?

  4. 4.

A coin is lying heads up. Isaias flips the coin. Sharonda chooses not to flip
the coin. Is the coin still heads up?

  5. 5.

A coin is tails up. Obie decides to flip the coin. Sonja does not flip the
coin. Is the coin still tails up?

  6. 6.

A coin is starting heads up. Mckenzie does not flip the coin. Katherine,
however, does flip the coin. Is the coin still heads up?

  7. 7.

A coin is tails up. Racheal did not flip the coin. Deion does flip the coin.
Is the coin still tails up?

  8. 8.

A coin is starting tails up. Lavon does not flip the coin, and Loretto also
does not flip the coin. Is the coin still tails up?

  9. 9.

A coin is heads up. Maddison begins by flipping the coin, but Waymon does not
flip the coin. Is the coin still heads up?

  10. 10.

A coin is tails up. Mittie does not flip the coin. Jamie does flip the coin.
Is the coin still tails up?

  11. 11.

A coin is heads up. Terell does not flip the coin. And, Christy also does not
flip the coin. Is the coin still heads up?

  12. 12.

A coin is heads up. Sara doesn’t flip the coin, but Tony does flip the coin.
Is the coin still heads up?

  13. 13.

A coin begins tails up. Winifred flips the coin. Trey also flips the coin. Is
the coin still tails up?

  14. 14.

A coin is heads up. Aurelio decides to flip the coin, and so does Tania. Is
the coin still heads up?

  15. 15.

A coin is heads up. Alexandra and Eddie both do not flip the coin. Is the coin
still heads up?

  16. 16.

A coin starts heads up. Arrie doesn’t flip the coin, but Glynn does. Is the
coin still heads up?

  17. 17.

A coin is tails up. Lourdes chooses to flip the coin, but Joycelyn does not
flip the coin. Is the coin still tails up?

  18. 18.

A coin is heads up. Both Spencer and Reuben flip the coin. Is the coin still
heads up?

  19. 19.

A coin is heads up. Pansy flips the coin, and then Dolores flips the coin. Is
the coin still heads up?

  20. 20.

A coin is tails up. Lillie and Aida both decide not to flip the coin. Is the
coin still tails up?

  21. 21.

A coin is tails up. Both Andrea and Matt flip the coin. Is the coin still
tails up?

  22. 22.

A coin is tails up. Both Anthony and Winnie choose not to flip the coin. Is
the coin still tails up?

  23. 23.

A coin is tails up, and Barrett and Adrienne both do not flip the coin. Is the
coin still tails up?

  24. 24.

A coin is heads up. Eleanor decides to flip the coin, but Harley does not. Is
the coin still heads up?

  25. 25.

A coin is tails up, and Rodney does not flip it. But, Berry does flip the
coin. Is the coin still tails up?

  26. 26.

A coin is heads up. Rupert flips it. Yoel does not flip it. Is the coin still
heads up?

  27. 27.

A coin is heads up. Earle flips the coin, but Cal chooses not to flip the
coin. Is the coin still heads up?

  28. 28.

A coin is heads up. Cecil does not flip the coin. But, Gaylord does flip the
coin. Is the coin still heads up?

  29. 29.

A coin is tails up. Neil chooses not to flip the coin. Ramiro does not flip
the coin. Is the coin still tails up?

  30. 30.

A coin is tails up. Both Otto and Ismael flip the coin. Is the coin still
tails up?

  31. 31.

A coin starts tails up. Elmira flips the coin. But, Retha does not flip the
coin. Is the coin still tails up?

  32. 32.

A coin is tails up. Donie flips the coin. After, Antoine also flips the coin.
Is the coin still tails up?

  33. 33.

A coin is tails up. Flora does not flip the coin, but Ella does flip it. Is
the coin still tails up?

  34. 34.

A coin is tails up. Giada flips the coin, but Aloysius does not. Is the coin
still tails up?

  35. 35.

A coin starts heads up. Chloe doesn’t flip the coin, but Kane does. Is the
coin still heads up?

  36. 36.

A coin is lying heads up. Carey chooses not to flip the coin. Armando flips
the coin. Is the coin still heads up?

  37. 37.

A coin is heads up. Korey flips the coin, but Myra chooses not to flip the
coin. Is the coin still heads up?

  38. 38.

A coin is tails up. Laney does not flip the coin. Neither does Gerard. Is the
coin still tails up?

  39. 39.

A coin is tails up. Both Mekhi and Mariah choose not to flip the coin. Is the
coin still tails up?

  40. 40.

A coin is tails up. Eloise flips the coin, and then Myrtice flips the coin. Is
the coin still tails up?

  41. 41.

A coin is heads up. Afterwards, both Marvin and Edd flip the coin. Is the coin
still heads up?

  42. 42.

A coin is heads up. Seeing this, Dalton flips the coin. But, Petra does not
flip the coin. Is the coin still heads up?

  43. 43.

A coin is heads up. Joesph starts by flipping the coin. Baby does not flip the
coin. Is the coin still heads up?

  44. 44.

A coin is heads up. Dayton flips the coin, but Ray does not. Is the coin still
heads up?

  45. 45.

A coin is tails up. Katheryn declines to flip the coin. Carmella also does not
flip the coin. Is the coin still tails up?

  46. 46.

A coin is heads up. Kali and Nat both flip the coin. Is the coin still heads
up?

  47. 47.

A coin is tails up. Dawn flips the coin, and afterwards, Danna flips the coin.
Is the coin still tails up?

  48. 48.

A coin is heads up. Alyson chooses to flip the coin. Magdalena chooses to flip
the coin. Is the coin still heads up?

  49. 49.

A coin is tails up. Thomas decides not to flip the coin. Fanny also does not
flip the coin. Is the coin still tails up?

  50. 50.

A coin is tails up. Diane doesn’t flip the coin, but Tabitha does. Is the coin
still tails up?

####  A.8.4 CSQA (Original)

  1. 1.

What would children do if they are bored?

    1. A. walk, B. change shoes, C. play tag, D. smell smoke, E. reach over.

  2. 2.

Behind what small door are canned goods usually kept in?

    1. A. cabinet, B. pantry, C. kitchen, D. cupboard, E. shelf.

  3. 3.

While laid up in the hospital she tried remembering good times, like that
massage on vacation that brought great what?

    1. A. dream, B. knowing, C. depression, D. pleasure, E. nostalgia.

  4. 4.

If I want to maximize my number of potential customers, where should I build a
new department store?

    1. A. idaho, B. shopping mall, C. big city, D. in town, E. new york.

  5. 5.

Where is a good place to get a ball?

    1. A. kitchen, B. child’s hand, C. street, D. toy store, E. soccer game.

  6. 6.

What common chemical reaction is often used as a heat source?

    1. A. coal or wood, B. fire, C. lava, D. house, E. solar energy.

  7. 7.

Where could you put a dollar that would make it hard to get out without
breaking?

    1. A. cash drawer, B. piggy bank, C. purse, D. box, E. pocket.

  8. 8.

I’m at a restaurant ans want a glass of milk, how do I find out if they serve
that?

    1. A. accompaniment to meal, B. waitress, C. cafeteria, D. home, E. menu.

  9. 9.

What could contain no more than around 100 people?

    1. A. apartment, B. classroom, C. buildings, D. car, E. audience.

  10. 10.

John joined the army because he wanted to see the world. When the fighting
started, he only saw what sort of things?

    1. A. fort, B. orphans, C. battlefield, D. other countries, E. war.

  11. 11.

where do children play a game?

    1. A. toy store, B. bar, C. casino, D. table, E. family room.

  12. 12.

What could happen after driving car that does not involve pain?

    1. A. automobile accidents, B. backache, C. eye strain, D. getting tired, E. car crash.

  13. 13.

In what place could you find air that has been breathed by only a few people
recently?

    1. A. train station, B. house, C. cupboard, D. cricket stadium, E. surface of earth.

  14. 14.

what happens to a company when it is bankrupt?

    1. A. liquidated, B. own resources, C. branch out, D. commit crime, E. game over.

  15. 15.

Blood isn’t usually spilled during what sport that doesn’t allow the use of
hands?

    1. A. soccer game, B. capillaries, C. ball, D. animals, E. person.

  16. 16.

The dog ran to the front window and barked, this is because of a presence at
the what?

    1. A. building, B. friend’s house, C. classroom, D. window, E. front door.

  17. 17.

Where would you find a sign with some people lined up next to it?

    1. A. bus stop, B. street corner, C. roadblock, D. fast food drive thru, E. city.

  18. 18.

Sometimes it felt like being able to pay bills was the only reason he was
actively what?

    1. A. pay debts, B. make money, C. having money, D. making money, E. keep house.

  19. 19.

John just made first violin. This is is a position on what?

    1. A. violin case, B. symphony orchestra, C. class, D. learning to play violin, E. music store.

  20. 20.

Where can someone keep a digital file?

    1. A. suitcase, B. filing cabinet, C. computer, D. desk drawer, E. cloud.

  21. 21.

What appliance uses water and soap?

    1. A. washing machine, B. space shuttle, C. blender, D. disneyland, E. surface of earth.

  22. 22.

I am looking to buy honey right from an apiary, where should I go?

    1. A. jar, B. stick to fingers, C. beehive, D. stick to fingers, E. farmer’s market.

  23. 23.

Where would someone likely bring an attache case?

    1. A. overhead compartment, B. chair, C. hospital, D. in shirt, E. business meeting.

  24. 24.

i like to examine different things,why do books the chapters on the back

    1. A. examine thing, B. go somewhere, C. knit, D. read book, E. see particular program.

  25. 25.

The whole family was playing cards, they were full of joy and what?

    1. A. sadness, B. happiness, C. amusement, D. losing money, E. anger.

  26. 26.

Jimmy wanted a beer but his wife was an alcoholic on probation. Where might he
go?

    1. A. neighbor’s house, B. bridge, C. home, D. homeless shelter, E. prison.

  27. 27.

What country has with the capital of Amsterdam has a Fortis bank?

    1. A. netherlands, B. utrecht, C. brussels, D. belgium, E. denmark.

  28. 28.

What building has a ot of glue bottles?

    1. A. art room, B. police station, C. classroom, D. desk drawer, E. school.

  29. 29.

The king was to meet an ambassador, where would the meet?

    1. A. court, B. dinning room, C. throne room, D. monarchy, E. castle.

  30. 30.

Where do kids play with a ball that is concrete?

    1. A. street, B. soccer field, C. soccer game, D. central park, E. pool table.

  31. 31.

Diving off a cliff for a cartoon character often ends in comedic what?

    1. A. exhilaration, B. dismemberment, C. spinal injuries, D. getting wet, E. splats.

  32. 32.

Jimmy didn’t like going to the party. It seemed like the guys there did
nothing but what?

    1. A. dance, B. getting drunk, C. meet new people, D. having fun, E. have fun.

  33. 33.

Where does a person need to get when they have more kids?

    1. A. compliments, B. understand themselves, C. life partner, D. second chances, E. larger house.

  34. 34.

Where would a special ficus be found?

    1. A. botanical garden, B. doctor’s office, C. in a sea, D. public place, E. arboretum.

  35. 35.

James is apply for a job that he really wants. He knows that it will be a good
fit and he has experience, but he doesn’t have all the qualifications that
they’re asking for, so he’s worried about what?

    1. A. acceptance, B. rejection, C. hope, D. less sleep, E. employment.

  36. 36.

What is a sign that you might not be living soon?

    1. A. growing older, B. reproducing, C. shuddering, D. perspiration, E. palpitations.

  37. 37.

Riding bike through rough terrain can be dangerous, one could find themselves
what?

    1. A. pleasure, B. accident, C. get somewhere, D. hurt, E. falling down.

  38. 38.

What would a document be if it is not obtuse?

    1. A. pointed, B. bright, C. easy, D. clear, E. acute angled.

  39. 39.

For convenience some customers want things straight to the doorstep and use
what?

    1. A. mail orders, B. demand satisfaction, C. place money on counter, D. pay merchants with money, E. spend money.

  40. 40.

What is a great, but non-specific, overall benefit of exercise?

    1. A. good health, B. heart attacks, C. relaxation, D. rest, E. fatigue.

  41. 41.

Where does a spoon go on one side of a plate?

    1. A. bowl of soup, B. serving dish, C. table setting, D. placemat, E. glass of iced tea.

  42. 42.

Where is a note likely to indicate a melody?

    1. A. fridge, B. a classroom, C. envelope, D. medical chart, E. sheet music.

  43. 43.

what can years of playing tennis lead to?

    1. A. tennis elbow, B. proffesional player, C. exercise, D. victory, E. becoming tired.

  44. 44.

Where would you be going if using a parking lot near a roller coaster?

    1. A. farm, B. business district, C. car, D. amusement park, E. urban area.

  45. 45.

What does a virus primarily do to a person?

    1. A. infect, B. reproduce, C. mutate, D. eat food, E. make computer malfunction.

  46. 46.

Where would you go if you didn’t feel like baking a cake yourself?

    1. A. theater, B. refrigerator, C. bakery, D. grocery store, E. oven.

  47. 47.

Where are floors kept clean for holy purposes?

    1. A. restaurant, B. basement, C. all buildings, D. carpet, E. synagogue.

  48. 48.

Where will a bullet projectile be found during a war?

    1. A. battle field, B. museum, C. weapon, D. canon, E. tornado.

  49. 49.

What is a place that might not always have a queen?

    1. A. bee hive, B. england, C. castle, D. deck of cards, E. chess game.

  50. 50.

What do you need to be to learn hard concepts?

    1. A. copy, B. intelligent, C. improved performance, D. empowering, E. have more knowledge.

####  A.8.5 GSM8K (Original)

  1. 1.

Marilyn’s first record sold 10 times as many copies as Harald’s. If they sold
88,000 copies combined, how many copies did Harald sell?

  2. 2.

A herd consists of camels and dromedaries. There are 180 heads and 304 bumps.
How many dromedaries are there if camels have two humps each and dromedaries
have one hump each?

  3. 3.

Harold sleeps for 10 hours a night. He works 2 hours less than he sleeps and
he walks his dog for an hour each day. How much free time does Harold have
left in his day?

  4. 4.

Cho hiked 14 kilometers per hour for 8 hours. Chloe hiked 9 kilometers per
hour and stopped after 5 hours. How many kilometers farther did Cho hike?

  5. 5.

The rug is 5 feet wider than the chair. The couch is 2 feet longer than twice
the width of the rug. If the chair is 3 feet wide. How many feet long is the
couch?

  6. 6.

A family of parents and a child go to the cinema. The cost of an adult ticket
is $12 and a child ticket is $8. Then they buy 2 popcorns for $3 each. How
many dollars do they pay in total?

  7. 7.

A hand-painted wallpaper costs $400 at the market. A DIY will saves 20% after
considering the materials cost. If Ethan made his own hand-painted wallpaper,
how much was the total cost?

  8. 8.

Ben bought a car for $20000 in 2007. The price of the car depreciates at a
constant rate of 21% per year. Find the price of the car in the year 2010.

  9. 9.

In 10 years, Melanie will be 18 years old. In how many years will her age be
thrice her present age?

  10. 10.

The number of songs in a playlist is 300. If John has 20 such playlists, and
each song is 10 hours long, how many hours will the 20 playlists last in
total?

  11. 11.

Bailey starts with a certain amount of money. Then she receives a weekly
allowance of $5 for 8 weeks. At the end of the 8 weeks, if she has a total of
$100, how much money did Bailey start with?

  12. 12.

Last Friday, 13 of the 82 teachers at Rydell Elementary School were sick.
There were 9 substitute teachers called in to help. How many teachers were at
school that day?

  13. 13.

John is a carpenter. For his friend Ali, he manufactured 4 wooden tables for
$20 each and 2 roof frames for $10 each. How much does Ali have to pay John?

  14. 14.

In a company of 50 employees, 20% of the employees are management. Out of this
20%, only 30% oversee the entire company. How many employees oversee the
company?

  15. 15.

James buys a ring for his bride-to-be. The diamond cost $600 and the gold cost
$300. He pays a 30% premium for it to be made. How much did he pay?

  16. 16.

Richard, Jerry, and Robert are going to share 60 cherries. If Robert has 30
cherries, and has 10 more than Richard, how many more cherries does Robert
have than Jerry?

  17. 17.

Farmer Brown’s farm is 200 acres, and Farmer Smith’s farm is 100 acres more
than twice that. How many acres do the two farms have, together?

  18. 18.

The vending machines sell chips for 40 cents and candy bars for 75 cents.
George spent $5 and got 3 bags of chips and had 1% of his money left. How many
candy bars did he buy?

  19. 19.

Ginger and Amy both start with 100 pieces of Halloween candy. Ginger eats 4
pieces a day and Amy eats 3 pieces a day. How much more candy does Amy have
after two weeks?

  20. 20.

Two cups of flour are needed to make a dozen cookies. Carla is making 36
cookies today and 30 cookies tomorrow. How many cups of flour will Carla need
to bake the cookies today and tomorrow?

  21. 21.

Theo bought a pen for $2 and a piece of paper that cost $1 less than three
times the price of the pen. She gave the cashier $10. What is the amount of
change Theo received?

  22. 22.

Tom plants 10 trees a year. Every year he also chops down 2 trees a year. He
starts with 50 trees. After 10 years 30% of the trees die. How many trees does
he have left?

  23. 23.

James has 30 teeth. His dentist drills 4 of them and caps 7 more teeth than he
drills. What percentage of James’ teeth does the dentist fix?

  24. 24.

Mico and Marco wanted to get to know each other. They realized that the sum of
their ages is 20. What will be the sum of their ages in 10 years?

  25. 25.

Joseph had 3 times as many notebooks as Martha. Martha decided she needed more
notebooks and then bought 5 more for a total of 7 notebooks. How many more
than Joseph does she now have?

  26. 26.

Rani has ten more crabs than Monic, who has 4 fewer crabs than Bo. If Bo has
40 crabs, calculate the total number of crabs the three have together.

  27. 27.

Jen decides to travel to 3 different countries. He has to pay $400 for the
supplies he needs, in total. The tickets for travel cost, in total, 50% more
than the supplies. How much does travel cost?

  28. 28.

Tommy is fundraising for his charity by selling brownies for $3 a slice and
cheesecakes for $4 a slice. If Tommy sells 43 brownies and 23 slices of
cheesecake, how much money does Tommy raise?

  29. 29.

Tim grows 5 trees. Each year he collects 6 lemons from each tree. How many
lemons does he get in a decade?

  30. 30.

A bus travels 60 miles per hour for 5 hours. A car travels 30 miles per hour
for 8 hours. How much farther did the bus go than the car, in miles?

  31. 31.

Henry and 3 of his friends order 7 pizzas for lunch. Each pizza is cut into 8
slices. If Henry and his friends want to share the pizzas equally, how many
slices can each of them have?

  32. 32.

Poppy is solving a 1000-piece jigsaw puzzle. She places a quarter of the
pieces on the board, then her mom places a third of the remaining pieces. How
many jigsaw pieces are left to be placed?

  33. 33.

A farm has 10 2-legged animals and 15 4-legged animals. If all animals are
healthy, how many pairs of animal legs in total are on the farm?

  34. 34.

To make a call from a payphone, you must pay $0.25 for each minute of your
call. After 16 minutes, that price drops to $0.2 per minute. How much would a
36-minute call cost?

  35. 35.

Farmer Brown has 60 animals on his farm, all either chickens or cows. He has
twice as many chickens as cows. How many legs do the animals have, all
together?

  36. 36.

John fills a 6 foot by 4 foot pool that is 5 feet deep. It cost $.1 per cubic
foot to fill. How much does it cost to fill?

  37. 37.

Jean is two years older than Mark. Two years ago Mark was 5 years older than
half Jan’s age. If Jan is 30 how old is Jean?

  38. 38.

Vince can staple 30 reports every 15 minutes. If he was stapling reports from
8:00 AM until 11:00 PM, how many reports did he staple altogether?

  39. 39.

In November, a toy was $40. In December, the price increased by 80%. In
January, the price decreased by 50%. What was the price of the toy after it
was discounted in January?

  40. 40.

Cody eats three times as many cookies as Amir eats. If Amir eats 5 cookies,
how many cookies do both of them eat together?

  41. 41.

Susan earns $5 every 10 minutes for an online task she does. If she works
between 8 a.m. and 11 a.m. and pauses in between for half an hour, how much
money does she earn for the online task?

  42. 42.

Carrie was given ten twenties and 140 quarters by her aunt to use for lunch.
If she spent all the quarters and 3/5 of the twenties, calculate the total
amount of money she paid for the lunch.

  43. 43.

A football team played 22 games. They won 8 more than they lost. How many did
they win?

  44. 44.

The combined age of Peter, Paul and Jean is 100 years old. Find the age of
Peter knowing that Paul is 10 years older than John and that Peter’s age is
equal to the sum of Paul and John’s age.

  45. 45.

Mia and Emma are currently 16 years apart in age. If Mia, who is younger than
Emma, is 40 years old, what’s the average of their ages?

  46. 46.

Jan has three times the number of pets as Marcia. Marcia has two more pets
than Cindy. If Cindy has four pets, how many total pets do the three have?

  47. 47.

There is one set of twins and one set of triplets. One twin is 7 years older
than one triplet. If their combined ages are 44, how old is one of the twins?

  48. 48.

Four friends ordered four pizzas for a total of 64 dollars. If two of the
pizzas cost 30 dollars, how much did each of the other two pizzas cost if they
cost the same amount?

  49. 49.

Jerry is twice as old as he was 5 years ago. How old will Jerry be in 3 years?

  50. 50.

A bag has a 5% discount. If it is marked $140, how much will you pay after the
discount?

####  A.8.6 GSM8K (Reworded)

  1. 1.

Marilyn’s first record sold 10 times as many copies as Harald’s. They sold
88,000 copies combined. How many copies did Harald sell?

  2. 2.

A herd consists of camels and dromedaries. Camels have two humps each and
dromedaries have one hump each. There are 180 heads and 304 humps. How many
dromedaries are there?

  3. 3.

Harold works 2 hours less than he sleeps and he walks his dog for an hour each
day. He sleeps for 10 hours a night. How much free time does Harold have left
in his day?

  4. 4.

Cho hiked for 8 hours at 14 kilometers per hour. Chloe hiked 9 kilometers per
hour and stopped after 5 hours. How many kilometers farther did Cho hike?

  5. 5.

The couch is 2 feet longer than twice the width of the rug. The rug is 5 feet
wider than the chair. The chair is 3 feet wide. How long, in feet, is the
couch?

  6. 6.

A family of parents and a child go to the cinema. The cost of a child ticket
is $8 and an adult ticket is $12. Then they buy 2 popcorns, which cost $3
each. How many dollars do they pay in total?

  7. 7.

At the market, a hand-painted wallpaper costs $400. After considering the
materials cost, a DIY will save 20%. If Ethan made his own hand-painted
wallpaper, how much was the total cost?

  8. 8.

In 2007, Ben bought a car for $20000. The price of a car depreciates at a
constant rate of 21% per year. What is the price of the car in the year 2010?

  9. 9.

Melanie will be 18 years old in 10 years. In how many years will her age be
thrice her present age?

  10. 10.

There are 300 songs in a playlist. John has 20 such playlists, with each song
being 10 hours long. How many hours will the 20 playlists last in total?

  11. 11.

Bailey starts with a certain amount of money. Then, for 8 weeks, she receives
a weekly allowance of $5. At the end of the 8 weeks, she has a total of $100.
How much money did Bailey start with?

  12. 12.

At Rydell Elementary School, 13 of the 82 teachers were sick last Friday.
There were 9 substitute teachers called in to help. How many teachers were at
school that day?

  13. 13.

John is a carpenter. For his friend Ali, he manufactured 2 roof frames for $10
each and 4 wooden tables for $20 each. How much does Ali have to pay John?

  14. 14.

20% of the employees are management in a company of 50 employees. Out of this
20%, only 30% oversee the entire company. How many employees oversee the
company?

  15. 15.

James buys a ring for his bride-to-be. The gold cost $300 and the diamond cost
$600. He pays a 30% premium for it to be made. How much did he pay?

  16. 16.

Robert, Jerry, and Richard are going to share 60 cherries. If Robert has 30
cherries, and has 10 more than Richard, how many more cherries does Robert
have than Jerry?

  17. 17.

Farmer Brown’s farm is 200 acres. Farmer Smith’s farm is 100 acres more than
twice that. Together, how many acres do the two farms have?

  18. 18.

The vending machines sell candy bars for 75 cents and chips for 40 cents.
George spent $5 and got 3 bags of chips and had 1% of his money left. How many
candy bars did he buy?

  19. 19.

Ginger and Amy both start with 100 pieces of Halloween candy. Amy eats 3
pieces a day. Ginger eats 4 pieces a day. How much more candy does Amy have
after two weeks?

  20. 20.

A dozen cookies requires two cups of flour to make. Carla is making 30 cookies
tomorrow and 36 cookies today. How many cups of flour will Carla need to bake
the cookies today and tomorrow?

  21. 21.

A pen costs $2. A piece of paper costs $1 less than three times the price of
the pen. Theo bought a pen and a piece of paper. She gave the cashier $10.
What is the amount of change Theo received?

  22. 22.

Tom plants 10 trees a year and chops down 2 trees a year. He starts with 50
trees. After 10 years 30% of the trees die. How many trees does he have left?

  23. 23.

James has 30 teeth. His dentist drills 4 of them. His dentist also caps 7 more
teeth than he drills. What percentage of James’ teeth does the dentist fix?

  24. 24.

Mico and Marco are good friends. The sum of their ages is 20. What will be the
sum of their ages in 10 years?

  25. 25.

Joseph had 3 times as many notebooks as Martha. Martha decided she needed more
notebooks and then bought 5 more. Now, Martha has a total of 7 notebooks. How
many more notebooks than Joseph does she now have?

  26. 26.

Rani has ten more crabs than Monic. Monic has 4 fewer crabs than Bo. Bo has 40
crabs. Calculate the total number of crabs that the three have together.

  27. 27.

Jen decides to travel to 3 different countries. The supplies he needs costs
$400 in total. The tickets for travel cost, in total, 50% more than the
supplies. How much does travel cost?

  28. 28.

Tommy is fundraising for his charity by selling cheesecakes for $4 a slice and
brownies for $3 a slice. If Tommy sells 43 brownies and 23 slices of
cheesecake, how much money does Tommy raise?

  29. 29.

Tim grows 5 trees. Each year, each tree gives him 6 lemons. How many lemons
does he get in a decade?

  30. 30.

A car travels 30 miles per hour for 8 hours. A bus travels 60 miles per hour
for 5 hours. How much farther did the bus go than the car, in miles?

  31. 31.

Each pizza is cut into 8 slices. Henry and 3 of his friends order 7 pizzas for
lunch. If Henry and his friends want to share the pizzas equally, how many
slices can each of them have?

  32. 32.

Poppy is working on a 1000-piece jigsaw puzzle. She places a quarter of the
pieces on the board. Her mom places a third of the remaining pieces on the
board. How many jigsaw pieces are left to be placed?

  33. 33.

A farm has 15 4-legged animals and 10 2-legged animals. If all animals are
healthy, how many pairs of animal legs in total are on the farm?

  34. 34.

Each minute of a call from a payphone costs $0.25. After 16 minutes, that
price drops to $0.2 per minute. How much would it cost to make a 36-minute
call?

  35. 35.

Farmer Brown has 60 animals on his farm. All of his animals are either
chickens or cows. He has twice as many chickens as cows. How many legs do the
animals have, all together?

  36. 36.

John fills a pool that is 5 feet deep, 6 feet long, and 4 feet wide. It costs
$.1 per cubic foot to fill. How much does it cost to fill?

  37. 37.