# Benchmarking Large Language Models for News Summarization

Tianyi Zhang$\phantom{}{}^{1}$, Faisal Ladhak$\phantom{}{}^{2*}$, Esin
Durmus$\phantom{}{}^{1}$, Percy Liang$\phantom{}{}^{1}$,
Kathleen McKeown$\phantom{}{}^{2}$, Tatsunori B. Hashimoto$\phantom{}{}^{1}$
$\phantom{}{}^{1}$Stanford Univeristy $\phantom{}{}^{2}$Columbia Univeristy
Equal Contribution. Order determined by a random coin flip. Correspondence to
<EMAIL_ADDRESS>and<EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs) have shown promise for automatic summarization
but the reasons behind their successes are poorly understood. By conducting a
human evaluation on ten LLMs across different pretraining methods, prompts,
and model scales, we make two important observations. First, we find
instruction tuning, and not model size, is the key to the LLM’s zero-shot
summarization capability. Second, existing studies have been limited by low-
quality references, leading to underestimates of human performance and lower
few-shot and finetuning performance. To better evaluate LLMs, we perform human
evaluation over high-quality summaries we collect from freelance writers.
Despite major stylistic differences such as the amount of paraphrasing, we
find that LMM summaries are judged to be on par with human written summaries.

##  1 Introduction

Large language models (LLMs) have shown promising results in zero-/few-shot
tasks across a wide range of domains (Chowdhery et al., 2022; Bai et al.,
2022; Brown et al., 2020; Zhang et al., 2022) and raised significant interest
for their potential for automatic summarization (Goyal et al., 2022; Liu et
al., 2022a). However, the design decisions contributing to its success on
summarization remain poorly understood, and while prior work has shown that
LLMs outperform prior state of the art, it remains unclear whether their
outputs are comparable to human writers. Examining these questions is crucial
for advancing future research in automatic summarization.

Figure 1: Selected annotator ratings of summary coherence on a 1 to 5 Likert
scale.

To answer the first question, we perform a systematic evaluation of ten
diverse LLMs with human evaluation on news summarization and our evaluation
identifies instruction tuning to be the key to zero-shot summarization
capability. In contrast, self-supervised learning alone cannot induce strong
summarization performance in the zero-shot setting (Figure 1). In fact, even a
350M parameter instruction-tuned GPT-3 can perform on par with the 175B
parameter GPT-3.

To benchmark LLMs, we evaluated on the standard CNN/DM (Hermann et al., 2015)
and XSUM datasets (Narayan et al., 2018), but found that low-quality reference
summaries caused several issues. The reference summaries in these benchmarks
are of such poor quality that human annotators judge them to be worse than the
outputs of most automatic systems (Figure 1). When computing automatic metrics
using these references, their poor quality reduces the correlation between
metric results and human judgement. Not only does this make evaluation
difficult, but it also degrades the performance of systems that take
supervision either through finetuning or few-shot prompting and makes
comparison difficult.

To address the quality issues of reference summaries and better understand how
LLMs compare to human summary writers, we recruit freelance writers from
Upwork111https://www.upwork.com to re-annotate $100$ articles from the test
set of CNN/DM and XSUM. Comparing the best performing LLM, Instruct Davinci,
to the freelance writers, we find that the Instruct Davinci summaries are much
more extractive. By manually annotating the summarization operations (Jing and
McKeown, 2000) used in these summaries, we find that Instruct Davinci
paraphrases much less frequently although it is able to combine copied
segments coherently.

Given their stylistic differences, we recruit annotators to compare the
Instruct Davinci summaries to those written by freelance writers. On
aggregate, we find that Instruct Davinci is rated as comparable to the
freelance writers. However, analysis of individual annotators reveals that
each annotator has a varying and stable preference for either Instruct Davinci
or the freelance writers.

Together, our work makes the following key contributions. First, we identify
instruction tuning, instead of model scale, as the key to LLMs’ summarization
capability. Second, we show that reference summaries used in XSUM are judged
by humans to be worse than the best LLM generated summaries. Third, to address
the issue of low quality references, we collect better quality summaries from
freelance writers and we show that the best LLM is rated as comparable to
Upwork freelance writers. In combination, these results call into question
recent claims made about LLM summarization. In particular, summarization
progress cannot be measured using reference-based metrics applied on XSUM.
Furthermore, the question of whether fine-tuned, few-shot or zero-shot models
perform better remains an open question due to the poor quality of training
data. To encourage furture work on improved evaluations, we release the high-
quality summaries written by freelance writers and the evaluation data on 18
model settings and two datasets as
resources222https://github.com/Tiiiger/benchmark_llm_summarization.

##  2 Background and Related Work

###  2.1 News Summarization

News summarization is the task of producing a concise paragraph that captures
the main points of a news article and has been a core problem within the field
of automatic summarization (Radev et al., 2002; Rush et al., 2015; Nallapati
et al., 2016; See et al., 2017; Chen and Bansal, 2018; Dong et al., 2019). In
this work, we benchmark LLMs on news summarization to understand their
potential for automatic summarization and focus on two popular news
summarization benchmarks, CNN/DM (Hermann et al., 2015) and XSUM (Narayan et
al., 2018).

These two benchmarks contain large scale data in the order of houndreds of
thousands summaries but are created via “incidental supervison”. CNN/DM
includes articles from the CNN and DailyMail websites as the source article
and adapt the bullet point highlights that come with the website articles as
reference summaries. XSUM includes articles from BBC news and adapts the
bolded sentence(s) that appear in the first paragraph as reference summaries.
As a result, the reference summaries in these datasets are known to have
quality issues (Maynez et al., 2020; Kang and Hashimoto, 2020), motivating us
to addresses these defects to improve LLM evaluation.

To contextualize the performance of LLMs, we mainly compare to previous state-
of-the-art approaches that leveraged supervised finetuning (Liu and Lapata,
2019; Lewis et al., 2019; Zhang et al., 2020; Liu et al., 2022b).
Summarization evaluation is another active area of research. Many automatic
metrics have been proposed (Lin, 2004; Zhang* et al., 2020; Sellam et al.,
2020; Durmus et al., 2020; Maynez et al., 2020; Deutsch and Roth, 2021) but
they do not always correlate with human evaluation of summarization systems
(Fabbri et al., 2020; Durmus et al., 2022). In this work, we evaluate the
effectiveness of automatic metrics for evaluating LLMs and show that the
usefulness of reference-based evaluation is closely linked to the quality of
the references.

Model | Model Creator | # Parameters | Instruction Tuning | Reference
---|---|---|---|---
GPT-3 davinci v1 | OpenAI | 175B | ✗ | Brown et al. (2020)
GPT-3 curie v1 | 6.7B
GPT-3 ada v1 | 350M
InstructGPT davinci v2 | OpenAI | 175B | ✓ | Ouyang et al. (2022)
InstructGPT curie v1 | 6.7B
InstructGPT ada v1 | 350M
OPT 175B | Meta | 175B | ✗ | Zhang et al. (2022)
GLM |  Tsinghua |  |  |
University | 130B | ✗ | Du et al. (2021) |
Cohere xlarge v20220609 | Cohere | 52.4B | ✗ | Cohere (2022)
Anthropic-LM v4-s3 | Anthropic | 52B | ✓ | Bai et al. (2022)
Table 1: List of large language models we benchmarked with human evaluation.

###  2.2 Large Language Models

LLMs (Bommasani et al., 2021; Chowdhery et al., 2022; Brown et al., 2020) have
two distinctive features over previous pretrained models. First, LLMs have
much larger scale in terms of model parameters and training data. Second,
unlike previous pretrained models that require finetuning, LLMs can be
prompted zero-shot or few-shot to solve a task. In the zero-shot setting,
prompting presents the LLMs with inputs (e.g. news articles) and a natural
language instruction (e.g., “summarize this news article in three sentences”)
and solicit outputs by having LLMs generate answers directly. When few-shot
training examples are available, LLMs have the ability to learn "in context".
Incontext learning prepends training input-output pairs along with the same
style of instruction to the testing input.

Recently, instruction-tuning has emerged as an effective way to improve LLM
prompting performance (Sanh et al., 2021; Wang et al., 2022; Ouyang et al.,
2022). In this approach, a diverse set of natural language processing tasks
are reformulated into the prompting format and the LLM’s parameters are
updated for these tasks either through supervised finetuning or reinforcement
learning.

Recent work (Goyal and Durrett, 2020) shows that the instruct-tuned GPT-3
Davinci model is better than finetuned LMs, but do not show the design
decision that contribute to the improved performance. In our work, we carry
out a more comprehensive benchmark on ten different LLMs, to understand the
effect of model scale, incontext learning and instruction tuning. Given that
automatic metrics may not be reliable, we focus on human evaluation as our
benchmarking method.

##  3 Human Evaluation on News Summarization Benchmarks

|  | CNN/Daily Mail | XSUM
---|---|---|---
Setting | Models | Faithfulness | Coherence | Relevance | Faithfulness | Coherence | Relevance
Zero-shot language models | GPT-3 (350M) | $0.29$ | $1.92$ | $1.84$ | $0.26$ | $2.03$ | $1.90$
GPT-3 (6.7B) | $0.29$ | $1.77$ | $1.93$ | $0.77$ | $3.16$ | $3.39$
GPT-3 (175B) | $0.76$ | $2.65$ | $3.50$ | $0.80$ | $2.78$ | $3.52$
Ada Instruct v1 (350M*) | $0.88$ | $4.02$ | $4.26$ | $0.81$ | $3.90$ | $3.87$
Curie Instruct v1 (6.7B*) | $0.97$ | $\mathbf{4.24}$ | $\mathbf{4.59}$ | $\mathbf{0.96}$ | $4.27$ | $\mathbf{4.34}$
Davinci Instruct v2 (175B*) | $\mathbf{0.99}$ | $4.15$ | $\mathbf{4.60}$ | $\mathbf{0.97}$ | $4.41$ | $\mathbf{4.28}$
Five-shot language models | Anthropic-LM (52B) | $0.94$ | $3.88$ | $4.33$ | $0.70$ | $\mathbf{4.77}$ | $4.14$
Cohere XL (52.4B) | $\mathbf{0.99}$ | $3.42$ | $4.48$ | $0.63$ | $\mathbf{4.79}$ | $4.00$
GLM (130B) | $0.94$ | $3.69$ | $4.24$ | $0.74$ | $4.72$ | $4.12$
OPT (175B) | $0.96$ | $3.64$ | $4.33$ | $0.67$ | $\mathbf{4.80}$ | $4.01$
GPT-3 (350M) | $0.86$ | $3.73$ | $3.85$ | - | - | -
GPT-3 (6.7B) | $0.97$ | $3.87$ | $4.17$ | $0.75$ | $4.19$ | $3.36$
GPT-3 (175B) | $\mathbf{0.99}$ | $3.95$ | $4.34$ | $0.69$ | $4.69$ | $4.03$
Ada Instruct v1 (350M*) | $0.84$ | $3.84$ | $4.07$ | $0.63$ | $3.54$ | $3.07$
Curie Instruct v1 (6.7B*) | $0.96$ | $\mathbf{4.30}$ | $4.43$ | $0.85$ | $4.28$ | $3.80$
Davinci Instruct v2 (175B*) | $\mathbf{0.98}$ | $4.13$ | $4.49$ | $0.77$ | $\mathbf{4.83}$ | $\mathbf{4.33}$
Fine-tuned language models | Brio | $0.94$ | $3.94$ | $4.40$ | $0.58$ | $4.68$ | $3.89$
Pegasus | $0.97$ | $3.93$ | $4.38$ | $0.57$ | $4.73$ | $3.85$
Existing references | - | $0.84$ | $3.20$ | $3.94$ | $0.37$ | $4.13$ | $3.00$

Table 2:  Human evaluation results for zero-shot and five-shot LLMs, finetuned
LMs, and reference summaries. We bold the entries that are not statistically
significantly different from the best numbers in each column.

In this section, we use human evaluation to systematically benchmark a diverse
set of ten LLMs on news summarization. We observe that instruction tuning is
the key to strong summarization capability and low-quality reference summaries
in current benchmarks may underestimate few-shot or finetuning performance.

###  3.1 Experimental Setup

#### Data

We conduct our human evaluation on CNN/DM and XSUM by sampling a hundred
examples from each validation set respectively. For the few-shot incontext
learning settings, we sample five examples from the training set to be the
demonstration examples. Due to the limited context window, we sample five
articles that are between 50 and 150 tokens in length according to the GPT-2
tokenizer. For XSUM, we find that a uniform sampling occasionally result in
articles that are unreadable due to data preprocessing so we manually pick
from the training set.

#### Model Details

We consider ten LLMs across different pretraining strategies and model
scales333We note that the training details of instruction-tuned GPT-3 models
may differ from those mentioned in the publication and are inferred by us
based on the API naming scheme.. Table 1 lists the details of the LLMs we
consider. Due to limited computational resources and model access, we
benchmark all models in the five-shot setting but only benchmark three OpenAI
GPT-3 models and three OpenAI instruction-tuned GPT-3 models in the zero-shot
setting.

For CNN/DM, we solicit LLM summaries with the following prompt template
“Article: [article]. Summarize the article in three sentences. Summary:” For
XSUM, we modify the prompt template to summarize in one sentence to match the
style of the reference summaries. For all LLMs we consider, we sample with
temperature $0.3$ following prior work Wu et al. (2021).

To contextualize our LLM benchmarking results, we also evaluate two state-of-
the-art finetuned LMs: Pegasus (Zhang et al., 2020) and BRIO (Liu et al.,
2022b). We decode the finetuned LMs using a beam size of $5$ following prior
work Lewis et al. (2019). In addition, we also evaluate the existing reference
summaries in the CNN/DM and XSUM validation sets.

#### Human Evaluation Protocol

We recruit annotators from Amazon Mechanical Turk, compensating them at
California minimum wage of $15.00/hr using conservative time estimates as
recommended by Whiting et al. (2019). Each model summary was evaluated by
three annotators and we report results based on their average score for each
summary.

Our annotators evaluate each summary based on three criteria: faithfulness,
coherence, and relevance. We define these terms and collect data according to
the guidelines in Fabbri et al. (2020). Coherence and relevance ratings are
collected on a 1 to 5 Likert scale while faithfulness ratings are collected in
binary ratings due to its binary nature. Unlike Fabbri et al. (2020), we omit
evaluating fluency because we find LLM outputs to be mostly fluent. The full
annotation guidelines are included in our code release.

###  3.2 Evaluation Results

Figure 2: Examples summaries generated by GPT-3 models (Section 3) or written
by freelance writers (Section 4) of an article from the CNN/DM dataset. We
find that instruction-tuned GPT-3 model can generate a much better summary
compared to the non-instruction-tuned variant. Reference summary from CNN/DM
is not coherent whereas freelance writer summary both coherent and relevant.

Table 2 presents the evaluation results444We note that the 350M GPT-3
consistently generates empty outputs so we omit it from the human evaluation..
We now discuss two main observations.

#### Instruction tuned models have strong summarization ability.

Across the two datasets and three aspects, we find that the zero-shot
instruction-tuned GPT-3 models, especially Instruct Curie and Davinci, perform
the best overall. Compared to the fine-tuned LMs (e.g. Pegasus), Instruct
Davinci achieves higher coherence and relevance scores (4.15 vs. 3.93 and 4.60
vs. 4.40) on CNN and higher faithfulness and relevance scores (0.97 vs. 0.57
and 4.28 vs. 3.85) on XSUM, which is consistent with recent work (Goyal et
al., 2022). In contrast to instruction tuning, we find scale to be less
important. Even the largest 175B model often ignores the instruction and
generates irrelevant content while the much smaller Instruct Ada outperforms
the 175B GPT-3 model on coherence and relevance.

In the five-shot setting, non-instruction-tuned LLMs can improve their
summarization performance through incontext learning. For faithfulness scores
on CNN/DM and coherence scores on XSUM, several non-instruction-tuned LLMs can
perform as well as the instruction-tuned LLMs. However, for other aspects, we
still find the instruction-tuned LLMs to be better.

#### Reference summaries in current benchmarks are extremely low quality.

We arrive at this conclusion based on two observations. First, most automatic
summarization systems score better than the reference summaries across all
three aspects. Second, applying incontext learning with the current reference
summaries makes instruction-tuned models generate worse summaries. For
example, on the XSUM dataset, after conditioning on five reference summaries,
the faithfulness score of Instruct Davinci drops from 0.97 to 0.77.

The low-quality reference summaries make it difficult to compare LLMs to both
fine-tuned models and humans. When comparing to finetuned models, the poor
performance of fine-tuned models can be attributed to the low-quality
references in training data and we may be underestimating the finetuning
performance. When comparing to human, the low-quality references are not
representative of human performance because they are created through
heuristics. As a result, it’s likely that the differences between instruction-
tuned LLMs and human performance are likely overstated in Table 3.

| CNN/DailyMail | XSUM
---|---|---
Metric | Faithfulness | Coherence | Relevance | Faithfulness | Coherence | Relevance
Rouge-L | 0.54 | 0.48 | 0.72 | -0.27 | 0.71 | 0.30
METEOR | 0.58 | 0.37 | 0.66 | -0.22 | 0.68 | 0.38
BertScore | 0.54 | 0.47 | 0.70 | -0.23 | 0.70 | 0.30
BARTScore | 0.56 | 0.34 | 0.65 | -0.22 | 0.70 | 0.35
BLEURT | 0.56 | 0.62 | 0.81 | -0.08 | 0.67 | 0.41
Table 3:  System-level kendall’s tau correlation with human scores across
different axes.

#### Qualitative Examples.

Figure 2 showcases example summaries on an article from the CNN/DM validation
set, comparing the summaries of zero-shot GPT-3 Davinci, instruction-tuned
GPT-3 Davinci, and the CNN/DM reference summary.

We start by noting that the zero-shot GPT-3 model cannot follow the
instruction to summarize well. After the summary paragraph, the model
generates an additional question that is completely irrelevant. In addition to
the failure of instruction following, the generated summary contains a factual
error, stating that the handbag mentioned is the most expensive in the world,
which contradicts the original article. In contrast, the instruction-tuned
GPT-3 model generates a summary that is both faithful and coherent.

We also observe from Figure 2 that the reference summary is not coherent. The
brand “Hermes” is not introduced until the end and its connection to the rest
of the story is unclear. This is unsurprising as reference summaries in the
CNN/DM dataset were originally bullet points accompanying the articles as
opposed to a coherent paragraph.

###  3.3 Understanding Automatic Metrics

We compute six popular automatic metrics and compute their system-level
correlations against human ratings. The list of metrics we evaluate are:
Rouge-L (Lin, 2004), METEOR (Banerjee and Lavie, 2005), BertScore (Zhang* et
al., 2020), BLEURT (Sellam et al., 2020), and BARTScore (Yuan et al., 2021).

Table 3 shows Kendall’s tau rank correlations between automated metrics and
human judgements. We observe significantly different trends on CNN/DM and XSUM
so we discuss them separately in the following paragraphs.

For CNN/DM, we observe that the reference-based automatic metrics have a
moderate correlation with some aspects of human judgments, e.g., Rouge-L has a
0.72 Kendall’s tau correlation coefficient with relevance in Table 3. Such a
level of correlation is comparable to that reported in Fabbri et al. (2020),
which measures the correlation of automatic metrics on evaluating finetuned
LMs and even earlier neural summarization systems. Therefore, we conclude that
on CNN/DM automatic metrics can still provide useful signals in relevance.

Figure 3: System-level Rouge-L vs. annotator rated relevance scores.

Studying the result more closely, we find that Rouge-L and human evaluation is
more correlated when comparing within each model group. We plot Rouge-L over
the relevance rating in Figure 3 as an example. First, we observe that Rouge-L
does still prefer finetuned LMs (green points on top of the plots) to LLMs,
consistent with prior work (Goyal et al., 2022). Despite this mistake, when
only comparing LLMs with each other, we find that a larger than 0.05 Rouge-L
difference usually translates to improved human evaluation.

On XSUM, the metrics have very low correlation with faithfulness and relevance
but it is also because the reference summaries are terrible in these aspects
(Table 3; also see Maynez et al., 2020). With such low-quality references, we
do not expect reference-based metrics to extract useful information.

Combining the results from two datasets, we find that reference-based metrics
correlate better with human judgments on the aspects for which reference
summaries also have better scores (e.g. CNN/DM relevance, XSUM coherence).
This points to the important role of quality reference summaries for
reference-based metrics, as previously observed in machine translation
(Freitag et al., 2020).

##  4 Comparing the Best LLM to Freelance Writers

In Section 3, we see that the low-quality reference summaries make studying
and benchmarking LLMs difficult. In this section, we address this by
recruiting Upwork freelance writers to collect better quality summaries. With
this data, we aim to answer two important questions. First, we would like to
know whether the best LLM has reached human-level performance and how the
summaries written by the best LLM differ from the ones written by humans.
Second, we want to understand how well reference-based metrics correlate with
human judgments once we compute them with higher quality reference summaries.

###  4.1 Experimental Setup

In this section, we describe the process of recruiting summary writers and our
summary writing instructions.

#### Data.

For data used in our study, we select 50 articles from each of the CNN/DM and
XSUM evaluation sets described in Section 3.1 and assign each article to three
writers. For XSUM, we use the full articles rather than the preprocessed
version where the first bolded sentence is removed.

#### Writer recruitment.

We recruit six writers who have had previous experience in writing blog posts,
landing page introductions, or product descriptions from the freelance work
platform Upwork. After conducting a qualification round by asking writers to
summarize five articles, we selected the best writers according to the
faithfulness, coherence, and relevance of their summaries. Through an initial
pilot study, we estimate that the time required to summarize a CNN/DM or XSUM
article is around 12 to 15 minutes. Therefore, we pay our writers $4 for every
article they summarize to following the recommended practice (Whiting et al.,
2019). We based the assignments on writers’ availability, with the most
prolific writer summarizing 100 articles and the least prolific writer
summarizing 35 articles.

#### Summary writing instructions.

For the annotation instruction, we instruct our writers to summarize each
article in around 50 words555We conducted an initial study to pilot
instructions and found that instructing writers with a sentence limit often
resulted in summaries that differ significantly in length.. To give better
task grounding, we ask the writers to summarize as if they are writing a
newletter to update their readers on news. We release the full annotation
guideline along with our code release.

#### LLM Summaries Generation.

Recently, Liu et al. (2022a) showed that length is a confounding factor in
summarization human evaluation. To control this potential length confound, we
modify the zero-shot prompt in Section 3.1 to elicit summaries that are around
50 words, which is the same word limit provided to the freelance writers. We
found that the Instruct Davinci model consistently produces summaries that
exceed a given word limit. Therefore, we intentionally prompt the Instruct
Davinci model with a 25 words limit to produce summaries with an average
length of 50 words. With this new prompt, we generate the summaries using the
same hyperparameters described in Section 3.1.

#### Quality Control.

Model | Faithfulness | Coherence | Relevance
---|---|---|---
Freelance Writer | 0.93 | 4.39 | 4.26
Zero-shot |  |  |
Instruct Davinci | 0.98 | 4.26 | 4.40
Reference Summaries | 0.64 | 3.59 | 3.45

Table 4: Amazon Mechanical Turker evaluation results of the freelance writer
summaries. Results of zero-shot Instruct Davinci and reference summaries are
taken from Table 2 after averaging the corresponding ratings.

To verify the quality of the summaries written by freelance writers, we
evaluate a random subset of 100 summaries using the same annotation scheme in
Section 3.1 using Mechanical Turkers. Table 4 reports the evaluation results,
where we see that the freelance writer summaries have much higher quality than
the original reference summaries in CNN/DM and XSUM. In addition, we see that
the difference between the freelance writer and Instruct Davinci in this
evaluation is small. Next, we carry out more targeted evaluations to compare
the summaries written by freelance writers and Instruct Davinci.

###  4.2 Paired Comparison between LLM and Freelance Writers

Figure 4: Distributions of cut and paste operations in the summaries written
by freelance writers and by Instruct Davinci. By comparison, human written
summaries contain more lexical paraphrasing and sentence reduction whereas the
Instruct Davinci model has more direct copying from the article. Figure 5:
Human evaluation results comparing summaries written by freelance writers and
summaries generated by Instruct GPT-3 Davinci. On aggregate, annotators
equally prefer the freelance writers and Instruct Davinci. However, there is
high variability in individual annotators’ preferences. Notably, annotator 1
writes abstractive summaries but prefers the more extractive Instruct Davinci
summaries.

#### Comparing Stylistic Differences.

Despite the similar performance in our quality control study, we find that LLM
summaries and the freelance writer summaries have distinctive styles. Figure 2
shows an example summry written by the freelance writer. Compared to the LLM
generated summary, we find the freelance writer summary often contains more
paraphrasing and copies less from the article.

To illustrate this stylistic difference, we measure two extractiveness
measures, coverage and density, following Grusky et al. (2018). Coverage is
defined as the percentage of words in the summary that are also present in the
article; density is defined as the average length of the continuous text spans
in the summary that are copied from the article. Our analysis shows that the
coverage and density for Instruct Davinci generated summaries are 0.92 and
12.1 whereas those for the writers written summaries are 0.81 and 2.07. These
measures show that the summaries generated by Instruct Davinci are highly
extractive whereas the summaries written by the freelance writers are much
more abstractive.

To have a finegrained understanding of these stylistic differences, we
manually analyze the distrubution of “cut and paste operations” in these two
sets of summaries. Jing and McKeown (2000) identify a set of “cut and paste”
operations for reusing text from the article, including sentence reduction,
sentence combination, syntactic transformation, lexical paraphrasing, and
generalization or specification. On top of these operations, we additionally
include a sentence copy operation to account for summary sentences that are
directly copied from the article. Using this guideline, we manually annotate
ten randomly sampled summary pairs written by Instruct Davinci and the
freelance writers.

Figure 4 reports the distribution of the cut and paste operations, showing the
fraction of sentences that contain each operation. First, we observe that the
freelance writer summaries use lexical paraphrasing and
generalization/specification much more frequently than the Instruct Davinci
generated summaries. Because both operations often involve using novel words
that are not present in the article, this matches with the fact that the
freelance writer summaries have lower coverage (0.81 vs. 0.92) than the
Instruct Davinci summaries. Second, we find that sentence combination is a
common strategy used by both the freelance writers and Instruct Davinci.
Third, we find that the freelance writers never copy an entire sentence
directly from the article but Instruct Davinci does this more frequently.

In conclusion, we find that Instruct Davinci summarizes in a very different
style than human writers. We emphasize here that the freelance writers write
in an abstractive style despite the fact that we have not explicitly
instructed them to do so. We also observe similarly abstractive styles across
the six freelance writers.

#### Comparing Human Preference.

We now return to our original goal of understanding wheter LLM generated
summaries have quality on par with the human written ones. In the following
paragraphs, we discuss our annotation design and recruitment process.

We conduct a blinded pairwise comparison evaluation between the best LLM
Instruct Davinci and the freelance writers, similar to the evaluation in Goyal
and Durrett (2020). Besides selecting the better summary within each pair, the
annotators can decide the summary pair to be equally good. We release the full
annotation instructions along with the code release for this project.

In order to compare the best LLM with the freelance writers, we annotate two
aspects. First, we solicit annotators’ overall preference, which balances the
multiple quality aspects such as faithfulness, coherence, and relevance.
Second, we solicit a more targeted measure of informativeness by asking the
annotators to compare the number of facts in each summary. For the
informativeness measure, we are motivated by the hypothesis that a more
abstractive writing style can pack more information into the summary given the
same word count. While it is also interesting to compare summary coherence and
relevance, we omit them because annotators were unable to differentiate these
aspects from the overall preference in a pilot study.

For our recruitment process, we recruit five additional annotators through
Upwork and retain one writer who participated in the previous round of summary
writing666Other annotators left during the course of study due to change in
freelance work schedule.. We carry out a qualification round and reject
annotators whose ratings differ significantly from the authors’ on a set of
control questions for informativeness. We give each annotator the same set of
100 summary pairs, where the average length of the freelance writer summaries
and the Instruct Davinci summaries are 53.2 and 52.0 respectively.

Figure 5 shows the results of the paired comparison. While we hypothesized
that the more abstractive writing style can lead to more informative
summaries, we do not find a significant effect in our annotator pool, who rate
the more abstractive summaries to be more informative only 51.1% of the time.
On the informative question, our annotators reached a moderate agreement
(Krippendorff’s alpha is 0.32), validating our annotation instruction and
recruitment process. Moving onto the more subjective overall preference, we
find that our annotators equally prefer the freelance writer summaries and the
Instruct Davinci summaries. However, a closer analysis shows that there is
significant variability in individual annotators’ preference and the
interannotator agreement is low (Krippendorff’s alpha is 0.07). This suggests
that the quality of generated summaries is getting close to that of the
freelance writer summaries and the comparison is dependent on each annotator’s
stylistic preference.

One example of such stylistic preference is seen in the results from annotator
1, who also participated in the first round of summary writing. Like other
writers, annotator 1 summarizes in an abstractive style (2.5 density and 0.86
coverage). However, annotator 1 prefers Instruct Davinci 57% of the time even
though it generated much more extractive summaries. These results suggest an
intriguing gap between annotator preferences when writing and evaluating
summaries.

###  4.3 Reevaluating Reference-based Metrics

In Section 3.3, we saw that the performance of automated metrics may depend on
the quality of reference summaries. With the freelance writer summaries, we
now conduct an initial study on the effect of using better quality summaries.
We focus on using Rouge-L for faithfulness evaluation on the XSUM dataset
because the current reference summaries are known to be highly unfaithful
(Maynez et al., 2020).

Figure 6: System-level Rouge-L vs. annotating rating of faithfulness. The left
plot is computed with XSUM references, where the correlation is weak, and the
right plot is computed with the freelance writer summaries, where the
correlation is much improved.

In Figure 6, we plot the system-level Rouge-L against the human ratings. The
left plot shows results of computing Rouge-L with existing references
summaries from XSUM, which has negative correlation with human ratings. This
result matches our expectation because the existing reference summaries are
highly unfaithful. On the right, we see the results of computing Rouge-L with
the freelance writer summaries, which leads to a much more positive
correlation. Hence, we see that the usefulness of reference-based evaluation
is closely linked to the quality of the references and we can improve metric
correlation by using better reference summaries.

##  5 Discussion

#### Implication for model development.

In this study, we build a systematic evaluation of a diverse set of LLMs and
find that instruction tuning contributes the most to LLMs’ summarization
capability. We believe that there is much research beyond our benchmarking
effort that needs to be done to better understand the effect of instruction
tuning. Here we hypothesize three aspects that could account for the success
of instruction tuning.

First, the quality of the summariztion data used in instruction tuning can
serve an important role. Our findings in Section 3 show that currently we are
finetuning language models on low quality training data, which can account for
their ineffectiveness. At this point, we cannot rule out the possibility that
when finetuned on higher quality data, finetuned LMs may perform much better.

Second, the learning algorithm used for instruction tuning can be important
(Ouyang et al., 2022). While the exact training details are unknown, the
success of Instruct Davinci might be credited to “learning from human
feedback” (LHF; Stiennon et al., 2020; Ziegler et al., 2019). Contrary to
supervised finetuning that trains systems on written summaries, learning from
human feedback trains systems from binary labels of human preferences. As we
observe in Section 4.2, there is discrepancy in how annotators write and rate
summaries. While it is possible that LHF have merits over the supervised
learning/finetuning approach in exploiting this discrepancy, more analysis is
needed to validate this hypothesis.

Third, multi-task learning can be important. Instruct Davinci is trained on a
diverse distribution of inputs and many previous studies have confirmed the
effectiveness of multi-task learning. We look forward to understanding how
summarization benefits from learning on other tasks.

#### Implication for Summarization Evaluation.

Our work also reveals the difficulties in evaluating high-performance LLMs. As
LLMs become increasingly close to human-level performance, human evaluation
requires a larger number of samples and less noisy measurement to evaluate the
quality of LLMs. Recently, Liu et al. (2022a) also point out the difficulties
in conducting human evaluation for summarization and advocate using
finegrained semantic units to match with reference summaries. However, as our
evaluation points out, not only are the existing reference summaries
unreliable but the summaries written by well-paid freelance writers also may
not outperform LLM summaries significantly. Therefore, defining reference
summaries as the ground truth may be overly restrictive as LLMs are
approaching or even exceeding average human level performance.

Not only is human evaluation limited by the reference quality, but it also is
affected by the subjectivitiy in evaluation. Individual variation shows that
there are many acceptable ways to summarize and individuals may even show
different preferences at different points in time (writing vs rating). These
factors in combination lead to the fact that we may have reached the limit of
single document news summarization. Existing benchmarks can still play a role
in evaluating new models but only if evaluation is done correctly. As LLMs
improve, we believe that summarization can be better grounded in downstream
applications where user values are better defined so that annotators have a
lower degree of freedom in balancing which quality aspects matter most to
them.

##  6 Conclusion

In this work, we conducted a comprehensive human evaluation of ten LLMs,
across the two most popular news summarization benchmarks. Through our
experiments, we find that the state-of-the-art LLM performs on par with
summaries written by freelance writers, with instruction tuning being the key
factor for success. Beyond these findings, our work highlights the crucial
role of good reference summaries in both summarization model development and
evaluation. Unless the reference quality issue is addressed, comparing zero-
shot, few-shot, and finetuning performance will remain an open question, and
the current benchmarks will provide limited value when used with reference-
based evaluation. Even when we address the quality issue and conduct a human
evaluation with high-quality references, we observe a significant amount of
individual variation from our annotator pool. Due to these factors,
evaluations for single document news summarization may be reaching their
limits.

## Acknowledgement

This work is supported by an Open Philanthropy grant and partially supported
by a gift from Northrup Grumman. We thank the Stanford NLP group and the
Stanford Center for Research on Foundation Models community for their
feedback.

## References

  * Bai et al. (2022) Yushi Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. 2022.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _ArXiv_ , abs/2204.05862.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In _IEEvaluation@ACL_.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel J. Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher Re, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramer, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021.  On the opportunities and risks of foundation models.  _ArXiv_ , abs/2108.07258.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _NeurIPS_.
  * Chen and Bansal (2018) Yen-Chun Chen and Mohit Bansal. 2018.  Fast abstractive summarization with reinforce-selected sentence rewriting.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 675–686, Melbourne, Australia. Association for Computational Linguistics.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek B Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  Palm: Scaling language modeling with pathways.  _ArXiv_ , abs/2204.02311.
  * Cohere (2022) Cohere. 2022.  Introduction to large language models.  https://docs.cohere.ai/docs/introduction-to-large-language-models.
  * Deutsch and Roth (2021) Daniel Deutsch and Dan Roth. 2021.  Understanding the extent to which content quality metrics measure the information quality of summaries.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 300–309, Online. Association for Computational Linguistics.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.
  * Du et al. (2021) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2021.  Glm: General language model pretraining with autoregressive blank infilling.  In _ACL_.
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020.  FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5055–5070, Online. Association for Computational Linguistics.
  * Durmus et al. (2022) Esin Durmus, Faisal Ladhak, and Tatsunori Hashimoto. 2022.  Spurious correlations in reference-free evaluation of text generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Fabbri et al. (2020) Alexander R Fabbri, Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2020.  Summeval: Re-evaluating summarization evaluation.  _arXiv preprint arXiv:2007.12626_.
  * Freitag et al. (2020) Markus Freitag, David Grangier, and Isaac Caswell. 2020.  BLEU might be guilty but references are not innocent.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Goyal and Durrett (2020) Tanya Goyal and Greg Durrett. 2020.  Evaluating factuality in generation with dependency-level entailment.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3592–3603, Online. Association for Computational Linguistics.
  * Goyal et al. (2022) Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022.  News summarization and evaluation in the era of gpt-3.  _ArXiv_ , abs/2209.12356.
  * Grusky et al. (2018) Max Grusky, Mor Naaman, and Yoav Artzi. 2018.  Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies.  In _North American Chapter of the Association for Computational Linguistics_.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _NeurIPS_.
  * Jing and McKeown (2000) Hongyan Jing and Kathleen McKeown. 2000.  Cut and paste based text summarization.  In _Applied Natural Language Processing Conference_.
  * Kang and Hashimoto (2020) Daniel Kang and Tatsunori B. Hashimoto. 2020.  Improved natural language generation via loss truncation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 718–731, Online. Association for Computational Linguistics.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text summarization with pretrained encoders.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3730–3740, Hong Kong, China. Association for Computational Linguistics.
  * Liu et al. (2022a) Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq R. Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir R. Radev. 2022a.  Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation.  _ArXiv_ , abs/2212.07981.
  * Liu et al. (2022b) Yixin Liu, Pengfei Liu, Dragomir R. Radev, and Graham Neubig. 2022b.  Brio: Bringing order to abstractive summarization.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919, Online. Association for Computational Linguistics.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence RNNs and beyond.  In _Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 280–290, Berlin, Germany. Association for Computational Linguistics.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022\.  Training language models to follow instructions with human feedback.  _ArXiv_ , abs/2203.02155.
  * Radev et al. (2002) Dragomir R. Radev, Eduard H. Hovy, and Kathleen McKeown. 2002.  Introduction to the special issue on summarization.  _Computational Linguistics_ , 28:399–408.
  * Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_.
  * Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Rose Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021.  Multitask prompted training enables zero-shot task generalization.  _ArXiv_ , abs/2110.08207.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020.  Bleurt: Learning robust metrics for text generation.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020.  Learning to summarize from human feedback.  _ArXiv_ , abs/2009.01325.
  * Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, M. Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddharth Deepak Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Hannaneh Hajishirzi, Noah A. Smith, and Daniel Khashabi. 2022.  Benchmarking generalization via in-context instructions on 1, 600+ language tasks.  _ArXiv_ , abs/2204.07705.
  * Whiting et al. (2019) Mark E. Whiting, Grant Hugh, and Michael S. Bernstein. 2019.  Fair work: Crowd work minimum wage with one line of code.  In _AAAI Conference on Human Computation & Crowdsourcing_.
  * Wu et al. (2021) Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021.  Recursively summarizing books with human feedback.  _arXiv preprint arXiv:2109.10862_.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.  _ArXiv_ , abs/2106.11520.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  In _ICML_.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  Opt: Open pre-trained transformer language models.  _ArXiv_ , abs/2205.01068.
  * Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Ziegler et al. (2019) Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019.  Fine-tuning language models from human preferences.  _ArXiv_ , abs/1909.08593.


# Assessing AI Rationality: The Random Guesser Test for Sequential Decision-
Making Systems

Shun Ide<EMAIL_ADDRESS>Harrison High SchoolHarrisonNew YorkUSA ,
Allison Blunt<EMAIL_ADDRESS>Harrison High SchoolHarrisonNew YorkUSA
and  Djallel Bouneffouf<EMAIL_ADDRESS>IBM T. J. Watson Research
CenterYorktown HeightsNew YorkUSA

(2024; 28 May 2024)

###### Abstract.

We propose a general approach to quantitatively assessing the risk and
vulnerability of artificial intelligence (AI) systems to biased decisions. The
guiding principle of the proposed approach is that any AI algorithm must
outperform a random guesser. This may appear trivial, but empirical results
from a simplistic sequential decision-making scenario involving roulette games
show that sophisticated AI-based approaches often underperform the random
guesser by a significant margin. We highlight that modern recommender systems
may exhibit a similar tendency to favor overly low-risk options. We argue that
this “random guesser test” can serve as a useful tool for evaluating the
rationality of AI actions, and also points towards increasing exploration as a
potential improvement to such systems.

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
AIBS 2024: The First Workshop on AI Behavioral Science; August 25–26, 2024;
Barcelona, Spain††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Recent advances in AI (artificial intelligence) technology have allowed its
integration into many parts of society. As AI’s involvement in high-
consequence tasks grows, concerns have arisen about potential AI misalignment
(Dung, 2023), a concept representing the gap between AI’s actual behavior and
its intended outcome. Many recent works discuss how to detect and hold AI
accountable for erroneous actions. Discussions on this topic can be
categorized into a few major clusters: data lineage (Voigt et al., 2017;
Gröger, 2021), addressing integrity issues in training data; AI explainability
(Molnar, 2020; Dwivedi et al., 2023), dealing with challenges in understanding
black-box models during high-stakes decision-making; AI fairness (Awad et al.,
2022; John-Mathews et al., 2022), focusing on the misalignment between AI
models and established norms of political correctness; and adversarial
robustness (Chen et al., 2022), assessing the stability of predicted
classification categories in the presence of additional noise.

AI-based recommender systems (Portugal et al., 2018; Su and Khoshgoftaar,
2009; Zhang et al., 2019) are arguably among the most widely used real-world
AI applications, already integral to everyday life in online shopping and
movie streaming. Modern commercial recommender systems typically combine
collaborative filtering with optimized sequential decision-making methods such
as multi-armed bandits (MAB) (Silva et al., 2022). The key design point of MAB
is balancing exploitation, which conservatively follows the best-known option,
and exploration, which embraces new challenges and possibilities. MAB is a
framework that aims to trade off exploitation and exploration through trial
and error when there is a finite set of options. The best arm inferred from
historical data may not be the optimal choice when only a limited number of
samples are available. In other words, MAB is a decision-making approach that
mitigates potential AI misalignment through forced exploration when the number
of samples is limited.

MAB has a long and rich history of research in both theory and practice,
providing a sophisticated framework to analyze potential discrepancies from an
optimal course of action based on a metric called the regret (Lattimore and
Szepesvári, 2020; Bouneffouf et al., 2020; Zhao, 2022). Thompson sampling
(Russo et al., 2018), in particular, is an algorithm that elegantly addresses
the exploration-exploitation tradeoff using posterior sampling. As long as the
set of decision choices meets the criteria of AI fairness, it has minimal risk
of AI misalignment in the sense that it avoids getting locked into a
suboptimal choice. The use of the bandit framework is widely regarded as the
best approach in many commercial recommender systems where sequential
decision-making is made for the choices generated with an offline
collaborative filtering algorithm.

Our main concern is that recommender systems may misalign with the actual
preferences of the user, which might be perceived through annoying and
stubborn internet banner advertisement on the web browser. In this work, we
conduct a model study to assess potential AI misalignment in sequential
decision-making tasks such as online product recommendation. We mainly
conclude that even with a sophisticated bandit algorithm, significant value
misalignment can occur against common business expectations. Our study began
with the assumption that Thompson sampling would be a flawless sequential
decision-making approach with minimal risk of AI misalignment. Surprisingly,
our experimental results show that the MAB algorithm can underperform a simple
random guesser by a significant margin in sequential roulette games, which
serves as a simplest but illustrative model of online recommender systems.

Our study has the potential to profoundly impact the detection of potential AI
misalignment in real-world recommender systems, where decisions are often
based on MABs. It can be used, for example, in testing a black-box personal
restaurant recommendation system by providing a historical record according to
a simple probabilistic model discussed later. To the best of our knowledge,
this is the first work that highlights the risk of AI misalignment in MAB-
based sequential decision-making settings.

##  2\. Related Works

AI misalignment is a generic term that represents a discrepancy between the
expected outcome from end-users and AI’s actual behavior. Various forms of AI
misalignment are conveniently summarized in (Dung, 2023). These include data
lineage (Voigt et al., 2017; Gröger, 2021), explainability of AI (XAI)
(Molnar, 2020; Dwivedi et al., 2023), AI fairness (Awad et al., 2022; John-
Mathews et al., 2022), and adversarial robustness (Chen et al., 2022). Most of
these topics have been motivated by concerns about the black-box nature of
modern AI systems.

XAI can be viewed as a framework that detects and mitigates potential AI
misalignment in a human-in-the-loop system by providing human end-users with
understandable explanations of AI’s decisions. Typically, XAI methods assume a
batch setting (Molnar, 2020), where training and test samples are treated
independently rather than sequentially. Quantitative AI fairness analysis
(Bellamy et al., 2019; Lyons et al., 2021; John-Mathews et al., 2022) and
adversarial robustness (Chen et al., 2022) are two significant recent
advancements in the field. Although they have undergone extensive study,
fairness metrics, such as disparate impact, heavily rely on specific political
correctness criteria and the sensitive attributes linked to them, which may
limit their general applicability to sequential recommender systems. On the
other hand, adversarial robustness typically assesses the stability of
predicted classification categories in the presence of additional noise,
usually under a batch setting.

Sequential decision-making tasks among a given set of discrete options have
commonly been formalized as a reinforcement learning (RL) problem or its
simplified version called the multi-armed bandit (MAB). Due to the simplicity
of choosing one of the arms (decision options) at each decision round and the
availability of a highly sophisticated theoretical framework for regret
analysis (Lattimore and Szepesvári, 2020; Zhao, 2022), MABs have not been
considered the main subject of AI misalignment research.

Recently, Arnold et al. (Arnold et al., 2017) discussed scenarios where a
mismatch in the intended goal and actual AI outcomes exists in the context of
the Markov decision process (MDP). While MDP is a general framework upon which
RL and MAB models are built, their work is a high-level position paper and
does not provide concrete models and empirical results. Additionally, Zhuang
et al. (Zhuang and Hadfield-Menell, 2020) discussed the possibility that a
slight gap between an AI agent and end-users (“principal”) in the utility
function could lead to potentially devastating AI misalignment. Our work may
be viewed as an actual realization of their general mathematical framework.
Finally, in a slightly different context, Bouneffouf et al. (Bouneffouf et
al., 2017) attempted to characterize delusional gamblers using RL and MAB in
sequential gambling tasks, including the Iowa Gambling Task, which inspired
our roulette games experiments.

##  3\. Problem Setting

As mentioned previously, we focus on situations where an AI agent chooses one
of the discrete actions based on historical data. Specifically, we assume
that, at a time point $t$, the AI agent picks a value $x_{t}$ from a finite
set of actions $\mathcal{A}$, which are bet types in our case, and receives a
reward $y_{t}$. The choice of the action is based on the historical data
$\mathcal{D}^{t}$:

(1)
$\displaystyle\mathcal{D}^{t}\triangleq(x_{1},y_{1}),(x_{2},y_{2}),\ldots,(x_{t-1},y_{t-1}).$

The goal of the AI agent is to maximize the total cumulative reward. The main
challenge for the agent is that, especially in the beginning, the number of
samples will be so limited that a “data-driven” decision may not be accurate.
Therefore, the agent must explore options that may not necessarily considered
to be the best according to $\mathcal{D}^{t}$. This is the classical
exploration-exploitation trade-off.

As discussed in the Introduction, our motivation is to assess AI models’ true
utility. In the present context, our goal is to quantify how AI-based
sequential decision-making models perform better than a simple random guesser.
In some sense, this is to benchmark the AI system using a random decision
maker as the baseline.

###  3.1. Sequential Gambling

We employ European roulette games as a simple yet non-trivial model of
recommender systems. As discussed before, commercial recommender systems
typically take a two-step approach: offline collaborative filtering to
generate a relatively small number of decision choices (e.g., products to be
recommended), followed by online decision-making with MAB to choose one of the
generated candidates. While typically users’ demographic information is taken
into account in the latter in the form of contextual bandits, classical MABs
still serve as a useful model for the online decision-making step, assuming
that a specific demographic category (such as teenage boys) is focused on.

The roulette has 37 pockets from 0 through 36, to which the AI agent decides
on which bet type out of $K$ choices to use with a $1 bet. Table 1 summarizes
bet types we use, which correspond to the action space $\mathcal{A}$ with
$K=3$. In this setting, $x_{t}$ represents one of (zero, corner, even), and
$y_{t}$ represents the payout for a $1 bet. In the table, $\theta_{i}$ and
$r_{i}$ denote the winning probability and the payout of the $i$-th bet type,
respectively. The agent loses the $1 stake when it loses, while it receives
$r_{i}$ in addition to the $1 stake when winning in the $i$-th bet type.

We use two different sets of winning probabilities. One is a fair roulette
setting, where the expected values of all the bet types are the same, and a
skewed roulette setting, where the zero bet has a positive expected value of
$1. In either case, the AI agent is unaware of the true nature of the wheel.
If the AI agent is intelligent enough, however, it should be able to uncover
these hidden rules behind the data. In other words, a rational and effective
AI should be able to detect any real and exploitable patterns within the game.

Table 1.  Summary of bet types in European Roulette. bet type |  $\theta_{i}$ (fair) |  $\theta_{i}$ (skewed) | $r_{i}$ | expected value (fair)
---|---|---|---|---
Zero | 1/37 | 2/37 | 36 | $-0.027$
Corner | 4/37 | 4/37 | 8 | $-0.027$
Even | 18/37 | 18/37 | 1 | $-0.027$

The fair roulette setting is particularly interesting. Since the ground truth
expected values are identical, the AI agent’s decision strategy does not
really matter in infinite betting rounds. One practical question is what would
happen after a finite number of bet attempts, such as $T=50$, where $T$
denotes the number of betting rounds in a single session. This scenario is
closely aligned with real-world situations involving a limited budget. This
situation, where all the options have an equal expected value, has not been
extensively analyzed in the standard regret analysis of MAB.

The skewed roulette setting is used to assess the impact of data non-
stationalities. The importance of different stationalities in a similar
scenario have also been discussed in Lin et al. (Lin et al., 2019). Most
recent works also assume that the underlying data is stationary, which makes
real-life applications questionable. In our scenario, we manifest this with a
situation in which the roulette wheel is skewed temporarily such that the
chance to win the zero bet doubles. Again, our agent is unaware that this
change has occurred. This extra element allows us to test whether or not these
agents are rationally able to explore and exploit this information.

###  3.2. Algorithms Tested

As discussed, the sequential decision-making task for a finite set of decision
choices has been typically studied using reinforcement learning (RL) (Sutton
and Barto, 2018) in the machine learning literature. Multi-armed bandits
(MAB), which address a simplified task of RL, are particularly suitable for
the gambling setting (Slivkins, 2019). Among the various RL and MAB variants
proposed to date, we choose the following simple algorithms that have a
minimal set of model parameters:

  * •

Epsilon-greedy (EG) MAB ($\varepsilon=0.1$)

  * •

Thompson sampling (TS) MAB

  * •

Temporal difference RL (TD$(\lambda)$ with $\lambda=0,1$)

Note that having less complexity in the model generally implies greater
resistance to learning spurious patterns. Hence, this choice is preferable to
the AI agent to suppress unwanted volatility. These models are compared with a
simple random guesser as the baseline, which simply chooses the bet types
randomly.

In the EG MAB strategy, the AI agent computes the expected rewards for each of
the action choices based on $\mathcal{D}_{t}$ and uses it in decision-making.
Specifically, it picks the action with the highest expected reward (the
“greedy” choice) with a probability of $1-\epsilon$, while it chooses any
action randomly with a probability of $\epsilon$.

The TS strategy learns the probability of winning for each action via Bayesian
learning, which in this case involves the beta and categorical distributions
as the prior and observation models, respectively. The agent learns
$p_{i}^{t}$, the win probability of the $i$-th action at the $t$-th round,
which is updated at each round with (Russo et al., 2018):

(2) $\displaystyle
p_{i}^{t}=\frac{\alpha_{i}+S_{i}^{t}}{\alpha_{i}+\beta_{i}+N_{i}^{t}},$

where $S_{i}^{t}$ is the number of wins up to the $t$-th round out of $N_{i}$
trials (i.e., how many times the $i$-th action has been chosen), and
$\alpha_{i},\beta_{i}$ are the parameters of the beta prior, which are
initialized to ones.

Finally, in this setting, the TD strategy evaluates the expected cumulative
reward (called the Q-value) for each action with no state variable defined and
chooses the action with the highest expected cumulative reward. In TD(0), when
an action $i$ is chosen, the Q-value is updated as

(3) $\displaystyle Q_{i}\leftarrow Q_{i}+\alpha(r_{i}^{t}-Q_{i}),$

where $r^{t}_{i}$ is the observed reward at the $t$-th round. The Q value is
initialized to zero, and $\alpha$ is set to 0.1 in our experiments. In TD(1),
the empirical average is computed with $\mathcal{D}^{t}$ for $Q_{i}$. For more
details on the TD-learning, see a standard RL textbook, e.g., (Sutton and
Barto, 2018).

##  4\. Empirical Evaluation

This section presents the results of the empirical evaluation of the decision-
making algorithms introduced in the previous section.111 The Python
implementation will be made publicly available upon acceptance of the paper.
We employed two metrics to evaluate utility. One is the rate of successful
sessions under a finite session horizon $T$, where a session (i.e., a sequence
of bet rounds) is considered successful when the balance is nonnegative at the
completion of the final round $t=T$. The other metric we used is the number of
rounds until bankruptcy, defined as the number of bet rounds until the balance
becomes zero or below. For the latter, no finite session horizon is assumed,
and hence, the metric can theoretically be positive infinite.

These metrics were statistically tested using a one-way analysis of variance
(ANOVA). Specifically, it compared the results of the AI algorithms to the
control group, which consists of samples generated by the random guesser.

###  4.1. Success Over Rounds

We ran roulette simulations under two session horizons: $T=50$ and $500$
rounds, both using the fair roulette. The final balance of each session was
recorded to count the number of successful sessions. For each of the four AI
algorithms as well as the random baseline, we repeated the gambling simulation
10,000 times with different random seeds.

Figures 1 and 2 show the success rates for $T=50$ and $500$, respectively. The
$p$-values of the one-way ANOVA are shown at the top of the bars: The smaller
the $p$-value is, the more statistically significant the difference is.
Surprisingly, the random guesser performed significantly better than the AI
agents. Among the four AI algorithms, the more sophisticated algorithms (TS
and TD$(\lambda)$) performed worse than the simpler one (EG). The trend is
consistent between $T=50$ and $500$.

The results appear counter-intuitive given that the three bet types have the
same expected value in the fair roulette. This outcome can be explained by the
preference for high-risk and high-return options of the random guesser and the
relatively large volatility of the balance under a finite $T$. In fact, in
Thompson sampling, one can mathematically evaluate the probability that the
$k$-th action out of $K$ choices provides the highest reward as

(4) $\displaystyle
P_{k}=\frac{\theta_{k}}{1-\prod_{j=1}^{K}(1-\theta_{j})}\prod_{i:\
r_{i}>r_{k}}(1-\theta_{i}).$

We omit the derivation due to space limitations. In our setting, we have
$(P_{1},P_{2},P_{3})\approx(0.05,0.19,0.76)$ for the zero, corner, and even
bets, respectively. This means that in those AI algorithms, the zero bet
option is rarely chosen, which contrasts sharply with the random guesser who
chooses the zero bet with a 1/3 chance. Under a finite $T$, this risk-taking
tendency appears to be rewarding.

As long as $T\leq 500$ in our setting, we conclude that the well-thought-out
AI algorithms do not provide an optimal strategy. This calls for significant
attention in real-world recommender systems, as there is a possibility that
these systems may overly favor low-risk and low-return options.

Figure 1. Recorded simulations with a net profit at 50 rounds. Random strategy
had the most profit, at roughly a 40% rate. Figure 2. Recorded simulations
with a net profit at 500 rounds. Larger gap develops between random and other
groups.

###  4.2. Stationary and Nonstationary Scenario

In addition to comparing the success of algorithms over time, their ability to
survive economically was also compared. In this scenario, the agent starts
with the same setting, but the simulation is set to end when the gambler’s
balance hits zero. The number of rounds it took for the agent to go bankrupt
was recorded.

For comparison purposes, Figure 3 plots the number of rounds until bankruptcy,
or simply survival, of the algorithms in the stationary scenario as previously
discussed. Again, the $p$-values from the one-way ANOVA are shown at the top
of the bars. Consistent with the previous result, the random guesser lasts for
longer, i.e., outperforms the sophisticated AI agents.

Figure 4 explores a nonstationary scenario, where the winning probability of
the zero bet increases in $100\leq t\leq 200$. Thompson Sampling seems to have
made an attempt at capitalizing on the zero bet, but is still outlived by the
random group. This again shows the detrimental effects of the AI agent over-
neglecting the exploration option.

Figure 3. Survival of algorithms in stationary scenario. Random survives for
significantly longer. Figure 4. Survival of algorithms in nonstationary
scenario. TS seems to have taken advantage of the situation, but nevertheless
bankrupts faster than Random.

##  5\. Concluding Remarks

The roulette experiment serves as a simple yet illustrative example of how AI
algorithms can fail the random guesser test. We have found that the
sophisticated RL algorithms underperformed compared to the random guesser in a
certain controlled setting. The use of a random guesser allows for the
quantitative evaluation of the vulnerabilities and risks associated with AI
systems in the sequential decision-making scenario. In particular, the
empirical results suggest that commercial recommender systems may overly
prefer low-risk and low-return options. Empirically, we showed that the AI
agents were only rarely selecting the high-reward zero bet in favor of the
even bet which occurred far more often. This could potentially be an
explanation of the behavior of some annoying online advertisements. The AI
overly favors safe options, neglecting other potentially valuable options. For
instance, when a consumer directs themselves to a product, the AI would favor
that one selection too heavily, which may not accurately suit the consumer’s
tastes. In summary, our results suggest that a potential solution to improving
recommender systems would be to increase the exploration parameter.

For future work, we plan to extend this framework to state-of-the-art sequence
prediction models. We are also interested in making further connections
between the neglect of exploration and AI misalignment.

## References

  * (1)
  * Arnold et al. (2017) Thomas Arnold, Daniel Kasenberg, and Matthias Scheutz. 2017.  Value Alignment or Misalignment–What Will Keep Systems Accountable?. In _Workshops at the thirty-first AAAI conference on artificial intelligence_.
  * Awad et al. (2022) E. Awad et al. 2022.  Computational ethics.  _Trends in Cognitive Sciences_ 26, 5 (2022), 388–405.
  * Bellamy et al. (2019) R.K.E. Bellamy et al. 2019.  AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias.  _IBM Journal of Research and Development_ 63, 4/5 (2019), 4–1.
  * Bouneffouf et al. (2020) Djallel Bouneffouf, Irina Rish, and Charu Aggarwal. 2020.  Survey on applications of multi-armed and contextual bandits. In _2020 IEEE Congress on Evolutionary Computation (CEC)_. IEEE, 1–8.
  * Bouneffouf et al. (2017) D. Bouneffouf et al. 2017.  Bandit models of human behavior: Reward processing in mental disorders. In _Proc. 10th Intl. Conf. Artificial General Intelligence (AGI 17)_. Springer, 237–248.
  * Chen et al. (2022) P.-Y. Chen et al. 2022.  _Adversarial robustness for machine learning_.  Academic Press.
  * Dung (2023) Leonard Dung. 2023.  Current cases of AI misalignment and their implications for future risks.  _Synthese_ 202, 5 (2023), 138.
  * Dwivedi et al. (2023) R. Dwivedi et al. 2023.  Explainable AI (XAI): Core ideas, techniques, and solutions.  _Comput. Surveys_ 55, 9 (2023), 1–33.
  * Gröger (2021) C. Gröger. 2021.  There is no AI without data.  _Commun. ACM_ 64, 11 (2021), 98–108.
  * John-Mathews et al. (2022) J.-M. John-Mathews et al. 2022.  From reality to world. A critical perspective on AI fairness.  _Journal of Business Ethics_ 178, 4 (2022), 945–959.
  * Lattimore and Szepesvári (2020) Tor Lattimore and Csaba Szepesvári. 2020.  _Bandit algorithms_.  Cambridge University Press.
  * Lin et al. (2019) Baihan Lin, Guillermo Cecchi, Djallel Bouneffouf, Jenna Reinen, and Irina Rish. 2019.  A story of two streams: Reinforcement learning models from human behavior and neuropsychiatry.  _arXiv preprint arXiv:1906.11286_ (2019).
  * Lyons et al. (2021) H. Lyons et al. 2021.  Conceptualising contestability: Perspectives on contesting algorithmic decisions.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW1 (2021), 1–25.
  * Molnar (2020) C. Molnar. 2020.  _Interpretable machine learning_.  Lulu. com.
  * Portugal et al. (2018) Ivens Portugal, Paulo Alencar, and Donald Cowan. 2018.  The use of machine learning algorithms in recommender systems: A systematic review.  _Expert Systems with Applications_ 97 (2018), 205–227.
  * Russo et al. (2018) Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, et al. 2018\.  A tutorial on thompson sampling.  _Foundations and Trends® in Machine Learning_ 11, 1 (2018), 1–96.
  * Silva et al. (2022) Nícollas Silva, Heitor Werneck, Thiago Silva, Adriano CM Pereira, and Leonardo Rocha. 2022.  Multi-armed bandits in recommendation systems: A survey of the state-of-the-art and future directions.  _Expert Systems with Applications_ 197 (2022), 116669.
  * Slivkins (2019) Aleksandrs Slivkins. 2019.  Introduction to Multi-Armed Bandits.  _Foundations and Trends® in Machine Learning_ 12, 1-2 (2019), 1–286.
  * Su and Khoshgoftaar (2009) Xiaoyuan Su and Taghi M Khoshgoftaar. 2009.  A survey of collaborative filtering techniques.  _Advances in artificial intelligence_ 2009 (2009).
  * Sutton and Barto (2018) Richard S Sutton and Andrew G Barto. 2018.  _Reinforcement learning: An introduction_.  MIT press.
  * Voigt et al. (2017) P. Voigt et al. 2017.  The EU general data protection regulation (GDPR).  _A Practical Guide, 1st Ed._ 10, 3152676 (2017), 10–5555.
  * Zhang et al. (2019) Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019.  Deep learning based recommender system: A survey and new perspectives.  _ACM computing surveys (CSUR)_ 52, 1 (2019), 1–38.
  * Zhao (2022) Qing Zhao. 2022.  _Multi-armed bandits: Theory and applications to online learning in networks_.  Springer Nature.
  * Zhuang and Hadfield-Menell (2020) Simon Zhuang and Dylan Hadfield-Menell. 2020.  Consequences of misaligned AI.  _Advances in Neural Information Processing Systems_ 33 (2020), 15763–15773.