# Decoding Speculative Decoding

Minghao Yan  Saurabh Agarwal  Shivaram Venkataraman

###### Abstract

Speculative Decoding is a widely used technique to speed up inference for
Large Language Models (LLMs) without modifying its outcome. When performing
inference on an LLM, speculative decoding uses a smaller draft model which
generates speculative tokens and then uses the _target_ LLM to verify those
draft tokens. The speedup provided by speculative decoding heavily depends on
the choice of the draft model. It has been widely suggested to select a draft
model that provides a high probability of the generated token being accepted
by the LLM to achieve the highest throughput. However, our experiments
indicate the contrary with throughput diminishing as the probability of
generated tokens to be accepted by the target model increases. To understand
this phenomenon, we perform extensive experiments to characterize the
different factors that affect speculative decoding and how those factors
interact and affect the speedups. Based on our experiments we describe an
analytical model which can be used to decide the right draft model for a given
workload. Further, using our insights we design a new draft model for
LLaMa-65B which can provide 30% higher throughput than existing draft models.

(a) OPT Series

(b) LLaMa Series

(c) OPT Series

(d) LLaMa Series

Figure 1: This figure shows the throughput scaling of different draft models
from the OPT series and the LLaMa series on MMLU. As model size increases,
throughput generally decreases due to significantly higher inference latency,
despite consistent increases in TAR. Figure 2: This figure shows the
speculative decoding process. In vanilla LLM inference, after the prompt is
processed into KV caches (Prefill phase), LLM generates the output token by
token in an autoregressive manner (Autoregressive generation phase). In
speculative decoding, a draft model is first used to generate $n$ candidate
tokens at each step (Draft token generation phase), then the LLM verifies the
candidate tokens and accepts $k$ ($k\leq n$) tokens (LLM verification phase).
Since LLM knows all $n$ candidate tokens in advance, this step is identical to
a prefill step of length $n$. In both cases, this process is repeated until
either an end-of-sequence (EOS) token is generated or if maximum generation
limit has been reached.

##  1 Introduction

In recent years, Large Language Models (LLMs) have emerged as a cornerstone of
modern computational linguistics, offering unprecedented capabilities in
generating and interpreting human language. As the demand for faster and more
efficient language processing grows, understanding and optimizing the
throughput of these models becomes increasingly crucial. Decoder-only LLMs
(Brown et al., 2020; Touvron et al., 2023a, b) use auto-regressive decoding to
perform inference. Auto-regressive decoding is known to be hardware
inefficient (Miao et al., 2023; Liu et al., 2023a) leading to poor resource
utilization and low throughput while serving.

Several methods (Yu et al., 2022; Wang et al., 2020; Kwon et al., 2023; Dao et
al., 2023; Hong et al., 2023) have been studied to optimize the serving of
LLMs. One promising approach to improve the throughput for serving LLMs
without accuracy loss is speculative decoding (Stern et al., 2018; Xia et al.,
2023a; Leviathan et al., 2023). When using speculative decoding to serve an
LLM (usually 100s of billion parameters), a draft model (a significantly
smaller LLM) is used to generate speculative tokens. The target LLM model then
verifies the output of the draft model and only outputs tokens that match its
output. In the case of speculative decoding, the target LLM for inference acts
as a _verifier_ for the draft model. By leveraging faster inference of smaller
draft models, speculative decoding turns autoregressive decoding on the target
LLM into a more hardware-friendly batched operation (similar to “prefill”)
while preserving accuracy.

Performing speculative decoding requires choosing a draft model (Su et al.,
2023). The choice of the draft model is usually based on how many tokens
generated by the draft model are accepted by the target LLM. This is captured
by the Token Acceptance Rate (TAR), which represents the average number of
tokens generated by the draft model that the target LLM accepts. Prior work
for speculative decoding (Leviathan et al., 2023; Chen et al., 2023)
encourages choosing draft models that provide a high TAR. For instance, prior
work (Liu et al., 2023a; Miao et al., 2023; Su et al., 2023) used OPT-125M and
LLaMa-165M to serve OPT-6.7B and LLaMa-7B, with draft models having $\approx
50\times$ fewer parameters than the target model. These choices were made
based on the assumption that a higher TAR correlates with better throughput
(Chen et al., 2023; Liu et al., 2023a).

In this work, we aim to better understand how should a draft model be chosen
to maximize throughput. We perform a detailed experimental study to capture
the relationship between TAR and throughput. In our experiments, we use LLMs
like LLaMa-65B and OPT-66B as target models and wider variants of LLaMa and
OPT family as draft models, ranging from $\approx 5\times$ to $528\times$
fewer parameters than the target models. Surprisingly, our experimental
results (Figure 1) show that smaller models with _lower TAR_ provide _higher
throughput_ , indicating that higher TAR does not always provide the highest
throughput. Based on this observation, we analyze how TAR, latency of the
draft model and latency of the target model interact with each other to affect
the throughput of serving a model using speculative decoding (Section 4.2). We
further analyze how latency changes resulting from hardware changes or TAR
changes due to new workloads might impact the throughput of speculative
decoding (Section 4.3). Our analysis also indicates that redesigning draft
model architecture to make them tailored for speculative decoding can provide
significant gains in throughput. As a first step, we introduce a new 1.3B
parameter draft model for serving LLaMa-65B which provides up to 30% higher
throughput than existing draft models.

Overall, we perform real-world experiments on 12 different models and 3
popular datasets. Our analyses reveal a delicate balance between the latency
of a draft model and its token acceptance rate (TAR), which needs to be
carefully managed to maximize the throughput of speculative decoding.

Our Contributions:

  * •

To the best of our knowledge, we are the first work to conduct comprehensive
experiments on serving the open source LLaMa-65B and OPT-66B models utilizing
speculative decoding, conducting more than 340 experiments to elucidate the
factors one needs to consider while selecting a draft model.

  * •

Based on our analysis we propose an analytical model (Section 4.2) to predict
the throughput of speculative decoding. The model highlights how different
factors like model latency and TAR interact and their effect on the throughput
of speculative decoding. We also show how users can use our analytical model
to make informed choices for draft model selection (Section 4.3).

  * •

Further, we show how users can design better draft models for speculative
decoding (Section 4.4), building a pruned version of the popular Sheared-
LLaMa-1.3B (Xia et al., 2023b), which we call NoFT-wide-LLaMa-1.3b. Our model
provides up to $30\%$ higher throughput than Sheared-LLaMa-1.3B while using
only $0.8\%$ of tokens (0.42B vs 50.42B) used to train Sheared-LLaMa-1.3B.

##  2 Background and Related Work

We first provide a high-level overview of LLM inference and the use of
speculative decoding.

###  2.1 Background

A decoder-only LLM performs inference in two phases: a prefill phase and an
autoregressive-decoding phase. In the prefill phase, the LLM is initialized
with a context or prompt, formulated as $C=\\{c_{1},c_{2},...,c_{n}\\}$, where
$C$ represents the input context and $n$ the length of the prefill. In the
prefill phase, the model processes the whole input context in parallel to
perform next-word prediction. During the autoregressive-decoding phase, the
model generates new text sequentially, one token at a time, building upon the
context provided in the prefill phase. The autoregressive decoding phase is
widely known to be memory bandwidth bound on modern GPUs (Leviathan et al.,
2023) due to its sequential nature.

To improve hardware utilization and throughput, Schuster et al. (2022); Chen
et al. (2023) proposed _speculative decoding_ , where a significantly smaller
_draft_ model generates multiple tokens, and the _target_ LLM performs
verification on the generated tokens in parallel. The verification is akin to
_prefill_ stage in LLM inference and as long as more than one token is
accepted on average, speculative decoding has the potential to provide
speedups. Figure 2 shows how inference using speculative decoding differs from
auto-regressive decoding. It is widely reported (Miao et al., 2023; Liu et
al., 2023a) that the number of tokens accepted by the target model influences
the speedup provided by speculative decoding.

In this work, we study if there are other factors beyond the number of
accepted tokens, like the latency of the draft model or the number of tokens
generated by the draft model, that would influence the speedup provided by
speculative decoding.

###  2.2 Related Work

#### LLM Inference

There has been significant amount of work on improving LLM serving including
work in Orca (Yu et al., 2022), LightSeq (Wang et al., 2020), DeepSpeed
Inference (Aminabadi et al., 2022), PagedAttention (Kwon et al., 2023),
FlashDecoding (Dao et al., 2023) and FlashDecoding++ (Hong et al., 2023).
These works seek to improve LLM inference by better utilization of hardware.
There is another line of work that has looked at pruning LLMs based on input
context to speed up inference (Liu et al., 2023b). However, in this work, we
focus on speculative decoding (Leviathan et al., 2023; Chen et al., 2023)
which has been inspired by speculative execution in hardware (Hennessy &
Patterson, 2011).

#### Speculative Decoding

Several prior works have studied ways to improve speculative decoding. Liu et
al. (2023a) seeks to continuously train the draft model on the output of the
target model to improve the token acceptance rate. However, training on the
same hardware during inference can be challenging depending on the inference
request rate and hardware utilization. Predictive Pipeline Decoding (PPD)
(Yang et al., 2023), was one of the first methods to introduce the use of
early exit (Schuster et al., 2022) from the target model to obtain draft
tokens. Similar to PPD, Draft&Verify (Zhang et al., 2023) seeks to combine the
use of early exit with speculative decoding, where the early exit (Schuster et
al., 2022) from the target model acts as a draft token. A drawback of these
methods is that the maximum benefit in latency is capped. For example, in
speculative decoding we can use draft models which are orders of magnitude
(e.g., $\approx$100x-1000x) smaller than the target model, while early exit
methods usually exit after performing inference over at least a fourth of the
model (Schuster et al., 2022), thus, limiting the gain in throughput.

In this work, we aim to understand how the choice of draft model affects the
throughput provided by speculative decoding. We show that we need to look at
other factors beyond just TAR when choosing a draft model.

Figure 3: Performance Breakdown of OPT speculative decoding, lookahead length
is set to be 8.

##  3 Benchmarking Speculative Decoding

To study the effects of the choice of the draft model we first perform a
detailed study on serving OPT-65B and LLaMa-65B (two popular LLMs) using
speculative decoding.

#### Setup.

We implement speculative decoding in the Microsoft Deepspeed library
(Microsoft, 2023). We use the same setup as SpecInfer (Miao et al., 2023),
first using the draft model to generate draft tokens and then using the target
model to verify the output of the draft model. We set the batchsize to 1 and
use greedy decoding. For all our experiments we use 4 Nvidia 80GB A100 GPUs.
We perform our experiment on the OPT and LLaMa base models (Zhang et al.,
2022; Touvron et al., 2023a) on MMLU (Hendrycks et al., 2020), Hellaswag
(Zellers et al., 2019), and Chatbot Arena datasets (Zheng et al., 2023). For
MMLU, we use the standard 5-shot setup. The remaining datasets were evaluated
in a zero-shot setting.

#### Metrics.

To quantify the performance of different draft models when performing
inference on a target model, we measure throughput (tokens generated per
second) and TAR. We note that the primary goal of speculative decoding is to
improve throughput.

#### Results.

In Figure 1 we plot the throughput while using speculative decoding against
the TAR for serving OPT-66B using OPT-125M, OPT-350M, OPT-1.3B, OPT-2.7B, and
OPT-6.7B as draft models on MMLU. Similarly, we also plot the throughput
against the TAR for serving LLaMa-65B using Sheared-LLaMa-1.3B, Sheared-
LLaMa-2.7B, LLaMa-7B, and LLaMa-13B as draft models. More results can be found
in the Appendix (Figure 9). We observe a surprising result: _larger models
with higher TAR provide lower overall throughput_ , i.e., OPT-125M provides
higher throughput than OPT-6.7B. This is contrary to prior work (Miao et al.,
2023; Liu et al., 2023a) which have suggested choosing draft models with
higher TAR for speculative decoding.

In the next section, we delve into the reasons behind the observed results and
provide a more comprehensive understanding of speculative decoding.

(a) Performance verification for the OPT series.

(b) Performance verification for the llama series.

Figure 4: This figure shows that our performance model correctly captures the
real performance of speculative decoding.

##  4 Decoding Speculative Decoding

We begin our analysis with the performance breakdown of speculative decoding,
followed by the development of an analytical model aimed at forecasting the
throughput of speculative decoding. We then explore the optimal approach for
selecting and designing draft models to enhance the efficiency of speculative
decoding.

###  4.1 Latency Breakdown of Speculative Decoding

To understand the throughput of LLMs we first plot a latency breakdown of
speculative decoding in Figure 3, where we show the latency breakdown between
the draft token generation phase and the target model verification phase for
serving OPT-66B model using using different smaller variants of OPT. A similar
figure for LLaMa models (Figure 10) can be found in the Appendix. In Figure 3,
as expected, the time taken by the draft model for token generation increases
with an increase in model sizes, going from 6.23 ms for OPT-125M to 18.56 ms
for OPT-6.7B. However, to achieve throughput benefits, a larger draft model
should provide a higher token acceptance rate to compensate for the higher
token generation cost.

Combining the two observations, (i) In Figure 1 where throughput decreases
even with an increase in TAR for larger models (ii) Increased latency (Figure
3) for larger draft models, we can conclude that a larger draft model, such as
OPT-350M, does not provide sufficient gain in TAR to compensate for increased
latency. This remains true for other models in the OPT family as well.
Appendix A shows more experiments that our observation holds for multiple
different datasets.

Therefore, a larger draft model is only useful if _it provides a sufficient
increase in TAR to compensate for additional latency in generating draft
tokens._ However, this insight does not provide any quantitative information,
e.g., for what magnitude of change in TAR should one switch draft models, nor
does it provide any understanding of how should draft models be designed to
improve the throughput of speculative decoding.

To provide such a tool for researchers and engineers deploying speculative
decoding, we next build an analytical model that can be used to determine the
expected throughput for a given draft model and its average TAR.

###  4.2 Analytical model

Based on our insights on speculative decoding we next propose an analytical
model that can capture the throughput of a speculative decoding model. As
shown in Figure 2, a cycle in speculative decoding can be decomposed into two
phases, draft model autoregressive generation and target model verification.
The target model verification step is identical to the prefill step. Assuming
a setup similar to prior work (Leviathan et al., 2023; Chen et al., 2023; Miao
et al., 2023) where speculative execution of the draft model and target model
verification phases happen sequentially, the performance of speculative
decoding can be decomposed into the following factors,

$\normalsize
Tput=\begin{cases}\displaystyle\frac{TAR}{(t^{d}_{target}+t^{d}_{draft})}&\text{if
}TAR>1,\\\ \displaystyle\frac{1}{(t^{d}_{target}+t^{d}_{draft})}&\text{if
}TAR\leq 1.\end{cases}$

Considering a case where in each iteration, $d$ tokens are generated by the
draft model, $t^{d}_{draft}$ depicts the time taken by draft models to
generate $d$ draft tokens, while $t^{d}_{target}$ is the time taken by the
target model for verifying those $d$ draft tokens. TAR is used to denote the
average number of tokens that were matched across a query or a dataset. This
allows us to determine the throughput of LLM inference using speculative
decoding with a particular draft and target model configuration.

#### Verifying Analytical Model

In Figure 4 we compare the throughput predicted by our model with throughput
measured on real hardware for two different model configurations within the
Llama series: LLaMa-7B and LLaMa-13B as draft models to serve LLaMa-65B for
MMLU, and for five different model configurations within the OPT series:
OPT-125M, OPT-350M, OPT-1.3B, OPT-2.7B, and OPT-6.7B. We run these experiments
on 4 Nvidia 80GB A100 GPUs. We run 100 iterations on the real server, and the
error bars in Figure 4 represent the standard deviation of the measurement.
For the performance model, we collect $t^{d}_{draft}$ and $t^{d}_{target}$ on
a real cluster with a single iteration. For TAR we collect the average Token
Acceptance rate on the MMLU dataset. The maximum deviation we observed between
our proposed analytical model and the results obtained is $3.5\%$. The close
correspondence between our performance model and real measurements shows that
our performance model is accurate in predicting the throughput of speculative
decoding.

#### Discussion.

Despite its simplicity, our analytical model provides a very useful tool.
Users can use the analytical model to accurately determine which draft and
target model pair should be used. They can get this information by just having
access to a sample dataset and the latency for potential candidate models.
Further, as we show in the next couple of sections, our performance model can
be used to evaluate the design space for draft models to be used for
speculative decoding. This is especially helpful as researchers often do not
have access to the production environment and usually just have access to
sample datasets. Using our models, researchers can reason about the
performance of speculative decoding and understand why there is a lack of
speedup and how they can achieve higher speedups.

In the next several sections we will use the performance model to study how to
choose the best draft model among available models and how to design an
optimal draft model. We discuss additional modeling choices and how to
incorporate other works in Section 5.2.

###  4.3 Choosing Draft Models

In this section, with the aid of our analytical model, we provide quantitative
answers to several questions about choosing draft models for a given target
model. First, we study the improvement in TAR a larger draft model needs to be
provided to compensate for the additional inference cost. Next, we study how
much improvement in latency is required to change the choice of the draft
model. Finally, we study the absolute TAR value a model needs to provide to
achieve a specific throughput.

#### Improvement in TAR needed to switch to a larger draft model model.

In our motivating example in Figure 1, we observed that with existing datasets
and models, we are better off with the smallest model as the draft model,
e.g., OPT-125M as the draft model than choosing a larger model. However, there
is a possibility that for new datasets the TAR difference becomes greater.
Using our performance model we can determine how much extra TAR is needed for
larger models to match the throughput of a smaller model. In Figure 5 we plot
the improvement in TAR (extra TAR) which larger models in the OPT model family
should provide to match the throughput of the smallest model (OPT-125M) for
MMLU. We find that for a new workload, if say a 1.3B model can achieve a TAR
advantage greater than 2 over OPT-125M, we would choose the 1.3B model
instead. Furthermore, given that in our scenario the maximum TAR is capped at
8 due to the length of draft token generation, it becomes unfeasible for
OPT-2.7B and OPT-6.7B to surpass OPT-125M in performance. This is because the
improvement needed in TAR for OPT-6.7B to match the throughput of OPT-125M
would exceed this maximum limit.

Figure 5: This figure shows the extra TAR needed for each model to achieve parity throughout with OPT-125M. This figure shows that on MMLU, it is impossible for OPT-2.7B and OPT-6.7B to ever achieve parity throughput with OPT-125M due to the large latency difference. We set draft generation length to 8. Table 1: This table shows the latency reduction needed for larger draft models to achieve parity throughput with OPT-125M on MMLU.  Model | Latency (ms) | Parity Latency | Reduction ($\%$)
---|---|---|---
OPT-125M | 43.7 | 43.7 | 0
OPT-350M | 79.8 | 50.6 | 36.6
OPT-1.3B | 87.1 | 58.7 | 32.6
OPT-2.7B | 114.3 | 49.8 | 56.4
OPT-6.7B | 139.5 | 68.2 | 51.1

#### Improvement in latency for switching to higher TAR model.

As hardware evolves, latency scaling patterns may change with more computing
power and memory bandwidth. Therefore, conclusions drawn on specific hardware
(e.g., A100) may not hold for newer or older hardware (e.g., H100 or V100). To
account for changing hardware, we study how much draft model latency
improvement is needed to achieve throughput parity. To demonstrate this, we
first compute the latency reduction needed for different members in OPT family
to reach the same throughput as the smallest draft model in Table 1. We find
that up to $56\%$ of latency reduction is needed to achieve the same
throughput. For instance, for OPT-1.3B to achieve parity throughput with
OPT-125M, its latency needs to be reduced by 32.9%. Overall this reinforces
our finding that latency reduction provided by the smaller models has
significantly more benefit than the extra TAR provided by a larger draft
model.

#### Required TAR to match throughput.

Using our analytical model we can also predict the TAR necessary for different
models to achieve a target throughput. This can be useful in scenarios where
developers are deploying speculative decoding-based LLMs and have to meet a
certain throughput goal. In Figure 6 we plot TAR needed by existing models to
achieve a specific throughput.

Figure 6: This figure shows the required TAR to achieve a given throughput.

The figure shows that the gap between draft models at each given throughput is
much larger than what we observed in Figure 1. When the throughput requirement
is high, a large draft model, such as OPT-6.7B can’t achieve the desired
throughput. This will allow model designers to quickly judge which draft and
target model pair allows them to meet throughput requirements.

In summary, the above results indicate that to improve the throughput of
speculative decoding, it is necessary to improve the throughput of draft
models, i.e., can we design a model which provides a similar TAR at a lower
inference cost? In the next section, we study the possibility of such a
design.

(a) Model accuracy vs TAR for OPT models on Hellaswag

(b) Model accuracy vs TAR for LLaMa models on Hellaswag

Figure 7: This figure shows the task accuracy versus TAR for OPT and LLaMa
models on Hellaswag. The accuracy numbers are obtained from OpenLLM
Leaderboard (HuggingFace, 2023). Figure 8: Throughput scaling with varying
lookahead length for Hellaswag on OPT series.

###  4.4 Draft Model design

Prior work in language model scaling (Hoffmann et al., 2022; Kaplan et al.,
2020) indicates that model capability is a function of the number of
parameters. However, different model designs with the same number of
parameters can lead to dramatic differences in inference latency. For example,
in Figure 1 we observe a Sheared-LLaMa-2.7B (Xia et al., 2023b) has the same
throughput as LLaMa-7B. Similarly Table 1 we observed that OPT-350M has a
latency of 79.8ms, close to OPT-1.3B’s latency of 87.1ms.

On further analysis, we hypothesize that this unexpected latency pattern could
be an artifact of model design. For example, the designers of Sheared-
LLaMa-2.7B pruned it to have the same number of layers as LLaMa-7B but reduced
the number of attention heads from 32 to 20, intermediate size from 11008 to
6912, and hidden size from 4096 to 2560. Similarly, OPT-350M has the same
number of layers as OPT-1.3B, but the number of attention heads was reduced
from 32 to 16, feed-forward network dimension from 8192 to 4096, hidden size
from 2048 to 1024, and embedding dimension from 2048 to 512. Other design
choices exist such as creating wider models which could better utilize the
hardware compared to deeper models but with the same number of parameters.
This prompted us to test if we can build a wider and shallower network and
study how it affects latency and TAR.

To perform this experiment we used Sheared-LLaMa (Xia et al., 2023b), which
provides a framework to prune larger models to a specific smaller
configuration. Our goal was to start with LLaMa-7B and produce a wider version
of Sheared-LLaMa-1.3B while keeping the number of parameters the same as
Sheared-LLaMa-1.3B. We use the two configurations: the first configuration was
provided by the Sheared-LLaMa authors (NoFT-1.3B-Deep), and the second
configuration (NoFT-1.3B-Wide) was designed by us to optimize for better
speculative decoding throughput. We pruned NoFT-1.3B-Deep and NoFT-1.3B-Wide
from LLaMa-7B using $0.4B$ tokens sampled from the RedPajama Dataset
(Computer, 2023) (the same as in Sheared LLaMA (Xia et al., 2023b)), but
skipped the expensive fine-tuning step on $50B$ more tokens (and hence the
name NoFT). Table 2 shows the detailed configuration of the two models. Using
only $0.8\%$ of tokens used to train Sheared-LLaMa-1.3B, we achieved up to
$30\%$ higher speculative decoding throughput, as shown in Figure 1(b), Figure
1(d), and Figure 9(b).

Table 2: This table shows the model configuration of the two pruned models. Model | # Params | # Layers | # Heads | Inter. size | Dim Model
---|---|---|---|---|---
Deep | 1.345B | 24 | 16 | 5504 | 2048
Wide | 1.334B | 12 | 20 | 9280 | 2560

Table 3 shows the latency and TAR of the two sheared models on MMLU. The deep
variant of the model can achieve $3\%$ higher TAR, but the wide variant
reduces draft latency by $49\%$, improving overall throughput improvement by
$41\%$. We found results are very similar for other datasets, such as Chatbot
Arena (Figure 9(b) in the Appendix) and Hellaswag (Figure 1(d)) This
experiment shows that there is a need to rethink the model design space for
speculative decoding, where we should specifically design models for higher
speculative decoding throughput. Existing model design is more geared towards
providing high accuracy on modeling tasks and in the next section we show how
this goal leads to sub-optimal models for speculative decoding.

Table 3: This table shows the speculative decoding throughput and the latencies to generate 8 tokens using the two pruned draft models. Model | TAR | Latency(ms) | Tput(tokens/s)
---|---|---|---
1.3B Deep | 3.81 | 105.1 | 23.10
1.3B Wide | 3.70 | 53.5 | 32.59

###  4.5 Accuracy vs TAR

To understand the correlation between the performance of a model on an NLP
task and its correlation to TAR, we plot the performance of a model against
the TAR it achieves in Figure 7. Surprisingly, we found that TAR has little
correlation to the model’s performance on a task. We believe this lack of
correlation is due to the majority of tokens in a sentence not being content
words and thus do not affect the accuracy of the model on a specific task. For
example, if a user asks a model: What is the capital of Uruguay? An LLM may
correctly answer: The capital of Uruguay is Montevideo. But a draft model,
without retaining this much knowledge may respond incorrectly: The capital of
Uruguay is Paris. For model accuracy evaluation, this would be a failure.
However, in the context of speculative decoding, this would be a good set of
candidate tokens as the first five words are generated correctly. Therefore,
as shown in Figure 7, TAR increases sub-linearly with an increase in model
size, irrespective of its performance on the task. These results also
highlight the need to design new models focused on speculative decoding.
Results on more datasets can be found in the Appendix (Figure 14).

##  5 Discussion

In this section, we discuss additional details and salient features of our
approach.

###  5.1 Speculative token generation length

The number of speculative tokens generated in a round of speculative decoding
determines the number of tokens the draft models should generate in one round
for verification. The number of speculative tokens generated can also affect
the speedup provided by speculative decoding, e.g., if the draft model can
provide a TAR of six but we are only generating two speculative tokens it will
limit the observed TAR to two. Typically designers choose the number of draft
tokens by trial and error, and we experimentally study how the choice of draft
token length affects throughput. In Figure 8 and Figures 11\- 12 in the
Appendix, we plot the throughput for serving OPT-66B for OPT-125M, OPT-350M,
OPT-1.3B, OPT-2.7B, and OPT-6.7B as draft models on MMLU, Chatbot Arena, and
Hellaswag for different draft token lengths. We observe that the scaling
pattern is very similar across datasets. Appendix A contains more results on
other datasets and LLaMa model families. In all of our other experiments, we
use draft generation length 8, which we find to be optimal for various
scenarios.

###  5.2 Analytical Model

Next, we highlight several different aspects of our performance model
including its limitations.

#### Modeling Choice.

For our analytical model, we chose to use actual runtime on hardware and an
estimated value of TAR. An alternative to using inference time on hardware
would have been the use of FLOPs to estimate time similar to prior works
(Geoffrey et al., 2021; Narayanan et al., 2023). However, these approaches
often lead to sub-optimal estimates of time for new models, and more often
than not FLOPs are not indicative of actual runtime. This is also exemplified
by almost 2x latency disparity between two similarly sized models, NoFT-
deep-1.3b and NoFT-wide-1.3b (Table 3) Consequently, token-wise inference
latency is required to more accurately model speculative decoding performance,
thus trading off accuracy for ease of use.

#### Extending analytical model.

The analytical model described in Section 4.2 currently models the common
setup for speculative decoding described in prior work (Leviathan et al.,
2023; Xia et al., 2023a) where the draft model and target model are run
serially. However, our analytical model can be easily extended to other
techniques that aim to improve speculative decoding. For techniques that aim
to boost TAR (such as OSD (Liu et al., 2023a)), we can rerun our model with
the new TAR. To model methods using multiple draft models in parallel, we can
update the latency to be the maximum latency of all the draft models. For
tree-building techniques, such as SpecInfer (Miao et al., 2023), we can update
the draft latency to be the latency of tree traversal and update the TAR
accordingly. Similarly, for speculative decoding models based on early exit
like Draft&Verify (Zhang et al., 2023), we can update the draft latency to be
the average measured latency across a query or a dataset and update the TAR
accordingly.

##  6 Conclusion

In this work, we conduct a large-scale experimental study to understand how we
can optimize the throughput of speculative decoding. Using our experiments we
outline the various factors which affect the throughput of speculative
decoding. Further, we propose an analytical model to characterize the
performance of speculative decoding and analyze how improvements in model and
hardware would affect the choice of the draft model. Using our analytical
model, we also show how to best design draft model architecture and describe a
modified draft model that outperforms existing draft models by $30\%$ with
just a fraction of the training costs.

## Broader Impact

Our work aims to analyze an LLM inference technique that would boost the
efficiency and throughput of LLM inference. Our work would contribute to
reducing the cost of LLM inference, making a positive environmental impact by
using fewer computational resources. Our work would not alter any LLM output,
and therefore would not generate any additional harm an LLM may cause that we
feel must be specifically highlighted here.

## References

  * Aminabadi et al. (2022) Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al.  Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale.  In _SC22: International Conference for High Performance Computing, Networking, Storage and Analysis_ , pp. 1–15. IEEE, 2022.
  * Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chen et al. (2023) Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J.  Accelerating large language model decoding with speculative sampling.  _arXiv preprint arXiv:2302.01318_ , 2023.
  * Computer (2023) Computer, T.  Redpajama: An open source recipe to reproduce llama training dataset, 2023.  URL https://github.com/togethercomputer/RedPajama-Data.
  * Dao et al. (2023) Dao, T., Haziza, D., Massa, F., and Sizov, G.  Flashdecoding.  https://pytorch.org/blog/flash-decoding/, 2023.  Accessed: January 26, 2024.
  * Geoffrey et al. (2021) Geoffrey, X. Y., Gao, Y., Golikov, P., and Pekhimenko, G.  Habitat: A $\\{$Runtime-Based$\\}$ computational performance predictor for deep neural network training.  In _2021 USENIX Annual Technical Conference (USENIX ATC 21)_ , pp. 503–521, 2021.
  * Hendrycks et al. (2020) Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_ , 2020.
  * Hennessy & Patterson (2011) Hennessy, J. L. and Patterson, D. A.  _Computer architecture: a quantitative approach_.  Elsevier, 2011.
  * Hoffmann et al. (2022) Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_ , 2022.
  * Hong et al. (2023) Hong, K., Dai, G., Xu, J., Mao, Q., Li, X., Liu, J., Chen, K., Dong, H., and Wang, Y.  Flashdecoding++: Faster large language model inference on gpus.  _arXiv preprint arXiv:2311.01282_ , 2023.
  * HuggingFace (2023) HuggingFace.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023.  Accessed: January 26, 2024.
  * Kaplan et al. (2020) Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Kwon et al. (2023) Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I.  Efficient memory management for large language model serving with pagedattention.  In _Proceedings of the 29th Symposium on Operating Systems Principles_ , pp. 611–626, 2023.
  * Leviathan et al. (2023) Leviathan, Y., Kalman, M., and Matias, Y.  Fast inference from transformers via speculative decoding.  In _International Conference on Machine Learning_ , pp. 19274–19286. PMLR, 2023.
  * Liu et al. (2023a) Liu, X., Hu, L., Bailis, P., Stoica, I., Deng, Z., Cheung, A., and Zhang, H.  Online speculative decoding.  _arXiv preprint arXiv:2310.07177_ , 2023a.
  * Liu et al. (2023b) Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Re, C., et al.  Deja vu: Contextual sparsity for efficient llms at inference time.  In _International Conference on Machine Learning_ , pp. 22137–22176. PMLR, 2023b.
  * Miao et al. (2023) Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Wong, R. Y. Y., Chen, Z., Arfeen, D., Abhyankar, R., and Jia, Z.  Specinfer: Accelerating generative llm serving with speculative inference and token tree verification.  _arXiv preprint arXiv:2305.09781_ , 2023.
  * Microsoft (2023) Microsoft.  Deepspeed.  https://github.com/microsoft/deepspeed, 2023.  Accessed: January 26, 2024.
  * Narayanan et al. (2023) Narayanan, D., Santhanam, K., Henderson, P., Bommasani, R., Lee, T., and Liang, P.  Cheaply estimating inference efficiency metrics for autoregressive transformer models.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Schuster et al. (2022) Schuster, T., Fisch, A., Gupta, J., Dehghani, M., Bahri, D., Tran, V., Tay, Y., and Metzler, D.  Confident adaptive language modeling.  _Advances in Neural Information Processing Systems_ , 35:17456–17472, 2022.
  * Stern et al. (2018) Stern, M., Shazeer, N., and Uszkoreit, J.  Blockwise parallel decoding for deep autoregressive models.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Su et al. (2023) Su, Q., Giannoula, C., and Pekhimenko, G.  The synergy of speculative decoding and batching in serving large language models.  _arXiv preprint arXiv:2310.18813_ , 2023.
  * Touvron et al. (2023a) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. (2023b) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Wang et al. (2020) Wang, X., Xiong, Y., Wei, Y., Wang, M., and Li, L.  Lightseq: A high performance inference library for transformers.  _arXiv preprint arXiv:2010.13887_ , 2020.
  * Xia et al. (2023a) Xia, H., Ge, T., Wang, P., Chen, S.-Q., Wei, F., and Sui, Z.  Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pp. 3909–3925, 2023a.
  * Xia et al. (2023b) Xia, M., Gao, T., Zeng, Z., and Chen, D.  Sheared llama: Accelerating language model pre-training via structured pruning.  _arXiv preprint arXiv:2310.06694_ , 2023b.
  * Yang et al. (2023) Yang, S., Lee, G., Cho, J., Papailiopoulos, D., and Lee, K.  Predictive pipelined decoding: A compute-latency trade-off for exact llm decoding.  _arXiv preprint arXiv:2307.05908_ , 2023.
  * Yu et al. (2022) Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G.  Orca: A distributed serving system for $\\{$Transformer-Based$\\}$ generative models.  In _16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)_ , pp. 521–538, 2022.
  * Zellers et al. (2019) Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y.  Hellaswag: Can a machine really finish your sentence?  _arXiv preprint arXiv:1905.07830_ , 2019.
  * Zhang et al. (2023) Zhang, J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., and Mehrotra, S.  Draft & verify: Lossless large language model acceleration via self-speculative decoding.  _arXiv preprint arXiv:2309.08168_ , 2023.
  * Zhang et al. (2022) Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ , 2022.
  * Zheng et al. (2023) Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _arXiv preprint arXiv:2306.05685_ , 2023.

##  Appendix A More experiment results

In this section, we show more experimental analysis of speculative decoding.
In Figure 9, we plot the throughput of OPT and LLaMa models against its TAR on
Chatbot Arena. This figure shows that as model size increases, throughput
generally decreases due to significantly higher inference latency, despite
consistent increases in TAR.

(a) Throughput scaling with increasing TAR in OPT series on Chatbot Arena

(b) Throughput scaling with increasing TAR in LLaMa series on Chatbot Arena

Figure 9: This figure shows the throughput scaling against TAR for Chatbot
Arena.

In Figure 10, we plot the throughput of OPT and LLaMa models against its TAR
on Chatbot Arena. This figure shows that draft latency occupies a large chunk
of time in a speculative decoding iteration, opening up new avenues for
designing draft models optimal for speculative decoding.

Figure 10: Performance Breakdown of LLaMa speculative decoding, lookahead
length is set to be 8.

In Figure 11, we plot the throughput of OPT and LLaMa models against varying
draft model lookahead lengths on MMLU. This figure shows that throughput
plateaus when lookahead length reaches $6$ to $8$, depending on the model.
Larger draft models tend to have a smaller optimal lookahead length due to
higher latency at each token generation step.

(a) Throughput scaling with varying lookahead length for 5-shot MMLU on LLaMa
series.

(b) Throughput scaling with varying lookahead length for 5-shot MMLU on OPT
series.

Figure 11: This figure shows the throughput scaling with varying lookahead
lengths on MMLU.

In Figure 12, we plot the throughput and TAR of OPT models against varying
draft model lookahead lengths on Chatbot Arena. This figure shows that while
TAR continues increasing at higher lookahead lengths, throughput plateaus when
lookahead length reaches $6$ to $8$, depending on the model.

(a) Throughput scaling with varying lookahead length for Chatbot Arena on OPT
series.

(b) TAR scaling with varying lookahead length for Chatbot Arena

Figure 12: This figure shows the throughput and TAR of OPT models against
varying draft model lookahead lengths on Chatbot Arena.

In Figure 13, we plot the TAR of OPT models against varying draft model
lookahead lengths on MMLU and Hellaswag. This figure shows that TAR increases
close to linearly when the lookahead length is low and becomes sublinear on a
higher lookahead length.

(a) TAR scaling with varying lookahead length for 5-shot MMLU.

(b) TAR scaling with varying lookahead length for Hellaswag.

Figure 13: This figure shows the TAR of OPT models against varying draft model
lookahead lengths on MMLU and Hellaswag.

In Figure 14, we plot the task accuracy versus TAR for OPT and LLaMa models on
MMLU. The accuracy numbers are obtained from OpenLLM Leaderboard (HuggingFace,
2023). This figure shows that task accuracy is irrelevant to TAR.

(a) Model accuracy vs TAR for OPT models on MMLU and Hellaswag

(b) Model accuracy vs TAR for LLaMa models on MMLU and Hellaswag

Figure 14: This figure shows the task accuracy versus TAR for OPT and LLaMa
models on MMLU. The accuracy numbers are obtained from OpenLLM Leaderboard
(HuggingFace, 2023).