# Draft & Verify: Lossless Large Language Model Acceleration via Self-
Speculative Decoding

Jun Zhang1, Jue Wang1, Huan Li1, Lidan Shou1,
Ke Chen1, Gang Chen1, Sharad Mehrotra2
1College of Computer Science and Technology, Zhejiang University
2School of Information and Computer Science at University of California,
Irvine
{zj.cs, zjuwangjue, lihuan.cs, should, chenk<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

We present a novel inference scheme, self-speculative decoding, for
accelerating Large Language Models (LLMs) without the need for an auxiliary
model. This approach is characterized by a two-stage process: drafting and
verification. The drafting stage generates draft tokens at a slightly lower
quality but more quickly, which is achieved by selectively skipping certain
intermediate layers during drafting. Subsequently, the verification stage
employs the original LLM to validate those draft output tokens in one forward
pass. This process ensures the final output remains identical to that produced
by the unaltered LLM, thereby maintaining output quality. The proposed method
requires no additional neural network training and no extra memory footprint,
making it a plug-and-play and cost-effective solution for inference
acceleration. Benchmarks with LLaMA-2 and its fine-tuned models demonstrated a
speedup up to 1.73$\times$.111Code is available at https://github.com/dilab-
zju/self-speculative-decoding.

##  1 Introduction

Transformer-based Large Language Models (LLMs), such as GPT-3/4, PaLM, and
LLaMA, have been widely adopted in various real-world applications Bommasani
et al. (2021); Liang et al. (2022); Brown et al. (2020); Min et al. (2022);
Chan et al. (2022); Touvron et al. (2023). However, their substantial
inference costs have raised significant concerns, especially for latency-
sensitive scenarios Pope et al. (2022). The main efficiency bottleneck is the
autoregressive decoding process. This process decodes each output token
sequentially, leading to a high number of Transformer calls; furthermore, each
Transformer call is typically memory bandwidth-bound, resulting in low
computation utility and thus longer wall-clock time Shazeer (2019). For
instance, decoding 128 tokens autoregressively using LLaMA-2-13B on an A100
GPU can take up to 100$\times$ longer than a sequence-level forward pass on
the same number of tokens, highlighting the substantial inefficiency inherent
in the current decoding process.

Established model compression techniques such as quantization Han et al.
(2015), pruning Molchanov et al. (2016), and distillation Hinton et al. (2015)
have been employed to alleviate these costs. While these solutions have proven
extremely effective, they usually require changing the model architecture,
changing the training-procedure, re-training or fine-tuning the models, and do
not maintain identical outputs.

In parallel to model compression, speculative execution is being explored to
accelerate the autoregressive decoding process Leviathan et al. (2023); Chen
et al. (2023). These methods train an auxiliary draft model that can quickly
generate some draft output tokens. Subsequently, the original LLM, referred to
as the verify model, then checks the acceptability of these draft tokens with
one single forward pass. This verification step ensures that the outputs are
derived from the original LLM’s probability distribution.

However, an essential issue of existing speculative execution methods is the
need to identify or train a suitable draft model that can generate outputs
consistent with the verify model. It becomes more tricky when the LLM is
already a fine-tuned model, e.g. LLaMA-2-Chat Touvron et al. (2023), CodeLLaMA
Rozière et al. (2023). How to find or train a draft model that can effectively
mimic the outputs of such a tailored model is a formidable task, with no
straightforward or guaranteed solutions. Furthermore, the introduction of an
additional draft model escalates the GPU memory overhead, increasing
deployment challenges particularly on devices with restricted memory capacity.

Figure 1: Visualization of the self-speculative decoding process. The blue
line represents the inference path of the original model, while the green line
depicts the path taken during the drafting stage. The verification stage
evaluates all drafted tokens in a single forward pass, with accepted tokens
marked in green and rejected tokens highlighted in red. Each verification step
also involves the prediction of a token, which is denoted in blue.

In this paper, we present self-speculative decoding, a novel approach to
accelerate the inference of LLMs. This method builds on the principles of
speculative execution, but with a unique twist: it utilizes one LLM for both
drafting and verification stages. The key insight driving our approach is the
observation that skipping some layers in LLMs does not significantly
compromise the quality of generation Liu et al. (2023). Therefore, by
selectively bypassing some of its intermediate layers, we can use the LLM
itself to generate draft tokens. Following this, the original LLM verifies
these draft tokens in a single forward pass. Figure 1 illustrates this two-
stage decoding process. The blue arrow indicates the inference path of the
original model, while the green arrow depicts the inference path during the
drafting stage. Notably, both inference paths share the same model so we do
not need to deploy a standalone draft model with additional memory overhead.

Figure 2: Illustration of the impact of the number of draft tokens ($K$) and
acceptance rate of draft tokens on end-to-end speedup. We assume that the
draft model is 2$\times$ faster than the verify model.

Implementing self-speculative decoding poses two main challenges: (a)
determining which layers and the number of layers to skip during drafting, and
(b) deciding the timing to stop generating draft tokens. By framing the first
challenge as an optimization problem, we leverage Bayesian optimization Jones
et al. (1998) to systematically select a subset of layers to skip. This
approach allows us to strike a balance between computational efficiency and
output quality, ensuring that the skipped layers do not significantly
compromise the drafting quality. The second challenge pertains to determining
the optimal number of draft tokens ($K$) to generate. As shown in Figure 2,
the choice of $K$ significantly influences the end-to-end speedup: for an
acceptance rate below 80%, $K=1$ is optimal, and for rates above 80%, a larger
$K$ is necessary. This observation underscores that a static $K$ is not
universally applicable. To tackle this variability, we introduce an adaptive
draft-exiting mechanism, which stops generating draft tokens once its
confidence level drops below a threshold. This intervention prevents
unnecessary computation and potential discard of additional draft tokens,
thereby enhancing efficiency.

To summarize, our main contributions are: (1) Inference scheme: we propose
self-speculative decoding, a practical, plug-and-play solution for inference
acceleration that does not require further neural network training and avoids
additional memory overhead; (2) Optimization strategies: we adopt Bayesian
optimization to select which layers to skip during drafting and propose a
simple yet effective method to adaptively determine the number of draft
tokens; (3) Evaluation: we evaluate our method on text summarization and code
generation tasks, and the experimental results indicate that our method can
achieve up to 1.73$\times$ in end-to-end speedup.

##  2 Related Work

##### Transformer-based LLM inference.

As LLMs continue to evolve rapidly, we are seeing a surge of systems
specifically engineered for LLM inference, including Faster Transformer NVIDIA
, Orca Yu et al. (2022), LightSeq Wang et al. (2021), PaLM inference Pope et
al. (2022), TurboTransformers Fang et al. (2021), Deepspeed Inference
Aminabadi et al. (2022), FlexGen Sheng et al. (2023), Text Generation
Inference HuggingFace (2023), etc. The token generation phase typically takes
up the majority of the end-to-end inference time compared to the prompting
encoding phase. Despite the introduction of system optimizations by those
state-of-the-art systems to improve the inference speed, there is still a gap
in the careful co-design of algorithms and systems. This is necessary to fully
harness the potential of hardware efficiency during LLM inference computation.

##### Model Compression.

Various model compression methods have been studied for model inference. For
example, quantization Han et al. (2015); Jacob et al. (2018); Nagel et al.
(2019); Zhao et al. (2019); Yao et al. (2022); Park et al. (2022); Dettmers et
al. (2022); Xiao et al. (2022); Frantar et al. (2022), pruning or
sparsification Molchanov et al. (2016); Liu et al. (2018); He et al. (2019);
Hoefler et al. (2021); Frantar and Alistarh (2023); Liu et al. (2023); Bansal
et al. (2022), and distillation Hinton et al. (2015); Cho and Hariharan
(2019); Tang et al. (2019); Touvron et al. (2021) have been applied to speed
up the inference of the machine learning model, particularly LLMs. While these
solutions are extremely effective, they often necessitate modifications to the
model architecture and the training procedure. This usually involves re-
training or fine-tuning the models. And it is important to note that these
methods do not result in identical outputs.

##### Speculative Execution.

Speculative execution Burton (1985); Hennessy and Patterson (2011) is an
optimization technique employed in computer architecture where a system
performs some task in advance if that task is known to be required after the
previous task. Speculative decoding Chen et al. (2023); Leviathan et al.
(2023) has been proposed as an effective strategy to boost inference speed of
LLMs. Previously, Stern et al. (2018) proposed to use block-wise parallel
decoding to accelerate greedy decoding of attention models. However, these
methods need to train or select a high-quality draft model, and also result in
increased memory overhead. Yang et al. (2023) designed a method for LLMs that
copies reference text tokens to the decoder, validating them via output
probabilities. However, its dependence on repetitiveness restricts its use in
general scenario generation. In contrast, our approach does not incur
additional memory overhead and does not hinge on explicit assumptions about
data distribution.

##  3 Method

In this section, we first provide a concise overview of the standard
autoregressive decoding. Subsequently, we provide a detailed depiction of our
proposed method, including an explanation of our technique for selectively
skipping layers during drafting, and our strategy for adaptively determining
the number of draft tokens to generate.

###  3.1 Standard Autoregressive Decoding

Algorithm 1 Autoregressive Decoding (Greedy)

Given model $p(x|x_{1},...,x_{t})$, prompt $x_{1},...,x_{t}$ and target
sequence length $T$.

for i = t, …, T-1 do

$x_{i+1}\leftarrow\operatorname*{arg\,max}{p(x|x_{1},...,x_{i})}$

end for

return $x_{1},...,x_{T}$

Transformer-based LLMs typically follows an autoregressive decoding process.
Given a user prompt sequence $x_{1},...,x_{t}$, the model calculates the
probability distribution of the next token, denoted as $p(x|x_{1},...,x_{t})$.
We present a greedy decoding process in Algorithm 1. In practice, instead of
always choosing the token with the highest probability (as in greedy
decoding), we can sample tokens based on their predicted probability
distribution, which introduces some randomness and generates more diverse
outputs.

Theoretically, the computational load of autoregressive decoding is comparable
to that of sequence-level forward processing for an equivalent number of
tokens.222 In fact, due to the causal nature of language modeling,
autoregressive decoding could potentially save some attention computation.
However, this decoding process is significantly memory bandwidth-bound. When
decoding each token, all the model parameters need to pass through the
accelerator chip. So the model size divided by the memory bandwidth gives a
hard ceiling on the decoding speed, resulting in a much longer inference time.

###  3.2 Self-Speculative Decoding

To mitigate the inherent inefficiency of autoregressive decoding, speculative
decoding can be employed to enhance the inference speed of LLMs. This strategy
involves two models: an LLM that we want to optimize, and a draft model that
runs faster, albeit potentially at a lower quality. The speculative decoding
process can be explained as a two-stage process: (1) drafting: the draft model
first generates $K$ draft tokens from a given prompt sequence
$x_{1},...,x_{i}$, denoted as $x_{i+1},...,x_{i+K}$. (2) verification:
following the drafting stage, the original LLM is then employed to validate
these draft tokens. This validation is accomplished in a single forward pass,
where the LLM predicts the probability distributions for each draft token and
assesses whether they align with the draft tokens. Once a draft token $x_{j}$
is not validated, we use the original LLM’s prediction to override $x_{j}$,
and start the next round of drafting beginning from token $x_{j+1}$.

The above process is based on the observation that computing the forward pass
of a short continuation of tokens in parallel has a very similar latency to
that of sampling a single token. Consequently, the verification stage could be
significantly more efficient than decoding tokens using the original LLM in
standard autoregressive decoding.

Algorithm 2 Self-Speculative Decoding (Greedy)

LLM $p(x|\boldsymbol{z}^{*},x_{1},...,x_{t})$ where $x_{1},...,x_{t}$ is the
prompt, $\boldsymbol{z}^{*}$ is a vector that represents the specific layers
to bypass; target sequence length $T$; max draft tokens to generate $K$. We
denote the original LLM as $p(x|\vec{0},x_{1},...,x_{t})$, where $\vec{0}$ is
a zero vector, indicating all layers are used in inference.

$i\leftarrow t$

while $i<T$ do

for $j\leftarrow i,...,i+K$ do $\triangleright$ Drafting Stage

$x_{j+1}\leftarrow\operatorname*{arg\,max}{p(x|\boldsymbol{z}^{*},x_{1},...,x_{j})}$

if need to exit drafting (§3.4) then

Break

end if

end for

for $i\leftarrow i,...,j$ do $\triangleright$ Verification Stage

if $x_{i+1}\neq\operatorname*{arg\,max}{p(x|\vec{0},x_{1},...,x_{i})}$ then

$x_{i+1}\leftarrow\operatorname*{arg\,max}{p(x|\vec{0},x_{1},...,x_{i})}$

Break

end if

end for

$i\leftarrow i+1$

If all draft tokens are accepted, generate next token
$x_{i+1}\leftarrow\operatorname*{arg\,max}{p(x|\vec{0},x_{1},...,x_{i})}$ and
$i\leftarrow i+1$

end while

return $x_{1},...,x_{T}$

In contrast to existing methods that use a standalone draft model to obtain
draft tokens, our paper proposes a novel ‘self-speculative’ approach. We
employ the original LLM itself for both the drafting and verification stages.
During the drafting stage, the LLM selectively skips some of its intermediate
layers so as to generate draft tokens quicker. Subsequently, these draft
tokens are verified by the original LLM. Algorithm 2 presents a detailed
description of the greedy decoding process. A sampling-based decoding process
is elaborated in the Appendix.

Despite the simplicity of the main idea of self-speculative decoding, it poses
several challenges:

Challenge 1: First, it is non-trivial to determine which layers and the number
of layers to skip during drafting. If an excessive number of layers are
skipped, the quality of the draft could be significantly compromised. This
could result in a low acceptance rate in the verification stage, consequently
increasing the overall inference time. On the other hand, if fewer layers are
skipped, it ensures a higher acceptance, but also caps the maximum speedup
that could be achieved.

Challenge 2: It is hard to decide when to stop the generation of draft tokens.
In speculative decoding, if a draft token is rejected, all subsequent draft
tokens are discarded. Therefore, generating an excessive number of draft
tokens could lead to unnecessary computational effort, thereby increasing the
end-to-end inference time.

In sections 3.3 and 3.4, we will detail our approach to address these two
challenges respectively.

###  3.3 Selection of Skipped Layers

While skipping more layers can expedite the drafting process, it also carries
the risk of lowering the token acceptance rate in the verification stage,
consequently increasing the overall end-to-end inference time. In this
subsection, we frame the layer selection process as an optimization problem,
with an objective of minimizing the average inference time per token.

Objective Function: The black box function, which we aim to minimize, is the
average inference time per token. This function takes as input the number and
specific layers to skip and returns the average inference time per token on a
development set. We represent this function as $f(\boldsymbol{z})$, where
$\boldsymbol{z}$ is a vector that represents the number and specific layers to
skip.

Input Space: The input space is the set of all possible combinations of layers
that can be skipped. If there are $L$ layers in the model, including both
attention and MLP layers, the input space is the power set of the set
$\\{1,2,...,L\\}$.

Optimization Goal: The goal of the optimization is to find the input
$\boldsymbol{z}^{*}$ that minimizes the objective function
$f(\boldsymbol{z})$, which means we want to find the combination of layers to
skip that results in the lowest inference time. The optimization problem can
be formally expressed as:

$\displaystyle\boldsymbol{z}^{*}=\operatorname*{arg\,min}_{\boldsymbol{z}}f(\boldsymbol{z})\quad
s.t.\ \boldsymbol{z}\in\\{0,1\\}^{L}$ (1)

For smaller models with a manageable solving space, a brute force search could
easily reach the globally optimal solution. However, for LLMs with numerous
layers ($L=160$ for LLaMA-2-70B), this method becomes prohibitively expensive.

Figure 3: Illustration of using Bayesian optimization to search the best
combination of skip layers that results in the lowest average token inference
time.

In such cases, Bayesian optimization can be employed to tackle this problem
Jones et al. (1998). As shown in Figure 3, it iteratively selects new inputs
$\boldsymbol{z}^{*}$ for evaluation, based on a surrogate model of the
objective function and an acquisition function. The latter balances
exploration (testing inputs where the model’s prediction is uncertain) and
exploitation (testing inputs where the model anticipates a favorable result).
Gaussian process is utilized as the surrogate model Rasmussen et al. (2006).
This procedure continues until a predetermined number of iterations is
reached. We use the obtained $\boldsymbol{z}^{*}$ to accelerate text
generation, and $\boldsymbol{z}^{*}$ is fixed for each model without further
updating.

Our empirical findings validate the effectiveness of this approach.
Furthermore, this entire procedure only involves forward passes, making it
highly memory and compute-efficient.

##### Discussion

We use skipping intermediate layers as a straightforward method to accelerate
the drafting stage. However, it is worth noting that there are other potential
strategies to accelerate drafting, such as quantization and structured
pruning. While such compression techniques could potentially enhance the
drafting stage’s speed, they do not provide a speed-up proportional to the
compression ratio. Moreover, since a standalone copy of the quantized or
pruned model parameters need be stored, they might require additional memory
overhead. This requirement contradicts a key advantage in our approach, namely
no extra memory overhead. Therefore, we adopt layer skipping in our approach.
Future research could explore other drafting techniques that uphold these
benefits while enhancing speed.

###  3.4 Adaptive Draft-Exiting Mechanism

Our self-speculative decoding approach incorporates an adaptive draft-exiting
mechanism to enhance computational efficiency during the drafting stage. In
speculative decoding, if a draft token is rejected, all subsequent draft
tokens are discarded accordingly. The draft-exiting mechanism prevents the
wasteful allocation of computational resources towards draft tokens that are
less likely to be accepted in the verification stage.

This mechanism evaluates the predicted probability of each draft token against
a threshold $\gamma$. If the predicted probability falls below this threshold
such that $p(x_{t+1}|x_{1},...,x_{t})<\gamma$, indicating a low confidence
score, it immediately stops drafting. This strategic approach ensures optimal
use of computing by focusing on the generation and verification of high-
quality tokens, thereby improving the overall efficiency of the decoding
process.

Moreover, it is worth noting that a static threshold may not accurately
reflect the actual acceptance rate between the drafting and verification
stages. For example, more challenging examples with a lower acceptance rate
would be better served by a higher $\gamma$. To avoid the need for case-by-
case threshold determination, we use an adaptive threshold which adjusts
dynamically according to an updating rule, thereby allowing for an accurate
reflection of the acceptance rate and better handling of examples in different
difficulties. We denote the acceptance rate at $e$-th drafting stage as
$AR_{e}$. Consequently, the update rule is defined as follows:

$\displaystyle{AR}_{\text{ma}}$
$\displaystyle=\beta_{1}{AR}_{\text{ma}}+(1-\beta_{1})AR_{e}$ (2)
$\displaystyle\tilde{\gamma}$
$\displaystyle=\begin{cases}\gamma+\epsilon,&\text{if}\
{AR}_{\text{ma}}\leq\alpha\\\ \gamma-\epsilon,&\text{otherwise}\end{cases}$
(3) $\displaystyle\gamma$
$\displaystyle=\beta_{2}\gamma+(1-\beta_{2})\tilde{\gamma}$ (4)

where $\alpha$ represents a target acceptance rate, $\epsilon$ is the update
step-size, and $\beta_{1}$ and $\beta_{2}$ are factors designed to mitigate
fluctuations of $\gamma$ and $AR$ respectively. Notably, when $e$ is 1,
$\beta_{1}=0$. We update $\gamma$ after each verification stage. This updating
rule ensures that the acceptance rate remains in close proximity to a target
acceptance rate $\alpha$.

##  4 Evaluation

###  4.1 Setup

We evaluate a diverse range of models including LLaMA-2-13B, LLaMA-2-13B-Chat,
CodeLLaMA-13B, and LLaMA-2-70B, enabling us to assess the performance across
various domains and tasks.

We perform Bayesian optimization333 We used the implementation of
https://github.com/bayesian-optimization/BayesianOptimization.  for 1000
iterations to select the intermediate layers to skip in the drafting stage
444We report the offline Bayesian optimization time of different models in
Appendix B.. We randomly select 8 instances from the train set and use them to
evaluate the inference time per token for Bayesian optimization. For the
adaptive draft-exiting mechanism, we set the initial threshold $\gamma=0.6$,
$\epsilon=0.01$, $\beta_{1}=0.5$, $\beta_{2}=0.9$, and $\alpha$ is slightly
tuned for the data and model, as detailed in Table 6. For random sampling-
based inference, we by default use $top\\_p=0.85$ for text summarization
tasks, and $0.95$ for code generation tasks.

We use an A100-40GB to conduct experiments for LLaMA-2-13B, LLaMA-2-13B-Chat,
and CodeLLaMA-13B. We use two A100-80GB with HuggingFace’s accelerate555
https://github.com/huggingface/accelerate to conduct experiments for
LLaMA-2-70B.

####  4.1.1 Data

The datasets that we have selected for evaluation are CNN/Daily Mail (CNN/DM),
Extreme Summarization (XSum), and HumanEval. These tasks cover the evaluation
of text and code generation capabilities. We perform 1-shot evaluation for
CNN/DM and XSum, and compare the ROUGE-2. We compare pass@1 and pass@10 for
HumanEval. We randomly sample 1000 instances from the testset for CNN/DM and
XSum.

###  4.2 Main Results

Model | Decoding Method | Temperature | CNN/DM | XSum
---|---|---|---|---
ROUGE-2 | Speedup | ROUGE-2 | Speedup
LLaMA-2-13B | Autoregressive | 0.0 | 0.107 | 1.000$\times$ | 0.125 | 1.000$\times$
LLaMA-2-13B | Self-Speculative | 0.0 | 0.107 | 1.557$\times$ | 0.126 | 1.478$\times$
LLaMA-2-13B | Autoregressive | 0.2 | 0.111 | 1.000$\times$ | 0.117 | 1.000$\times$
LLaMA-2-13B | Self-Speculative | 0.2 | 0.111 | 1.529$\times$ | 0.117 | 1.433$\times$
LLaMA-2-13B-Chat | Autoregressive | 0.0 | 0.143 | 1.000$\times$ | 0.109 | 1.000$\times$
LLaMA-2-13B-Chat | Self-Speculative | 0.0 | 0.142 | 1.411$\times$ | 0.109 | 1.241$\times$
LLaMA-2-13B-Chat | Autoregressive | 0.2 | 0.143 | 1.000$\times$ | 0.105 | 1.000$\times$
LLaMA-2-13B-Chat | Self-Speculative | 0.2 | 0.144 | 1.392$\times$ | 0.107 | 1.220$\times$
LLaMA-2-70B | Autoregressive | 0.0 | 0.130 | 1.000$\times$ | 0.118 | 1.000$\times$
LLaMA-2-70B | Self-Speculative | 0.0 | 0.130 | 1.731$\times$ | 0.118 | 1.475$\times$
LLaMA-2-70B | Autoregressive | 0.2 | 0.131 | 1.000$\times$ | 0.108 | 1.000$\times$
LLaMA-2-70B | Self-Speculative | 0.2 | 0.131 | 1.722$\times$ | 0.110 | 1.435$\times$
Table 1: Evaluation on text generation tasks. ‘Speedup’ represents the acceleration of average inference time per token compared to the ‘Autoregressive’ baseline on the same setting. Model | Decoding Method | HumanEval | Speedup
---|---|---|---
CodeLLaMA-13B | Autoregressive | pass@1 | 0.311 | 1.000$\times$
CodeLLaMA-13B | Self-Speculative | pass@1 | 0.317 | 1.456$\times$
CodeLLaMA-13B | Autoregressive | pass@10 | 0.659 | 1.000$\times$
CodeLLaMA-13B | Self-Speculative | pass@10 | 0.659 | 1.345$\times$
Table 2: Evaluation on code generation tasks. We use greedy decoding for
pass@1 and random sampling with a temperature of 0.6 for pass@10.

We evaluate the performance of our decoding scheme, denoted as ‘Self-
Speculative’, with both greedy decoding (temperature = 0.0) and random
sampling (temperature = 0.2/0.6) versions, across text generation and code
generation. The baseline is ‘Autoregressive’, which uses the original model to
perform standard autoregressive decoding. The experiments involve models
spanning various scales of LLaMa-2 and its fine-tuned models. The summarized
results can be found in Tables 1 and 2.

For text generation tasks, the results presented in Table 1 demonstrate that
our method, when applied with temperature settings of 0.0 and 0.2 achieves
considerable speedups ranging from 22% to 73%. Another important observation
from these results is the minimal to nonexistent loss in ROUGE-2 rouge666 We
attribute any slight differences observed in the case of greedy decoding to
numerical rounding errors. , which verifies one of the core advantages of our
decoding scheme, namely consistent output quality. In particular, our approach
can be effectively applied on LLaMA-2-13B-Chat, a fine-tuned LLaMA-2-13B for
conversation scenarios, indicating the compatibility of our method with fine-
tuned models. This effectively addresses the dependency of the original
speculative decoding method on high-quality draft models, which can be
challenging to train and obtain, especially for fine-tuned models.
Furthermore, the higher speedup achieved on LLaMA-2-70B suggests that larger
models introduce more redundancy. This allows the drafting stage to skip a
larger percentage of intermediate layers, thereby enhancing the overall
speedup.

To evaluate the performance of our method in code generation tasks, we
utilized CodeLLaMA-13B, another fine-tuned variant of LLaMA-2-13B optimized
for code generation. The assessment was carried out using the HumanEval
benchmark. Table 2 shows that our variants achieve speedups of 1.345$\times$
and 1.456$\times$, respectively, while maintaining similar task scores in
terms of pass@1 and pass@10. This further validates the model compatibility of
our decoding scheme in the field of coding.

###  4.3 Impact of Skipped Layer Selection

Figure 4: Speedup vs the number of skipped layers. These results are derived
from the Bayesian optimization process.

In this subsection, we investigate the relationship between the number of
skipped layers during the drafting stage and the resultant end-to-end speedup
in our method. Our experiments are conducted on the LLaMA-2-13B model, which
comprises 80 layers. Throughout the Bayesian optimization process, we tracked
the number of layers skipped, denoted as $||\boldsymbol{z}^{*}||$, and the
corresponding end-to-end speedup relative to the autoregressive baseline.

The results are presented in Figure 4, where a dashed line represents the
maximum speedup for runs that skip the same number of layers. We have several
observations: (1) The peak end-to-end speedup is observed when about half of
the layers are skipped during the drafting stage; (2) The specific combination
of layers skipped also plays a significant role. In particular, an
inappropriate combination of skipped layers can actually result in a decrease
in the end-to-end inference speed. (3) There is a noticeable drop in speedup
when more than 42 layers are skipped. This suggests that the quality of
drafting significantly deteriorates when an excessive number of layers are
omitted.

These findings indicate the importance of careful layer selection in the
implementation of our self-speculative decoding.

###  4.4 Effectiveness of Draft-Exiting

Here we explore the effectiveness of the adaptive draft exit mechanism,
specifically examining whether a threshold needs to be set and whether a
static threshold is sufficient. Our settings are LLaMA-2-13B, CNN/DM, and
greedy decoding.

Fixed Number of Draft Tokens. We first evaluate a variant self-speculative
decoding where the number of tokens generated at each drafting stage is always
equal to $K$. Table 3 illustrates the speedup of these variants for different
max draft token values $K$, showing an initial increase and then a decrease.
This trend can be attributed to the fact that an excessively large $K$ ($K=8$)
generates a substantial number of tokens that are likely to fail in the
verification stage. This is demonstrated by its lower acceptance rate (AR) of
only 0.760, which results in squandered computational resources during the
drafting stage and a consequent reduction in speedup.

Although choosing an appropriate $K$, such as $K=4$, can alleviate the above
problem to some extent, this static setting cannot fully utilize the
capabilities of the draft model and only achieves a limited speed
(1.45$\times$). For example, we should use a larger $K$ for simple instances
and a smaller $K$ for difficult instances. In addition, Table 3 also shows
that the acceptance rate and speedup are not directly proportional. When $K$
is 2, the acceptance rate reaches the highest 0.925, but the acceleration is
only 1.38$\times$. We believe this is because $K$ is too small, which is a
serious underestimate of the ability of the draft model, resulting in cases
where more valid draft tokens could have been generated but were not, thereby
limiting the overall speedup.

Draft-Exiting with Static Threshold. Another variant is to stop generating
draft tokens if the confidence score falls below a predefined static
threshold. Table 4 shows that different static thresholds have large
differences in acceleration (1.40$\times$~1.55$\times$). This indicates the
importance of adaptively determining the appropriate threshold to optimize the
speedup. Specifically, we observe that high thresholds ($\gamma$=0.8) tend to
underestimate the capabilities of the drafting model. Despite a high
acceptance rate (AR=0.930), this does not necessarily result in the best
speedup due to a reduced number of drafting tokens (1.55$\times$). Conversely,
a lower threshold ($\gamma$=0.2) tends to overestimate the drafting model’s
capabilities, leading to a significantly lower acceptance rate (AR=0.765),
wasting computational resources during the drafting stage, and thereby leading
to slower inference speed (1.40$\times$).

$K$ | 2 | 4 | 6 | 8 | Adaptive
---|---|---|---|---|---
ROUGE-2 | 0.105 | 0.106 | 0.106 | 0.105 | 0.107
AR | 0.925 | 0.866 | 0.813 | 0.760 | 0.911
Speedup | 1.38$\times$ | 1.45$\times$ | 1.43$\times$ | 1.38$\times$ | 1.56$\times$
Table 3: Drafting with different $K$ values.  $\gamma$ | 0.2 | 0.4 | 0.6 | 0.8 | Adaptive
---|---|---|---|---|---
ROUGE-2 | 0.106 | 0.106 | 0.106 | 0.107 | 0.107
AR | 0.765 | 0.843 | 0.897 | 0.930 | 0.911
Speedup | 1.40$\times$ | 1.51$\times$ | 1.55$\times$ | 1.55$\times$ | 1.56$\times$
Table 4: Static draft-exiting threshold $\gamma$ with $K=12$.

Draft-Exiting with Adaptive Threshold. To address the issue of optimal
threshold determination, we propose an adaptive draft-exiting mechanism.
Specifically, we evaluate the acceptance rate and compare it to a target
acceptance rate. The threshold is updated with an updating rule depicted in
section 3.3. Table 4 shows that the speedup achieved by our adaptive threshold
update method (1.56$\times$) is comparable to, if not superior to, the speedup
achieved with careful tuning of static thresholds. This indicates that dynamic
threshold updating yields efficient and stable inference acceleration. This is
mainly because the acceptance rate gets closer to the target acceptance rate
by adjusting the $\gamma$ value in a timely manner for instances of varying
difficulties.

Figure 5: Threshold $\gamma$ varies with models and data. We calculate a
moving average for every 64 cycles and plot the standard deviation. The
initial $\gamma$ is set to 0.4.

###  4.5 Trend of Threshold Change

In Figure 5, we record the changing trend of the threshold in our decoding
method under different models and data during about 5,000 drafting and
verification processes. We can observe that our strategy can adaptively adjust
the threshold to an appropriate range to achieve effective acceleration; In
addition, differences in models and data sets bring drastically different
ranges of variation, which further highlights the limitations of static
threshold settings and the need for adaptive updates.

###  4.6 Breakdown of Computation

Operation | Autoregressive | Self-Speculative
---|---|---
Drafting | - | 22.9$\pm$1.28 ms
Verification | - | 8.88$\pm$2.30 ms
$\gamma$ update | - | 0.544$\pm$0.138 µs
End-to-end * | 50.6$\pm$2.67 ms | 32.3$\pm$3.04 ms
Table 5: Breakdown of computation. * denotes the average end-to-end inference
time for 10 instances randomly sampled from the CNN/DM test set after
inference testing on LLaMA-2-13B.

Table 5 presents a computation breakdown comparing the baseline with our
‘Self-Speculative’ decoding method. Our ‘Self-Speculative’ approach exhibits
significant speedup in end-to-end inference time compared to ‘Autoregressive’.
This speedup is primarily attributed to two key techniques: the selection of
skipped layers and the adaptive draft-exiting. Additionally, it is evident
that the drafting stage consumes the majority of inference time, emphasizing
the need for specific optimizations in the draft model to improve overall
inference speed. Importantly, our adaptive exit mechanism ($\gamma$ update)
incurs negligible computational cost as it does not employ any neural network
calculations.

##  5 Conclusion

In this paper, we have introduced self-speculative decoding, a novel and
efficient inference scheme that accelerates Transformer-based LLMs. Our
proposed method does not depend on additional neural network training and its
zero extra memory footprint, making it a highly practical and cost-effective
solution for inference acceleration. Moreover, we used Bayesian optimization
to search the skip layers in drafting and proposed an adaptive draft-exiting
mechanism to improve the end-to-end inference speed. Through benchmark tests
with LLaMA-2 and its fine-tuned models, we have demonstrated a speedup of up
to 1.73$\times$. In the future, we will try sophisticated model compression
strategies to further accelerate the drafting stage.

## References

  * Aminabadi et al. (2022) Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. 2022\.  Deepspeed-inference: Enabling efficient inference of transformer models at unprecedented scale.  In _2022 SC22: International Conference for High Performance Computing, Networking, Storage and Analysis (SC)_ , pages 646–660. IEEE Computer Society.
  * Bansal et al. (2022) Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. 2022.  Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale.  _arXiv preprint arXiv:2212.09095_.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Burton (1985) F Warren Burton. 1985.  Speculative computation, parallelism, and functional programming.  _IEEE Transactions on Computers_ , 100(12):1190–1193.
  * Chan et al. (2022) Stephanie CY Chan, Adam Santoro, Andrew Kyle Lampinen, Jane X Wang, Aaditya K Singh, Pierre Harvey Richemond, James McClelland, and Felix Hill. 2022.  Data distributional properties drive emergent in-context learning in transformers.  In _Advances in Neural Information Processing Systems_.
  * Chen et al. (2023) Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023.  Accelerating large language model decoding with speculative sampling.  _arXiv preprint arXiv:2302.01318_.
  * Cho and Hariharan (2019) Jang Hyun Cho and Bharath Hariharan. 2019.  On the efficacy of knowledge distillation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 4794–4802.
  * Dettmers et al. (2022) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022.  Llm. int8 (): 8-bit matrix multiplication for transformers at scale.  _arXiv preprint arXiv:2208.07339_.
  * Fang et al. (2021) Jiarui Fang, Yang Yu, Chengduo Zhao, and Jie Zhou. 2021.  Turbotransformers: an efficient gpu serving system for transformer models.  In _Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming_ , pages 389–402.
  * Frantar and Alistarh (2023) Elias Frantar and Dan Alistarh. 2023.  Massive language models can be accurately pruned in one-shot.  _arXiv preprint arXiv:2301.00774_.
  * Frantar et al. (2022) Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022.  Gptq: Accurate post-training quantization for generative pre-trained transformers.  _arXiv preprint arXiv:2210.17323_.
  * Han et al. (2015) Song Han, Huizi Mao, and William J Dally. 2015.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.  _arXiv preprint arXiv:1510.00149_.
  * He et al. (2019) Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. 2019.  Filter pruning via geometric median for deep convolutional neural networks acceleration.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4340–4349.
  * Hennessy and Patterson (2011) John L Hennessy and David A Patterson. 2011.  _Computer architecture: a quantitative approach_.  Elsevier.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2(7).
  * Hoefler et al. (2021) Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. 2021\.  Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks.  _J. Mach. Learn. Res._ , 22(241):1–124.
  * HuggingFace (2023) HuggingFace. 2023.  Text generation inference: Fast optimized inference for LLMs.  https://github.com/huggingface/text-generation-inference.
  * Jacob et al. (2018) Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2704–2713.
  * Jones et al. (1998) Donald R Jones, Matthias Schonlau, and William J Welch. 1998.  Efficient global optimization of expensive black-box functions.  _Journal of Global optimization_ , 13:455–492.
  * Leviathan et al. (2023) Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023.  Fast inference from transformers via speculative decoding.  In _International Conference on Machine Learning_ , pages 19274–19286. PMLR.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_.
  * Liu et al. (2018) Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. 2018.  Rethinking the value of network pruning.  _arXiv preprint arXiv:1810.05270_.
  * Liu et al. (2023) Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. 2023.  Deja vu: Contextual sparsity for efficient llms at inference time.  In _International Conference on Machine Learning_ , pages 22137–22176. PMLR.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work?  _arXiv preprint arXiv:2202.12837_.
  * Molchanov et al. (2016) Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2016.  Pruning convolutional neural networks for resource efficient inference.  _arXiv preprint arXiv:1611.06440_.
  * Nagel et al. (2019) Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. 2019.  Data-free quantization through weight equalization and bias correction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1325–1334.
  * (28) NVIDIA.  Fastertransformer.  https://github.com/NVIDIA/FasterTransformer.
  * Park et al. (2022) Gunho Park, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. 2022.  nuqmm: Quantized matmul for efficient inference of large-scale generative language models.  _arXiv preprint arXiv:2206.09557_.
  * Pope et al. (2022) Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2022\.  Efficiently scaling transformer inference.  _arXiv preprint arXiv:2211.05102_.
  * Rasmussen et al. (2006) Carl Edward Rasmussen, Christopher K Williams, et al. 2006.  Gaussian processes for machine learning, vol. 1.
  * Rozière et al. (2023) Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, I. Evtimov, Joanna Bitton, Manish P Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre D’efossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023.  Code llama: Open foundation models for code.  _arXiv preprint arXiv:2308.12950_.
  * Shazeer (2019) Noam Shazeer. 2019.  Fast transformer decoding: One write-head is all you need.  _arXiv preprint arXiv:1911.02150_.
  * Sheng et al. (2023) Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y Fu, Zhiqiang Xie, Beidi Chen, Clark Barrett, Joseph E Gonzalez, et al. 2023.  High-throughput generative inference of large language models with a single gpu.  _arXiv preprint arXiv:2303.06865_.
  * Stern et al. (2018) Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018.  Blockwise parallel decoding for deep autoregressive models.  _Advances in Neural Information Processing Systems_ , 31.
  * Tang et al. (2019) Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019\.  Distilling task-specific knowledge from bert into simple neural networks.  _arXiv preprint arXiv:1903.12136_.
  * Touvron et al. (2021) Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021.  Training data-efficient image transformers & distillation through attention.  In _International Conference on Machine Learning_ , pages 10347–10357. PMLR.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2021) Xiaohui Wang, Ying Xiong, Yang Wei, Mingxuan Wang, and Lei Li. 2021.  Lightseq: A high performance inference library for transformers.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers_ , pages 113–120.
  * Xiao et al. (2022) Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. 2022.  Smoothquant: Accurate and efficient post-training quantization for large language models.  _arXiv preprint arXiv:2211.10438_.
  * Yang et al. (2023) Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023.  Inference with reference: Lossless acceleration of large language models.  _arXiv preprint arXiv:2304.04487_.
  * Yao et al. (2022) Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022.  Zeroquant: Efficient and affordable post-training quantization for large-scale transformers.  _arXiv preprint arXiv:2206.01861_.
  * Yu et al. (2022) Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022\.  Orca: A distributed serving system for $\\{$Transformer-Based$\\}$ generative models.  In _16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)_ , pages 521–538.
  * Zhao et al. (2019) Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. 2019.  Improving neural network quantization without retraining using outlier channel splitting.  In _International conference on machine learning_ , pages 7543–7552. PMLR.

##  Appendix A Data

The datasets that we have selected for evaluation are CNN/Daily Mail (CNN/DM),
Extreme Summarization (XSum), and HumanEval. These tasks cover a broad
spectrum of language processing capabilities, including text and code
generation capabilities. We perform 1-shot evaluation for CNN/DM and XSum, and
compare the ROUGE-2. We compare pass@10 for HumanEval. For efficiency issues,
we randomly sample 1000 instances from the testset for CNN/DM and XSum.

##### CNN/Daily Mail (CNN/DM):

This task involves summarizing news articles from the CNN and Daily Mail
websites. The models are required to generate a concise summary of each
article, testing their ability to understand and condense complex information.

##### Extreme Summarization (XSum):

In the XSum task, models are asked to produce a single-sentence summary of a
news article. This task tests the models’ capability to extract the most
salient information from a text and express it in a single, coherent sentence.

##### HumanEval:

The HumanEval task is a benchmark for Python programming. This task challenges
the models with a variety of coding problems that require a wide range of
skills, from basic programming to complex problem-solving abilities. It serves
to evaluate the models’ understanding of Python syntax, their ability to
implement algorithms, and their proficiency in problem-solving using code.
This benchmark provides a unique perspective on the models’ capabilities in
the realm of programming, complementing the language-focused tasks.

##  Appendix B Setup

We present the hyperparameter settings of the experiments in the Table 6,
including the parameters involved in the decoding process, adaptive draft-
exiting mechanism, and random sampling. The offline Bayesian optimization time
for 1000 iterations is about 2.5 hours for LLaMA-2-13B, LLaMA-2-13B-Chat and
CodeLLaMA-13B, and about 6 hours for LLaMA-2-70B.

Model | Data | Decoding | Adaptive Draft-Exiting | Random Sampling
---|---|---|---|---
$T$ | $K$ | $\alpha$ | $\epsilon$ | $\beta_{1}$ | $\beta_{2}$ | $\gamma_{0}$ | $top\\_p$ | temperature
LaMA-2-13B | CNN/DM | 512 | 12 | 0.90 | 0.01 | 0.50 | 0.90 | 0.60 | 0.85 | 0.20
XSum | 512 | 12 | 0.85 | 0.01 | 0.50 | 0.90 | 0.60 | 0.85 | 0.20
LaMA-2-13B-Chat | CNN/DM | 512 | 12 | 0.85 | 0.01 | 0.50 | 0.90 | 0.60 | 0.85 | 0.20
XSum | 512 | 12 | 0.70 | 0.01 | 0.50 | 0.90 | 0.60 | 0.85 | 0.20
LLaMA-2-70B | CNN/DM | 512 | 12 | 0.90 | 0.01 | 0.50 | 0.90 | 0.60 | 0.85 | 0.20
XSum | 512 | 12 | 0.90 | 0.01 | 0.50 | 0.90 | 0.60 | 0.85 | 0.20
CodeLLaMA-13B | HumanEval | 512 | 12 | 0.90 | 0.01 | 0.50 | 0.90 | 0.60 | 0.95 | 0.60
Table 6: Hyperparameter settings. $\gamma_{0}$ represents the default initial
value of $\gamma$.

##  Appendix C Algorithm with Sampling

In addition to the greedy version of self-speculative decoding that we have
presented in the main paper, we also explore a variant that incorporates
random sampling, as shown in Algorithm 3. This approach introduces an element
of randomness into the selection of tokens for speculative decoding, as
opposed to the deterministic nature of the greedy version. This could
potentially lead to diverse decoding paths and outcomes, which may be
beneficial in certain scenarios.

Algorithm 3 Self-Speculative Decoding

LLM $p(x|\boldsymbol{z}^{*},x_{1},...,x_{t})$ where $x_{1},...,x_{t}$ is the
prompt, $\boldsymbol{z}^{*}$ is a vector that represents the specific layers
to bypass; target sequence length $T$; max draft tokens to generate $K$. We
denote the original LLM as $p(x|\vec{0},x_{1},...,x_{t})$, where $\vec{0}$ is
a zero vector, indicating all layers are used in inference.

$i\leftarrow t$

while $i<T$ do

for $j\leftarrow i,...,i+K$ do $\triangleright$ Drafting Stage

$x_{j+1}\leftarrow\operatorname*{\text{sample}}{p(x|\boldsymbol{z}^{*},x_{1},...,x_{j})}$

if need to exit drafting (§3.4) then

Break

end if

end for

for $i\leftarrow i,...,j$ do $\triangleright$ Verification Stage

$r\leftarrow\operatorname*{\text{sample}}$ from a uniform distribution
$U[0,1]$

if
$r\geq\min(1,\frac{p(x|\vec{0},x_{1},...,x_{i})}{p(x|\boldsymbol{z}^{*},x_{1},...,x_{i})})$
then

$x_{i+1}\leftarrow\operatorname*{\text{sample}}$ from
$\frac{\max(0,p(x|\vec{0},x_{1},...,x_{i})-p(x|\boldsymbol{z}^{*},x_{1},...,x_{i}))}{\sum_{x}{\max(0,p(x|\vec{0},x_{1},...,x_{i})-p(x|\boldsymbol{z}^{*},x_{1},...,x_{i}))}}$

Break

end if

end for

$i\leftarrow i+1$

If all draft tokens are accepted, generate next token
$x_{i+1}\leftarrow\operatorname*{\text{sample}}{p(x|\vec{0},x_{1},...,x_{i})}$
and $i\leftarrow i+1$

end while

return $x_{1},...,x_{T}$


# Generative Adversarial Networks for Urban Block Design

Stanislava Fedorova<EMAIL_ADDRESS>Politecnico di
MilanoPiazza Leonardo da Vinci, 32MilanMIItaly20133

(2021)

###### Abstract.

Development and diffusion of machine learning and big data tools provide a new
tool for architects and urban planners that could be used as analytical or
design instruments. The topic investigated in this paper is the application of
Generative Adversarial Networks to the design of an urban block. The research
presents a flexible model able to adapt to the morphological characteristics
of a city. This method does not define explicitly any of the parameters of an
urban block typical for a city, the algorithm learns them from the existing
urban context. This approach has been applied to the cities with different
morphology: Milan, Amsterdam, Tallinn, Turin, and Bengaluru in order to see
the performance of the model and the possibility of style translation between
different cities. The data are gathered from Openstreetmap and Open Data
portals of the cities. This research presents the results of the experiments
and their quantitative and qualitative evaluation.

generative design; generative adversarial networks; urban morphology;
artificial intelligence; machine learning.

††copyright: acmcopyright††journalyear: 2021††conference: SimAUD 2021:
Symposium on Simulation for Architecture and Urban Design; April 15–17, 2021;
Online††booktitle: SimAUD 2021: ASymposium on Simulation for Architecture and
Urban Design, April 15–17, 2021, Online††price: 15.00††isbn:
978-1-4503-XXXX-X/21/04††submissionid: 38††ccs: Applied computing Computer-
aided design††ccs: Computing methodologies Neural networks

##  1\. Introduction

Urban design solutions determine many important aspects of urban life:
economical, ecological, social aspects, transportation, pedestrian flows,
consequently, multiple parameters are involved. The necessity to consider many
different factors interacting with each other made this problem a challenge in
the field of generative design. The result of an urban project has a long-
lasting effect on various spheres of life, which makes finding an optimal
solution even more important. The issue arises when it is necessary to define
a set of parameters determining an optimal solution for a given area. Some of
them are dictated by the local regulations, some by common knowledge, some by
the sense of aesthetics; sometimes it is difficult to formulate them using
words while algorithms would require formulae or numeric representation.

One of the problems involving a manifold of parameters is a design of an urban
block. This problem takes into consideration the shapes of the buildings,
their proportions, relationships with the adjacent buildings, the composition
of the open space, relationships of buildings with the open space, with the
urban block itself, relationships with the buildings of the adjacent blocks.
Block design involves the knowledge of the typologies of the blocks in a city,
surroundings (streets, squares, railways nearby), etc. Moreover, it should
smoothly integrate into the existing urban context, which means taking into
consideration the disposition and typology of the adjacent blocks. All of the
parameters mentioned above would be difficult to define for each block a
design should be produced to.

With the recent introduction of Generative Adversarial Networks
(GANs)(Goodfellow et al., 2014), it became possible to use the neural
networks’ ability to autonomously learn visual features in order to generate
new content. The main advantage of this technique applied to generative design
is the elimination of the necessity to determine any parameters of the desired
design solution and learning the visual features determining the existing city
structure and consequently generating blocks imitating this structure.

##  2\. Related Work

As the concept of GANs has been introduced quite recently(Goodfellow et al.,
2014) there has not been much research on the topic concerning architecture
and urban design, although it is well-known in the field of computer science.

Prior work in the generative urban design field is mostly based on a set of
defined rules and constraints according to which several design options are
generated(Beirão et al., 2010). Nagy, Villaggi, and Benjamin(Nagy et al.,
2018) explore optimization of the problem complexity with the set of two
principal goals and the use of genetic algorithms. However, their approach,
too, needs a set of constraints and requirements; their model outputs a design
space out of which, after optimization, only a few designs are chosen as
final.

The use of GANs has been explored in interior design as well as in
visualization(Wang et al., 2018) and simulation. Stanislas Chaillou(Chaillou,
2020) explored the functionality of the method applied to the design of the
apartments generating their shape, internal structure, functional zoning, and
furniture placement. The same approach at the same architectural
scale(Galanos, 2020) was used for the simulation of natural illumination in an
apartment provided its plan. Another work published in 2020 used a GAN with a
different architecture with respect to ours for landuse planning(Wang et al.,
2020) with the use of predefined parameters that a planner considers important
for his solution and the zone design. However, this approach produces only a
large-scale distribution of the functional zoning in the city and is not
applicable to the urban design.
The present work tackles a different scale problem: urban block level design.
Instead of using a set of rules exploited by the generative urban design
models previously, the GAN learns the set of parameters present in the city
block design from the visual features of the images. A work that uses GANs in
architecture on a scale that is similar to ours is the project This Map Does
Not Exist(Boates, 2020). However, this work uses a different architecture that
does not allow the production of the designs based on the existing input: the
maps are not attached to any existing point in the real world and the existing
conditions of an urban environment are not taken into account when producing
an image. One more work(Chu et al., 2019) uses generative networks for the
generation of an urban street layout. This approach is the closest to our work
both in terms of urban scale and the possibility of conditional generation.
The problem is explored from a different point of view: the authors use graph
generation to produce a road design while our research tackles the generation
of the urban blocks.
Importantly, our research combines both the challenges faced in the previously
mentioned works using GANs: production of a design solution based on the
chosen parameters of the surrounding environment and the integrity of the
solution within its urban block.

##  3\. Data sampling and acquisition

Firstly, the data of the city of Milan was used to make a dataset. This city
has been chosen due to its characteristic morphology and relatively consistent
urban fabric inside the external ring road and within a 1 km distance from it.

The dataset consists of images that are diagrammatic representations of the
city zones. It has been decided to start with the basic parameters determining
the composition of a building block: building footprints, building heights’
categories, road structure, and railways. The heights of the buildings have
been divided into 3 ranges: low-rise structures, medium-rise structures, and
high-rise structures depending on the distribution of the heights in the city.
Heights in the diagrammatic representation are defined as three gray values.
In case of height not having been mentioned for a building, a mean value for
the city was used to denote its height.

The dataset has been generated with the use of QGIS software (QGIS Development
Team, 2009) with the condition to have more than 12 percent of a block of
interest occupied. Urban blocks containing exclusively industrial buildings
have been excluded from the datasets as their presence introduces additional
variance. Since the information about the landuse and the building type is not
provided to the algorithm the experiment is concentrated on the buildings with
residential and mixed functions. Moreover, the proportions of the industrial
buildings are similar across different geographical regions and do not
introduce any characteristics typical to a specific city.

The images were produced in scale 1:3000 in RGB colorspace; these images were
scaled down from 2100x2100 to 256x256 pixels to be fed into the model.

The dataset consists of two sets: set B includes the diagrammatic
representations of the city and set A holds the same images with an empty
polygon in the center of the image, destined to be filled with the buildings
as in Figure 1. Code for dataset generation is available in the
repository(Fedorova, 2020a).

The data was acquired from open sources: Open Data Portal of Comune di Milano
(Comune di Milano, 2017) available under CC License (cc_, [n.d.]) and
Openstreetmap.org (OpenStreetMap contributors, 2017) available under the Open
Database Licence. The absent parameters have been considered as a mean value
for the city. No modifications have been made to the datasets provided.

Figure 1. Sample from Milan Dataset of urban blocks: on the left an image from
set A, on the right a corresponding image from set B. The white block in the
center of the left image is the block to be designed.

Two images showing a diagrammatic representation of the top view on a part of
the city of Milan (Piazzale Segesta). The image on the left is the input
condition, the image on the right is the ground truth.

Figure 2. The results were obtained on the test set of Milan images with the
model trained on Milan urban blocks. The images are presented in sets of 3:
input, output, output with the zone of interest highlighted (for the clarity
of reading the results). The zones have been taken from the Riformare Milano
project(rif, [n.d.]). 1 - Dergano zone; 2 - Oriani zone; 3 - Caserma
Perucchetti; 4 - Dergano zone; 5 - Mecenate zone; 6 - San Cristoforo zone

A 3x6 matrix showing 6 samples from the Milan test set results: the first
column is the input image, the second is the output from the model and the
third is the ground truth. The images are diagrammatic representations of the
city plan.

##  4\. Architecture

The architecture used for this research is a GAN for Image-to-Image
translation (Isola et al., 2017). The algorithm consists of two models:
Generator Network based on Unet architecture and Discriminator Network based
on PatchGAN Convolutional Network. The principle is in mutual learning: the
generator generates an output image based on the condition (input image). The
discriminator model receives generated images and target images and learns to
distinguish between them, classifying each of them into a ’real’ or a ’fake’
class. In the case of PatchGAN, the model learns to classify patches of a
given image as real or fake instead of giving a result for the whole image
sample, so it outputs the result over which areas of an image should be
improved by the generator.

Three different implementations of the algorithm were used, which are:
Tensorflow-based (pix, [n.d.]b), PyTorch-based without Unet (pix, [n.d.]a),
Pytorch-based with Unet (Fedorova, 2020a) (Deeplearning.ai, 2020). The use of
adaptive learning rate tends to give better results. An algorithm compares the
mean loss from 250 iterations with the previous accumulated losses. In case
the latest loss is larger than the largest of the previous three losses, the
learning rate reduces. This technique results in more distinct shapes of the
building contours and more efficient training in terms of time. The algorithm
has been trained using Google Colab environment with a single NVIDIA Tesla K80
GPU.

##  5\. Generation of Milan Urban Style

The specifics of Milan morphology and consistency in the urban fabric of the
center of the city make it a good location for the initial experiments. The
relative homogeneity of the urban morphology reduces the variance in the data.
The model has been trained for 20 epochs with a dataset of 1200 images.

After the training, the model has been tested on a new set of images that were
not included in the training set. The areas chosen for the test have been
previously defined as ’problematic’(rif, [n.d.]) and requiring a
transformation. However, the main point is not a proposal of a solution to
these areas, but a demonstration of the possibilities of GAN-based generative
urban design, even in the complex urban contexts that are different from the
residential ones. The results in Figure 2 show designs of the urban blocks
coherent with the traditional Milanese urban blocks(Biraghi et al., 2018) that
tend to have a closed structure and an internal courtyard system in their
majority. The model successfully recognizes the zone of interest to construct
the block within and manages to adapt the learned visual characteristics of
the city into a new context. Samples 1, 2, 4 are the samples with the
irregular urban fabric and sample 5 is an example of an urban context with a
high presence of industrial buildings differing in proportions from the forms
in the training set.

At the same time model fails to produce a clear solution in a few samples (3,
6 in Figure 2). This behavior occurs in the samples having a low urban density
of the surroundings or their absence. Another reason is connected to the
absence of the surroundings: their small proportion in the image to the zone
of interest (e.g. when the zone occupies more than 30 percent of an image).
This factor is an indication of the fact that the model produces solutions
based on the visible surroundings; at the same time, it introduces a
limitation of its use as well as a constraint for the input images: the model
is not capable of producing design without any surroundings. This behavior is
consistent on different urban scales and is related mostly to the proportion
of the block to the image.

Figure 3. Illustration of the model’s performance on the urban contexts in
different scales. The images are on scales 1:2500, 1:5600, and 1:8500
respectively; the upper row represents the city of Milan, Mecenate zone; the
second row is the city of Bengaluru.

A 2x3 matrix with a diagrammatic representation of Milan and Bengaluru top
views in different scales (1:2500, 1:5600, 1:8500) and with a different area
of the zone of interest within one scale with the results from the generative
models.

Figure 4. Illustration of the model’s performance on the zone of interest with
different proportions with respect to the image within the same Bengaluru
urban context.

4 images with a diagrammatic representation of the city top view in scale
1:2500 with different proportion of the area defined as a block of interest.
This shows that the model fails when the proportion of the block with respect
to the image is large.

Furthermore, the model was tested on a set of images from the same locations
but with different scales to see whether the design capabilities depend on the
scale and the proportions of the zone’s size with respect to the surroundings.
As it can be observed in Figure 3 performance of the model differs based on
the scale. At the same time, it should be noticed that difference in
performance is not related only to the scale but also to the proportion of the
zone of interest: the block of Bengaluru (see Figure 3 second row, first
column) at the scale of 1:2500 causes model’s failure as the area of the block
is too large with respect to the surroundings’ area. On the scale of 1:5600,
however, the performance is close to the one of Milan’s sample at the scale of
1:2500. The areas of the respective blocks are similar. The same observation
can be made for the last sample of Bengaluru and the second sample of Milan.
Besides, the model was checked on an image on a large scale cropping the block
to different proportions with respect to surroundings (Figure 4). The
experiment demonstrates that the performance of the model depends on the
proportion of the surroundings in an input image not on the urban scale and
the fact that the design is conditioned by the context. This is the reason the
importance of the presence of the context should be noted: firstly, the empty
area is likely to be treated as the construction site and, secondly, there
will be no surrounding blocks to condition the design of the current block of
interest (see model’s behavior in Figure 2 in samples 3, 6).
However, there is another limitation related to the chosen training process
and the generated images. The dimensions of the generated buildings remain the
same independently from the scale of the input image (for instance, in Figure
3, row 2): the width of the building volumes does not change with the
modification of the scale of the surroundings. This behavior was expected, as
all the images in the training set were produced on one scale, thus the
distances and the units in the images are perceived by the model within this
scale only.

##  6\. Style Translation

Having obtained the results from the model trained on the city of Milan, it
was decided to experiment further and see which features the model will learn
from a completely different morphology. For this reason, several cities with a
morphology that is distant from the morphology of Milan have been chosen:

  * •

Amsterdam

  * •

Bengaluru

  * •

Tallinn

  * •

Turin

Figure 5. Illustration of the difference in urban morphology of the chosen
cities. 1 - Milan, 2 - Amsterdam, 3 - Tallinn, 4 - Turin, 5 - Bengaluru

Five images showing the morphology of the following cities: Amsterdam,
Bengaluru, Tallinn, Turin, Milan. The morphology is shown by drawing white
building footprints from a part of each city on the black background.

The datasets for these cities are built with the use of scripting and QGIS
software in the same way as the dataset of Milan urban blocks. The cities of
Tallinn and Bengaluru had a condition of having more than 8 buildings in the
block of interest due to the composition of their urban fabric.

As can be seen in Figure 5 the cities vary greatly in urban density,
buildings’ types (isolated structures in Bengaluru and Tallinn; in connected
structures and internal courtyard forming in Milan, Turin, and Amsterdam),
relationships between the buildings and the block they belong to, the
composition of the blocks. For instance, Milan, Turin, and Amsterdam have
closed block structure often with an internal courtyard; at the same time the
density of the blocks vary from city to city as well as the composition of the
blocks: in Amsterdam typically a closed block is formed by many buildings
while in Turin just by a few and the shape of a footprint of a block is more
clear and geometrical. The differences in the urban fabric of the cities are
helpful in the estimation of the learning results of the model and comparison
of the results produced.

After the training and testing of these models on the corresponding cities
each model was tested on the urban contexts from the other cities. This test
gives an understanding of whether it is possible to translate an urban style
from one city to another with the use of GAN, whether the model is able to
adapt to completely different surroundings, and the coherence of the results
with the urban blocks of training. The results were presented as a confusion
matrix in Figure 7.
The resulting models are published(Fedorova, 2020b) and are available in open
access under MIT License. The code for model testing is provided as well.

##  7\. Estimation

The result of training for the case of urban block design should be evaluated
both quantitatively and qualitatively. The qualitative approach is based on
the use of a neural network predicting the class for the given images.
Unfortunately, there was no possibility to have a survey based on the experts’
opinion to obtain the qualitative estimation. This method serves as an
alternative to experts’ judgment on the blocks’ design as it exploits the
visual comprehension and appearance of an urban block in general that allows
to attribute it to a particular city. This neural network classifier is
trained independently of GAN and it represents a Convolutional Neural Network
with a resnet50 backbone. This classifier is trained to distinguish among the
urban blocks of all the cities the models were trained on.

The quantitative approach is based on the analysis of the block design from
the urban and morphological perspective relative to the city it is situated
in. The numerical parameters determining the morphology of an urban block are
taken into account: relationships between the buildings within a block,
relationships between the buildings and the block, average individual
parameters of the buildings.

###  7.1. Qualitative Estimation

A separate convolutional neural network with a resnet50 backbone (He et al.,
2015) was trained to classify between the images of the urban blocks of all
the five cities. The training and validation sets were composed of the images
from the real cities’ structures (Fedorova, 2020a) which can be seen in Figure
6 while the test images were the images generated by the GAN model. The
results of the classification are reported in the Table 3. The accuracy
achieved on real samples is compared with the accuracy achieved on the
generated samples.

Figure 6. Samples from the dataset used for urban blocks classification.

Four samples from the dataset used for classification purposes. Each sample
contains a white mask of an urban block on the black background.

Figure 7. Results obtained on the test set. Rows represent the existing urban
context of the city, columns represent the cities the models were trained on
(the expected morphology of the urban block). E.g. cell [1, 3], row 1 column 3
represents the model trained on the city of Amsterdam tested within Milan
urban context: Amsterdam urban block design in Milan. The diagonal cells
represent the models trained and tested on the same city.

A 5x5 matrix where every cell has an image of a city plan as a diagrammatic
representation with different urban block designs produced by the generative
models.

As in the case with the datasets for GAN, the classification dataset is
divided into training and validation sets. Training images are the images the
CNN learns from while validation images are used to see the results of the
training process, these are the data the algorithm has not seen before. From
the confusion matrix of results produced from the training set as seen in
Table 1 and validation set (Table 2), it can be seen that urban blocks of some
cities are confused more often with some specific cities. For example,
Bengaluru and Turin do not have a large amount of misclassified images while
the samples from Milan are often predicted to be from Tallin and vice versa.

Table 1. Classification Confusion Matrix from Training set results. Columns represent the predicted classes, rows represent the actual classes. E.g. cell [1, 3] row Amsterdam column Milan shows the number of Amsterdam sample images classified as Milan images. |  Amster. |  Bengaluru |  Milan |  Turin |  Tallinn
---|---|---|---|---|---
Amsterdam | 926 | 21 | 47 | 20 | 6
Bengaluru | 16 | 951 | 24 | 14 | 7
Milan | 55 | 22 | 866 | 14 | 38
Tallinn | 54 | 26 | 32 | 484 | 10
Turin | 19 | 11 | 47 | 8 | 570
Table 2. Classification Confusion Matrix from Validation set results. Columns represent the predicted classes, rows represent the actual classes. |  Amster. |  Bengaluru |  Milan |  Turin |  Tallinn
---|---|---|---|---|---
Amsterdam | 146 | 3 | 0 | 17 | 13
Bengaluru | 1 | 167 | 0 | 12 | 3
Milan | 8 | 13 | 102 | 31 | 44
Tallinn | 19 | 2 | 4 | 82 | 1
Turin | 4 | 0 | 8 | 0 | 77
Table 3. Statistics from the classification model trained on the images of the urban blocks of Milan, Amsterdam, Bengaluru, Tallinn, Turin. The test accuracy is the confidence of prediction over the generated images from the Generator’s model training set. Classification |  Training Accuracy |  Validation Accuracy |  Accuracy on Test Gen samples |  Confidence On Test Gen Samples
---|---|---|---|---
Amsterdam | 0.815 | 0.907 | 0.952 | 0.791
Bengaluru | 0.913 | 0.94 | 0.352 | 0.799
Milan | 0.515 | 0.87 | 0.619 | 0.877
Tallinn | 0.759 | 0.799 | 0.523 | 0.85
Turin | 0.865 | 0.87 | 0.5 | 0.795
Overall | 0.758 | 0.885 | 0.566 | 0.822

This classification model has been tested on the images generated by GAN. The
test set consists of 50 images, 10 images per city. The results produced from
Classifier over this set are reported in Table 3. As can be seen, the
generated images are classified with more than 0.5 accuracy to the correct
class apart from Bengaluru samples with a confidence of more than 0.75.
Bengaluru model sometimes fails to produce a solution in Tallinn blocks due to
the reason of low density.

###  7.2. Quantitative Estimation

Taking into consideration the city of Milan it can be noticed that from the
morphological point of view its blocks could be divided into a few groups
sharing some common features across the city based on the following
characteristics: shape, dimension, number of parcels, and topography(Biraghi
et al., 2018). While the first two are the conditions determined by the input
image, the focus is on the latter two. The number of parcels and the
topography of the blocks could be described numerically as parameters
describing the relationships between the urban forms. These relationships have
the same importance for the other cities as well. For this evaluation, the
factors considered are the relationships between the buildings within a block
(distance between the neighboring buildings), the relationships between a
block and its buildings (building density, the proportion of built space in a
block; the distance between the street and the closest building in a block)
and the individual parameters of the buildings (average area of a building
footprint in a block). We acknowledge that these parameters vary across one
city, however, it is possible to deduce the most common characteristics shared
by the majority of the blocks by taking the median value across the city.
These most common characteristics represent the urban fabric, the feature that
distinguishes one city from another 5. Moreover, in our case, the ”inter-city”
similarity is higher than the ”intra-city” similarity as the cities have been
chosen to have a ’contrasting’ morphology. However, this research is limited
by the number of comparison parameters for urban blocks; it would be a
valuable continuation of the work to develop a complete set of comparison
criteria.

Table 4. Quantitative estimation of the solutions produced by 5 trained GAN models: comparison of the average parameters by city and the parameters in the samples produced by the models. City |  City Dataset: Median Density |  Generated Blocks: Median Density |  City Dataset: Median Building Area, px |  Generated Blocks: Median Building Area, px
---|---|---|---|---
Amsterdam | 0.537 | 0.331 | 104 | 124
Bengaluru | 0.484 | 0.307 | 112 | 69
Milan | 0.489 | 0.521 | 258 | 215
Tallinn | 0.236 | 0.307 | 13 | 31
Turin | 0.553 | 0.594 | 283 | 389
Table 5. Quantitative estimation of the solutions produced by 5 trained GAN models: comparison between the offset of the buildings from the street by city and the parameters in the samples produced by the models. Distance from the street is a minimum distance between the buildings in a block and the surrounding streets; the resulting value is a median value for the dataset samples and the samples produced by the models. City |  City Dataset: Median Distance Between the Buildings and the Street, px |  Generated Blocks: Median Distance Between the Buildings and the Street, px
---|---|---
Amsterdam | 1.14 | 1.414
Bengaluru | 1.01 | 3.4
Milan | 0.346 | 2.0
Tallinn | 1.85 | 2.4
Turin | 1.23 | 2.8
Table 6. Quantitative estimation of the solutions produced by 5 trained GAN models: comparison between the median distance between the adjacent buildings by city and the parameters in the samples produced by the models. The distance between the adjacent buildings is an average of minimum distances between the buildings in a block; the resulting value is a median value. City |  City Dataset: Median Distance Between the Adjacent Building |  Generated Blocks: Median Distance Between the Adjacent Buildings
---|---|---
Amsterdam | 2.6 | 2.5
Bengaluru | 2.8 | 3.0
Milan | 1.5 | 2.2
Tallinn | 3.1 | 3.4
Turin | 0.5 | 0.0

These parameters have been measured for all the blocks in each city dataset,
then the median value has been calculated for each city. The median has been
chosen over the mean value as it takes into consideration the presence of the
outliers which are likely to be present in the datasets due to the urban
variety in each city. The metrics have been calculated from the images with
the use of Computer Vision(Fedorova, 2020a) and QGIS; the units of absolute
values such as the median building area or the distance between adjacent
buildings are pixels.

From Tables 4, 6 it can be seen that urban blocks produced by the GAN in
different urban contexts are coherent with the city of training: the
parameters of building density, building area as in Table 4 are close to the
actual values, which confirms the observations made in the confusion matrix
(Figure 7). At the same time, the distance from the street to the closest
building in a block differs as in Table 5: it is a constraint taken from the
surroundings present in the input image (Figure 8).

##  8\. Contribution

This paper aims to contribute to the field of generative design by:

  * •

introducing a new approach towards the context-based generative design of an
urban block without a predefined set of constraints, by learning the visual
parameters from the existing environment instead;

  * •

introducing an application of an Image-to-Image translation GAN to the field
of urban design and evaluating the produced design solutions from quantitative
and qualitative points of view.

Figure 8. Samples from the test set. The block design is aligned with the
surrounding blocks’ street lines.

Six results from the test set with the annotations showing an alignment of the
generated buildings to the existing buildings.

##  9\. Results

The study shows that it is possible to achieve a generation of an urban block
without the explicit definition of its parameters with the use of GAN. Not
only does this method allow for learning the parameters from the existing
urban context without their explicit definition, but it also produces results
coherent with the urban surroundings of a block as in Figure 7. Moreover, this
method produces a design solution with the characteristics of a generated
urban block very close to the actual parameters of the blocks of a city of
interest.

The experiments have shown that the model is applicable to diverse urban
contexts with different urban morphology. Besides, urban parameters and
morphology learned by a model can be translated to another city with a
completely different context. The algorithm adapts the parameters learned
previously to a new urban surrounding.

Some limitations of the algorithm include the size of an urban block within
the image, the presence of the urban surroundings in the image, the scale of
the buildings in the image. The size of a block is directly connected to the
presence of the urban surroundings: the block design is shaped by the context,
furthermore, the algorithm learns from the visible context. Thus it is
important to have it present in a considerable portion of the image
(determined experimentally). The scale of an urban block is limited due to the
specifics of the current training preset: all the images from the training set
were of the same scale, which means it is reasonable to test the algorithm
with the images of the same scale. The preset is limited to a certain scale
(which may vary from city to city based on the average block size) due to the
necessity of having a sufficient part of the image filled with urban
surroundings of a block of interest.

Besides, it should be noted that the design solutions produced do not have the
same clarity as the vector images produced by the architects. However, urban
shapes in the generated images are clearly visible and give a good
understanding of the blocks’ design.

Another obvious limitation is the production of an urban block design that is
similar to the existing blocks in the city. It is an approach to design that
might be considered non-creative or lacking other important factors of urban
design theory. We do not insist on the application of the suggested approach
’as-is’. This work highlights the ability of the algorithm to autonomously
learn from the visual features, which could be further explored to take into
account the other planning criteria and optimize the block’s design from
different points of view.

##  10\. Conclusion

The experiment and evaluation of the GAN-based approach to the generative
design of an urban block confirmed the possibility of the generation of a
piece of urban fabric consistent with the surroundings and the desired urban
style with the use of neural networks. Nonetheless, the images lack clarity of
the vector images, the urban forms are compliant with the actual forms present
in the city. What is more, these results have been achieved without image
annotations, explicit definition of the parameters needed for each typology
and shape of an urban block, typical characteristics of a city: all of these
attributes are learnt by the neural network during the training process, from
the experience having seen multiple urban surroundings.

In addition, the time and effort that are consumed for the block’s design
generation are much less compared to the human-generated design. It takes 2-4
hours to train one model after which it is able to generate the design
solutions within 10 seconds.

This approach can not, obviously, substitute human participation in a city
design process: the algorithm imitates the existing urban environment after
having learned from it. It does not invent new space organization principles,
does not rethink the comprehension of space by its users, and does not plan
important landmarks of a city. However, considering that cities majorly
consist of residential buildings and not landmarks, this approach could be
used as a support tool for a planner or an architect by giving insights about
the possible solutions and existing building composition of a city.
Alternatively, it could be a design solution meant for typical blocks’ design
which would have a consistent integration into the existing surroundings.
Moreover, as already mentioned, this approach could become a base for the
consequent integration of the other urban design parameters into the model
training which would result in a design that accustoms to multiple factors.

###### Acknowledgements.

I would like to express my very great appreciation to Professor Luigi Mussio
and Professor Federico Deambrosis who dedicated their time and effort to
guiding and directing the work and encouraging me in the doubtful moments. My
thanks also go to the professors of Architecture and Urban Design of
Politecnico di Milano: Professor Lemes, Professor Caja, Professor Montedoro
who had found time to give me their valuable suggestions from the urbanistic
and architectural point of view as well as insights on the morphology of the
cities. I would also like to thank Dr. Carlo Biraghi for the constructive
advice.
My gratitude is extended to Andrey Baran, Alberto Floris, Ram Kummamuru and
Viktor Petukhov for the productive discussions and inspiration.

## References

  * (1)
  * cc_([n.d.]) [n.d.].  Creative Commons License. Attribution 2.5 Italy (CC BY 2.5 IT).  hhttps://creativecommons.org/licenses/by/2.5/it/deed.en.
  * pix ([n.d.]a) [n.d.]a.  Pix2Pix Implementation. Pytorch.  https://github.com/mrzhu-cool/pix2pix-pytorch.
  * pix ([n.d.]b) [n.d.]b.  Pix2Pix Implementation. Tensorflow.  https://github.com/affinelayer/pix2pix-tensorflow.
  * rif ([n.d.]) [n.d.].  Riformare Milano.  http://www.riformaremilano.polimi.it/?page_id=1754.
  * Beirão et al. (2010) José Beirão, Gelly Mendes, Jose Duarte, and Rudi Stouffs. 2010.  Implementing a Generative Urban Design Model: Grammar-based design patterns for urban design.  _FUTURE CITIES [28th eCAADe Conference Proceedings / ISBN 978-0-9541183-9-6] ETH Zurich (Switzerland) 15-18 September 2010, pp.265-274_.
  * Biraghi et al. (2018) Carlo A. Biraghi, Michele Caja, and Sotirios Zaroulas. 2018\.  Urban Blocks and Architectural Typology in the Milanese Context. In _Proc. XXV ISUF International Conference_. Siberian Federal University Library, 859–868.
  * Boates (2020) Isaac Boates. 2020\.  This Map Does Not Exist project.  https://github.com/iboates/thismapdoesnotexist.
  * Chaillou (2020) Stanislas Chaillou. 2020\.  ArchiGAN: Artificial Intelligence x Architecture. In _Architectural Intelligence, Selected Papers from the 1st International Conference on Computational Design and Robotic Fabrication (CDRF 2019)_. 117–127.  https://doi.org/10.1007/978-981-15-6568-7_8
  * Chu et al. (2019) Hang Chu, Daiqing Li, David Acuna, Amlan Kar, Maria Shugrina, Xinkai Wei, Ming-Yu Liu, Antonio Torralba, and Sanja Fidler. 2019.  Neural Turtle Graphics for Modeling City Road Layouts.  (10 2019).
  * Comune di Milano (2017) Comune di Milano. 2017\.  Milano Geoportale.  https://geoportale.comune.milano.it/sit/open-data/.
  * Deeplearning.ai (2020) Deeplearning.ai. 2020\.  GANs Specialization.  https://www.deeplearning.ai/generative-adversarial-networks-specialization/.
  * Fedorova (2020a) Stanislava Fedorova. 2020a.  Github Repository for urban datasets generation.  https://github.com/STASYA00/urban_datasets.
  * Fedorova (2020b) Stanislava Fedorova. 2020b.  Github repository with published models.  https://github.com/STASYA00/UrbanGen.
  * Galanos (2020) Theodore Galanos. 2020\.  DaylightGAN.  https://github.com/TheodoreGalanos/DaylightGAN.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative Adversarial Nets. In _Advances in Neural Information Processing Systems_ , Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (Eds.), Vol. 27. Curran Associates, Inc., 2672–2680.
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015\.  Deep Residual Learning for Image Recognition.  _CoRR_ abs/1512.03385 (2015).  arXiv:1512.03385 http://arxiv.org/abs/1512.03385
  * Isola et al. (2017) P. Isola, J. Zhu, T. Zhou, and A. A. Efros. 2017\.  Image-to-Image Translation with Conditional Adversarial Networks. In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 5967–5976.  https://doi.org/10.1109/CVPR.2017.632
  * Nagy et al. (2018) Danil Nagy, Lorenzo Villaggi, and David Benjamin. 2018\.  Proc. SIMAUD 2018, Generative Urban Design: Integrating Financial and Energy Goals for Automated Neighborhood Layout. 25\.  https://doi.org/10.22360/simaud.2018.simaud.025
  * OpenStreetMap contributors (2017) OpenStreetMap contributors. 2017\.  Data retrieved from https://www.openstreetmap.org .  https://www.openstreetmap.org.
  * QGIS Development Team (2009) QGIS Development Team. 2009\.  QGIS Geographic Information System.  http://qgis.org),.
  * Wang et al. (2020) Dongjie Wang, Yanjie Fu, Pengyang Wang, Bo Huang, and Chang-Tien Lu. 2020.  _Reimagining City Configuration: Automated Urban Planning via Adversarial Learning_.  Association for Computing Machinery, New York, NY, USA, 497–506.  https://doi.org/10.1145/3397536.3422268
  * Wang et al. (2018) Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. 2018\.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs.  https://tcwang0509.github.io/pix2pixHD/.