# A Novel Memory-Efficient Deep Learning Training Framework via Error-Bounded
Lossy Compression

Sian Jin  Washington State University PullmanWAUSA<EMAIL_ADDRESS>,
Guanpeng Li  University of Iowa Iowa CityIAUSA<EMAIL_ADDRESS>,
Shuaiwen Leon Song  University of Sydney SydneyNSWAustralia
<EMAIL_ADDRESS>and  Dingwen Tao  Washington State University
PullmanWAUSA<EMAIL_ADDRESS>
###### Abstract.

Deep neural networks (DNNs) are becoming increasingly deeper, wider, and non-
linear due to the growing demands on prediction accuracy and analysis quality.
Training wide and deep neural networks though requires a large amount of
processing time and computing resources such as memory, storage, and I/O. When
training a DNN model, the intermediate activation data must be saved in the
memory during forward propagation and then restored for backward propagation.
However, state-of-the-art accelerators such as GPUs are only equipped with
very limited memory capacities due to hardware design constraints, which
significantly limits the maximum batch size and hence performance speedup when
training large-scale DNNs. Traditional memory saving techniques such as data
recomputation and migration either suffers from a high performance overhead or
is constrained by specific interconnect technology and limited bandwidth.

In this paper, we propose a novel memory-driven high performance DNN training
framework that leverages error-bounded lossy compression to significantly
reduce the memory requirement for training in order to allow training larger
neural networks. Different from the state-of-the-art solutions that adopt
image-based lossy compressors such as JPEG to compress the activation data,
our framework purposely designs error-bounded lossy compression with a strict
error-controlling mechanism. Specifically, we provide theoretical analysis on
the compression error propagation from the altered activation data to the
gradients, and then empirically investigate the impact of altered gradients
over the entire training process. Based on these analyses, we then propose an
improved lossy compressor and an adaptive scheme to dynamically configure the
lossy compression error-bound and adjust the training batch size to further
utilize the saved memory space for additional speedup. We evaluate our design
against state-of-the-art solutions with four widely-adopted DNNs and the
ImageNet dataset. Results demonstrate that our proposed framework can
significantly reduce the training memory consumption by up to 13.5$\times$ and
1.8$\times$ over the baseline training and state-of-the-art framework with
compression, respectively, with little or no accuracy loss.

††copyright: none

##  1\. Introduction

Deep neural networks (DNNs) have rapidly evolved to the state-of-the-art
technique for many artificial intelligence (AI) tasks in various science and
technology domains, including image and vision recognition (Simonyan and
Zisserman, 2014), recommendation systems (Wang et al., 2015), and natural
language processing (NLP) (Collobert and Weston, 2008). DNNs contain millions
of parameters in an unparalleled representation, which is efficient for
modeling complexity nonlinearities. Many works (Krizhevsky et al., 2012;
Szegedy et al., 2015; He et al., 2016) have suggested that using either deeper
or wider DNNs is an effective way to improve analysis quality, and in fact
many recent DNNs have gone significantly deeper and/or wider (Wang et al.,
2018; Jin et al., 2019). For instance, OpenAI recently published their new
DNN-based NLP model GPT-3 (Brown et al., 2020) with 175 billion parameters,
which is the largest NLP model that is ever trained. Compared with its
predecessor GPT-2, GPT-3 expands the capacity by three orders of magnitudes
without significant modification to the model architecture, instead just
adopting deeper and/or wider layers (Brown et al., 2020).

DNN training is an optimization process to minimize the target loss function,
which reflects the event or values into a real number intuitively to represent
the cost or difference. Training modern DNNs requires a large amount of
computation and resources such as memory, storage, and I/O. A typical strategy
to implement efficient DNN training is to rely on advanced high-performance
computing (HPC) or data center architectures equipped with high-throughput
accelerators, such as general-propose accelerators (e.g., GPUs) or customized
hardware (e.g., FPGA or ASIC designs), to perform floating-point operations in
a massively parallel manner. However, under the current large-scale network
models, efficient training is hard to achieve. For example, training GPT-3
model will take a high-end NVIDIA Tesla V100 GPU 355 years, and cost $4.6
million on a cost-effective cloud instance (Li, 2020). This significantly
hinders the exploration and adoption of new AI algorithms and DNN designs.
Furthermore, designing new DNN architectures and training algorithms for
various AI tasks require numerous trial-and-error fine-tuning instances,
worsening the training cost challenge.

In this paper, we explore a general memory-driven approach for enabling
efficient DNN training. Specifically, the ultimate goal is to drastically
reduce the memory requirement for training in order to reduce the max batch
size limitation for large training speedup. When training a DNN model, the
intermediate activation data (i.e., the input of all the neurons) is typically
saved in the memory during forward propagation, and then restored during
backpropagation to calculate gradients and update weights accordingly (Hecht-
Nielsen, 1992). However, taking into account the deep and wide layers in the
current large-scale nonlinear DNNs, storing these activation data from all the
layers requires large memory spaces which are not available in the state-of-
the-art training accelerators such as GPUs. For instance, training
Inception-V4 (Szegedy et al., 2017) with the batch size of 32 on the
ImageNet-2012 dataset (Challenge, 2020) requires more than 40 GBs of memory,
which is larger than the memory available on the state-of-the-art GPUs such as
NVIDIA Tesla V100 with 32 GBs. Furthermore, modern DNN model designs trades-
off memory requirement for higher accuracy. For example, Gpipe (Huang et al.,
2019) increases the memory requirement by more than 4$\times$ for achieving a
5% top-1 accuracy improvement from Inception-V4.

Evolving in recent years, model-parallel (Ben-Nun and Hoefler, 2019)
techniques that distribute a model into multiple nodes can reduce the memory
consumption of each node but introduce high communication overheads; on the
other hand, data-parallel techniques (Sergeev and Del Balso, 2018) remain the
same model in every node but distribute the training data to different nodes,
thereby suffering from high memory consumption to fully utilize the
computational power. Several techniques such as recomputation, migration, and
lossless compression for activation data have been proposed to address the
memory consumption challenge for training large-to-large-scale DNNs. For
example, GeePS (Cui et al., 2016) and vDNN (Rhu et al., 2016) have developed
data migration techniques for transferring the intermediate data from GPU to
CPU to alleviate the memory burden. However, the performance of data migration
approaches is limited by the specific in-node interconnect technology (e.g.,
PCIe and NVLinks (Foley and Danskin, 2017)) and its available bandwidth. Some
other approaches proposed to recompute the activation data (Chen et al., 2016;
Gomez et al., 2017) but they often incur large performance degradation,
especially for computationally intensive layers such as convolutional layers.
Moreover, memory compression approaches based on lossless compression on the
activation data (Son et al., 2014)) suffers from the limited compression
ratio, e.g., only around 2:1 for most floating-point data. Alternatively,
recent works (Evans et al., 2020; Choukse et al., 2020) proposed to develop
compression offloading accelerators for reducing the activation data size
before transferring it to the CPU DRAM. However, adding a new dedicated
hardware component to the existing GPU architecture requires tremendous
industry efforts and is not ready for immediate deployment. This design may
not be general enough to accommodate future DNN models and accelerator
architectures.

To tackle these challenges, we propose an memory-efficient deep neural network
training framework by compressing the activation data using flexible error-
bounded lossy compression. Compared to lossy compression approaches such as
JPEG (Wallace, 1992) and JPEG2000 (Taubman and Marcellin, 2012), error-bounded
lossy compression can provide a more strict control over the errors occurred
to the floating point activation data; compared to lossless compression such
as GZIP (Deutsch, 1996) and Zstd (Zstandard, 2020), it can offer a much higher
compression ratio to gain higher memory consumption reduction and performance
improvement. The key insights explored in this work include: (i) the impact of
compression errors occurred in the activation data on the gradients and the
entire DNN training process under the strict error-controlling lossy
compression can be theoretically and experimentally analyzed, and (ii) the
training accuracy can be well maintained based on a dynamic fine-grained
control over error-bounded lossy compression (i.e., compression error). To the
best of our knowledge, this is the first work to theoretically investigate the
compression error propagation during DNN training and leverage this analysis
to significantly reduce the memory consumption for training large DNNs while
maintaining the high training accuracy. In summary, this paper makes the
following contributions:

  * •

We propose a novel memory-efficient DNN training framework via dynamically
compressing the intermediate activation data through fine-grained error-
bounded lossy compression.

  * •

We provide a thorough analysis on the impact of compression error propagation
during DNN training from both theoretical and empirical perspectives.
Moreover, we improve the lossy compression algorithm to avoid the significant
alteration (vanish or explosion) of gradients due to the errors introduced to
the activation data.

  * •

We propose an adaptive scheme to dynamically configure the error-bounded lossy
compression based on a series of current training status data.

  * •

We evaluate our proposed training framework on four widely-adopted DNN models
(AlexNet, VGG-16, ResNet-18, ResNet-50) with the ImageNet-2012 dataset and
compare it against state-of-the-art solutions. Experimental results show that
our design can reduce the memory consumption by up to 13.5$\times$ and
1.8$\times$ compared to the original training framework and the state-of-the-
art method, respectively, under the same batch size.

##  2\. Background and Motivation

In this section, we first present the background information on training
large-scale DNNs and error-bounded lossy compression for floating-point data.
We then discuss the motivation of this work and research challenges.

###  2.1. Training Large-Scale DNNs

DNNs have been widely studied in recent years, which have proven their
capabilities in various science domains such as vision detection (Russakovsky
et al., 2015) and natural language processing (NLP) (Young et al., 2018). Each
neural network contains many connected layers that allow input data flow to
compute through and provide the associated results. There are different types
of layers that have been developed for DNNs, among which convolutional layer
and its optimization have become one of the research focuses for the deep
learning (DL) community in the recent years, e.g., the invention of TPU
(Google, 2018). They consist of a set of filters and kernels that perform
convolutions with a sliding window on the input data. Convolutional neural
networks (CNNs) typically consist of convolutional, activation, pooling, batch
normalization, local response normalization, fully connected, and dropout
layers. To further improve the state-of-the-art accuracy on large real-world
datasets, recent works (He et al., 2016; Szegedy et al., 2017) have proposed
deeper and wider nonlinear CNN architectures, e.g., ResNet (He et al., 2016)
and Inception V4 (Szegedy et al., 2017). For instance, ResNet-50 with 50
convolutional layers can provides 71.49% top-1 accuracy for the ImageNet-12
dataset, but the early linear CNN architecture like AlexNet (Krizhevsky et
al., 2012) with just five sequential layers can only provide 57.41% top-1
accuracy.

Figure 1. Data flow in a sample iteration of training CNNs. Figure 2. Memory
consumption and top-1 accuracy of different state-of-the-art neural networks.

Training deep and wide neural networks has become increasingly challenging.
While many state-of-the-art deep learning frameworks such as TensorFlow (Abadi
et al., 2016) and PyTorch (Paszke et al., 2019) can provide high computation
throughput by leveraging the massive parallelism on general-purpose
accelerators such as GPUs, one of the most common bottlenecks remains to be
the high memory consumption during the training process, especially
considering the limited on-chip memory available on modern DNN accelerators.
This is mainly due to the ever-increasing size of the activation data computed
in the training process. Training a neural network involves many training
iterations/epochs to fine tune the model weights. Each iteration includes a
forward and backward propagation, as shown in Figure 1. The intermediate
activation data (the output from each neuron) generated by every layer are
commonly kept in the memory until the backpropagation reaches this layer
again. Several works (Chen et al., 2016; Gomez et al., 2017; Rhu et al., 2016;
Cui et al., 2016; Evans et al., 2020) have pointed out that there would be a
huge gap between the time when the activation data is generated in the forward
propagation and the time when the activation data is reused in the back
propagation, especially when training very deep and wide neural networks.
Figure 2 shows the memory consumption of various neural networks. For these
CNNs, comparing with the model/weight size, activation data size is much
larger, since the convolution kernels are relatively small compared to the
activation tensors. In summary, we are facing two main challenges due to the
high memory consumption in today’s deep learning training: (1) it is
challenging to scale up the training process under a limited GPU memory
capacity; and (2) a limited batch size leads to low training performance. The
performance challenge is further exacerbated when the error-and-trail based
approach is typically used in practice for developing new DNN models and
algorithms.

In recent years, several works have been proposed to reduce the memory
consumption for DNN training, including activation data recomputation(Chen et
al., 2016; Gomez et al., 2017), migration (Rhu et al., 2016; Cui et al.,
2016), and compression (Evans et al., 2020). Recomputation takes advantage of
the layers with low computational cost, such as pooling layer. Specifically,
it would deallocate the activation data for those layers and recompute them
based on their prior layer during the back propagation when needed. This
method can reduce some unnecessary memory cost, but it can only be applied to
limited types of layers with low performance overhead. Layers such as
convolutional layers often cannot be recomputed efficiently with acceptable
performance overhead.

Another type of methods are proposed around data migration (Rhu et al., 2016;
Cui et al., 2016), which sends the activation data from the accelerator to the
CPU host when generated, and then loads it back from the host when needed.
However, the performance of data migration heavily depends on the interconnect
bandwidth available between the host and the accelerator(s), and the in-node
interconnect technology applied. For example, NVLink (Foley and Danskin, 2017)
technology is currently limited to high-end NVIDIA AI nodes (e.g., DGX series)
and IBM power series. This paper targets to develop a general technique that
can be applied to all types of HPC and datacenter systems.

Last but not least, data compression is another efficient approach to reduce
the memory consumption, especially for conserving the memory bandwidth (Evans
et al., 2020; Lal et al., 2017; Ekman and Stenstrom, 2005). The basic idea
using data compression here is to compress the activation data when generated,
hold the compressed data in the memory, and decompress it when needed.
However, using lossless compression (Choukse et al., 2020) can only provide a
relatively low memory reduction ratio (i.e., compression ratio), e.g.,
typically within 2$\times$. Some other studies such as JPEG-ACT (Evans et al.,
2020) leverages the similarity between activation tensors and images for
vision recognition tasks and apply a modified JPEG lossy compressor to
activation data. This approach provides a higher compression ratio (e.g.,
typically around 7$\times$) compared to the lossless compression based
solution. But it suffers from two main drawbacks. First, it introduces
uncontrollable compression error to activation data and eventually could lose
the control of the overall training accuracy, since JPEG is mainly designed
for images and is an integer based lossy compression. The JPEG based solution
(or other image based lossy compression) may not be suitable for DNNs applied
in a more general and larger scientific context than image processing to
analyze extremely large amounts of data, e.g., large-scale HPC scenarios for
unprecedented discoveries (Wozniak et al., 2018) via deep learning. Second,
the JPEG based solution (Evans et al., 2020) needs a support from a dedicated
hardware component to be added in GPU hardware. Hardware innovation typically
takes 5 to 10 years for adaptation and it cannot be directly deployed to
today’s systems.

We note that all the three methods above are orthogonal to each other, which
means they could be deployed together to maximize the compression ratio and
training performance. Thus, in this paper, we mainly focus on designing an
efficient data compression, more specifically lossy compression based
solution, to achieve the memory reduction ratio beyond the state-of-the-art
approach on CNN models. In addition, since convolutional layers are the most
difficult type of layers for efficient recomputation, our solution focuses on
the convolutional layers to provide high compression ratio with minimum
performance overhead and accuracy loss.

###  2.2. Error-Bounded Lossy Compression

Floating-point data compression has been studied for decades. There are two
main categories: lossless compression and lossy compression. Lossless
compressors such as FPZIP (Lindstrom and Isenburg, 2006) and FPC (Burtscher
and Ratanaworabhan, 2008) can only provide limited compression ratios
(typically up to 2:1 for most scientific data) due to the significant
randomness of the ending mantissa bits (Son et al., 2014).

Lossy compression, on the other hand, can compress data with little
information loss in the reconstructed data. Compared to lossless compression,
lossy compression can provide a much higher compression ratio while still
maintaining useful information for scientific discoveries. Different lossy
compressors can provide different compression modes, such as error-bounded
mode and fixed-rate mode. Error-bounded mode requires users to set an error
bound, such as absolute error bound and point-wise relative error bound. The
compressor ensures the differences between the original data and the
reconstructed data do not exceed the user-set error bound. Fixed-rate mode
means that users can set a target bitrate, and the compressor guarantees the
actual bitrate of the compressed data to be lower than the user-set value.

In recent years, a new generation of lossy compressors for scientific data
have been proposed and developed, such as SZ (Di and Cappello, 2016; Tao et
al., 2017; Liang et al., 2018) and ZFP (Lindstrom, 2014). Unlike traditional
lossy compressors such as JPEG (Wallace, 1992) which are designed for images
(in integers), SZ and ZFP are designed to compress floating-point data and can
provide a strict error-controlling scheme based on user’s requirements. In
this work, we chose to use SZ instead of ZFP because the GPU version of
SZ—cuSZ (Tian et al., 2020)111Compared to CPU SZ, cuSZ can provide much higher
compression and decompression throughput on GPUs.—provides a higher
compression ratio than ZFP and offers the absolute error bound mode that ZFP
does not support (but necessary for our error control). Specifically, SZ is a
prediction based error-bounded lossy compressor for scientific data. SZ has
three main steps: (1) predict each data point’s value based on its neighboring
points by using an adaptive, best-fit prediction method; (2) quantize the
difference between the real value and predicted value based on the user-set
error bound; and (3) apply a customized Huffman coding and lossless
compression to achieve a higher compression ratio.

###  2.3. Research Goals and Challenges

In this paper, we propose a novel memory-driven high performance CNN training
framework that leverages error-bounded lossy compression to significantly
reduce the memory requirement for training in order to allow larger neural
network training and developing. To achieve this goal, there are several
critical challenges to be addressed. First, since we plan to use an error-
bounded lossy compressor, a strictly controlled compression error would be
introduced to the activation data. Second, in order to maintain the training
accuracy curve with a minimum impact to the performance and final model
accuracy, _we must understand how the introduced error would propagate through
the whole training process._ In other words, we must theoretically and/or
experimentally analyze the error propagation, which is challenging. To the
best of our knowledge, there is no prior investigation on this.

Third, once we understand the connection between the controlled error and
training accuracy, how to balance the compression ratio and accuracy
degradation in a fine granularity is also challenging. In other words, a more
aggressive compression can provide a higher compression ratio but also
introduces more errors to activation data, which may significantly degrades
the final model accuracy or training performance (cant converge). Thus, we
must find a balance to offer as high compression ratio as possible to
different layers across different iterations while maintaining minimum impact
to the accuracy.

##  3\. Compression Error Propagation Analysis

In this section, we present the theoretical support of our proposed training
framework—analyzing compression error propagation (1) from activation data to
gradient and (2) from gradient to overall training for convolutional layers.

###  3.1. Modeling Compression Error

cuSZ (Tian et al., 2020) is a prediction based error-bounded lossy compressor
for floating-point data on GPUs. It first uses a dual-quantization technique
to quantize the floating-point input data based on user-set error bound. Then,
it applies Lorenzo based predictor to efficiently predict the value of each
data point based on its neighboring points. After that, a quantization code
(integer) is generated for each value. Finally, a customized Huffman coding is
applied to all the quantization codes. Similar to the original SZ, the error
introduced to the input data after decompression usually forms an uniform
distribution. This is mainly because of the linear-scaling quantization
technique adopted. We refer readers to (Lindstrom, 2017) for more details
about the error distribution of SZ from a statistical perspective.

Figure 3 illustrates an example error distribution when compressing and
decompressing an activation data by cuSZ. The activation data used here for
demonstration is extracted from Conv-5 layer of AlexNet (Krizhevsky et al.,
2012) in a certain iteration. Note that in fact we plot the error distribution
every 50 iterations, and observe that all the error distributions are quite
similar and follow uniform distribution, which is consistent with the
conclusion drawn in the prior work (Lindstrom, 2017). Thus, we propose to use
the uniformly distributed error model for the theoretical analysis and an
error injection based approach to demonstrate the effectiveness of our
theoretical derivation in this section. Note that for the purpose of
theoretical analysis, we inject the error, rather than actually compressing
and decompressing activation data, to demonstrate how uniformly distributed
error propagates from activation data to gradient and then to the whole
training process. We leave the evaluation of actual compression to Section 5.

Figure 3. Sample error distribution of activation data compressed by cuSZ
lossy compression with error bound $10^{-4}$.

###  3.2. Modeling Error Impact on Gradient

Next, we theoretically derive how error propagates from activation data to
gradient and provide an experimental proof based on statistical analysis using
error injection.

As shown in Figure 1, the compressed activation data needs to be decompressed
when the back propagation reaches the corresponding layer. During the back
propagation, each layer computes the gradient to update the weights and the
loss to be propagated to the previous layer (back propagation). As shown in
Figure 4, on the one hand, the loss of activation data for the previous layer
only depends on the current layer’s loss and weight. On the other hand, the
gradient depends not only on the loss of the current layer, but also on the
activation data. We note that the activation data can be compressed during the
forward propagation to reduce the memory consumption. In a conclusion, in
order to understand the impact of compressing the activation data, we must
first understand how compression error introduced to the activation data would
propagate to the gradient.

Figure 4. Data dependencies in one convolutional layer during backpropagation.

(a) Forward pass

(b) Backward pass

Figure 5. Forward and backward computation in a sample convolutional layer.
Kernel size is 2$\times$2, stride is 1, input channel count is 3, and output
channel count is 1.

In the forward pass, multiple kernels perform convolutions on the input
activation data. As shown in Figure 5(a), the kernel is performed on the
activation data (marked in brown) and generates the output value as shown in
red. Similar to forward pass, the backward pass reverses the computation,
where the parameter’s gradient is computed based on the loss (with the same
dimension of the output data in the forward pass) and the original data in the
kernel, as shown in Figure 5(b). Similarly, the activation data and the loss
compute the gradient value as shown in red. More specifically, this value is
computed as

(1)
$G_{k,i}=A_{k,0}{\times}L_{0}+A_{k,1}{\times}L_{1}+...+A_{k,n^{{}^{\prime}}}{\times}L_{n},$

where $G$ is the gradient, $A$ is the activation data, $L$ is the loss, $k$ is
the current channel, $i$ is the value index of the channel, $n$ is the number
of values in the loss matrix, $n^{{}^{\prime}}$ is the corresponding index of
activation data to loss matrix. Note that for simplicity, we ignore all the
$i$s on the right side of Equation (1). Note that here, if the number of
output channels is greater than 1, which is true for most convolutional
layers, the same process will be used for multiple kernels, as shown in Figure
5(b), and the above formula still holds in this case.

Based on our analysis in Section 3.1, the error introduced to the activation
data is uniformly distributed, thus, in the backpropagation, we can have

(2) $\displaystyle G^{{}^{\prime}}_{k,i}$
$\displaystyle=A^{{}^{\prime}}_{k,0}{\times}L_{0}+A^{{}^{\prime}}_{k,1}{\times}L_{1}+...+A^{{}^{\prime}}_{k,n^{{}^{\prime}}}{\times}L_{n},$
$\displaystyle A^{{}^{\prime}}_{k,j}$ $\displaystyle=A_{k,j}+e,e\in[-eb,+eb],$

where $A^{{}^{\prime}}$ is the decompressed activation data, $G^{{}^{\prime}}$
is the gradient altered by the compression error, $e$ is the error, and $eb$
is the user-set absolute error bound. After a simple transformation, we can
have

$\displaystyle G^{{}^{\prime}}_{k,i}$
$\displaystyle=A_{k,0}{\times}L_{0}+A_{k,1}{\times}L_{1}+...+A_{k,n^{{}^{\prime}}}{\times}L_{n}$
$\displaystyle\quad+e_{0}{\times}L_{0}+e_{1}{\times}L_{0}+...+e_{n}{\times}L_{n}$
(3) $\displaystyle=G_{k,i}+E,$ $\displaystyle E$
$\displaystyle=e_{0}{\times}L_{0}+e_{1}{\times}L_{1}+...+e_{n}{\times}L_{n},$

where $E$ is the gradient error.

Although it is not possible to calculate or predict the exact value of every
element $E$, we can predict its distribution based on our previous assumption.
We also note that the batch size of a typical neural network is usually
relatively large during the training process, such as 128$\sim$256, since
larger batch size results in higher training performance in general. As a
result, the final gradient for updating weights can be computed as follows.

$\displaystyle G^{{}^{\prime}}_{final}$
$\displaystyle=Average(G^{{}^{\prime}}_{0},G^{{}^{\prime}}_{1},...,G^{{}^{\prime}}_{N}),$
(4) $\displaystyle E_{final}$ $\displaystyle=Average(E_{0},E_{1},...,E_{N})$
$\displaystyle=(e_{0,0}{\times}L_{0,0}+e_{0,1}{\times}L_{0,1}+...+e_{0,n}{\times}L_{0,n}$
$\displaystyle\quad+e_{1,0}{\times}L_{1,0}+e_{1,1}{\times}L_{1,1}+...+e_{1,n}{\times}L_{1,n}$
$\displaystyle\quad\dots$
$\displaystyle\quad+e_{N,0}{\times}L_{N,0}+e_{N,1}{\times}L_{N,1}+...+e_{N,n}{\times}L_{N,n})/N,$
$\displaystyle e$ $\displaystyle\in[-eb,+eb],$

where $N$ is the number of batch size. Note that all $e$ are independently and
uniformly distributed as discussed in Section 3.1. Although $L$ can be related
with each other in the same batch, they are still independently distributed
across different batches. According to (Sum of normally distributed random
variables, [n. d.]), the sum of a series independent random variables with the
same distribution follows a normal distribution, which means the error
distribution of the gradient can be expected to be be normally distributed. We
identify that the distribution of loss $L$ for one input (i.e., one image) is
highly concentrated in zero, where the highest value in the loss is usually
much larger than the average of $L$. Thus, we can simplify our Equation 4 to

(5) $\displaystyle E_{final}$ $\displaystyle\approx
e_{0}{\times}L_{max,0}+e_{1}{\times}L_{max,1}+...+e_{N}{\times}L_{max,N},$
$\displaystyle e$ $\displaystyle\in[-eb,+eb],$

where $L_{max}$ is the maximum value in the loss for each input alone. In
order to reduce the complexity, we can greatly improve the performance with
fewer parameters that need to be collected.

(a) Original zeros been compressed

(b) Original zeros remain zero

Figure 6. Distributions of gradient error when injecting modeled compression
error to activation data. Note that number of elements is normalized to ensure
different layers under the same scale. Data is collected every 100 iterations.

Next, we inject error (to simulate the compression error) to the activation
data in convolutional layers based on our error model discussed in Section
3.1. Then, we collect the error of the gradient in the backpropagation.

Figure 6(a) illustrates the normalized error distribution of gradients
collected from different layers, all of which follow the normal distribution
as expected. In fact, by calculating the percentage of the area within
$\pm\sigma$ of each curve, we can get a value close to 68.2%, which confirms
our theoretical derivation. After that, we need to figure out how to predict
$\sigma$ before compression in order to calculate the desired and acceptable
error bound for each layer’s activation data.

First, we note that $\sigma$ is highly related to the number of elements that
are combined together. In general, more elements result in larger $\sigma$,
vice versa. A $2\times$ increase of elements results in $\sqrt{2}\times$
increase of $\sigma$, which means that more uncertainties has been added to
the system. Second, $\sigma$ is also related to the value scale, in this case,
the average of the loss $L$ at the current layer. Note that it is not
necessary to compute the average value of the loss every iteration. Instead,
we can compute it every $W$ iterations to reduce the computation overhead,
since this value is relatively stable in a fixed period of time. Based on
these two parameters, we can estimate $\sigma$ by the following equation:

(6) $\sigma\approx a\bar{L}\sqrt{N}eb,$

where $L$ is the loss matrix, $N$ is the batch size, and $a$ is an empirical
coefficient. Note that this coefficient $a$ is unchanged for different neural
networks because it is essentially a simplified value of the previous
equation.

Finally, we note that there is a noticeable fraction of activation data to be
zeros, however, our above analysis so far does not cover it. When compressing
a series of continuous zeros, cuSZ may change them into a small value within
the user-set error bound instead of following the error distribution. In such
case, we manually set these values (lower than the error bound) to zeros when
decompression, meaning no error would be introduced when facing continuous
zeros. Moreover, in some cases, the activation data may contain many zeros due
to the activation function layer (such as the ReLU layer) before the current
convolutional layer. If this happens, the activation data of convolutional
layer can be quickly recomputed through the activation function instead of
being saved, which will essentially erase the negative values to zeros. Since
lossy compression such as cuSZ is unlikely to change the sign of the
activation data value, these data will remain at zero.

Figure 6(b) shows the distribution of the gradient error after we inject the
error to the activation data (maintaining zeros unchanged). Compared with
Figure 6(a), we can observe a decrease of $\sigma$, but it still holds a
normal distribution. This decrease is partly because of the reduction of the
number of elements in Equation 4, since those zeros do not have any error. In
these cases, we can revise the prediction of $\sigma$ accordingly by the
following equation:

(7) $\sigma^{{}^{\prime}}=\sigma\sqrt{R},$

where $R$ is the ratio of non-zero elements percentage in the activation data.
Again, in practice, we do not need to compute this ratio every iteration but
every $W$ iterations, since this ratio is relatively stable in a fixed period
of time.

###  3.3. Error Propagation Analysis

Finally, we discuss the error propagated from gradient to overall training
curve using an experimental analysis.

Our goal is to identify the maximum acceptable gradient error that would cause
little or no accuracy loss. According to our theoretical and experimental
proofs in Section 3.2, the gradient error can be modeled as normally
distributed error. In this section, we follow the same strategy used in the
last section to inject error to the gradient that follows our error model and
perform the analysis and evaluation.

We propose to use momentum for alleviating the impact of the gradient error.
Momentum has been widely used in most of neural network training (Dong et al.,
2018). Actually, in order to update the weights, it is based not only on the
gradient computed from the current iteration but also on the momentum. In
other words, both the gradient and the momentum (with the same dimension as
the weights and gradient) take up a portion of the updated data for weights.
Thus, it is critical to maintain an accurate momentum vector similar to the
error-free one to guide the direction of weight update.

Figure 7. Overview of our proposed memory-efficient CNN training framework.

In fact, the momentum is computed based on the current layer’s gradient, and
the momentum error is directly correlated to the gradient error. While thanks
to the normally distributed gradient error, which is centralized and symmetric
in the original direction, the momentum error is relatively low compared to
the gradient error. Therefore, this does help the training towards the correct
direction—even a few iterations may generate the undesired gradient, they can
be offset quickly through the momentum based on the estimated gradient error.

##  4\. Design Methodology

In this section, we describe the overall design of our proposed lossy
compression supported CNN training framework and analyze the performance
overhead.

Our proposed memory driven framework is shown in Figure 7 and Figure 1. We
iteratively repeat the process shown in Figure 7 for each convolutional layer
in every iteration. Our proposed framework mainly include four phases: (1)
parameter collection of current training status for adaptive compression, (2)
gradient assessment to determine the maximum acceptable gradient error, (3)
estimation of compression configuration (e.g, absolute error bound), and (4)
compression/decompression of activation data with cuSZ.

###  4.1. Parameter Collection

First, we collect the parameters of current training status for the following
adjustment of lossy compression configurations. Our framework mainly collects
two types of parameters: (1) offline parameters in CNN architecture, and (2)
semi-online parameters including activation data samples, gradient, and
momentum.

First of all, we collect multiple static parameters including batch size,
activation data size of each convolutional layer, and the size of its output
layer. We need these parameters because they affect the number of elements
combined into each value in the gradient and hence affect $\sigma$ in its
normal error distribution, as discussed in Section 3. It would also help the
framework collects corresponding online parameters.

For the semi-online parameters, we collect the sparsity of activation data and
the its average loss in backpropagation to estimate how the compression error
would propagate from the activation data to the gradient. For the gradient, we
compute the average value of its momentum. Note that in many DNN training
framework such as Caffe (Jia et al., 2014) and TensorFlow (Abadi et al.,
2016), momentum is naturally supported and activated, so it can be easily
accessed. The data collection phase is shown as the dark thin arrows in Figure
7. Note that we still use decompressed data in training for collecting these
parameters here.

Moreover, an active factor $W$ needs to be set in the beginning of training
process to adjust the overall activeness in our framework. $W$ is used to
determine the activeness of our parameter extraction. We only extract semi-
online parameters every $W$ iterations to reduce the computation overhead and
improve the overall training performance. As discussed in Section 3, these
parameters vary relatively slowly during the training, thus, we only need to
estimate the error impact in a fixed iteration interval in our framework. In
this paper, we set $W$ to 1000 as default, which provides high accuracy and
low overhead in our evaluation.

###  4.2. Gradient Assessment

Next, we estimate the limit of the gradient error that would result in little
or no accuracy loss to the training curve, as shown in Figure 7. Even with the
help of the offset from momentum, we still want to keep the gradient of each
iteration as close as possible to the original one. Based on our previous
analysis in Section 3, we need to determine the acceptable $\sigma$ in the
gradient error distribution that minimizes the impact to the overall training
accuracy curve. We use 1% as the acceptable error rate based on our empirical
study, i.e., the $\sigma$ in the momentum error model needs to be

(8) $\displaystyle\sigma=0.01M_{Average}$

where $M_{Average}$ is the average value of the momentum. Note that here we
use the average instead of the modulus length of the momentum because we focus
on each individual value of the gradient, and the average value can be more
representative. Also, note that we do not want to estimate the average value
of gradient and momentum; instead, we seek an empirical solution to directly
monitor the average values. Based on our experiment, the gradient does not
tend to vary dramatically in a short time period during training.

###  4.3. Activation Assessment

After that, we dynamically configure the lossy compression for activation data
based on the gradient assessment in the previous phase and the collected
parameters as shown in Figure 7. As discussed in Section 3, we need $\sigma$
(from gradient error model), $R$ (sparsity of activation data), $\bar{L}$
(average value of current loss), and $N$ (batch size) to determine the
acceptable error bound for compressing the activation data at current layer.
We simplify our estimator to the following:

(9) $eb=\frac{\sigma}{a\bar{L}\sqrt{NR}}$

where $eb$ is the absolute error bound for activation data, $\sigma$ describes
the acceptable error distribution in the gradient, $a$ is the empirical
coefficient, $\bar{L}$ is the average value of current layer’s loss, $N$ is
the batch size, and $R$ is the sparsity ratio of activation data.

###  4.4. Adaptive Compression

In the last phase, we deploy the lossy compression with our optimized
configuration to the corresponding convolutional layers. We also monitor the
compression ratio for analysis. We decompress the compressed activation data
in the backpropagation when needed. As discussed in Section 3, we need to
force zeros in activation data to remain unchanged in compression algorithm.
However, the current cuSZ algorithm may result in a series of small values
within the error bound when encountering continuous zeros.

To solve this issue, we propose a way to bypass this, which is to take
advantage of the data recomputation technique. Specifically, we recalculate
the activation function layer when this happens, so that these zeros can
remain zero after decompression, while the other values are still decompressed
values. For a more general solution, we propose to modify cuSZ for the case of
compressing continuous zeros. More specifically, we add a filter to the
decompression process to re-zero those values within the error bound. Note
that we propose to modify the decompression algorithm instead of the
compression algorithm because it would not affect our compression ratio but
only introduce little extra overhead.

##  5\. Experimental Evaluation

In this section, we evaluate our proposed framework from four aspects,
including (1) evaluation of compression error impact on gradient, (2)
evaluation of error propagation from gradient to training curve, (3)
comparison between our proposed framework and the state-of-the-art method, and
(4) performance evaluation.

###  5.1. Experimental Setup

Our evaluation are conducted with Caffe (Jia et al., 2014) and Tensorflow
(Abadi et al., 2016). We choose Caffe for single-node experiment due to its
easy-to-modify and choose Tensorflow for multi-node evaluation due to its wise
use. Our experiment platform is the TACC Longhorn system (Longhorn subsystem,
2020), of which each GPU node is equipped with 4 Nvidia Tesla V100 GPUs (GPU,
2020) per node. Our evaluation dataset is the ImageNet-2012 (Krizhevsky et
al., 2012). We use the CNN models for image classification including AlexNet
(Krizhevsky et al., 2012), VGG-16 (Simonyan and Zisserman, 2014), ResNet-18
and ResNet-50 (He et al., 2016).

###  5.2. Error Propagation Evaluation

First, we evaluate our proposed theoretical analysis in Section 3.2. Based on
Equation 6 and Equation 7, we can estimate $\sigma$ which stands for how error
is distributed in the gradient. After implementing our estimation, we identify
that coefficient $a$ in Equation 6 is $0.32$ based on our experiment. This is
reasonable because if we consider the extreme condition that the batch size
$N=1$, the error distribution in the gradient will be the same as the SZ lossy
compression to uniformly distribute and result in $a=1/3$. We also evaluate
our estimation on different layers of AlexNet and VGG-16 using the batch size
of 256, as shown in Figure 8. We can clearly observe that the coefficient and
how our our estimated value aligns with the actual error distribution. This
means that we can not only estimate the error propagation, but also determine
the error bound based on a given acceptable $\sigma$ error distribution.

Figure 8. Comparison of $\sigma$ from real measured distribution and from
predicted distribution.

Next, we evaluate the error propagation from gradient to overall training
curve in terms of accuracy. As we target to cause little or no accuracy loss,
we focus on the iterations close the end of training in this evaluation, since
the accuracy is hard to be increased when the training is close to the end. To
reduce the training time and find an empirical solution, we pre-train the
model without the help of our framework and save the snapshot every epoch.
Then, we perform our evaluation of error propagation from different iterations
from different snapshots. Figure 9 shows our experiment with AlexNet starting
from the iteration of 190,000 with the batch size of 256. On the one hand, we
can observe from the zoomed in sub-figure that $\sigma=0.05$ would result in
an unacceptable error loss that cannot be eventually recovered. On the other
hand, $\sigma=0.02$ can provide better accuracy and higher compression ratio,
but it does affect the accuracy a bit in some cases. Thus, considering that
our goal is a general solution for all convolutional layers, we eventually
choose $\sigma=0.01$ as default in our framework; in other words, the target
$\sigma$ is 1% of the average of gradient.

Figure 9. Evaluation on the overall training accuracy curve with different
fraction of error introduced to the gradient. $G$ represents for the average
value of gradient with the coefficient value used in our framework.

###  5.3. Memory Reduction Evaluation

We then test our framework on various popular CNNs with the ImageNet dataset
and evaluate its memory reduction capability. Figure 10 illustrates the result
with AlexNet. The red and blue lines are the training accuracy curves of
baseline training and our framework. We can observe that these two curves are
very close to each other, meaning our framework does not obviously affect the
training accuracy. We also illustrate the curve of compression ratio to
iteration in yellow in this figure. In the early stage of the training,
compression ratio can be slightly unstable because of the relatively large
change to the model. Note that the compression ratio will change slightly when
the learning rate changes, because the learning rate only matters when
updating the gradient to the weights. Moreover, for some layers, although the
average maximum loss of each input should be decreased and result in a higher
error bound, the corresponding activation data value is actually increased.
Thus, the compression ratio would not increase even with a higher error bound.

Figure 10. Training accuracy curve comparison between the baseline and our
proposed framework (batch size = 256).

Table 1 shows the compression ratio of convolutional layers that our framework
can provide. There is almost no accuracy loss or only little, with up to
$0.31$% . This thanks to our careful control of compression error and thorough
theoretical analysis and modeling of error impact. Our framework can deliver a
promising compression ratio without heavy efforts of fine-tuning any parameter
for different models. Overall, our proposed framework can provide up to
13.5$\times$ compression ratio with little or no accuracy loss.

Compared with the lossless compression based solution (Rhu et al., 2018),
which reduces the memory usage by within 2$\times$, our framework outperforms
it by over 9$\times$; compared with the the current state-of-the-art lossy
compression based solution (Evans et al., 2020), which uses an image based
lossy compressor to provide up to 7$\times$ compression ratios, our framework
outperforms it by 1.5$\times$ and 1.8$\times$ on ResNet-18 and ResNet-50,
respectively.

Table 1. Comparison of accuracy and activation size between baseline training
and our proposed framework.

Neural Network |  | Top-1
---
Accuracy
| Convolutional
---
Act. Size
| Compress
---
Ratio
baseline | 57.41% |  407 MB |
AlexNet compressed | 57.42% | 30 MB |  13.5$\times$
baseline | 68.05% |  9.30 GB |
VGG-16 compressed | 68.02% | 0.83 GB |  11.1 $\times$
baseline | 67.57% |  3.42 GB |
ResNet-18 compressed | 67.43% | 0.32 GB |  10.7 $\times$
baseline | 71.49% |  10.28 GB |
ResNet-50 compressed | 71.18% | 0.93 GB |  11.0 $\times$

###  5.4. Performance Evaluation and Analysis

Our framework introduces relatively small overhead to the training process
while can greatly reduce the memory utilization and allow larger and wider
neural networks to be trained with limited GPU memory. Moreover, the saved
memory can also be further utilized for larger batch size, which improves the
overall performance. Figure 11 shows the improvement of raw performance (i.e.,
images per second) with increasing batch size on both single-GPU and multi-
node/multi-GPU cases. We can clearly observe the raw performance can be
improved with both use cases.

Another potential way to improve the performance from increasing the batch
size is faster convergence speed to well trained status (You et al., 2017).
More batch size can lead to a more precise direction for the gradient instead
of just rely on methods such as the momentum to reduce the impact of gradient
uncertainty. Considering the optimized memory utilization from our framework,
we can provide a up to 1.27$\times$ raw performance improvement, while
providing the benefits from larger batch size.

Figure 11. Training performance on ResNet-50 with different $N$.

Regarding the performance overhead of our framework, on the one hand, it needs
to extract usable parameters and compute the compression configuration every
1000 iteration, while the amortized overhead is almost negligible; on the
other hand, thanks to its high working efficiency, cuSZ can provide an
extremely high compression and decompression throughput on GPU (Tian et al.,
2020). The only modification made to cuSZ is the filter that changes all the
values under the error bound to zeros, as discussed in Section 4.4. This helps
us to keep zeros in the activation data unchanged while only causing little
overhead to the framework. Overall, our proposed framework introduces about
17% overhead the training process while keeping the same training batch size.
Moreover, our framework can further utilize the saved memory to increase the
batch size and improve the training performance to offset this performance
overhead. For example, our framework can achieve as low as 7% overhead on
VGG-16 by increasing the batch size from 32 to 256 with the similar memory
consumption. In comparison, the state-of-the-art migration solution such as
Layrub on average achieves a memory reduction of 2.4$\times$ but with a high
training performance overhead of 24.1% (Liu et al., 2018). We also acknowledge
that our framework may introduce high overheads to the networks that contain
convolutional layers with many 1$\times$1 kernels. This is because 1$\times$1
kernels take little time to recompute but requires a relatively high overhead
to compress and decompress. This is because calculating such layers is very
efficient, compared with the GPU (de)compression on similar sizes of
activation data. Thus, our framework is more suitable for the CNNs composed of
larger convolution kernels.

##  6\. Conclusion and Future Work

In this paper, we propose a novel memory-efficient deep learning training
framework. We utilize the SZ error-bounded lossy compressor to reduce the
memory consumption of convolutional layers. We develop an error propagation
model and prove its accuracy. We evaluate our proposed framework on several
popular CNNs with the ImageNet dataset. The result shows that our framework
significantly reduces the memory usage by up to 13.5$\times$ with little or no
accuracy loss. Compared with the state-of-the-art compression based approach,
our framework can provide an performance improvement of up to 1.8$\times$. We
plan to implement those orthogonal methods such as data migration and
recomputation into the framework for higher performance and more memory
reduction. Moreover, we will further optimize the compression and
decompression performance on the activation data.

## Acknowledgments

This research is supported by the National Science Foundation under Grants
OAC-2034169 and OAC-2003624. The authors acknowledge the Texas Advanced
Computing Center (TACC) at The University of Texas at Austin for providing HPC
resources that have contributed to the research results reported within this
paper.

## References

  * (1)
  * Abadi et al. (2016) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. 2016\.  Tensorflow: Large-scale machine learning on heterogeneous distributed systems.  _arXiv preprint arXiv:1603.04467_ (2016).
  * Ben-Nun and Hoefler (2019) Tal Ben-Nun and Torsten Hoefler. 2019.  Demystifying parallel and distributed deep learning: An in-depth concurrency analysis.  _ACM Computing Surveys (CSUR)_ 52, 4 (2019), 1–43.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ (2020).
  * Burtscher and Ratanaworabhan (2008) Martin Burtscher and Paruj Ratanaworabhan. 2008.  FPC: A high-speed compressor for double-precision floating-point data.  _IEEE Trans. Comput._ 58, 1 (2008), 18–31.
  * Challenge (2020) Large Scale Visual Recognition Challenge. 2020\.  http://www.image-net.org/challenges/LSVRC/.
  * Chen et al. (2016) Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016\.  Training deep nets with sublinear memory cost.  _arXiv preprint arXiv:1604.06174_ (2016).
  * Choukse et al. (2020) Esha Choukse, Michael B Sullivan, Mike O’Connor, Mattan Erez, Jeff Pool, David Nellans, and Stephen W Keckler. 2020.  Buddy compression: Enabling larger memory for deep learning and HPC workloads on gpus. In _2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA)_. IEEE, 926–939.
  * Collobert and Weston (2008) Ronan Collobert and Jason Weston. 2008.  A unified architecture for natural language processing: Deep neural networks with multitask learning. In _Proceedings of the 25th International Conference on Machine Learning_. ACM, 160–167.
  * Cui et al. (2016) Henggang Cui, Hao Zhang, Gregory R Ganger, Phillip B Gibbons, and Eric P Xing. 2016.  Geeps: Scalable deep learning on distributed gpus with a gpu-specialized parameter server. In _Proceedings of the Eleventh European Conference on Computer Systems_. ACM, 4.
  * Deutsch (1996) Peter Deutsch. 1996\.  _GZIP file format specification version 4.3_.  Technical Report.
  * Di and Cappello (2016) Sheng Di and Franck Cappello. 2016.  Fast error-bounded lossy HPC data compression with SZ. In _2016 IEEE International Parallel and Distributed Processing Symposium_. IEEE, 730–739.
  * Dong et al. (2018) Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. 2018.  Boosting adversarial attacks with momentum. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 9185–9193.
  * Ekman and Stenstrom (2005) Magnus Ekman and Per Stenstrom. 2005.  A robust main-memory compression scheme. In _32nd International Symposium on Computer Architecture (ISCA’05)_. IEEE, 74–85.
  * Evans et al. (2020) R David Evans, Lufei Liu, and Tor M Aamodt. 2020.  JPEG-ACT: Accelerating Deep Learning via Transform-based Lossy Compression. In _2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA)_. IEEE, 860–873.
  * Foley and Danskin (2017) Denis Foley and John Danskin. 2017.  Ultra-performance Pascal GPU and NVLink interconnect.  _IEEE Micro_ 37, 2 (2017), 7–17.
  * Gomez et al. (2017) Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. 2017.  The reversible residual network: Backpropagation without storing activations. In _Advances in neural information processing systems_. 2214–2224.
  * Google (2018) Google. 2018.  https://cloud.google.com/tpu.
  * GPU (2020) NVIDIA V100 TENSOR CORE GPU. 2020\.  https://www.nvidia.com/en-us/data-center/v100/.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_. 770–778.
  * Hecht-Nielsen (1992) Robert Hecht-Nielsen. 1992\.  Theory of the backpropagation neural network.  In _Neural networks for perception_. Elsevier, 65–93.
  * Huang et al. (2019) Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. 2019\.  Gpipe: Efficient training of giant neural networks using pipeline parallelism. In _Advances in neural information processing systems_. 103–112.
  * Jia et al. (2014) Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014\.  Caffe: Convolutional architecture for fast feature embedding. In _Proceedings of the 22nd ACM international conference on Multimedia_. ACM, 675–678.
  * Jin et al. (2019) Sian Jin, Sheng Di, Xin Liang, Jiannan Tian, Dingwen Tao, and Franck Cappello. 2019\.  Deepsz: A novel framework to compress deep neural networks by using error-bounded lossy compression. In _Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing_. 159–170.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  Imagenet classification with deep convolutional neural networks. In _Advances in neural information processing systems_. 1097–1105.
  * Lal et al. (2017) Sohan Lal, Jan Lucas, and Ben Juurlink. 2017.  E^ 2MC: Entropy Encoding Based Memory Compression for GPUs. In _2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS)_. IEEE, 1119–1128.
  * Li (2020) Chuan Li. 2020.  OpenAI’s GPT-3 Language Model: A Technical Overview.  https://lambdalabs.com/blog/demystifying-gpt-3/.
  * Liang et al. (2018) Xin Liang, Sheng Di, Dingwen Tao, Sihuan Li, Shaomeng Li, Hanqi Guo, Zizhong Chen, and Franck Cappello. 2018\.  Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets.  (2018).
  * Lindstrom (2014) Peter Lindstrom. 2014\.  Fixed-rate compressed floating-point arrays.  _IEEE Transactions on Visualization and Computer Graphics_ 20, 12 (2014), 2674–2683.
  * Lindstrom (2017) Peter Lindstrom. 2017\.  _Error distributions of lossy floating-point compressors_.  Technical Report. Lawrence Livermore National Lab.(LLNL), Livermore, CA (United States).
  * Lindstrom and Isenburg (2006) Peter Lindstrom and Martin Isenburg. 2006.  Fast and efficient compression of floating-point data.  _IEEE Transactions on Visualization and Computer Graphics_ 12, 5 (2006), 1245–1250.
  * Liu et al. (2018) Bo Liu, Wenbin Jiang, Hai Jin, Xuanhua Shi, and Yang Ma. 2018.  Layrub: layer-centric GPU memory reuse and data migration in extreme-scale deep learning systems. In _Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming_. 405–406.
  * Longhorn subsystem (2020) Longhorn subsystem. 2020\.  https://www.tacc.utexas.edu/systems/longhorn.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library. In _Advances in neural information processing systems_. 8026–8037.
  * Rhu et al. (2016) Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W Keckler. 2016\.  vDNN: Virtualized deep neural networks for scalable, memory-efficient neural network design. In _The 49th Annual IEEE/ACM International Symposium on Microarchitecture_. IEEE Press, 18.
  * Rhu et al. (2018) Minsoo Rhu, Mike O’Connor, Niladrish Chatterjee, Jeff Pool, Youngeun Kwon, and Stephen W Keckler. 2018.  Compressing DMA engine: Leveraging activation sparsity for training deep neural networks. In _2018 IEEE International Symposium on High Performance Computer Architecture (HPCA)_. IEEE, 78–91.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015\.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ 115, 3 (2015), 211–252.
  * Sergeev and Del Balso (2018) Alexander Sergeev and Mike Del Balso. 2018.  Horovod: fast and easy distributed deep learning in TensorFlow.  _arXiv preprint arXiv:1802.05799_ (2018).
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ (2014).
  * Son et al. (2014) Seung Woo Son, Zhengzhang Chen, William Hendrix, Ankit Agrawal, Wei-keng Liao, and Alok Choudhary. 2014\.  Data compression for the exascale computing era-survey.  _Supercomputing Frontiers and Innovations_ 1, 2 (2014), 76–88.
  * Sum of normally distributed random variables ([n. d.]) Sum of normally distributed random variables. [n. d.].  https://en.wikipedia.org/wiki/Sum_of_normally_distributed_random_variables.
  * Szegedy et al. (2017) Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. 2017.  Inception-v4, inception-resnet and the impact of residual connections on learning. In _Thirty-first AAAI conference on artificial intelligence_.
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015.  Going deeper with convolutions. In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_. 1–9.
  * Tao et al. (2017) Dingwen Tao, Sheng Di, Zizhong Chen, and Franck Cappello. 2017\.  Significantly improving lossy compression for scientific data sets based on multidimensional prediction and error-controlled quantization. In _2017 IEEE International Parallel and Distributed Processing Symposium_. IEEE, 1129–1139.
  * Taubman and Marcellin (2012) David Taubman and Michael Marcellin. 2012.  _JPEG2000 image compression fundamentals, standards and practice: image compression fundamentals, standards and practice_. Vol. 642.  Springer Science & Business Media.
  * Tian et al. (2020) Jiannan Tian, Sheng Di, Kai Zhao, Cody Rivera, Megan Hickman Fulp, Robert Underwood, Sian Jin, Xin Liang, Jon Calhoun, Dingwen Tao, and Franck Cappello. 2020.  cuSZ: An Efficient GPU-Based Error-Bounded Lossy Compression Framework for Scientific Data.  (2020), 3–15.
  * Wallace (1992) Gregory K Wallace. 1992\.  The JPEG still picture compression standard.  _IEEE Transactions on Consumer Electronics_ 38, 1 (1992), xviii–xxxiv.
  * Wang et al. (2015) Hao Wang, Naiyan Wang, and Dit-Yan Yeung. 2015.  Collaborative deep learning for recommender systems. In _Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining_. ACM, 1235–1244.
  * Wang et al. (2018) Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, and Tim Kraska. 2018.  Superneurons: dynamic GPU memory management for training deep neural networks. In _Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming_. ACM, 41–53.
  * Wozniak et al. (2018) Justin M Wozniak, Rajeev Jain, Prasanna Balaprakash, Jonathan Ozik, Nicholson T Collier, John Bauer, Fangfang Xia, Thomas Brettin, Rick Stevens, Jamaludin Mohd-Yusof, et al. 2018\.  CANDLE/Supervisor: A Workflow Framework for Machine Learning Applied to Cancer Research.  _BMC Bioinformatics_ 19, 18 (2018), 491.
  * You et al. (2017) Yang You, Igor Gitman, and Boris Ginsburg. 2017.  Scaling sgd batch size to 32k for imagenet training.  _arXiv preprint arXiv:1708.03888_ 6 (2017).
  * Young et al. (2018) Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2018.  Recent trends in deep learning based natural language processing.  _ieee Computational intelligenCe magazine_ 13, 3 (2018), 55–75.
  * Zstandard (2020) Zstandard. 2020\.  http://facebook.github.io/zstd/.


Sewade Ogun, Vincent Colotte, Emmanuel Vincent

# Stochastic Pitch Prediction Improves the Diversity and Naturalness of Speech
in Glow-TTS

###### Abstract

Flow-based generative models are widely used in text-to-speech (TTS) systems
to learn the distribution of audio features (e.g., Mel-spectrograms) given the
input tokens and to sample from this distribution to generate diverse
utterances. However, in the zero-shot multi-speaker TTS scenario, the
generated utterances lack diversity and naturalness. In this paper, we propose
to improve the diversity of utterances by explicitly learning the distribution
of fundamental frequency sequences (pitch contours) of each speaker during
training using a stochastic flow-based pitch predictor, then conditioning the
model on generated pitch contours during inference. The experimental results
demonstrate that the proposed method yields a significant improvement in the
naturalness and diversity of speech generated by a Glow-TTS model that uses
explicit stochastic pitch prediction, over a Glow-TTS baseline and an improved
Glow-TTS model that uses a stochastic duration predictor.

Index Terms: text-to-speech, pitch prediction, generative models

##  1 Introduction

Multi-speaker text-to-speech (TTS) systems have significantly improved in
quality and naturalness over the years. Given the one-to-many mapping between
phonemes and a speaker's utterance, and other variabilities which improve the
perceived quality of speech, some systems model these variabilities by
explicitly learning the speaker's characteristics such as pitch, energy, or
rhythm [1, 2]. Instead, flow-based TTS systems learn the distribution of the
Mel-spectrogram (or audio), which includes all the speaker's characteristics,
given the input tokens and a speaker embedding [3, 4, 5, 6]. Flow-based
generative models are able to fit complex data distributions and generate
diverse utterances with high quality. In addition, they eliminate the problem
of over-smoothness in generated Mel-spectrograms [7], and therefore, they do
not require a post-processing stage for generated Mel-spectrograms, as is
found in many non-flow-based non-generative models, e.g., [8].

However, in the zero-shot multi-speaker TTS scenario, flow-based models are
unable to learn the diversity of speaking styles inherent to each speaker
given only the speaker embeddings. In particular, when these models are used
to generate speech for unseen speakers, the generated utterances often sound
monotonic and less natural. Our goal is to design an improved flow-based
architecture that is able to match the diversity of real speech, including for
unseen speakers.

Many methods have been proposed in the literature for controlling the
expressivity of speakers. These methods use global style tokens or embeddings
learned during training [9, 10, 11] to control the speaking style. Style tags
[12] have also been used to transfer speaking styles from text embeddings to
speech utterances, while other studies, e.g., [13], have attempted to transfer
emotions from one speaker to another using an expressivity encoder. These
methods generally aim to mimic the style of a reference utterance or text,
which is different from our goal.

Several TTS models also explicitly model speech properties such as duration,
energy, timbre, rhythm, or speaking rate. In FastSpeech2 [1] and FastPitch
[2], the duration, pitch contour, and energy of the input tokens are
explicitly learned. ZSM-SS [14] uses a similar approach with external speaker
embeddings, but it normalizes the prosodic features (e.g., energy and pitch)
in the affine layers of the model to improve speaker similarity in the zero-
shot multi-speaker setting. Since these systems are deterministic by design,
they are unable to output diverse renderings for a given text input and
speaker. Recently, Varianceflow [15] replaced the deterministic pitch and
energy predictors in a FastSpeech2 TTS model with normalizing flows. The work
showed the effectiveness of using normalizing flows to learn the prosodic
features, however, it differs from ours as it was proposed for a single-
speaker, non-generative TTS model.

SC-GlowTTS [16] is a zero-shot multi-speaker TTS system that conditions a
Glow-TTS model on external speaker embeddings. SNAC [17], also a flow-based
zero-shot multi-speaker TTS system, uses a coupling layer that explicitly
normalizes the input by the parameters predicted from a speaker embedding
vector. VITS [5], an end-to-end flow-based TTS model, introduced a stochastic
flow-based duration predictor to generate diverse rhythms from the input text,
which was found to improve the rhythm of generated utterances over a
deterministic duration predictor. Your-TTS extends VITS to the multi-speaker
multi-language scenario, enabling the generation of utterances for unseen
speakers in several languages. Although all these models inherently enable
sampling from the latent representation of the input tokens to generate
diverse utterances, the quality of generated utterances in the multi-speaker
setup is still worse than that of single-speaker TTS models [7].

The focus of this work is on generating realistic and diverse utterances for
speakers unseen during training by explicitly learning the pitch distribution.
To do so, we enhance a Glow-TTS system by integrating a stochastic duration
predictor and a stochastic pitch predictor into the model. The stochastic
duration predictor learns more realistic phoneme duration and gives better
speech rhythm to the utterances while the pitch predictor learns the
distribution of pitch contours in the dataset. These provide a method to
generate diverse and natural utterances for different unseen speakers at
inference time.

Section 2 presents the proposed improved Glow-TTS architecture. Sections 3 and
4 describe the experimental setup and the results. We conclude in Section 5.

##  2 Improved Glow-TTS Architecture

The classical Glow-TTS architecture comprises a Transformer-based encoder, a
flow-based decoder, and a deterministic duration predictor. The Transformer
encoder generates contextual phonetic embeddings from the input tokens, which
are then linearly projected into an 80-dimensional representation of the mean
$\mu$ of the prior distribution. Given $\mu$, a scalar noise temperature $T$,
and a random vector $\epsilon$ sampled from the standard normal distribution,
the latent representation $z$ sampled from the prior distribution can be
expressed as

$\displaystyle z$ $\displaystyle=\mu+T\,\epsilon.$ (1)

At training time, only the mean $\mu$ is predicted and the temperature $T$ is
fixed to $1$. At inference time, a certain value of $T$ (often smaller than
$1$) is chosen, and a latent representation is sampled from the prior
distribution as input to the decoder to generate a Mel-spectrogram.

Figure 1 illustrates the proposed improved architecture with arrows showing
the computation during inference. This architecture incorporates a stochastic
duration predictor and a novel stochastic pitch predictor, and it requires
changes to the decoder.

Figure 1: Proposed Improved Glow-TTS architecture including (a) a Stochastic
Duration Predictor and a Stochastic Pitch Predictor, and (b) an Affine
coupling layer with pitch and speaker conditioning. WN represents Weight
Normalization layers.

###  2.1 Decoder

The decoder is a stack of normalizing flow blocks, each consisting of an
activation normalization layer, an invertible 1x1 convolution layer, and an
affine coupling layer. Classically, in the multi-speaker TTS setup, the
decoder is conditioned on speaker embeddings through the affine coupling
layers. At training time, the decoder inverts the Mel-spectrogram into a
hidden representation, which is aligned to the latent representation from the
encoder using an alignment search algorithm called Monotonic Alignment Search
(MAS). MAS finds the most probable monotonic alignment between the hidden
representation and the mean of the prior distribution.

In this work, the decoder is additionally conditioned on the pitch
information. We use the log-F0 values extracted from the ground-truth Mel-
spectrograms as pitch targets, and set all unvoiced values to zero during
training. The log-F0 is first projected to match the dimension of the decoder
features using a 1D convolution layer, which is then used to condition the
decoder features through the affine coupling layers of the decoder. We squeeze
the projected log-F0 features to match the squeeze ratio of the original
decoder features. At inference, the predicted log-F0 values generated by the
stochastic pitch predictor are used for conditioning.

###  2.2 Duration Prediction

The classical Glow-TTS model predicts the speaker-dependent duration of each
token through conditioning on speaker embeddings. The predicted durations have
been found to yield unnatural rhythm and cannot express how a person speaks at
different speaking rates [5].

To generate a more human-like rhythm, we replace the deterministic predictor
by the stochastic flow-based duration predictor111We used the official
implementation at https://github.com/jaywalnut310/vits. in VITS [5]. This
model is composed of two normalizing flow stages that learn the discrete
duration estimated using the MAS alignment algorithm. At training time, a
first-stage set of flows (called post flows) inverts the discrete duration
which is conditioned on the speaker and input embeddings into a de-discretized
distribution of log-duration, then the second-stage set of flows inverts these
log-durations into a Gaussian noise.222A stop-gradient operator is applied to
the input embeddings when learning the duration and pitch. At inference time,
the post-flows are discarded. A Gaussian noise vector is simply sampled and
passed through the second-stage flows to generate log-durations.

###  2.3 Pitch Prediction

Utterances with diverse stress and intonation can be generated by sampling
different $\epsilon$ in (1). In the multi-speaker TTS scenario, the resulting
distribution of stress and intonation varies depending on the speaker
embedding. We found that this alone is not enough to produce diverse, natural,
and lively utterances for speakers unseen in training.

Thus, we use an explicit flow-based stochastic pitch predictor to learn pitch
contours that closely match the distribution of real contours. This model is
made up of 4 blocks of 1D convolution, dilated and depth-separable
convolutions and spline flows [18]. Variational data augmentation [19] is
applied to increase the dimension of the scalar pitch inputs, as this is
necessary to perform high-dimensional transformation. Specifically, an extra
vector of Gaussian noise is concatenated to the pitch vector or input Gaussian
noise as padding, before passing them through the block of flows. Our
architecture is borrowed from the second set of flow blocks of the stochastic
duration predictor.

During training, the speaker-conditioned input embeddings are expanded to
match the duration of each input token estimated by MAS. These are then used
as input to the flow-based stochastic pitch predictor to reverse the log-F0
values into Gaussian noise. The model is optimized jointly with other modules
via maximum likelihood estimation.2 At inference, the speaker-conditioned
input embeddings are expanded using the predicted duration values, while a
Gaussian noise is sampled to generate log-F0 values for the decoder, as shown
in Fig. 1.

##  3 Experimental Setup

We conducted objective and subjective evaluations on three variants of the
model: the baseline Glow-TTS (54.5 M parameters), Glow-TTS with stochastic
duration predictor (GlowTTS-STD, 55.3 M parameters), and Glow-TTS with
stochastic duration and pitch predictors (GlowTTS-STDP, 56.0 M parameters).

###  3.1 Dataset Preparation

The Common Voice dataset (English subset, version 7.0) [20] was used to train
all the models. Common Voice is a crowdsourced, read speech dataset with very
large speaker diversity. The stochastic pitch predictor can benefit from
learning a diverse distribution of F0 values for speakers in different
contexts from the dataset. The dataset was filtered by selecting only high-
quality speakers based on speaker-level Mean Opinion Scores (MOS) estimated by
a MOS estimator [21]. We used the resulting filtered 4,469-speaker, 230.75 h
dataset for model training and validation. 512 samples were randomly selected
from the dataset for validation. The dataset was resampled from 48 kHz or 32
kHz to 16 kHz, and silences and long pauses were removed using a voice
activity detection tool.333https://github.com/wiseman/py-webrtcvad

F0 contours were extracted from the training and validation sets using pYIN
[22] with 64 ms window size and 16 ms hop size to match the parameters of the
Mel-spectrogram. We found it suitable in order to aid the correct alignment of
the extracted F0 values to the input embeddings during model training. In
addition, a speaker embedding was pre-computed for each utterance from a
speaker verification model444https://github.com/resemble-ai/Resemblyzer
trained on Voxceleb [23]. The embeddings are l2-normalized, 256-dimensional
vectors. Phonemes (without stress) and characters were used as input tokens in
a mixed approach [24]. The input tokens were interspersed with a blank token.

###  3.2 Training and Inference Hyperparameters

All TTS models were trained in automatic mixed-precision mode using a global
batch size of 192 on 4 Nvidia RTX GPUs. Optimization was done using the RAdam
optimizer [25] with a learning rate of 0.001, and a cosine-annealing scheduler
with a linear warm-up of 6,000 steps. It took approximately 4 days for the
models with stochastic pitch prediction and stochastic duration prediction to
completely train while the baseline Glow-TTS was trained for 2 days. Each
model was trained for 200 epochs, and the best model was selected at the end
of the training run using the validation loss.

At inference time, the noise temperature of the prior distribution was set to
0.667 for all models as in Glow-TTS [3]. The noise temperature for the pitch
predictor was set to 0.8, similar to VITS [5]. The noise temperature for the
duration predictor was also set to 0.8 for speaking style evaluation and to
1.0 for reading style evaluation (see Section 3.3).

A 16 kHz HiFi-GAN V1 vocoder [26] trained on the LibriTTS dataset [27] was
used to convert the generated Mel-spectrograms into audio. The vocoder was not
finetuned on Mel-spectrograms generated by the models. All experiments were
performed using the NeMo toolkit [28].

###  3.3 Subjective Evaluation

For all the subjective evaluations, 3 unseen male (p245, p254, and p263) and 3
unseen female speakers (p228, p231, and p250) were selected from the VCTK
dataset [29].555 Speaker embeddings in all of our evaluations were extracted
from the fifth utterance (SpeakerID 005) of the VCTK speaker. Firstly, the
naturalness of the utterances generated by each model both in speaking style
(N-MOS) and reading style (NR-MOS) were evaluated subjectively. We selected 3
short (3–4 s) text sentences from VCTK for the N-MOS evaluation, and 3 long
(9–13 s) text sentences with punctuation from LibriTTS for the NR-MOS
evaluation to generate 18 short and 18 long utterances per model. The
evaluators were asked to rate how likely these are real speaking and read
speech, respectively. Secondly, the speaker similarity (S-MOS) of the
generated utterances to a reference utterance was evaluated. For this, 3
utterances that were recorded by each of the selected speakers in the VCTK
dataset were used. The VCTK reference utterances were re-synthesized using the
pretrained vocoder (VCTK-copy) and were added to the subjective evaluations to
determine the upper bound on the scores. Lastly, the diversity of generated
utterances (D-MOS) was evaluated for each model. For each speaker, 3
utterances were generated from the same text prompt and concatenated. The
evaluators were asked to rate the diversity of the utterances in terms of the
intonation (flow of pitch) of the speaker. Mean opinion score evaluations were
performed on a 1–5 Likert scale. In total, 26 volunteers participated in the
evaluation.

Table 1: N-MOS, NR-MOS, S-MOS of utterances generated for 6 unseen speakers by the baseline Glow-TTS, GlowTTS-STD, GlowTTS-STDP, and VCTK-copy. Bold numbers denote the best system in each row and the systems statistically equivalent to it. | Baseline | GlowTTS-STD | GlowTTS-STDP | VCTK-copy
---|---|---|---|---
N-MOS | Male | $2.92$ | $3.11$ | $\mathbf{3.40}$ | $4.00$
Female | $3.35$ | $\mathbf{3.51}$ | $\mathbf{3.51}$ | $4.40$
Total | $3.13$ | $3.31$ | $\mathbf{3.45}$ | $4.21$
WV-MOS | $4.11$ | $\mathbf{4.17}$ | $\mathbf{4.18}$ | $4.32$
NR-MOS | Male | $2.91$ | $\mathbf{3.24}$ | $\mathbf{3.25}$ | -
Female | $2.98$ | $3.28$ | $\mathbf{3.53}$ | -
Total | $2.95$ | $3.26$ | $\mathbf{3.39}$ | -
S-MOS | Male | $2.43$ | $2.90$ | $\mathbf{3.10}$ | $4.71$
Female | $2.14$ | $\mathbf{3.07}$ | $2.91$ | $4.85$
Total | $2.26$ | $\mathbf{2.98}$ | $\mathbf{2.99}$ | $4.79$
cos-sim | $0.8287$ | $\mathbf{0.8364}$ | $\mathbf{0.8319}$ | $0.8121$

###  3.4 Objective Evaluation

Objective evaluations were carried out similarly to subjective evaluations.
The overall quality (WV-MOS) of the utterances generated by each model was
evaluated using a MOS predictor [30, App. C].666pretrained model can be found
at https://github.com/AndreevP/wvmos Then, the cosine similarity (cos-sim)
between the generated utterances and the reference speaker embeddings was
computed. Furthermore, we compare the distribution of log-F0 values of the
generated utterances to the reference VCTK audio samples. For this evaluation,
10 utterances from each of 10 unseen male speakers and 10 unseen female
speakers were selected from the VCTK dataset.5

##  4 Results

Table 1 shows the results of subjective and objective evaluations. The
standard deviations of N-MOS, NR-MOS, and S-MOS scores lie between $1.00$ and
$1.24$, while those of WV-MOS and cos-sim scores are approximately $0.3$ and
$0.04$ respectively for each model. The best models have been selected (and
highlighted) in each row by computing the $95\%$ confidence intervals (CI) on
the difference between a pair of models. The difference is statistically
significant if the confidence interval lies fully on the positive side of the
real axis.

###  4.1 Naturalness and Quality of Utterances

We found a significant improvement in the naturalness (N-MOS) of speaking
utterances generated by GlowTTS-STD and GlowTTS-STDP over the baseline model.
This shows that the stochastic duration predictor more accurately learns the
speaking style of speakers than the baseline duration predictor. In addition,
stochastic pitch prediction does not degrade the naturalness of utterances,
and in particular, improves or maintains the naturalness of utterances. For
unseen male speakers, we see an improvement of 0.29 MOS points over GlowTTS-
STD.

For longer sentences, the utterances generated by GlowTTS-STD and GlowTTS-STDP
are rated higher than the baseline in general as measured by NR-MOS. Here,
utterances generated by GlowTTS-STDP for female speakers are rated higher than
utterances generated by GlowTTS-STD, but there is no significant difference in
the ratings for male speakers. In general, short utterances were rated higher
than longer utterances. Nevertheless, GlowTTS-STDP still shows significant
improvement in reading-style naturalness over the baseline. In addition, the
utterances generated by GlowTTS-STD and GlowTTS-STDP have higher overall
quality (WV-MOS), however, the ratings are still a distance away from the VCTK
utterances with copy synthesis, which were rated higher in general.

###  4.2 Diversity of Utterances

In Figure 2, we see an increasing trend in the diversity of utterances (D-MOS)
from the baseline to GlowTTS-STDP, with GlowTTS-STDP being able to generate
more diverse utterances than the other models without loss of naturalness. In
general, female utterances were rated higher in diversity than male ones.

Figure 3 shows the distribution of log-F0 values computed for the generated
utterances. Here, we see that the distribution of log-F0 values in GlowTTS-
STDP utterances better matches those of real speech for both male and female
speakers with more spread-out log-F0 values, while those of the baseline and
GlowTTS-STD are narrower, indicating less variability in their predicted
log-F0 values.

Figure 2: D-MOS of utterances generated by the evaluated models for both male
and female unseen speakers. Figure 3: Log-F0 distribution of utterances
generated with GlowTTS, GlowTTS-STD, GlowTTS-STDP, and the real VCTK
utterances for both male and female speakers. Outliers due to F0 estimation
errors have been removed.

###  4.3 Speaker Similarity

The utterances generated by models using a stochastic duration predictor
(GlowTTS-STD and GlowTTS-STDP) exhibit significantly higher speaker similarity
(S-MOS) than the baseline, especially for female speakers. We believe that the
rhythm of speech, controlled by the stochastic duration predictor,
significantly impacts the subjective speaker similarity ratings given by
volunteers. Similarly, the cos-sim values are higher for GlowTTS-STD and
GlowTTS-STDP, however this is not as significant as the S-MOS scores.

##  5 Conclusions

In this paper, the naturalness of utterances was successfully improved using a
stochastic duration predictor and stochastic pitch predictor. Similarly, we
significantly improve the naturalness of long, reading-form utterances. In
addition, we increase the diversity of generated utterances by integrating a
stochastic pitch predictor into Glow-TTS. We show that the distribution of
log-F0 values of our model better matches the distribution of log-F0 values of
real utterances. This work is important, as it provides a means to generate
more realistic and diverse utterances for downstream tasks e.g., data
augmentation for automatic speech recognition and long-form audiobooks. It
also enables us to perform procedural manipulation of the rhythm and the pitch
of generated utterances, given the prior distribution.

##  6 Acknowledgements

This is left blank to meet double-blind requirements.

## References

  * [1] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, ``FastSpeech 2: Fast and high-quality end-to-end text to speech,'' in _International Conference on Learning Representations_ , 2021.
  * [2] A. Łańcucki, ``Fastpitch: Parallel text-to-speech with pitch prediction,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 6588–6592.
  * [3] J. Kim, S. Kim, J. Kong, and S. Yoon, ``Glow-TTS: A generative flow for text-to-speech via monotonic alignment search,'' _Advances in Neural Information Processing Systems_ , vol. 33, pp. 8067–8077, 2020.
  * [4] R. Valle, K. J. Shih, R. Prenger, and B. Catanzaro, ``Flowtron: An autoregressive flow-based generative network for text-to-speech synthesis,'' in _International Conference on Learning Representations_ , 2021.
  * [5] J. Kim, J. Kong, and J. Son, ``Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,'' in _International Conference on Machine Learning_. PMLR, 2021, pp. 5530–5540.
  * [6] C. Miao, S. Liang, M. Chen, J. Ma, S. Wang, and J. Xiao, ``Flow-TTS: A non-autoregressive network for text to speech based on flow,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 7209–7213.
  * [7] Y. Ren, X. Tan, T. Qin, Z. Zhao, and T.-Y. Liu, ``Revisiting over-smoothness in text to speech,'' in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2022, pp. 8197–8213.
  * [8] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan _et al._ , ``Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 4779–4783.
  * [9] Y. Wang, D. Stanton, Y. Zhang, R.-S. Ryan, E. Battenberg, J. Shor, Y. Xiao, Y. Jia, F. Ren, and R. A. Saurous, ``Style Tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,'' in _International Conference on Machine Learning_. PMLR, 2018, pp. 5180–5189.
  * [10] R. Valle, J. Li, R. Prenger, and B. Catanzaro, ``Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 6189–6193.
  * [11] D. Min, D. B. Lee, E. Yang, and S. J. Hwang, ``Meta-stylespeech: Multi-speaker adaptive text-to-speech generation,'' in _International Conference on Machine Learning_. PMLR, 2021, pp. 7748–7759.
  * [12] M. Kim, S. J. Cheon, B. J. Choi, J. J. Kim, and N. S. Kim, ``Expressive text-to-speech using style tag,'' in _Proc. Interspeech 2021_ , 2021, pp. 4663–4667.
  * [13] A. Kulkarni, V. Colotte, and D. Jouvet, ``Improving transfer of expressivity for end-to-end multispeaker text-to-speech synthesis,'' in _29th European Signal Processing Conference (EUSIPCO)_ , 2021, pp. 31–35.
  * [14] N. Kumar, S. Goel, A. Narang, and B. Lall, ``Normalization driven zero-shot multi-speaker speech synthesis,'' in _Proc. Interspeech 2021_ , 2021, pp. 1354–1358.
  * [15] Y. Lee, J. Yang, and K. Jung, ``Varianceflow: High-quality and controllable text-to-speech using variance information via normalizing flow,'' in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 7477–7481.
  * [16] E. Casanova, C. Shulby, E. Gölge, N. M. Müller, F. S. de Oliveira, A. Candido Jr., A. da Silva Soares, S. M. Aluisio, and M. A. Ponti, ``SC-GlowTTS: An efficient zero-shot multi-speaker text-to-speech model,'' in _Proc. Interspeech 2021_ , 2021, pp. 3645–3649.
  * [17] B. J. Choi, M. Jeong, J. Y. Lee, and N. S. Kim, ``SNAC: Speaker-normalized affine coupling layer in flow-based architecture for zero-shot multi-speaker text-to-speech,'' _IEEE Signal Processing Letters_ , vol. 29, pp. 2502–2506, 2022.
  * [18] C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios, ``Neural spline flows,'' _Advances in neural information processing systems_ , vol. 32, 2019.
  * [19] J. Chen, C. Lu, B. Chenli, J. Zhu, and T. Tian, ``VFlow: More expressive generative flows with variational data augmentation,'' in _International Conference on Machine Learning_. PMLR, 2020, pp. 1660–1669.
  * [20] R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, ``Common Voice: A massively-multilingual speech corpus,'' in _Proceedings of the 12th Language Resources and Evaluation Conference_ , 2020, pp. 4218–4222.
  * [21] S. Ogun, V. Colotte, and E. Vincent, ``Can we use common voice to train a multi-speaker TTS system?'' in _IEEE Spoken Language Technology Workshop (SLT)_ , 2023, pp. 900–905.
  * [22] M. Mauch and S. Dixon, ``pYIN: A fundamental frequency estimator using probabilistic threshold distributions,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2014, pp. 659–663.
  * [23] A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, ``Voxceleb: Large-scale speaker verification in the wild,'' _Computer Science and Language_ , 2019\.
  * [24] K. Kastner, J. F. Santos, Y. Bengio, and A. Courville, ``Representation mixing for TTS synthesis,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019, pp. 5906–5910.
  * [25] L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han, ``On the variance of the adaptive learning rate and beyond,'' in _International Conference on Learning Representations_ , 2020.
  * [26] J. Kong, J. Kim, and J. Bae, ``Hifi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,'' _Advances in Neural Information Processing Systems_ , vol. 33, pp. 17 022–17 033, 2020.
  * [27] H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu, ``LibriTTS: A corpus derived from LibriSpeech for text-to-speech,'' in _Proc. Interspeech 2019_ , 2019, pp. 1526–1530.
  * [28] O. Kuchaiev, J. Li, H. Nguyen, O. Hrinchuk, R. Leary, B. Ginsburg, S. Kriman, S. Beliaev, V. Lavrukhin, J. Cook _et al._ , ``NeMo: A toolkit for building AI applications using neural modules,'' _arXiv preprint arXiv:1909.09577_ , 2019.
  * [29] C. Veaux, J. Yamagishi, K. MacDonald _et al._ , ``CSTR VCTK corpus: English multi-speaker corpus for CSTR voice cloning toolkit,'' _University of Edinburgh. The Centre for Speech Technology Research (CSTR)_ , 2017.
  * [30] P. Andreev, A. Alanov, O. Ivanov _et al._ , ``HiFi++: A unified framework for neural vocoding, bandwidth extension and speech enhancement,'' _arXiv:2203.13086_ , 2022.