# CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency
Model

Zhen Ye1, Wei Xue2${\dagger}$, Xu Tan3, Jie Chen1, Qifeng Liu4,2, Yike
Guo2${\dagger}$
1 Hong Kong Baptist University 2 Hong Kong University of Science and
Technology
3 Microsoft Research Asia
4 Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences
†Corresponding authors: Wei Xue<EMAIL_ADDRESS>Yike Guo<EMAIL_ADDRESS>
###### Abstract

Denoising diffusion probabilistic models (DDPMs) have shown promising
performance for speech synthesis. However, a large number of iterative steps
are required to achieve high sample quality, which restricts the inference
speed. Maintaining sample quality while increasing sampling speed has become a
challenging task. In this paper, we propose a Consistency Model-based Speech
synthesis method, CoMoSpeech, which achieve speech synthesis through a single
diffusion sampling step while achieving high audio quality. The consistency
constraint is applied to distill a consistency model from a well-designed
diffusion-based teacher model, which ultimately yields superior performances
in the distilled CoMoSpeech. Our experiments show that by generating audio
recordings by a single sampling step, the CoMoSpeech achieves an inference
speed more than 150 times faster than real-time on a single NVIDIA A100 GPU,
which is comparable to FastSpeech2, making diffusion-sampling based speech
synthesis truly practical. Meanwhile, objective and subjective evaluations on
text-to-speech and singing voice synthesis show that the proposed teacher
models yield the best audio quality, and the one-step sampling based
CoMoSpeech achieves the best inference speed with better or comparable audio
quality to other conventional multi-step diffusion model baselines. Audio
samples are available at https://comospeech.github.io/.

##  1 Introduction

Speech synthesis (Tan et al., 2021) aims to produce realistic audio of humans
and has broadly included text-to-speech (TTS) (Taylor, 2009) and singing voice
synthesis (SVS) (Nishimura et al., 2016) tasks due to the increasing
applications in human-machine interaction and entertainment. The mainstream of
speech synthesis has been dominated by the deep neural network (DNN)-based
methods (Wang et al., 2017) (Kim et al., 2021), and typically a two-stage
pipeline is adopted (Ren et al., 2019) (Lu et al., 2020), in which the
acoustic model first converts the textual and other controlling information
into acoustic features (e.g., mel-spectrogram) and then the vocoder further
transforms the acoustic features into audible waveforms. The two-stage
pipeline has achieved substantial success since the acoustic features, which
are expressed by frames, effectively act as the “relay” to alleviate the
dimension-exploding problem of converting short texts to long audios with a
high sampling frequency.

The quality of the acoustic feature produced by the acoustic model, typically
mel-spectrogram, crucially affects the quality of the synthesized speeches.
Approaches widely used in the industry, such as Tacotron (Wang et al., 2017),
DurIAN (Yu et al., 2019), and FastSpeech (Ren et al., 2019), generally adopt
the convolutional neural network (CNN) and Transformers to predict the mel-
spectrogram from the controlling factor. Diffusion model methods have
attracted much attention because their potential to produce high-quality
samples is well recognized.

A diffusion model (Ho et al., 2020), also named score-based model (Song et
al., 2020b), is based on two processes, a diffusion process that gradually
perturbs data to noise and a reverse process that progressively converts noise
back to data. A critical drawback (Song et al., 2020a) (Yang et al., 2022) of
the diffusion model is that it requires many iterations for the generation.
Several methods based on the diffusion model have been proposed for acoustic
modeling in speech synthesis. Most of these works still have the issue of slow
generation speed.

Grad-TTS (Popov et al., 2021) formulated a stochastic differential equation
(SDE) (Anderson, 1982) to gradually transform the noise to the mel-spectrogram
and a numerical ODE solver is used for solving reverse SDE (Song et al.,
2020b). Although yielding high audio quality, the inference speed is low due
to the large number of iterations ($10\sim 1000$ steps) in the reverse
process. Prodiff (Huang et al., 2022) was further developed to use progressive
distillation (Salimans and Ho, 2022) to reduce the sampling steps. In Liu et
al. (2022b), DiffGAN-TTS adopted an adversarially-trained model to approximate
the denoising function for efficient speech synthesis. In Chen et al. (2022b),
the ResGrad uses the diffusion model to estimate the prediction residual from
pre-trained FastSpeech2 (Ren et al., 2020) and ground truth. Apart from normal
speaking voice, recent studies also focus on voice with more complex
variations in pitch, timing, and expression. For example, Diffsinger (Liu et
al., 2022a) also shows that a well-designed diffusion model can achieve high
quality on synthesized singing voice through one hundred steps of iteration.

From the above discussion, the objectives of speech synthesis are three-fold:

  * •

High audio quality: The generative model should accurately express the nuances
of speaking voice which contribute to the naturalness and expressiveness of
the synthesized audio. Additionally, artefacts and distortions in the
generated audio should also be avoided.

  * •

Fast inference speed: Real-time applications, including communication,
interactive speech and music systems, require the fast generation speed of
audio. When considering making time for other algorithms in an integrated
system, simply being faster than real-time is insufficient for speech
synthesis.

  * •

Beyond speaking: Instead of normal speaking voice, more complex modeling of
voice on pitch, expression,rhythm, breach control and timbre is required such
as singing voice.

Although many efforts have been made, due to the mechanism of the denoising
diffusion process when performing sampling, the trade-off problem among the
synthesized audio quality, model capability and inference speed still exists
in TTS and particularly pronounced in SVS. Existing methods generally seek to
alleviate the slow inference problem rather than solve it fundamentally, and
their speed is still not comparable to conventional methods without relying on
diffusion models such as FastSpeech2 (Ren et al., 2020). Recently, by
expressing the stochastic differential equation (SDE) describing the sampling
process as an ordinary differential equation (ODE), and further enforcing the
consistency property of model on the ODE trajectory, the consistency model
(Song et al., 2023) has been developed, yielding high-quality images with only
one sampling step. However, despite such success in image synthesis, no speech
synthesis model based on the consistency model is known so far. This indicates
the potential of designing a consistency model based speech synthesis method
to achieve both high-quality synthesis and fast inference speed.

In this paper, we propose Consistency Model based method for speech synthesis,
namely CoMoSpeech, which achieves fast and high-quality audio generation. Our
CoMoSpeech is distilled from a pre-trained teacher model. More specifically,
our teacher model leverages the SDE to smoothly transform the mel-spectrogram
into the Gaussian noise distribution and learn the corresponding score
function. After training, we utilize the corresponding numerical ODE solvers
to construct the teacher denoiser function, which is used for further
consistency distillation. Through distillation, our CoMoSpeech with
consistency property is obtained. Ultimately, high-quality audio can be
produced by our CoMoSpeech with a single-step sampling.

We conducted experiments for both TTS and SVS, and the results show that the
CoMoSpeech can generate speeches with one sampling step, more than 150 times
faster than in real-time. The audio quality evaluation also shows that the
CoMoSpeech achieves better or comparable audio quality to other diffusion
model methods involving tens to hundreds of iterations. This makes the speech
synthesis based on the diffusion model truly practical for the first time.

##  2 Background of Consistency Model

Now we briefly introduce the consistency model. Supposing that we have a
dataset with distribution as ${p_{\textrm{data}}}(\mathbf{x})$. The diffusion
model performs sampling by progressively adding the Gaussian noise to diffuse
data, and then adopting a reverse denoising process to generate samples from
noise. For data $\\{\mathbf{x}\\}_{t=0}^{T}$ in the diffusion process where
$p_{0}(\mathbf{x})={p_{\textrm{data}}}(\mathbf{x})$, $p_{T}(\mathbf{x})$ is
Gaussian, and $T$ is the time constant, the forward diffusion process can be
expressed by a SDE (Song et al., 2020b) as

$\mathrm{d}\mathbf{x}=f(\mathbf{x},t)\mathrm{d}t+g(t)\mathrm{d}\mathbf{w},$
(1)

where $\mathbf{w}$ is the standard wiener process, $f(\cdot,\cdot)$ and
$g(\cdot)$ are drift and diffusion coefficients, respectively.
$f(\mathbf{x},t)$ always acts as a time-dependent scaling as
$f(\mathbf{x},t)=f(t)\mathbf{x}$ in previous work (Song et al., 2020b; Karras
et al., 2022), thus

$\mathrm{d}\mathbf{x}=f(t)\mathbf{x}\mathrm{d}t+g(t)\mathrm{d}\mathbf{w}.$ (2)

A notable property of the above SDE is that it corresponds to a probability
flow ODE which indicates the sampling trajectory distribution of SDE at time
$t$ (Song et al., 2020b; Karras et al., 2022), as

$\mathrm{d}\mathbf{x}=\left[f(t)\mathbf{x}-\frac{1}{2}g(t)^{2}\nabla\log
p_{t}(\mathbf{x})\right]\mathrm{d}t,$ (3)

where ${\nabla}\log{p_{t}}({\mathbf{x}})$ is the score function of
${p_{t}(\mathbf{x})}$ (Hyvärinen and Dayan, 2005). The probability flow ODE
eliminates the stochastic $\mathbf{w}$, thus generating a deterministic
sampling trajectory.

Figure 1:  An illustration of CoMoSpeech. Our CoMoSpeech distills the multi-
step sampling of the teacher model into one step utilizing the consistency
constraint.

As long as the score function ${\nabla}\log{p_{t}}({\mathbf{x}})$ is known,
the probability flow ODE in (3) can be used for sampling. Supposing
$D(\mathbf{x}_{t},t)$ is the “denoiser” which denoise the sample
$\mathbf{x}_{t}$ at step $t$, the score function can be obtained by minimizing
the denoising error $||D(\mathbf{x}_{t},t)-\mathbf{x}||^{2}$ (Karras et al.,
2022), yielding:

${\nabla}\log{p}_{t}({\mathbf{x}})=(D(\mathbf{x}_{t},t)-\mathbf{x}_{t})/\sigma_{t}^{2},$
(4)

where $\sigma_{t}^{2}=\int{g{{(t)}^{2}}}dt$. Further, the probability flow ODE
based sampling can be performed by first sampling from a noise distribution
and then denoising to the true sample by the numerical ODE solver such as
Euler and Heun solvers (Song et al., 2020b; Karras et al., 2022). However, the
ODE solvers normally involve many iterations causing a slow sampling.

To accelerate sampling (Song et al., 2023), consistency property has been
proposed to impose both:

$D(\mathbf{x}_{t},t)=D(\mathbf{x}_{t^{\prime}},t^{\prime})$ (5)

for any $t$ and $t^{\prime}$, and

$D(\mathbf{x}_{0},0)=\mathbf{x}_{0}.$ (6)

In this way, a consistency model can be obtained, and one-step sampling
$D(\mathbf{x}_{T},T)$ can be achieved since all points in the sampling
trajectory of probability flow ODE is directly linked to the original
distribution $p_{0}(\mathbf{x})$. The consistency model can be trained either
in isolation or by distilling from a pre-trained diffusion-based teacher
model, and the later approach generally produces better performances. Detailed
discussions can be referred to (Song et al., 2023). In our work, a
distillation-based consistency model for speech synthesis, called CoMoSpeech,
is proposed as below.

##  3 CoMoSpeech

This section presents the proposed CoMoSpeech, a one-step speech synthesis
model. The framework of the proposed method is shown in Figure. 1, which has
two main stages. The first stage trains a diffusion-based teacher model to
produce audios conditioned on the textual (for TTS and SVS) and musical score
inputs (for SVS). Then in the second stage, by forcing the consistency
property, we distill the CoMoSpeech from the teacher model to finally achieve
a one-step inference given the conditional inputs. How to design the teacher
model, perform consistency distillation and implement the training and
inference will be discussed.

###  3.1 Teacher Model

As a blossoming class of generative models, many speech synthesis systems
apply diffusion models and generate high-quality audio. However, specific
criteria must be met to be the teacher model. First, the model needs to meet
the theoretical requirement. As mentioned in Section 2, we aim to adopt the
denoiser to implement the one-step generation, which means this function
should point to the clean data instead of noise. In other words, we follow the
term in (Huang et al., 2022) that our teacher model should be a generator-
based rather than a gradient-based method. This restriction requires us to
modify the state-of-art model Grad-TTS (Popov et al., 2021) to be our teacher
model. We inherited the setting of learning from the distribution of the prior
mel-spectrogram , $\mathbf{\mu}$, to the distribution of ground truth mel-
spectrogram $gt_{\textrm{mel}}$, and the main architectures. In addition, we
also adopt the EDM(Karras et al., 2022) as our design choice for the diffusion
model to ensure further consistency distillation (Song et al., 2023).

Specifically, we set mel-spectrogram as $\mathbf{x}$ in (2) with the drift and
diffusion coefficients as $t$ and 1, respectively. Combined with (4), our ODE
can be formulated as

$\mathrm{d}\mathbf{x}_{t}=[(D_{\theta}(\mathbf{x}_{t},t,cond)-\mathbf{x}_{t})/t]\mathrm{d}t,$
(7)

where $cond$ is the conditional input that will be introduced in the following
section, and $D_{\theta}(\mathbf{x}_{t},t,cond)$ is designed to precondition
the neural network with a t-dependent skip connection as

$D_{\theta}(\mathbf{x}_{t},t,cond)=c_{\text{skip
}}(t)\mathbf{x}_{t}+c_{\text{out
}}(t)F_{\mathbf{\theta}}(\mathbf{x}_{t},t,cond)$ (8)

where $F_{\mathbf{\theta}}$ is the network to be trained whose architecture
can be flexibly chosen. For instance, the architectures of WaveNet (Liu et
al., 2022a)(Oord et al., 2016) or U-Net (Popov et al., 2021)(Ronneberger et
al., 2015) can be selected to construct $F_{\mathbf{\theta}}$. The
$c_{\text{skip }}(t)$ and $c_{\text{out }}(t)$ are used to modulate the skip
connection and scale the magnitudes of $F_{\theta}$, which can be given by
(Song et al., 2023)

$c_{\mathrm{skip}}(t)=\frac{\sigma_{\mathrm{data}}^{2}}{(t-\epsilon)^{2}+\sigma_{\mathrm{data}}^{2}},~{}~{}~{}~{}c_{\mathrm{out}}(t)=\frac{\sigma_{\mathrm{data}}(t-\epsilon)}{\sqrt{\sigma_{\mathrm{data}}^{2}+t^{2}}},$
(9)

where $\sigma_{\mathrm{data}}=0.5$ is used to balance the ratio between
$c_{\text{skip }}$ and $c_{\text{out }}$ and $\epsilon=0.002$ as the smallest
time instant during sampling. The first reason for choosing the above
formulation is it can meet (6) since $c_{\text{skip }}(\epsilon)=1$ and
$c_{\text{out }}(\epsilon)=0$. The second reason is that both scaling factors
can help the predicted results of $F_{\mathbf{\theta}}$ scale to the unit
variance, which avoids the large variation in gradient magnitudes at different
noise levels.

To train the $D_{\theta}$, the loss function can be formulated as

$\mathcal{L}_{\theta}=||D_{\theta}(\mathbf{x}_{t},t,cond)-\mathbf{x}_{0}||^{2},$
(10)

which is a weighted $\mathcal{L}_{2}$ loss between the predicted mel-
spectrogram $pred_{mel}$ and ground truth mel-spectrogram $gt_{\textrm{mel}}$,
and we also re-weight the loss function for different $t$ as the same as EDM
(Karras et al., 2022).

Finally, the teacher model can be trained, and the synthesized mel-spectrogram
can be sampled by Algorithm 1. During the inference on the teacher model, we
first sample $\mathbf{x}_{N}$ from $\mathcal{N}(\mu,I)$, and then iterative
the numerical ODE solver for $N$ steps. More specifically, we choose the Euler
solver, and the $N$ discrete-time points are designed as the same setting in
EDM (Karras et al., 2022).

Algorithm 1 Sampling procedure of the proposed teacher model

1:Input: The denoiser function $D_{\theta}$; the prior mel-spectrogram $\mu$;
a set of time points $t_{i\in\\{0,\ldots,N\\}}$

2:Sample $\mathbf{x}_{N}\sim\mathcal{N}(\mu,I)$

3:for $i=N$ to $1$ do

4: $d_{i}\leftarrow\mathbf{x}_{i}-D_{\theta}(\mathbf{x}_{i},t_{i},\mu)$

5: $\mathbf{x}_{i-1}\leftarrow\mathbf{x}_{i}+(t_{i}-t_{i-1})d_{i}$

6:end for

7:$\mathbf{x}\leftarrow\mathbf{x}_{0}$

8:Output: $\mathbf{x}$

Algorithm 2 Sampling procedure of the proposed CoMoSpeech

1:Input: The denoiser function $D_{\theta}$; the prior mel-spectrogram $\mu$;
a set of time points $t_{i\in\\{0,\ldots,N\\}}$

2:Sample $\mathbf{x}_{N}\sim\mathcal{N}(\mu,I)$

3:$\mathbf{x}\leftarrow D_{\theta}(\mathbf{x}_{N},t_{N},\mu)$

4:if one-step synthesis

5: Output: $\mathbf{x}$

6:else Optional: multi-step synthesis

7: for $i=N-1$ to $1$ do

8: Sample $\textbf{z}\sim\mathcal{N}(0,I)$

9: $\mathbf{x}_{i}\leftarrow\mathbf{x}+\sqrt{t_{i}^{2}-\epsilon^{2}}{\bf z}$

10: $\mathbf{x}\leftarrow D_{\theta}(\mathbf{x}_{i},t_{i},\mu)$

11: end for

12: Output: $\mathbf{x}$

###  3.2 Consistency Distillation

A one-step diffusion sampling-based model is further trained from the teacher
model based on consistency distillation, resulting in the proposed CoMoSpeech.
Now we re-examine the constraints defined in (5) and (6). We note that given
the choice of $c_{\text{skip }}(t)$ and $c_{\text{out }}(t)$ in (9), the
denoiser $D_{\theta}$ in the proposed teacher model already satisfies (6),
therefore, the remaining training objective is to fulfill the property in (5).

Inspired by (Song et al., 2023), we utilize the momentum-based distillation to
train the proposed CoMoSpeech. The consistency distillation loss is defined as

$\mathcal{L}_{\theta}=||D_{\theta}(\mathbf{x}_{i+1},t_{i+1},cond)-D_{{\theta}^{-}}(\hat{\mathbf{x}}^{\mathbf{\phi}}_{i},t_{i},cond)||^{2},$
(11)

where $\theta$ and $\theta^{-}$ are initialized weights of CoMoSpeech
inherited from the teacher model, $\mathbf{\phi}$ is the ODE solver of the
teacher model in section 3.1, and $i$ is a step-index uniformly sampled from
the total ODE steps from $N$ to 1. $\hat{\mathbf{x}}^{\mathbf{\phi}}_{i}$ is
estimated from $x_{i+1}$ and the ODE solver $\mathbf{\phi}$. During training,
gradient-based iterations directly optimize the weight $\theta$ over the loss
function, and $\theta^{-}$ is recursively updated by

$\displaystyle\theta^{-}\leftarrow\mathrm{{stopgrad}}(\alpha\theta^{-}+(1-\alpha)\theta),$
(12)

where $\alpha$ is a momentum coefficient empirically set as 0.95.

After distillation, the consistency property can be exploited so that the
original data point $\mathbf{x}_{0}$ can be transformed from any point
$\mathbf{x}_{t}$ on the ODE trajectory as shown in Figure 2. Therefore, we can
directly generate the target sample from the distribution
$\mathbf{x}_{N}\sim\mathcal{N}(\mu,I)$ at the step $t_{N}$, as:

$mel_{\textrm{pred}}=D_{\theta}(\mathbf{x}_{N},t_{N},cond).$ (13)

Therefore, one-step mel-spectrogram generation can be achieved. In addition,
multi-step synthesis can be conducted by Algorithm 2 as a trade-off between
audio quality and sampling speed, similar to other stochastic samplers.

Figure 2:  An illustration of consistency property. A function with
consistency property maps any points on the ODE trajectory to the original
data.

###  3.3 Conditional Input

A remaining problem in the framework shown in Figure 1 is how to obtain the
conditional input $cond$, which will be used throughout the algorithm design.
A well-designed speech synthesizer is expected to perform well not only on
reading speech synthesis (TTS) but also on other more complicated tasks, such
as SVS which additionally produces highly dynamic melodies. In producing the
conditional inputs, both TTS and SVS tasks are considered to examine the
proposed framework’s effectiveness comprehensively.

Concretely, we adopt the phoneme as the basic input for TTS and SVS. Then, a
simple lookup table is used for embedding the phoneme feature. Additionally,
for the SVS task, we add a music score that specifies the note levels time-
aligned to the phonemes. For note feature extraction, we use the embedding
method for both categorical feature note pitch and slur indicator and rely on
a linear layer for continuous feature note duration.

Summing all the feature sequences together, we utilize the encoder structure
and variance adaptor in FastSpeech (Ren et al., 2019). Specifically, $N$ feed-
forward transformer blocks (FFT blocks) are stacked to extract the phoneme
hidden sequence. A duration predictor is used to estimate the duration of each
phoneme $d_{\textrm{pred}}$, and the corresponding loss function is expressed
as

$\mathcal{L}_{\textrm{duration}}=||\log(d_{\textrm{pred}})-\log(d_{\mathrm{gt}})||^{2}$
(14)

where $d_{\textrm{gt}}$ indicates the ground-truth phoneme duration.

Further, the length regulator projects the phoneme hidden sequence into the
hidden sequence in the mel-spectrogram domain, with the phoneme duration
denoted as $hidden_{\textrm{mel}}$. Then, the prior mel-spectrogram $\mu$ is
predicted using the $hidden_{\textrm{mel}}$ with prior loss function as

$\mathcal{L}_{\textrm{prior}}=||\mu-gt_{\textrm{mel}}||^{2}.$ (15)

As shown in the bottom left part of Figure 1, since the expanded features
belonging to the same phoneme in $hidden_{\textrm{mel}}$ are repeated, the
predicted $prior_{\textrm{mel}}$ can only roughly approximate the time-
frequency structure of $gt_{\textrm{mel}}$ based on the phoneme sequence.
Nevertheless, it significantly reduces the distance to $gt_{\textrm{mel}}$ by
sampling from $\mathcal{N}(\mathbf{\mu},\mathbf{I})$ rather than
$\mathcal{N}(\mathbf{0},\mathbf{I})$.

For the neural network and conditional inputs in the denoiser, we investigated
different combinations and finally selected a) the WaveNet architecture(Oord
et al., 2016) and $hidden_{\textrm{mel}}$ as feature $cond$ in (13) for SVS
and b) U-Net architecture (Ronneberger et al., 2015) and $\mu_{\textrm{mel}}$
as $cond$ for TTS.

###  3.4 Training procedure

The whole process can be summarised as two stages which are the training of
the teacher model and the consistency distillation.

As for the training of the teacher model, the loss term can consist of three
parts which are duration loss (in (14)), prior loss (in (15)), and denoising
loss (in (10)). These three losses are summed up together without any extra
weight. The objective of this stage is to build a speech synthesis system that
can generate high-quality audios with multi-step synthesis and have the
potential for further consistency distillation.

The second stage is consistency distillation. There is only one loss function
as defined by (11), which helps the model learn the consistency property. The
parameters are initialized from the teacher model. During training, the
parameters of the encoder are fixed, which means only the weight in the
denoiser is updated. After distillation, high-quality recordings with one-step
synthesis (13) can be achieved.

##  4 Experiments

To evaluate the performance of the proposed CoMoSpeech, we conduct experiments
on both TTS and SVS.

###  4.1 Experimental Setup

####  4.1.1 Data and Preprocessing

We adopt the public LJSpeech (Ito and Johnson, 2017) as the TTS dataset, which
includes around $24$ hours of English female voice recordings sampled at
$22.05$ kHz. Similar to (Ren et al., 2019) (Chen et al., 2022b), we split the
dataset into three sets: $12,228$ samples for training, $349$ samples (with
document title LJ003) for validation, and $523$ samples (with document title
LJ001 and LJ002) for testing. Following the common practice in (Ren et al.,
2020)(Huang et al., 2022) for TTS, we extract the 80-bin mel-spectrogram with
the frame size of $1024$ and hop size of $256$.

For the SVS task, we use the Opencpop dataset (Wang et al., 2022) containing
100 Chinese pop songs which are split into $3,756$ utterances with a total
duration of around $5.2$ hours. All recordings are from a single female singer
and labeled with aligned phoneme and MIDI-pitch sequences. We follow the
official train/test split (Wang et al., 2022), i.e., $95$ songs and $5$ songs
for training and evaluation, respectively. Same as the setting in (Chen et
al., 2020)(Liu et al., 2022a) for SVS, the recordings are resampled at $24$kHz
rates with $16$-bit precision, and the $80$-bin mel-spectrogram is extracted
with a frame size of $512$ and hop size of $128$.

####  4.1.2 Implementation Details

For TTS, for a fair comparison, the encoder and duration predictor are exactly
the same as those in Grad-TTS (Popov et al., 2021). The encoder contains $6$
feed-forward transformer (FFT) blocks (Ren et al., 2019), and The hidden
channel is set to $192$. The duration predictor uses two convolutional layers
for prediction. Both the teacher model and CoMoSpeech are trained for $1.7$
million iterations on a single NVIDIA A100 GPU with a batch size of $16$. The
Adam optimizer (Kingma and Ba, 2014) is adopted with the learning rate 1e-4.

For SVS, we adopt almost the same architecture as in TTS with different
hyperparameters. The encoder adopts $4$ FFT blocks, and we set the hidden
channel to $256$ in the encoder. The duration predictor consists of $5$
convolutional layers to estimate the duration. The teacher model of SVS and
CoMoSpeech are trained on a single GPU for $250$k steps with the AdamW
(Loshchilov and Hutter, 2017) optimizer. The initial learning rate is 1e-3,
and the exponential decay strategy with a decreasing factor of $0.5$ every
$50$k steps is adopted.

####  4.1.3 Evaluation Metrics

we conduct both objective and subjective evaluations to measure the sample
quality (MOS & FD) and the model inference speed (RTF & NFE):

  * •

MOS (mean opinion score) (Chu and Peng, 2006) is used to measure the perceived
quality of the synthesized audio, which is obtained by presenting 10 listeners
with the test set and asking them to rate the quality of the synthesized audio
on a scale of 1 to 5.

  * •

FD (frechet distance)111https://github.com/haoheliu/audioldm_eval is similar
to the frechet inception distance (Heusel et al., 2017) in image generation.
We use frechet distance (Liu et al., 2023) in audio to measure the similarity
between generated samples and target samples utilizing the large-scale
pretrained audio neural networks PANNs (Kong et al., 2020b).

  * •

RTF (real-time factor) determines how quickly the system can synthesize audio
in real-time applications. It is defined as the ratio between the total time a
speech system takes to synthesize a given amount of audio and the duration of
that audio.

  * •

NFE (number of function evaluations) measures the computational cost, which
refers to the total number of times the denoiser function is evaluated during
the generation process.

###  4.2 Performances on Text-to-Speech

We compare the above four metrics of the samples generated by the teacher
model and CoMoSpeech with the following systems:

  * •

GT, the ground truth recordings.

  * •

GT (Mel+HiFi-GAN), using ground-truth mel-spectrogram to synthesize waveform
with HiFi-GAN vocoder (Kong et al., 2020a).

  * •

FastSpeech 2 (Ren et al., 2020), synthesizing high-quality speech at a fast
speed with FFT blocks and variance adaptor.

  * •

DiffGAN-TTS (Liu et al., 2022b) 222https://github.com/keonlee9420/DiffGAN-TTS,
applying an adversarially-trained model to approximate the denoising function
for efficient speech synthesis.

  * •

ProDiff (Huang et al., 2022) 333https://github.com/Rongjiehuang/ProDiff,
directly adopting progressive distillation (Salimans and Ho, 2022) to TTS for
fast generation speed.

  * •

DiffSpeech (Liu et al., 2022a)
444https://github.com/MoonInTheRiver/DiffSinger/blob/master/docs/README-
TTS.md, using an auxiliary acoustic model to generate mel-spectrogram and
injects K steps noise to a noisy mel-spectrogram. Then, the mel-spectrogram is
generated from the noisy mel-spectrogram by DDPM iteratively.

  * •

Grad-TTS (Popov et al., 2021) 555https://github.com/huawei-noah/Speech-
Backbones/tree/main/Grad-TTS, using stochastic differential equation modelling
for the mel-spectrogram and use corresponding ODE solver for audio generation.

METHOD |  NFE |  RTF (↓) |  FD (↓) |  MOS (↑)
---|---|---|---|---
GT |  / |  / |  / |  4.778
GT(Mel+HiFi-GAN) |  / |  / |  0.282 |  4.590
FastSpeech 2 (Ren et al., 2020) |  1 |  0.0017 |  10.48 |  4.034
DiffGAN-TTS (Liu et al., 2022b) |  4 |  0.0084 |  8.310 |  3.889
ProDiff (Huang et al., 2022) |  4 |  0.0097 |  3.503 |  3.374
DiffSpeech (Liu et al., 2022a) |  71 |  0.1030 |  2.349 |  4.103
Grad-TTS (Popov et al., 2021) |  50 |  0.1694 |  1.882 |  4.487
Teacher |  50 |  0.1824 |  0.748 |  4.538
CoMoSpeech |  1 |  0.0058 |  0.774 |  4.239
Table 1: Evaluation results on LJSpeech for TTS. METHOD |  NFE |  RTF (↓) |  FD (↓) |  MOS (↑)
---|---|---|---|---
GT |  / |  / |  / |  4.675
GT(Mel+HiFi-GAN) |  / |  / |  0.882 |  4.588
FFTSinger (Blaauw and Bonada, 2020) |  1 |  0.0032 |  7.867 |  2.769
HiFiSinger (Chen et al., 2020) |  1 |  0.0034 |  6.340 |  3.156
DiffSinger (Liu et al., 2022a) |  60 |  0.1338 |  3.466 |  3.506
Teacher-SVS |  50 |  0.1282 |  3.162 |  4.050
CoMoSpeech-SVS |  1 |  0.0048 |  3.571 |  3.794
Table 2: Evaluation Results on Opencpop for SVS.

The evaluation results of TTS are shown in Table 1. For audio quality, our
teacher model achieved the highest MOS and Grad-TTS ranked second because our
teacher model is based on the design of Grad-TTS, but we adopt better choices
on drift and diffusion coefficients in SDE. The proposed CoMoSpeech takes 3rd
place among all methods, but it is substantially better than other fast-
inference methods ProDiff, DiffGAN-TTS and Fastspeech2. This demonstrates the
effectiveness of the consistency distillation and the effectiveness of teacher
model selection. In addition, we also observe that our teacher model and
CoMoSpeech achieve the best frechet distance scores among all methods, further
demonstrating the proposed methods’ superior performance on modeling data
distribution.

Regarding inference speed, while Fastspeech2 obviously achieves the best, our
CoMoSpeech also yields a very low RTF, and is faster than all other baselines.
Compared with the diffusion-based methods involving a large number of
iterations including DiffSpeech, Grad-TTS and our teacher model, our method
achieves about 50 times faster with similar or even better audio quality. In
addition, our CoMoSpeech also achieves faster speed and better quality than
methods for speeding up diffusion sampling, i.e., DiffGAN-TTS and ProDiff.

###  4.3 Performances on Singing Voice Synthesis

To further examine the modeling capability of our methods, we compare the
proposed SVS-version models, teacher-SVS and CoMoSpeech-SVS, with several
baselines on SVS, and the baselines include:

  * •

GT, the ground truth recordings.

  * •

GT(Mel+HiFi-GAN), synthesizing song samples using HiFi-GAN (Kong et al.,
2020a) vocoder with ground truth mel-spectrogram inputs.

  * •

FFTSinger (Blaauw and Bonada, 2020), adopting FFT blocks to predict mel-
spectrogram, and using the HiFi-GAN vocoder to synthesize audio;

  * •

HiFiSinger (Chen et al., 2020), using a novel sub-frequency GAN (SF-GAN) to
generate the mel-spectrogram. Since our aim is to compare the acoustic model,
we modify the original vocoder to HiFi-GAN, being the same as other methods.

  * •

DiffSinger (Liu et al.,
2022a)666https://github.com/MoonInTheRiver/DiffSinger/blob/master/docs/README-
SVS-opencpop-cascade.md, using DDPM to generate the mel-spectrogram from noisy
mel-spectrogram. DiffSinger uses an auxiliary acoustic model to generate a
prior mel-spectrogram injecting K steps noise to a noisy mel-spectrogram.

The results of SVS are shown in Table 2. As for audio quality, it can be seen
that our CoMoSpeech and other diffusion model based methods can significantly
surpass all non-iterative methods including FFTSinger and HIFiSinger on
frechet distance and mean opinion score. Among diffusion models, our teacher
model achieves the best performance, and our student model CoMoSpeech has a
close performance to it. For the inference speed, with one-step inference, the
proposed CoMoSpeech could maintain a speed similar to non-iterative methods
and significantly outperform other diffusion model based methods.

In addition, we also compare the results between speaking voice and singing
voice synthesis. Based on two methods DiffSinger and DiffSpeech, which are
basically the same, we can observe that the singing voice has a greater FD
than the speaking voice, indicating that it is more difficult to model the
data. However, the proposed teacher model and CoMoSpeech still achieve the
best performances on audio quality and inference speed, respectively. This
shows the capability of CoMoSpeech for speech synthesis beyond speaking
voices. In addition, we can observe that our CoMoSpeech-SVS is faster than
CoMoSpeech because the denoiser function in SVS follows the WaveNet
architecture which is faster than U-Net architecture in TTS. This observation
inspires us that if a more efficient denoiser function that runs faster than
the decoder in FastSpeech2 can be designed, we can make CoMoSpeech even faster
than non-iterative methods in future work.

| Frechet Distance (↓)
---|---
NFE | Teacher model | CoMoSpeech
1 | 7.526 | 0.774
2 | 4.558 | 0.762
4 | 2.477 | 0.784
10 | 1.197 | 0.725
50 | 0.748 | 0.850
Table 3: Comparison between CoMoSpeech and its teacher model with different sampling steps for TTS. Figure 3:  Effect of consistency distillation: Compared to the denoiser of teacher model before consistency distillation, our CoMoSpeech can generate a high-quality mel-spectrogram instead of an over-smoothed mel-spectrogram by calling the denoiser function only one time. | Frechet Distance (↓)
---|---
NFE | Teacher model | CoMoSpeech
1 | 7.786 | 3.571
2 | 7.219 | 3.520
4 | 4.932 | 3.433
10 | 3.937 | 3.658
50 | 3.162 | 3.732
Table 4: Comparison between CoMoSpeech and its teacher model with different
sampling steps for SVS.

###  4.4 Ablation Studies of Consistency Distillation

In this part, we will show the importance of consistency distillation. As
shown in Figure 3, we visualize the differences before and after consistency
distillation in the results,in other words, the teacher model and our
CoMoSpeech. At $t_{N}$ steps, the denoiser function before distillation points
to a smooth mel-spectrogram, indicating a great distance between ground truth
mel-spectrogram. However, we can observe that the results after distillation
significantly improve the performances by enriching many details, resulting in
natural and expressive sounds.

In Table 3 and Table 4, we also conduct experiments using the frechet distance
metric to further demonstrate the effectiveness of consistency distillation.
For teacher models for both TTS and SVS tasks, the frechet distance decrease
when the iteration steps increase. This trade-off property between inference
speed and sample quality has also been observed in other diffusion model
methods. Surprisingly, we can find that our CoMoSpeech can achieve nearly the
best performance in one step, and the best performance can be achieved at 4
and 10 steps on TTS and SVS, respectively. However, the trade-off property
seems to disappear after $10$ steps. The issue that the model performance
improves in a few sampling steps and then declines slightly as the number of
steps increases is called the sampling drift challenge (Daras et al., 2023)
(Ji et al., 2023) (Chen et al., 2022a) (Saxena et al., 2023). We will leave
the exploration to future work.

##  5 Conclusions and Future Work

In this paper, we propose CoMoSpeech, a one-step acoustic model for speech
synthesis based on the consistency model. With different conditional inputs,
our CoMoSpeech can generate high-quality speech or singing voice by
transforming the noise mel-spectrogram into the predicted mel-spectrogram in a
single step.

However, there are still some limitations to our method. Since our CoMoSpeech
needs to distill from a teacher model for better performance, this makes the
pipeline of constructing a speech synthesis system more complicated.
Therefore, how to directly train the CoMoSpeech without distillation of the
teacher model is our next step to investigate. In addition, we show the
capability of CoMoSpeech on the SVS task. Even though CoMoSpeech achieves the
best result among all the methods, it still has a significant gap between the
ground truth recording.

## References

  * Anderson [1982] Brian DO Anderson.  Reverse-time diffusion equation models.  _Stochastic Processes and their Applications_ , 12(3):313–326, 1982.
  * Blaauw and Bonada [2020] Merlijn Blaauw and Jordi Bonada.  Sequence-to-sequence singing synthesis using the feed-forward transformer.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7229–7233. IEEE, 2020.
  * Chen et al. [2020] Jiawei Chen, Xu Tan, Jian Luan, Tao Qin, and Tie-Yan Liu.  Hifisinger: Towards high-fidelity neural singing voice synthesis.  _arXiv preprint arXiv:2009.01776_ , 2020.
  * Chen et al. [2022a] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang.  Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions.  _arXiv preprint arXiv:2209.11215_ , 2022a.
  * Chen et al. [2022b] Zehua Chen, Yihan Wu, Yichong Leng, Jiawei Chen, Haohe Liu, Xu Tan, Yang Cui, Ke Wang, Lei He, Sheng Zhao, et al.  Resgrad: Residual denoising diffusion probabilistic models for text to speech.  _arXiv preprint arXiv:2212.14518_ , 2022b.
  * Chu and Peng [2006] Min Chu and Hu Peng.  Objective measure for estimating mean opinion score of synthesized speech, April 4 2006.  US Patent 7,024,362.
  * Daras et al. [2023] Giannis Daras, Yuval Dagan, Alexandros G Dimakis, and Constantinos Daskalakis.  Consistent diffusion models: Mitigating sampling drift by learning to be consistent.  _arXiv preprint arXiv:2302.09057_ , 2023.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Huang et al. [2022] Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren.  Prodiff: Progressive fast diffusion model for high-quality text-to-speech.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 2595–2605, 2022.
  * Hyvärinen and Dayan [2005] Aapo Hyvärinen and Peter Dayan.  Estimation of non-normalized statistical models by score matching.  _Journal of Machine Learning Research_ , 6(4), 2005.
  * Ito and Johnson [2017] Keith Ito and Linda Johnson.  The lj speech dataset.  https://keithito.com/LJ-Speech-Dataset/, 2017.
  * Ji et al. [2023] Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo.  Ddp: Diffusion model for dense visual prediction.  _arXiv preprint arXiv:2303.17559_ , 2023.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _arXiv preprint arXiv:2206.00364_ , 2022.
  * Kim et al. [2021] Jaehyeon Kim, Jungil Kong, and Juhee Son.  Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech.  In _International Conference on Machine Learning_ , pages 5530–5540. PMLR, 2021.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kong et al. [2020a] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae.  Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.  _Advances in Neural Information Processing Systems_ , 33:17022–17033, 2020a.
  * Kong et al. [2020b] Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley.  Panns: Large-scale pretrained audio neural networks for audio pattern recognition.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:2880–2894, 2020b.
  * Liu et al. [2023] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley.  Audioldm: Text-to-audio generation with latent diffusion models.  _arXiv preprint arXiv:2301.12503_ , 2023.
  * Liu et al. [2022a] Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao.  Diffsinger: Singing voice synthesis via shallow diffusion mechanism.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 11020–11028, 2022a.
  * Liu et al. [2022b] Songxiang Liu, Dan Su, and Dong Yu.  Diffgan-tts: High-fidelity and efficient text-to-speech with denoising diffusion gans.  _arXiv preprint arXiv:2201.11972_ , 2022b.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Lu et al. [2020] Peiling Lu, Jie Wu, Jian Luan, Xu Tan, and Li Zhou.  Xiaoicesing: A high-quality and integrated singing voice synthesis system.  _arXiv preprint arXiv:2006.06261_ , 2020.
  * Nishimura et al. [2016] Masanari Nishimura, Kei Hashimoto, Keiichiro Oura, Yoshihiko Nankaku, and Keiichi Tokuda.  Singing voice synthesis based on deep neural networks.  In _Interspeech_ , pages 2478–2482, 2016.
  * Oord et al. [2016] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  _arXiv preprint arXiv:1609.03499_ , 2016.
  * Popov et al. [2021] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov.  Grad-tts: A diffusion probabilistic model for text-to-speech.  In _International Conference on Machine Learning_ , pages 8599–8608. PMLR, 2021.
  * Ren et al. [2019] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu.  Fastspeech: Fast, robust and controllable text to speech.  _Advances in neural information processing systems_ , 32, 2019.
  * Ren et al. [2020] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu.  Fastspeech 2: Fast and high-quality end-to-end text to speech.  _arXiv preprint arXiv:2006.04558_ , 2020.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Salimans and Ho [2022] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _arXiv preprint arXiv:2202.00512_ , 2022.
  * Saxena et al. [2023] Saurabh Saxena, Abhishek Kar, Mohammad Norouzi, and David J Fleet.  Monocular depth estimation using diffusion models.  _arXiv preprint arXiv:2302.14816_ , 2023.
  * Song et al. [2020a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Song et al. [2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020b.
  * Song et al. [2023] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models.  _arXiv preprint arXiv:2303.01469_ , 2023.
  * Tan et al. [2021] Xu Tan, Tao Qin, Frank Soong, and Tie-Yan Liu.  A survey on neural speech synthesis.  _arXiv preprint arXiv:2106.15561_ , 2021.
  * Taylor [2009] Paul Taylor.  _Text-to-speech synthesis_.  Cambridge university press, 2009.
  * Wang et al. [2022] Yu Wang, Xinsheng Wang, Pengcheng Zhu, Jie Wu, Hanzhao Li, Heyang Xue, Yongmao Zhang, Lei Xie, and Mengxiao Bi.  Opencpop: A high-quality open source chinese popular song corpus for singing voice synthesis.  _arXiv preprint arXiv:2201.07429_ , 2022.
  * Wang et al. [2017] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al.  Tacotron: Towards end-to-end speech synthesis.  _arXiv preprint arXiv:1703.10135_ , 2017.
  * Yang et al. [2022] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang.  Diffusion models: A comprehensive survey of methods and applications.  _arXiv preprint arXiv:2209.00796_ , 2022.
  * Yu et al. [2019] Chengzhu Yu, Heng Lu, Na Hu, Meng Yu, Chao Weng, Kun Xu, Peng Liu, Deyi Tuo, Shiyin Kang, Guangzhi Lei, et al.  Durian: Duration informed attention network for multimodal synthesis.  _arXiv preprint arXiv:1909.01700_ , 2019.