# RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for
Text-to-Speech Synthesis

Detai Xin1,2, Xu Tan1, Kai Shen1,3, Zeqian Ju1,4, Dongchao Yang5, Yuancheng
Wang6 Shinnosuke Takamichi2, Hiroshi Saruwatari2, Shujie Liu1, Jinyu Li1,
Sheng Zhao1
1Microsoft, 2The University of Tokyo
3Zhejiang University, 4University of Science and Technology of China
5The Chinese University of Hong Kong, 6The Chinese University of Hong Kong,
Shenzhen
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We present RALL-E, a robust language modeling method for text-to-speech (TTS)
synthesis. While previous work based on large language models (LLMs) shows
impressive performance on zero-shot TTS, such methods often suffer from poor
robustness, such as unstable prosody (weird pitch and rhythm/duration) and a
high word error rate (WER), due to the autoregressive prediction style of
language models. The core idea behind RALL-E is chain-of-thought (CoT)
prompting, which decomposes the task into simpler steps to enhance the
robustness of LLM-based TTS. To accomplish this idea, RALL-E first predicts
prosody features (pitch and duration) of the input text and uses them as
intermediate conditions to predict speech tokens in a CoT style. Second,
RALL-E utilizes the predicted duration prompt to guide the computing of self-
attention weights in Transformer to enforce the model to focus on the
corresponding phonemes and prosody features when predicting speech tokens.
Results of comprehensive objective and subjective evaluations demonstrate
that, compared to a powerful baseline method VALL-E, RALL-E significantly
improves the WER of zero-shot TTS from $5.6\%$ (without reranking) and $1.7\%$
(with reranking) to $2.8\%$ and $1.0\%$, respectively. Furthermore, we
demonstrate that RALL-E correctly synthesizes sentences that are hard for
VALL-E and reduces the error rate from $68\%$ to $4\%$.

##  1 Introduction

Large language models (LLMs) have demonstrated great progress in natural
language generation [20, 1]. With a sufficient model size LLMs emerge powerful
in-context learning abilities that can handle unseen tasks with a text
instruction (usually called prompt) in a zero-shot or few-shot manner [31].
Moreover, the simple yet effective next-token prediction task of LLMs makes it
easy to apply LLMs on other fields, such as vision [5] and speech synthesis
[29], as long as the data can be converted to discrete speech tokens. This
work focuses on the language modeling of text-to-speech (TTS) synthesis.
Recent work [29, 14] have shown that TTS can be modeled by a decoder-only
language model by using a neural codec [34, 4] to convert continuous waveforms
into discrete tokens. These methods, typically leverage tens of thousands of
hours of speech data, emerge in-context learning ability that can clone a
speaker’s voice by providing a short audio prompt to the language model, and
thus have impressive performance on zero-shot TTS.

Table 1: Performance of RALL-E and the baseline method VALL-E [29] on $50$ particularly hard sentences obtained from Ren et al. [21]. The result of NaturalSpeech 2 is from Shen et al. [25]. Model | Mispronunciation | Omission | Repetition | Hallucination | Error rate
---|---|---|---|---|---
NaturalSpeech 2 | $0$ | $0$ | $0$ | $0$ | $0\%$
VALL-E | $10$ | $19$ | $8$ | $7$ | $68\%$
RALL-E | $2$ | $0$ | $0$ | $0$ | $4\%$

However, due to the sequential generation property of language models, such
codec language models suffer from poor robustness. Although the autoregressive
(AR) prediction style of language models enables the model to generate speech
with diverse prosody patterns, they can also cause bad cases with unnatural
prosody. Moreover, since there is no strict alignment between text and speech,
the models can omit or repeat words in the input text. This is quite different
from TTS methods based on non-autoregressive (NAR) generative models [25, 16,
12], which predicts all tokens at the same time, thus have high robustness but
relatively low diversity. As suggested by previous work [33, 12], LLM-based
TTS have a higher word error rate (WER) than NAR TTS even if they have similar
performance on other metrics. To alleviate this problem, a simple but
effective method is to sample the same input text multiple times and select
the best one [14, 33]. However, such a reranking method further increases the
inference time.

Figure 1: Overview of RALL-E with CoT prompting. Symbols are all defined in
Section 3.1. The proposed CoT prompting of prosody tokens and duration-guided
masking are introduced in Section 3.2 and 3.3, respectively.

In this paper, we present RALL-E (the abbreviation of robust VALL-E), a method
to improve the robustness of LLM-based TTS. The core idea of RALL-E is
inspired from the chain-of-thought (CoT) prompting [32]. In CoT prompting, the
LLM is instructed to generate an intermediate result that is used as a
condition for the prediction of the final result. The CoT prompting breaks a
complex task into several simpler steps, so that can improve the robustness of
LLMs, especially on hard tasks like arithmetic [32]. To adapt CoT prompting to
LLM-based TTS, RALL-E predicts prosody tokens (pitch and duration) before
predicting speech tokens to stabilize the prosody. Given an input sentence,
RALL-E first predicts phoneme-level pitch and duration of the input, then
predicts speech tokens conditioning on both the input phonemes and the
predicted prosody tokens. Furthermore, RALL-E utilizes the predicted duration
to mask irrelevant phonemes and prosody tokens when computing self-attention
weights, so that the codec language model is enforced to concentrate on tokens
around the phoneme and prosody token the speech token corresponds to. We use
VALL-E [29], a recent powerful LLM-based TTS method, as the baseline, and
conduct experiments on a large dataset with $44$K hours speech data. Results
of comprehensive objective and subjective evaluations demonstrate that RALL
significantly improves the robustness of LLM-based TTS by reducing the WER on
the LibriSpeech [18] test-clean set from $5.6\%$ (w/o reranking) and $1.7\%$
(with reranking) to $2.8\%$ and $1.0\%$, respectively. Furthermore, we
evaluate RALL-E on $50$ particularly hard sentences. As demonstrated in Table
1, compared to VALL-E, RALL-E significantly reduces the error rate from $68\%$
to $4\%$ by eliminating almost all types of error, which demonstrates the
superior robustness of RALL-E (see Section 4.4 for more details). The
contributions of this work are summarized as follows:

  * •

We present RALL-E, a robust codec language modeling method with chain-of-
thought prompting for TTS. RALL-E improves the robustness of LLM-based TTS by
(1) incorporating prosody tokens as chain-of-thought prompting to stabilize
the generation of speech tokens and (2) using duration-guided masking to
enhance the alignment between phoneme and speech tokens.

  * •

We conduct comprehensive objective and subjective evaluations. Experimental
results demonstrate that RALL-E obtains significantly better robustness than
the baseline VALL-E and two previous works.

  * •

We further evaluate RALL-E on sentences that are particularly hard to
synthesize for LLM-based TTS. The results demonstrate that RALL-E correctly
synthesizes hard sentences and reduces the error rate from $68\%$ to $4\%$
compared to VALL-E, which closely approaches the performance of non-
autoregressive TTS.

Audio samples can be found at https://ralle-demo.github.io/RALL-E

##  2 Related work

#### LLM-based TTS

Inspired by the success of LLMs [20, 1], several recent works adopt language
models to model TTS [29, 33, 14] and begin to use decoder-only architecture
based on transformer [28]. In such models, text and speech tokens are
concatenated together and fed to a single transformer. The whole model is
trained on a next-token prediction task like a language model. The LLM-based
TTS systems are typically trained on tens of thousands of hours of speech data
and have hundreds of millions of parameters, hence can leverage the emergent
abilities of LLMs like in-context learning [31] to enable zero-shot TTS [29].
Besides, recent works [22, 30, 33] have shown the decoder-only architecture
can be used to learn multiple tasks, as the input and output are processed
jointly by a language model, and the model can be signaled to generate results
for different tasks by inputting pre-defined special tokens. RALL-E focuses on
the robustness problem of LLM-based TTS.

#### Robust autoregressive TTS

The robustness of AR TTS is a popular topic in the literature. For encoder-
decoder AR TTS, several previous works enforce the attention weights to be
monotonic [36, 9, 2] that can effectively improve the robustness. In addition,
Shen et al. [24] proposed a non-attentive Tacotron, in which the attention
module was replaced by a duration predictor to determine the alignment path
before decoding. For decoder-only TTS, a key difference is that the attention
weights are computed on text and context at the same time, hence the whole
attention weights should not be monotonic. Song et al. [26] proposed ELLA-V
that interleaves the speech tokens with phonemes by inserting a phoneme token
and a special EndOfPhone (EOP) token at the beginning and end of the speech
tokens corresponding to the phoneme, respectively. While the inserted phoneme
and the EOP token indicate the duration of each phoneme, such an implicit way
entangles the prediction of speech tokens and duration together. RALL-E
disentangles the predictions of duration and speech tokens by predicting the
duration of all phonemes before the speech tokens, hence has higher
controllability over the generation process. Du et al. [6] proposed VALL-T
that uses an unsupervised transducer loss [7] to implicitly model the duration
of phonemes. Compared to RALL-E, although VALL-T doesn’t rely on external
alignment tools during training, its training time is considerably decelerated
since the transducer loss requires the model to perform a forward process for
every phoneme. Besides, like ELLA-V, VALL-T also entangles the predictions of
duration and speech tokens, thus has weaker controllability than RALL-E.

##  3 RALL-E

The overview of RALL-E is illustrated in Figure 1. The core idea of RALL-E is
CoT prompting that generates intermediate results to assist and stabilize the
generation of speech tokens and improve the robustness of LLM-based TTS. To
accomplish this idea, we first propose to predict two kinds of phoneme-level
prosody tokens: pitch and duration before predicting the speech tokens. The
distributions of the prosody tokens are modeled together with speech tokens by
a single Transformer so that they can influence the duration and pitch of the
predicted speech tokens. To further utilize the predicted duration to guide
the generation and improve the robustness, we propose duration-guided masking
to enhance the alignment between speech tokens, phonemes, and prosody tokens
learned by the language model. At each decoding step of the speech tokens,
RALL-E masks phonemes and prosody tokens that are irrelevant to the synthesis
of the current speech token based on the duration information.

In the following sections, we first briefly introduce VALL-E since we apply
the proposed method to it in the experiments. We then formulate and introduce
RALL-E in detail. It should be stressed that, though we use VALL-E to
implement RALL-E, the proposed method can be applied in any decoder-only AR
TTS model.

###  3.1 Preliminary: VALL-E

We inherit most symbols from the original paper of VALL-E [29] for ease of
reading. Readers are recommended to refer to the original paper for more
details.

Generally, VALL-E is a decoder-only LLM-based TTS system that uses two
Transformers [28] to predict speech tokens from the text. The speech tokens
here are extracted from EnCodec [4], a neural audio codec based on residual
vector quantization (RVQ) [34] that can convert continuous speech signal into
discrete tokens. After predicting the discrete tokens, the waveforms can be
reconstructed by feeding the tokens into the decoder of EnCodec. An RVQ
typically contains $N$ quantization layers ($N=8$ in VALL-E), hence at each
time step the encoded speech has $N$ tokens. Formally, given speech
$\mathbf{y}$ and its transcription $\mathbf{x}$, the discrete speech token
matrix $\mathbf{C}$ encoded by the codec has a shape of $T\times N$, where $T$
is the total time step. In addition to $\mathbf{x}$, to clone a speaker’s
voice and utilize the in-context learning ability of LLMs, VALL-E receives a
short prompt $\tilde{\mathbf{C}}^{T^{{}^{\prime}}\times N}$ as input before
predicting $\mathbf{C}$. Hence, VALL-E models and maximizes the following
distribution:

$\mathbb{P}(\mathbf{C}\ |\ \mathbf{x},\tilde{\mathbf{C}}).$ (1)

VALL-E predicts speech tokens hierarchically where the speech tokens of the
$1$st layer of RVQ are first predicted by an AR Transformer, and the tokens of
the rest layers are predicted by a NAR Transformer. This is because RVQ uses a
residual quantization method, i.e. higher layers encode the information that
is not encoded by the lower layers, hence tokens of the $1$st layer contain
most information of the waveforms, and the information encoded by the rest
layers gradually decreases. The AR Transformer takes the phoneme sequence
$\mathbf{x}$, and speech tokens of the $1$st layer of the prompt
$\tilde{\mathbf{c}}_{:,1}$ as input to predict the target speech tokens of the
$1$st layer $\mathbf{c}_{:,1}$ sequentially, i.e. maximizes the following
distribution:

$\mathbb{P}(\mathbf{c}_{:,1}\ |\
\mathbf{x},\tilde{\mathbf{c}}_{:,1};\theta_{AR})=\prod_{t=1}^{T}\mathbb{P}(\mathbf{c}_{t,1}\
|\ \mathbf{x},\mathbf{c}_{<t,1},\tilde{\mathbf{c}}_{:,1};\theta_{AR}),$ (2)

where $\theta_{AR}$ is the trainable parameters of the AR Transformer. The NAR
Transformer predicts all target speech tokens $\mathbf{c}_{:,j}$ of the $j$th
layer at the same time with the phoneme sequence $\mathbf{x}$, the prompt
$\tilde{\mathbf{C}}$, and target speech tokens $\mathbf{c}_{:,<j}$ of all
layers less than $j$ as the conditions, i.e. maximizes the following
distribution:

$\mathbb{P}(\mathbf{c}_{:,2:N}\ |\
\mathbf{x},\tilde{\mathbf{C}};\theta_{NAR})=\prod_{j=2}^{N}\mathbb{P}(\mathbf{c}_{:,j}\
|\ \mathbf{x},\mathbf{c}_{:,<j},\tilde{\mathbf{C}};\theta_{NAR}),$ (3)

where $\theta_{NAR}$ is the trainable parameters of the NAR Transformer. By
combining Eq. 2 and Eq. 3 VALL-E breaks Eq. 1 into the following form:

$\mathbb{P}(\mathbf{C}\ |\
\mathbf{x},\tilde{\mathbf{C}})=\mathbb{P}(\mathbf{c}_{:,1}\ |\
\mathbf{x},\tilde{\mathbf{c}}_{:,1};\theta_{AR})\mathbb{P}(\mathbf{c}_{:,2:N}\
|\ \mathbf{x},\tilde{\mathbf{C}};\theta_{NAR}).$ (4)

It is noteworthy that in practice the two Transformers have the same
architecture but have different attention masks during computation.
Specifically, both the two Transformers use a bidirectional mask for the
phoneme sequence $\mathbf{x}$, which means every phoneme $x_{i}$ can attend to
all other phonemes $x_{\neq i}$. However, for the speech tokens, the AR
Transformers uses a unidirectional mask so that $\mathbf{c}_{t,1}$ can only
attend to previous tokens $\mathbf{c}_{<t,1}$, while the NAR Transformer still
uses a bidirectional mask.

###  3.2 Prosody tokens as chain-of-thought prompts

One of the problems of LLM-based TTS is that it directly generates speech from
phonemes with no restriction on the prosody, e.g. pitch, duration, etc, which
usually results in speech with unstable prosody. A similar problem is also
observed in Wei et al. [32] where the authors find LLMs cannot directly answer
a complex question like arithmetic and propose CoT prompting to solve this
problem. The idea of CoT prompting is breaking a complex task into several
simpler tasks so that LLMs can utilize the intermediate results to reach the
final answers. As shown in Wei et al. [32], by CoT prompting the correct rate
of LLMs on complex tasks can be significantly improved. This motivates us to
adapt CoT prompting to LLM-based TTS by generating intermediate prosody tokens
before generating speech tokens to alleviate the robustness problem of LLM-
based TTS. To incorporate pitch and duration in the AR Transformer of VALL-E,
we first get the alignment between phonemes and speech tokens and extract the
pitch value for each speech token. We then compute phoneme-level pitch value
based on the duration and linearly quantize it to $M_{p}$ buckets. We define a
maximal duration value $M_{d}$, and all duration values that exceed $M_{d}$
will be truncated to the maximum. RALL-E predicts the two prosody tokens
before the speech tokens in a CoT style. Formally, assume
$\mathbf{p},\mathbf{d}$ are the discrete pitch and duration sequences of the
target speech tokens $\mathbf{C}$, $\tilde{\mathbf{p}},\tilde{\mathbf{d}}$ are
the ones of the prompt $\tilde{\mathbf{C}}$, we model and maximize the
following distribution:

$\mathbb{P}(\mathbf{p},\mathbf{d}\ |\
\mathbf{x},\tilde{\mathbf{p}},\tilde{\mathbf{d}};\theta_{AR})=\prod_{t=1}^{L}\mathbb{P}(p_{t},d_{t}\
|\
\mathbf{x},\mathbf{p}_{<t},\mathbf{d}_{<t},\tilde{\mathbf{p}},\tilde{\mathbf{d}};\theta_{AR}),$
(5)

where $L$ is the length of $\mathbf{x}$. In practice, the model predicts
$p_{t}$ and $d_{t}$ with two separate heads, and their embeddings are summed
up and fed to the model for the prediction of the next step. RALL-E then
predicts the speech tokens with $\mathbf{p}$ and $\mathbf{d}$ as a new
condition, which makes Eq. 2 becomes:

$\mathbb{P}(\mathbf{c}_{:,1}\ |\
\mathbf{x},\tilde{\mathbf{c}}_{:,1},\mathbf{p},\tilde{\mathbf{p}},\mathbf{d},\tilde{\mathbf{d}};\theta_{AR})=\prod_{t=1}^{T}\mathbb{P}(\mathbf{c}_{t,1}\
|\
\mathbf{x},\mathbf{c}_{<t,1},\tilde{\mathbf{c}}_{:,1},\mathbf{p},\tilde{\mathbf{p}},\mathbf{d},\tilde{\mathbf{d}};\theta_{AR}).$
(6)

The above two equations can be jointly optimized by the AR Transformer.
Although the proposed method adds additional $L$ decoding steps, since $L\ll
T$, it intuitively has little influence on the efficiency.

For the NAR Transformer, we simply sum the embeddings of the phoneme, pitch,
and duration together as the input. This makes Eq. 3 becomes:

$\mathbb{P}(\mathbf{c}_{:,2:N}\ |\
\mathbf{x},\tilde{\mathbf{C}},\mathbf{p},\tilde{\mathbf{p}},\mathbf{d},\tilde{\mathbf{d}};\theta_{NAR})=\prod_{j=2}^{N}\mathbb{P}(\mathbf{c}_{:,j}\
|\
\mathbf{x},\mathbf{c}_{:,<j},\tilde{\mathbf{C}},\mathbf{p},\tilde{\mathbf{p}},\mathbf{d},\tilde{\mathbf{d}};\theta_{NAR}).$
(7)

Figure 2: A comparison between how speech token attends to phonemes in the AR
Transformer of VALL-E and RALL-E. Here $k$ is set to $1$.

###  3.3 Enhancing alignment with duration-guided masking

As the left side of Figure 2 illustrates, since the speech token attends to
all phonemes in the AR Transformer of VALL-E, the alignment between the
phonemes and the speech tokens is implicitly modeled by the self-attention of
VALL-E. This can be imprecise and causes errors like word omissions or
hallucinations. Though RALL-E introduces prosody CoT prompting to guide and
stabilize the generation, we still find the model can fail to align in the
experiments. We thus propose duration-guided masking to fully utilize the
intermediate duration results and boost the robustness.

As the right side of Figure 2 illustrates, in the proposed duration-guided
masking, the speech token is restricted to only attend on a phoneme (prosody
token) window centered at the phoneme (prosody token) it corresponds to. We
define the window size as $k$, thus each speech token can attend on $2k+1$
phonemes and $2k+1$ prosody tokens. All phonemes and prosody tokens at other
positions will be masked out, hence their attention weights are always zero.
When $k=0$ the speech token strictly attends to the phoneme it corresponds to.
If the alignment is perfect this should be enough. However, in the
experiments, we found that the alignment results obtained by our alignment
tool usually have errors. We thus loosen the restriction by also allowing the
speech token to attend at the near phonemes of the corresponding phoneme.
Another reason for this design is that the pronunciation of a phoneme is
usually dependent on near phonemes. As one will see in Section 4.3 and
Appendix A, the experimental results verify the effectiveness of this design.
For the NAR Transformer, we obtained almost no gain when applying the proposed
masking strategy to it in our preliminary experiments. Thus we only apply the
masking strategy on the AR Transformer.

The general inference procedure follows VALL-E [29] with two differences.
First, before sampling the speech tokens $\mathbf{c}_{:,1}$ the prosody tokens
$\mathbf{p}$ and $\mathbf{d}$ are sampled conditioning on the phoneme sequence
$\mathbf{x}$ and acoustic prompt $\tilde{\mathbf{p}},\tilde{\mathbf{d}}$.
Second, although normal language models depend on a special token <eos> as the
stop condition, since we know the total duration $D=\sum_{t=1}^{L}d_{t}$, we
propose a duration-guided inference method that forces the inference to stop
at the $D$-th step. This method ensures no phoneme is omitted or repeated as
it continues the inference if the <eos> token is predicted before the $D$-th
step and stops at the right step as guided by the predicted duration.

##  4 Experiments

###  4.1 Setup

#### Data

We use the English subset of the multilingual LibriSpeech (MLS) [19] corpus
containing about $44$K hours of speech data from $5490$ speakers as the
training data. The test-clean set of the LibriSpeech corpus [18] is used as
the test set. Following Wang et al. [29], we only select utterances with a
length of $4$-second to $10$-second, which results in $1205$ utterances. For
each utterance in the test set, we randomly select another one of the same
speaker and use the first $3$s of it as the prompt. All speech data are in
$16$ kHz. The transcriptions are converted into phonemes by a grapheme-to-
phoneme tool [27]. The frame-level pitch value is extracted by the WORLD
vocoder [17]. We get the alignments between the phoneme sequences and the
speech tokens with our internal alignment tool. The maximal duration value
$M_{d}$ is set to $32$. The phoneme-level pitch value is then computed based
on the alignments. The number of buckets $M_{p}$ for pitch quantization is set
to $256$.

#### Model configuration

We use SoundStream [34] as the speech codec to get speech tokens and decode
waveforms from the tokens. The architecture follows the original work and the
number of quantization layer $N$ of RVQ is set to $16$. The codec language
model follows VALL-E [29], where both the AR and NAR models are a $12$-layer
Transformer [28] with $1024$-dimensional token embeddings, sinusoidal
positional embeddings, $4096$-dimensional feed-forward layers, and a dropout
rate of $0.1$. The window size $k$ is set to $1$ without explicit statements
(see Appendix A for a detailed explanation of how we choose this value).

#### Training and inference

The SoundStream codec is trained on $8$ NVIDIA V100 GPUs with a batch size of
$200$ per GPU. We use Adam [15] as the optimizer with a learning rate of
$2e\mathchar 45\relax 4$. The model converges in about $440$K steps. The AR
and NAR Transformers are trained separately on $16$ AMD MI200 GPUs with a
batch size of $7000$ speech tokens per GPU. Adam is again used as the
optimizer. The scheduled inverse square root learning rate is used with $30$K
warm-up steps and a peak learning rate of $5e\mathchar 45\relax 4$. Both of
the two Transformers converge in about $500$K steps.

We adopt nucleus sampling [10] as the sampling method of the AR Transformer.
For the predicted probability distribution, nucleus sampling first selects a
token set with the highest probability whose accumulative probability exceeds
a hyperparameter $\rho$, and randomly samples from the set. Note that, $\rho$
can be different for the sampling of pitch, duration, and speech tokens, which
results in $3$ hyperparameters: $\rho_{p}$, $\rho_{d}$, $\rho_{c}$ for pitch,
duration, speech tokens, respectively. Without explicit statements we set
$\rho_{p}=\rho_{d}=\rho_{c}=0.9$ in the following experiments. We select the
token with the highest probability in the NAR Transformer without sampling.

#### Baseline methods

We use VALL-E [29] as the baseline method. We implement VALL-E and train it on
our training set. Besides, two previous works: VALL-T [6] and ELLA-V [26] are
also compared with RALL-E. VALL-T is trained on the LibriTTS [35] corpus with
$520$ hours of speech data, and we get $500$ synthesized samples selected from
the test-clean set of LibriTTS from the authors. ELLA-V [26] is trained on the
training set of the LibriSpeech [18] corpus with $960$ hours speech data. We
request the authors to run ELLA-V on our test set and get $904$ samples. We
don’t use the result of the original paper [26] since the continual generation
method adopted by ELLA-V can result in a significantly better WER than non-
continual generation, as suggested by Wang et al. [29].

Table 2: Main results of RALL-E on the LibriSpeech test set with $1205$ utterances. Bold indicates the best score. The WER shown in the parenthesis is obtained from the HuBERT model used in the original VALL-E paper [29]. † means the results of VALL-E trained on LibriLight. ‡ means the results of VALL-E trained on the English subset of MLS. We get the results of VALL-T ($500$ samples) [6] and ELLA-V ($904$ samples) [26] from the authors. RALL-E ($904$) means the results are computed on the same test set with ELLA-V including $904$ samples. | WER% ($\downarrow$) | WER-R% ($\downarrow$) | UTMOS ($\uparrow$) | SIM ($\uparrow$) | Sub ($\downarrow$) | Del ($\downarrow$) | Ins ($\downarrow$)
---|---|---|---|---|---|---|---
GT |  $1.8$ ($2.1)$ | - | $4.1$ | $0.69$ | $1.4$ | $0.2$ | $0.2$
VALL-E† | \- ($5.9$) | - | - | $0.58$ | - | - | -
VALL-E‡ |  $5.6$ ($6.3)$ | $1.7$ | $3.9$ | $0.49$ | $2.8$ | $1.5$ | $1.3$
ELLA-V |  $4.6$ ($6.5$) | - | $2.2$ | $0.40$ | $3.0$ | $1.2$ | $0.4$
VALL-T |  $3.9$ ($5.4$) | - | $\mathbf{4.0}$ | $0.46$ | $2.4$ | $1.3$ | $\mathbf{0.2}$
RALL-E (904) |  $2.7$ ($3.0$) | $0.8$ |  | $0.49$ | $1.6$ | $0.7$ | $0.3$
RALL-E |  $\mathbf{2.8}$ ($\mathbf{3.1}$) | $\mathbf{1.0}$ | $\mathbf{4.0}$ | $0.49$ | $\mathbf{1.7}$ | $\mathbf{0.8}$ | $0.3$

#### Objective metrics

We use the following objective metrics:

  * •

Word error rate (WER). We transcribe the synthesized samples by a large
Conformer-based [8] ASR
model111https://huggingface.co/nvidia/stt_en_conformer_transducer_xlarge that
is trained on a large collection of speech corpora including LibriSpeech [18].
The WER is then computed between the recognized and the ground truth (GT)
transcriptions. Besides, we also report WERs computed from transcriptions
recognized by a HuBERT222https://huggingface.co/facebook/hubert-large-ls960-ft
[11] model trained on Libri-Light [13] and finetuned on LibriSpeech [18]. We
regard the WERs obtained from the Conformer-based model as the primary scores
since it results in better performance than the HuBERT model, though the
HuBERT model is used in the original VALL-E paper [29].

  * •

Reranked WER (WER-R). For each test utterance, we sample it $5$ times and
select the one with the best edit distance to the GT transcription to compute
WER. This metric can be regarded as a performance upper bound, while the WER
represents average performance.

  * •

Substitution (Sub), Deletion (Del), and Insertion (Ins) computed by the edit
distance algorithm. These three metrics are by-products of the computing of
WER and can reveal specific error types made by the TTS model. Typically Sub
refers to mispronunciation, Del refers to word omission, and Ins refers to
word repetition or hallucination.

  * •

UTMOS [23], which is a powerful automatic speech quality assessment model that
evaluates speech naturalness.

  * •

Speaker similarity (SIM) defined as the cosine similarity between the speaker
embeddings of the prompt and the synthesized utterance. Following VALL-E [29],
we use the  wavlm_large_finetune checkpoint of WavLM-
TDNN333https://github.com/microsoft/UniSpeech/tree/main/downstreams/speaker_verification,
a speaker verification model based on WavLM [3], to extract speaker
embeddings.

#### Subjective metrics

We use two common subjective metrics: comparative mean opinion score (CMOS)
and similarity mean opinion score (SMOS) to evaluate speech naturalness and
speaker similarity, respectively. CMOS shows the performance difference
between two systems ranging from $-3$ (the new system is much worse than the
old system) to $3$ (the new system is much better than the old system). SMOS
has $5$ scales from $1$ to $5$, and higher values indicate better speaker
similarity between the synthesized and GT samples.

###  4.2 Main results

We first evaluate the overall performance of RALL-E on the full LibriSpeech
test set with $1205$ utterances. The results are demonstrated in Table 2. It
can be observed that RALL-E outperforms all other methods on WER. The reranked
WER (WER-R) of RALL-E is even better than the WER of GT. Compared to the
baseline VALL-E method, RALL-E obtains a $50\%$ relative improvement on WER
and a $41\%$ relative improvement on WER-R, showing the superior robustness of
the proposed method. This is also proved by the performance on the three error
types, where RALL-E consistently reduces all error types from
$2.8$/$1.5$/$1.3$ to $1.7$/$0.8$/$0.3$, respectively. In addition, the higher
UTMOS score of RALL-E than the one of VALL-E indicates RALL-E can synthesize
speech with better naturalness, showing the effectiveness of the proposed
method on stabilizing speech prosody. Regarding the speaker similarity, both
RALL-E and VALL-E show better performance than the previous two works which is
possibly because of the larger training set used by us. However, the original
VALL-E has a significantly higher SIM score ($0.58$) than other methods. One
possible reason is that the speaker similarity of the original VALL-E is
computed between the synthesized utterance and the prompt resynthesized by the
codec instead of the GT prompt. We notice that VALL-T has slightly fewer
insertion errors ($0.2$) than RALL-E ($0.3$), but we think this is reasonable
and doesn’t necessarily mean RALL-E is worse than VALL-T since the test set of
VALL-T only contains $500$ samples and fewer test samples usually result in a
better WER, as suggested by the result of RALL-E ($904$) computed on the $904$
samples used by ELLA-V.

Table 3: Results of subjective CMOS (v.s. RALL-E) and SMOS tests. Bold indicates the best score. | CMOS | SMOS
---|---|---
GT | -0.02 | $4.23$
VALL-E | -0.17 | $3.50$
RALL-E |  0.00 | $3.57$

Next, we conduct subjective tests to evaluate the performance of RALL-E. We
randomly select $20$ samples from the test set for the CMOS tests, and $10$
samples from distinct speakers for the SMOS test. Two CMOS tests with $6$
workers per test are conducted on two pairs: (GT, RALL-E) and (VALL-E,
RALL-E), respectively. Thus each utterance has $6$ answers. Similarly, an SMOS
test is conducted with $6$ workers. The results are shown in Table 3. It can
be seen that RALL-E obtains a better CMOS than VALL-E and even slightly better
performance than GT utterance, showing the effectiveness of RALL-E on
stabilizing prosody by incorporating prosody tokens as CoT prompting.
Regarding SMOS the two methods have similar performance, which is consistent
with the result of SIM scores shown in Table 2.

###  4.3 Ablation study

Table 4: Results of ablation studies. Bold indicates the best score. | WER%($\downarrow$) | UTMOS ($\uparrow$) | Sub ($\downarrow$) | Del ($\downarrow$) | Ins ($\downarrow$)
---|---|---|---|---|---
RALL-E | $\mathbf{2.8}$ | $\mathbf{4.00}$ | $\mathbf{1.7}$ | $0.8$ | $\mathbf{0.3}$
w/o pitch | $2.8$ | $3.96$ | $1.8$ | $\mathbf{0.7}$ | $0.3$
w/o window masking | $2.9$ | $3.84$ | $1.8$ | $0.8$ | $0.3$
w/o duration-guided masking | $3.4$ | $3.88$ | $2.0$ | $0.9$ | $0.5$
w/o duration CoT prompting | $13.4$ | $3.52$ | $7.8$ | $4.1$ | $1.5$

We conduct ablation experiments to study the contributions of each component
of RALL-E. Specifically, we study the following four settings: (1) w/o pitch
that removes the pitch prompt; (2) w/o window masking where the window size
$k$ is set to $0$; (3) w/o duration-guided masking that uses normal
unidirectional autoregressive attention masks; (4) w/o duration CoT prompting
that removes duration from the CoT and model it separately. In the w/o
duration CoT prompting setting we use an independent $8$-layer Transformer
with $256$-dimensinal token embeddings and $8$ self-attention heads to predict
duration, the masking strategy is still used based on the duration.

The results are shown in Table 4. First, the result of w/o pitch demonstrates
that pitch has little contribution to the performance. While pitch prompting
slightly improves speech naturalness and reduces mispronunciation, it causes
more word omissions. Second, it can be observed from the result of w/o window
masking that the model performance degrades when the window size is set to
$0$, which demonstrates the effectiveness of the window masking strategy. See
Appendix A for a comprehensive study on the effects of window size $k$. Third,
w/o duration-guided masking shows consistently worse performance than RALL-E
on all metrics, showing the effectiveness of the proposed duration-guided
masking strategy. Finally, w/o duration CoT prompting has the worst
performance even though it uses the proposed duration-guided masking. By
manually listening to the samples we find that the predicted duration of the
independent Transformer does not influence the synthesized speech, which
demonstrates the necessity of incorporating the duration with CoT prompting.
In summary, each component of RALL-E contributes to the robustness
improvements, and the CoT prompting is the most important component of RALL-E.

###  4.4 Evaluations on hard sentences

To further evaluate the robustness of RALL-E, we synthesize $50$ particularly
hard sentences (see Appendix B for the transcripts of these sentences) with
RALL-E and VALL-E. We manually evaluate the results since the WER computed on
these sentences with a lot of numbers and symbols is imprecise. We decompose
the possible errors into four types: mispronunciation, omission, repetition,
and hallucination. Each utterance is synthesized $5$ times and the best one is
selected. We count the frequency of all error types and compute the overall
sentence error rate. Each error type is only counted once in an utterance. The
results are shown in Table 1, in which we also add a powerful non-
autoregressive TTS method NaturalSpeech2 [25] for reference. It can be seen
that RALL-E substantially reduces the error rate from $68\%$ to $4\%$ with
only $2$ mispronunciation errors. This performance approaches the one of non-
autoregressive NaturalSpeech2 that has no error, which again demonstrates the
effectiveness of RALL-E in improving the robustness of LLM-based TTS.
Specifically, we observe that for extremely short sentences, e.g. a single
alphabet “A”, VALL-E usually synthesizes words that don’t exist in the input
sentence, causing a severe hallucination problem. Besides, if a word is
repeated many times in a sentence, e.g. “22222222”, VALL-E easily makes
mistakes by omitting or repeating the word. These problems together
demonstrate that LLM-based TTS like VALL-E has low controllability on the
duration of the synthesized speech and learns poor alignments between phonemes
and speech tokens. In contrast, RALL-E first introduces prosody tokens as CoT
prompting to improve duration controllability and then uses duration-guided
masking to enhance the alignment, which effectively alleviates the
aforementioned problems made by the language models. All in all, our RALL-E
demonstrates superior robustness in all evaluations.

##  5 Conclusions

This paper presents RALL-E, a robust codec language modeling method with CoT
prompting for TTS. To address the robustness problem of LLM-based TTS, RALL-E
(1) incorporates prosody features (pitch and duration) in the LLM as a CoT
prompting to assist and stabilize the generation of speech tokens, and (2)
proposes duration-guided masking that enforces the model to attend on relevant
phonemes (prosody features) corresponding to each speech token. We conduct
comprehensive objective and subjective evaluations and demonstrate that RALL-E
can significantly improve the robustness of LLM-based TTS compared to the
baseline VALL-E and two previous works. Furthermore, we show that RALL-E can
correctly synthesize sentences that are particularly hard to synthesize for
VALL-E with a $4\%$ error rate that even approaches the performance of non-
autoregressive TTS.

## Acknowledgments and Disclosure of Funding

We thank Yakun Song and Chenpeng Du for providing the test samples of ELLA-V
and VALL-T, respectively.

## References

  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chen et al. [2020] M. Chen, X. Tan, Y. Ren, J. Xu, H. Sun, S. Zhao, T. Qin, and T.-Y. Liu.  Multispeech: Multi-speaker text to speech with transformer.  _arXiv preprint arXiv:2006.04664_ , 2020.
  * Chen et al. [2022] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, et al.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  _IEEE Journal of Selected Topics in Signal Processing_ , 16(6):1505–1518, 2022.
  * Défossez et al. [2022] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi.  High fidelity neural audio compression.  _arXiv preprint arXiv:2210.13438_ , 2022.
  * Dehghani et al. [2023] M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, et al.  Scaling vision transformers to 22 billion parameters.  In _Proc. ICML_ , pages 7480–7512. PMLR, 2023.
  * Du et al. [2024] C. Du, Y. Guo, H. Wang, Y. Yang, Z. Niu, S. Wang, H. Zhang, X. Chen, and K. Yu.  Vall-t: Decoder-only generative transducer for robust and decoding-controllable text-to-speech.  _arXiv preprint arXiv:2401.14321_ , 2024.
  * Graves [2012] A. Graves.  Sequence transduction with recurrent neural networks.  _arXiv preprint arXiv:1211.3711_ , 2012.
  * Gulati et al. [2020] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang.  Conformer: Convolution-augmented Transformer for Speech Recognition.  In _Proc. Interspeech_ , pages 5036–5040, 2020.  doi: 10.21437/Interspeech.2020-3015.
  * He et al. [2019] M. He, Y. Deng, and L. He.  Robust sequence-to-sequence acoustic modeling with stepwise monotonic attention for neural tts.  _arXiv preprint arXiv:1906.00672_ , 2019.
  * Holtzman et al. [2019] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi.  The curious case of neural text degeneration.  In _Proc. ICLR_ , 2019.
  * Hsu et al. [2021] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460, 2021.
  * Ju et al. [2024] Z. Ju, Y. Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y. Liu, Y. Leng, K. Song, S. Tang, et al.  Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models.  _arXiv preprint arXiv:2403.03100_ , 2024.
  * Kahn et al. [2020] J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P. E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux.  Libri-light: A benchmark for asr with limited or no supervision.  In _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7669–7673, 2020.  https://github.com/facebookresearch/libri-light.
  * Kharitonov et al. [2023] E. Kharitonov, D. Vincent, Z. Borsos, R. Marinier, S. Girgin, O. Pietquin, M. Sharifi, M. Tagliasacchi, and N. Zeghidour.  Speak, read and prompt: High-fidelity text-to-speech with minimal supervision.  _arXiv preprint arXiv:2302.03540_ , 2023.
  * Kingma and Ba [2015] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In _Proc. ICLR_ , San Diego, USA, May 2015.
  * Le et al. [2024] M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokar, et al.  Voicebox: Text-guided multilingual universal speech generation at scale.  _Advances in neural information processing systems_ , 36, 2024.
  * Morise et al. [2016] M. Morise, F. Yokomori, and K. Ozawa.  World: a vocoder-based high-quality speech synthesis system for real-time applications.  _IEICE TRANSACTIONS on Information and Systems_ , 99(7):1877–1884, 2016.
  * Panayotov et al. [2015] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur.  Librispeech: an asr corpus based on public domain audio books.  In _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 5206–5210. IEEE, 2015.
  * Pratap et al. [2020] V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert.  MLS: A Large-Scale Multilingual Dataset for Speech Research.  In _Proc. Interspeech_ , pages 2757–2761, 2020.  doi: 10.21437/Interspeech.2020-2826.
  * Radford et al. [2019] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * Ren et al. [2019] Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu.  Fastspeech: Fast, robust and controllable text to speech.  _Proc. NeurIPS_ , 32, 2019.
  * Rubenstein et al. [2023] P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. d. C. Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov, et al.  Audiopalm: A large language model that can speak and listen.  _arXiv preprint arXiv:2306.12925_ , 2023.
  * Saeki et al. [2022] T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari.  Utmos: Utokyo-sarulab system for voicemos challenge 2022.  _arXiv preprint arXiv:2204.02152_ , 2022.
  * Shen et al. [2020] J. Shen, Y. Jia, M. Chrzanowski, Y. Zhang, I. Elias, H. Zen, and Y. Wu.  Non-attentive tacotron: Robust and controllable neural tts synthesis including unsupervised duration modeling.  _arXiv preprint arXiv:2010.04301_ , 2020.
  * Shen et al. [2023] K. Shen, Z. Ju, X. Tan, Y. Liu, Y. Leng, L. He, T. Qin, S. Zhao, and J. Bian.  Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers.  _arXiv preprint arXiv:2304.09116_ , 2023.
  * Song et al. [2024] Y. Song, Z. Chen, X. Wang, Z. Ma, and X. Chen.  Ella-v: Stable neural codec language modeling with alignment-guided sequence reordering.  _arXiv preprint arXiv:2401.07333_ , 2024.
  * Sun et al. [2019] H. Sun, X. Tan, J.-W. Gan, H. Liu, S. Zhao, T. Qin, and T.-Y. Liu.  Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion.  In _Proc. Interspeech 2019_ , pages 2115–2119, 2019.  doi: 10.21437/Interspeech.2019-1208.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2023a] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, et al.  Neural codec language models are zero-shot text to speech synthesizers.  _arXiv preprint arXiv:2301.02111_ , 2023a.
  * Wang et al. [2023b] T. Wang, L. Zhou, Z. Zhang, Y. Wu, S. Liu, Y. Gaur, Z. Chen, J. Li, and F. Wei.  Viola: Unified codec language models for speech recognition, synthesis, and translation.  _arXiv preprint arXiv:2305.16107_ , 2023b.
  * Wei et al. [2022a] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ , 2022a.
  * Wei et al. [2022b] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837, 2022b.
  * Yang et al. [2023] D. Yang, J. Tian, X. Tan, R. Huang, S. Liu, X. Chang, J. Shi, S. Zhao, J. Bian, X. Wu, et al.  Uniaudio: An audio foundation model toward universal audio generation.  _arXiv preprint arXiv:2310.00704_ , 2023.
  * Zeghidour et al. [2021] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi.  Soundstream: An end-to-end neural audio codec.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 30:495–507, 2021.
  * Zen et al. [2019] H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu.  Libritts: A corpus derived from librispeech for text-to-speech.  _Proc. Interspeech_ , 2019.
  * Zhang et al. [2018] J.-X. Zhang, Z.-H. Ling, and L.-R. Dai.  Forward attention in sequence-to-sequence acoustic modeling for speech synthesis.  In _Proc. ICASSP_ , pages 4789–4793. IEEE, 2018.

##  Appendix A Window size study

Figure 3: Results of the window size study. For simplicity, we use $\rho_{pd}$
to refer to $\rho_{p}$ and $\rho_{d}$ together. $k=\infty$ means the window
covers the whole phoneme sequence, i.e. no phoneme is masked. The model with
$k=\infty$ fails to generate speech tokens when $\rho_{c}=0.9$.

We study the window size hyperparameter $k$ used in the proposed duration-
guided masking method. Basically, $2k+1$ is the number of phonemes (prosody
features) the model can attend during decoding. As mentioned in Section 3.3,
the motivation of the window is to (1) increase context information received
by the model during decoding and (2) improve the robustness of the proposed
duration-based masking strategy since the extracted duration features can have
errors during training and the predicted duration may not strictly correspond
to the number of predicted speech tokens for each phoneme during inference. We
suppose $k=0$ will make RALL-E less robust, but large $k$ will also make it
difficult to learn the alignment between phonemes and speech tokens. Thus we
study the optimal value of $k$. We train RALL-E and compute WER on the test
set with $k=0,1,2,3,\infty$, in which $k=\infty$ means the window covers the
whole phoneme sequence, i.e. no phoneme is masked during decoding. We
hypothesize that diverse sampling will make the robustness problem more
obvious, thus we perform nucleus sampling on each model with four settings:
(1) $\rho_{p}=\rho_{d}=\rho_{c}=0.9$; (2)
$\rho_{p}=\rho_{d}=0.9,\rho_{c}=1.0$; (3)
$\rho_{p}=\rho_{d}=1.0,\rho_{c}=0.9$; and (4)
$\rho_{p}=\rho_{d}=1.0,\rho_{c}=1.0$. The results are illustrated in Figure 3.
First, it can be observed that in every sampling setting the WER can be
substantially improved by increasing $k$ from $0$ to $1$, showing the
effectiveness of the proposed window masking strategy. This observation also
verifies the hypothesis that the more the sampling becomes diverse, the more
the robustness problem becomes obvious. Second, the performance cannot be
further improved by increasing $k$ to values larger than $1$, which verifies
another hypothesis that large $k$ makes it difficult to learn the alignment.
When $k=\infty$ the model has to learn the alignment completely by itself,
thus resulting in the worst WERs. Combining all results we conclude that the
proposed window masking strategy can effectively improve WERs and the best
performance is obtained when $k=1$.

##  Appendix B Transcripts of the 50 hard sentences

We list the $50$ hard sentences used in Section 4.4 below:

  1. 1.

a

  2. 2.

b

  3. 3.

c

  4. 4.

H

  5. 5.

I

  6. 6.

J

  7. 7.

K

  8. 8.

L

  9. 9.

22222222 hello 22222222

  10. 10.

S D S D Pass zero - zero Fail - zero to zero - zero - zero Cancelled - fifty
nine to three - two - sixty four Total - fifty nine to three - two -

  11. 11.

S D S D Pass - zero - zero - zero - zero Fail - zero - zero - zero - zero
Cancelled - four hundred and sixteen - seventy six -

  12. 12.

zero - one - one - two Cancelled - zero - zero - zero - zero Total - two
hundred and eighty six - nineteen - seven -

  13. 13.

forty one to five three hundred and eleven Fail - one - one to zero two
Cancelled - zero - zero to zero zero Total -

  14. 14.

zero zero one , MS03 - zero twenty five , MS03 - zero thirty two , MS03 - zero
thirty nine ,

  15. 15.

1b204928 zero zero zero zero zero zero zero zero zero zero zero zero zero zero
one seven ole32 11

  16. 16.

zero zero zero zero zero zero zero zero two seven nine eight F three forty
zero zero zero zero zero six four two eight zero one eight

  17. 17.

c five eight zero three three nine a zero bf eight FALSE zero zero zero
bba3add2 - c229 - 4cdb -

  18. 18.

Calendaring agent failed with error code 0x80070005 while saving appointment .

  19. 19.

Exit process - break ld - Load module - output ud - Unload module - ignore ser
- System error - ignore ibp - Initial breakpoint -

  20. 20.

Common DB connectors include the DB - nine , DB - fifteen , DB - nineteen , DB
- twenty five , DB - thirty seven , and DB - fifty connectors .

  21. 21.

To deliver interfaces that are significantly better suited to create and
process RFC eight twenty one , RFC eight twenty two , RFC nine seventy seven ,
and MIME content .

  22. 22.

int1 , int2 , int3 , int4 , int5 , int6 , int7 , int8 , int9 ,

  23. 23.

seven _ ctl00 ctl04 ctl01 ctl00 ctl00

  24. 24.

Http0XX , Http1XX , Http2XX , Http3XX ,

  25. 25.

config file must contain A , B , C , D , E , F , and G .

  26. 26.

mondo - debug mondo - ship motif - debug motif - ship sts - debug sts - ship
Comparing local files to checkpoint files …

  27. 27.

Rusbvts . dll Dsaccessbvts . dll Exchmembvt . dll Draino . dll Im trying to
deploy a new topology , and I keep getting this error .

  28. 28.

You can call me directly at four two five seven zero three seven three four
four or my cell four two five four four four seven four seven four or send me
a meeting request with all the appropriate information .

  29. 29.

Failed zero point zero zero percent < one zero zero one zero zero zero zero
Internal . Exchange . ContentFilter . BVT ContentFilter . BVT_log . xml Error
! Filename not specified .

  30. 30.

C colon backslash o one two f c p a r t y backslash d e v one two backslash
oasys backslash legacy backslash web backslash HELP

  31. 31.

src backslash mapi backslash t n e f d e c dot c dot o l d backslash backslash
m o z a r t f one backslash e x five

  32. 32.

copy backslash backslash j o h n f a n four backslash scratch backslash M i c
r o s o f t dot S h a r e P o i n t dot

  33. 33.

Take a look at h t t p colon slash slash w w w dot granite dot a b dot c a
slash access slash email dot

  34. 34.

backslash bin backslash premium backslash forms backslash r e g i o n a l o p
t i o n s dot a s p x dot c s Raj , DJ ,

  35. 35.

Anuraag backslash backslash r a d u r five backslash d e b u g dot one eight
zero nine underscore P R two h dot s t s contains

  36. 36.

p l a t f o r m right bracket backslash left bracket f l a v o r right bracket
backslash s e t u p dot e x e

  37. 37.

backslash x eight six backslash Ship backslash zero backslash A d d r e s s B
o o k dot C o n t a c t s A d d r e s

  38. 38.

Mine is here backslash backslash g a b e h a l l hyphen m o t h r a backslash
S v r underscore O f f i c e s v r

  39. 39.

h t t p colon slash slash teams slash sites slash T A G slash default dot aspx
As always , any feedback , comments ,

  40. 40.

two thousand and five h t t p colon slash slash news dot com dot com slash i
slash n e slash f d slash two zero zero three slash f d

  41. 41.

backslash i n t e r n a l dot e x c h a n g e dot m a n a g e m e n t dot s y
s t e m m a n a g e

  42. 42.

I think Rich’s post highlights that we could have been more strategic about
how the sum total of XBOX three hundred and sixtys were distributed .

  43. 43.

64X64 , 8K , one hundred and eighty four ASSEMBLY , DIGITAL VIDEO DISK DRIVE ,
INTERNAL , 8X ,

  44. 44.

So we are back to Extended MAPI and C++ because . Extended MAPI does not have
a dual interface VB or VB .Net can read .

  45. 45.

Thanks , Borge Trongmo Hi gurus , Could you help us E2K ASP guys with the
following issue ?

  46. 46.

Thanks J RGR Are you using the LDDM driver for this system or the in the build
XDDM driver ? 12

  47. 47.

Btw , you might remember me from our discussion about OWA automation and OWA
readiness day a year ago .

  48. 48.

empidtool . exe creates HKEY_CURRENT_USER Software Microsoft Office Common
QMPersNum in the registry , queries AD , and the populate the registry with MS
employment ID if available else an error code is logged .

  49. 49.

Thursday, via a joint press release and Microsoft AI Blog, we will announce
Microsoft’s continued partnership with Shell leveraging cloud, AI, and
collaboration technology to drive industry innovation and transformation.

  50. 50.

Actress Fan Bingbing attends the screening of ’Ash Is Purest White (Jiang Hu
Er Nv)’ during the 71st annual Cannes Film Festival


# Superlinear transmission in an indirect signal production chemotaxis system

Xinru Cao<EMAIL_ADDRESS>North Renmin Road 2999, 201620 Shanghai, China
School of Mathematics and Statistics,
Donghua University

###### Abstract

In this paper, the indirect signal production system with nonlinear
transmission is considered

$\left\\{\begin{array}[]{lll}&u_{t}=\Delta u-\nabla\cdot(u\nabla v),\\\
\displaystyle&v_{t}=\Delta v-v+w,\\\ \displaystyle&w_{t}=\Delta
w-w+f(u)\end{array}\right.$

in a bounded smooth domain $\Omega\subset\mathbb{R}^{n}$ associated with
homogenous Neumann boundary conditions, where $f\in C^{1}([0,\infty))$
satisfies $0\leq f(s)\leq s^{\alpha}$ with $\alpha>0$. It is known from [7]
that the system possesses a global bounded solution if $0<\alpha<\frac{4}{n}$
when $n\geq 4$. In the case $n\leq 3$ and if we consider superlinear
transmission, no regularity of $w$ or $v$ can be derived directly. In this
work, we show that if $0<\alpha<\min\\{\frac{4}{n},1+\frac{2}{n}\\}$, the
solution is global and bounded via an approach based on the maximal Sobolev
regularity.
Keywords: chemotaxis, boundedness, nonlinear transmission
Math Subject Classification (2010): 35K55, 35B33, 92C17

##  1 Introduction

The pioneer chemotaxis model was established by Keller and Segel in 1970’s
[4], to describe cells’ motion which biased by a chemical substance secreted
by the cells themselves.

Recently, phenotype heterogeneity is assumed among the cells population, they
are divided into two groups; one group secrets chemical signal, the other
group is attracted by the signal. These abilities do not hold simultaneously,
some kind of trade-off is observed between two groups. This model can be seen
as an extension of the classical Keller-Segel model, with $u$ and $w$
presenting the density of “chemotaxis” and “secreting” groups, respectively,
and $v$ is the concentration of chemical substance. Under these assumptions,
the following system is proposed [6]:

$\left\\{\begin{array}[]{lllll}&u_{t}=\Delta u-\nabla\cdot(u\nabla
v)-k_{1}u+k_{2}w,&(x,t)\in\Omega\times(0,T),\\\\[2.0pt]
\displaystyle&v_{t}=\Delta v-v+w,&(x,t)\in\Omega\times(0,T),\\\\[2.0pt]
\displaystyle&w_{t}=\Delta w-w+f(u),&(x,t)\in\Omega\times(0,T),\\\\[2.0pt]
&\partial_{\nu}u=\partial_{\nu}v=\partial_{\nu}w=0,&(x,t)\in\par\Omega\times(0,T),\\\\[2.0pt]
&u(x,0)=u_{0}(x),v(x,0)=v_{0}(x),w(x,0)=w_{0}(x),&x\in\Omega,\end{array}\right.$
(1.1)

where $\Omega\subset\mathbb{R}^{n}$ ($n\geq 1$) is smooth and bounded, $T>0$,
$\nu$ denotes the outer normal vector on $\partial\Omega$, $k_{1},k_{2}\geq
0$, and

$\displaystyle f\in C^{1}([0,\infty))\text{ and }0\leq f(s)\leq
s^{\alpha}\text{ with some }{\alpha>0},$ (1.2) $\displaystyle u_{0}\in
C^{0}(\overline{\Omega}),v_{0}\in W^{2,\infty}(\Omega),w_{0}\in
W^{2,\infty}(\Omega)\text{ are nonnegative}.$ (1.3)

Compared to the classical Keller-Segel system, the indirect signal production
mechanism in (1.1) manifests a stronger tendency toward stabilization; it is
proven in [2, 3] that in the case $f(u)\equiv u$, the solution of (1.1) is
bounded for $n\leq 3$ , which is in contrast to the classical Keller-Segel
system that there exist finite time blowup solutions in two and three
dimensions [8]. Considering direct nonlinear signal production model:

$u_{t}=\Delta u-\nabla\cdot(u\nabla v),\quad v_{t}=\Delta v-v+f(u),$ (1.4)

where $f$ satisfies (1.2), the value $\alpha$ determines boundedness or blow
up of the solution; if $0<\alpha<\frac{2}{n}$, the solution is global and
bounded [5]; if $f(s)=s^{\alpha}$ with $\alpha>\frac{2}{n}$ and the second
equation is replaced by $\Delta
v-\frac{1}{|\Omega|}\int_{\Omega}f(u(\cdot,t))+f(u)=0$ in (1.4), finite time
blowup solution is constructed in radial symmetric setting [9]. In the present
work, we study indirect nonlinear signal production mechanism in (1.1), and
obtain the following:

######  Theorem 1.1.

Let $n\leq 3$, $k_{1}=k_{2}=0$, $\Omega\subset\mathbb{R}^{n}$ be bounded
domain with smooth boundary, $f$ satisfy (1.2) and let $(u_{0},v_{0},w_{0})$
comply with (1.3). If $0<\alpha<\min\\{\frac{4}{n},1+\frac{2}{n}\\}$, then the
system (1.1) admits a unique global classical solution $(u,v,w)$ satisfying

$(u,v,w)\in(C^{0}(\overline{\Omega}\times[0,\infty))\cap
C^{2,1}(\overline{\Omega}\times(0,\infty)))^{3},$

which is uniformly bounded.

Noting that the condition on $\alpha$ will allow superlinear transmission in
the third equation, no integrable regularity is known for the production term
$f(u)$, we can not gain any information about $w$. The strategy of the proof
is to track the evolution of $\int_{\Omega}u^{p}(\cdot,t)$, and run a loop
through each equation via the maximal Sobolev regularity.

##  2 Preliminaries

Let us first give local existence theory for (1.1). It can be proven by fix
point argument and Schauder theory.

######  Lemma 2.1.

Assume that $\Omega\subset\mathbb{R}^{n}$ $(n\geq 1)$ is a bounded domain with
smooth boundary, $f$ satisfies (1.2) and that the initial data
$(u_{0},v_{0},w_{0})$ satisfy (1.3). There exists $T_{max}\in(0,\infty]$ with
the property that the problem possess a unique classical solution $(u,v,w)$
satisfy

$(u,v,w)\in(C^{0}(\overline{\Omega}\times[0,T_{max}))\cap
C^{2,1}(\overline{\Omega}\times(0,T_{max})))^{3},{\text{ and }u,v,w>0\text{ in
}\overline{\Omega}\times(0,T_{max}).}$

Moreover, if $T_{max}<\infty$, then
$\|u(\cdot,t)\|_{L^{\infty}(\Omega)}\to\infty,\text{ as }t\nearrow T_{max}.$

The next lemma can be proven by simply integrating over $\Omega$. The second
result is of particular interest for the superlinear transmission case, that
no regularity for $w$ is known, however, its $L^{1}(\Omega)$-norm can be
estimated by $u$ non-locally in time.

######  Lemma 2.2.

Let $s\in(0,T_{max})$. There hold that

$\displaystyle\|u(\cdot,s)\|_{L^{1}(\Omega)}=\|u_{0}\|_{L^{1}(\Omega)},$ (2.5)
$\displaystyle\|w(\cdot,s)\|_{L^{1}(\Omega)}\leq{\int_{0}^{s}e^{-(s-\sigma)}\|u^{\alpha}(\cdot,\sigma)\|_{L^{1}(\Omega)}d\sigma+\|w_{0}\|_{L^{1}(\Omega)}}\qquad\mbox{for
all }s\in(0,T_{max}).$ (2.6)

The following result on maximal Sobolev regularity is the key tool for us to
treat the term with space-time integrations. The proof can be found in [1,
Lemma 2.3].

######  Lemma 2.3.

For any given $q,r\in(1,\infty)$ and for any fixed $\kappa\in(0,1)$, there
exists $C=C(q,r,\kappa)>0$ with the following property:
For all $T>0$, let $h\in C^{0}(\overline{\Omega}\times(0,T))$ and let $z$ be a
classical solution of

$\left\\{\begin{array}[]{lll}&z_{t}=\Delta
z-z+h,&(x,t)\in\Omega\times(0,T),\\\\[2.84526pt]
&\partial_{\nu}z=0,&(x,t)\in\partial\Omega\times(0,T),\\\\[2.84526pt]
&z(x,0)=z_{0}(x),&x\in\Omega,\end{array}\right.$ (2.7)

then we have

$\displaystyle\int_{0}^{t}e^{-\kappa r(t-s)}\|\Delta
z(\cdot,s)\|_{L^{q}(\Omega)}^{r}ds\leq C\int_{0}^{t}e^{-\kappa
r(t-s)}\|h(\cdot,s)\|_{L^{q}(\Omega)}^{r}ds+C\|z_{0}\|_{W^{2,q}(\Omega)}^{r}$
(2.8)

for all $t\in(0,T)$.

As the final preparation, we present a fundamental estimate to treat a term
with double integration in time.

######  Lemma 2.4.

Let $\delta\in(0,1)$, $q\geq 1$, $T>0$ and let $g\in L^{\infty}((0,T))$ be
nonnegative. It holds that

$\displaystyle~{}~{}\int_{0}^{t}e^{-\delta(t-s)}\left(\int_{0}^{s}e^{-(s-\sigma)}g(\sigma)d\sigma\right)^{q}ds\leq\frac{1}{1-\delta}\int_{0}^{t}e^{-\delta(t-s)}g^{q}(s)ds\text{
for all }t\in(0,T).$

###### Proof.

First we apply Hölder’s inequality to find that

$\displaystyle\left(\int_{0}^{s}e^{-(s-\sigma)}g(\sigma)d\sigma\right)^{q}\leq\left(\int_{0}^{s}e^{-(s-\sigma)}g^{q}(\sigma)d\sigma\right)\left(\int_{0}^{s}e^{-(s-\sigma)}d\sigma\right)^{q(1-\frac{1}{q})}\leq\int_{0}^{s}e^{-(s-\sigma)}g^{q}(\sigma)d\sigma.$

Thanks to the Fubini Theorem, we have

$\displaystyle~{}~{}~{}~{}\int_{0}^{t}e^{-\delta(t-s)}\left(\int_{0}^{s}e^{-(s-\sigma)}g(\sigma)d\sigma\right)^{q}ds$
$\displaystyle\leq\int_{0}^{t}e^{-\delta(t-s)}\int_{0}^{s}e^{-(s-\sigma)}g^{q}(\sigma)d\sigma
ds=\int_{0}^{t}e^{-\delta
t+\sigma}g^{q}(\sigma)\int_{\sigma}^{t}e^{-(1-\delta)s}dsd\sigma\leq\frac{1}{1-\delta}\int_{0}^{t}e^{-\delta(t-\sigma)}g^{q}(\sigma)d\sigma.$

∎

##  3 Proof of Theorem 1.1

In order to show boundedness of $\|u\|_{L^{p}(\Omega)}$ for arbitrarily large
$p$, we need to fix parameters properly, the choices of the parameters are
guaranteed by the condition on $\alpha$.

######  Lemma 3.1.

Let $n\leq 3$. Suppose that $1<\alpha<\min\\{\frac{4}{n},1+\frac{2}{n}\\}$. We
can fix

$\displaystyle\tau>1\text{ satisfying
}\alpha-\frac{2}{n}<\frac{1}{\tau}<\min\left\\{1,\frac{2}{n}\right\\},$ (3.9)
$\displaystyle\theta^{\prime}>1\text{ be such that
}\frac{1}{\tau}-\min\left\\{1,\frac{2}{n}\right\\}<\frac{1}{\theta^{\prime}}<\frac{1}{\tau}+\frac{2}{n}-\alpha,\text{
and }\quad\frac{1}{\theta}+\frac{1}{\theta^{\prime}}=1.$ (3.10) Let
$\displaystyle
b:=\frac{n-\frac{n}{\theta^{\prime}}}{2-\frac{n}{\tau}+n}\in(0,1),\quad
p>\max\left\\{1,\frac{\alpha(n-2)_{+}}{n},\frac{\alpha\tau(n-2)_{+}}{n},\frac{\frac{2}{n}-\frac{1}{\theta^{\prime}}}{1-b}+1-\frac{2}{n}\right\\},$
(3.11) $\displaystyle
a:=\frac{\frac{np}{2}-\frac{n}{2\theta}}{1-\frac{n}{2}+\frac{np}{2}}\in(0,1),\quad\lambda=2a\in(0,2),\quad$
(3.12) and that
$\displaystyle\eta_{1}:=\frac{n\frac{2}{2-\lambda}b(\alpha-\frac{1}{\tau})}{2-n+np}>0,\quad\eta_{2}:=\frac{n\frac{2}{2-\lambda}(1-b)(\alpha-1)}{2-n+np}>0,\quad\eta_{3}:=\frac{n\frac{2}{2-\lambda}(\alpha-1)}{2-n+np}\in(0,1).$
(3.13)

We can find $\beta,\beta^{\prime}\in(1,\infty)$ satisfying
$\frac{1}{\beta}+\frac{1}{\beta^{\prime}}=1$, and that

$\displaystyle
0<\eta_{1}+\eta_{2}<1,\quad\beta\eta_{1}<1,\quad\beta^{\prime}\eta_{2}<1.$
(3.14)

###### Proof.

First $n\leq 3$ and $1<\alpha<\min\left\\{\frac{4}{n},1+\frac{2}{n}\right\\}$
yields that $\alpha-\frac{2}{n}<\min\left\\{1,\frac{2}{n}\right\\}$. Hence we
can fix $\tau$ and $\theta^{\prime}$ as in (3.9) and (3.10). Moreover, we see
that $\frac{1}{\theta^{\prime}}<\frac{2}{n}$, and
$\frac{1}{\theta}>(1-\frac{2}{n})_{+}$, which implies that $a\in(0,1)$, as
well as $\lambda\in(0,2)$. Noting that (3.10) ensures

$\displaystyle\alpha-\frac{2}{n}<\frac{1}{\tau}-\frac{1}{\theta^{\prime}}<\min\left\\{1,\frac{2}{n}\right\\}.$
(3.15)

Hence $b\in(0,1)$. And (3.15) also implies
$\alpha<1+\frac{2}{n}-\frac{1}{\theta^{\prime}}$ because of $\tau>1$, then
$\eta_{3}\in(0,1)$. Furthermore, we have
$0<\alpha-\frac{1}{\tau}<\frac{2}{n}-\frac{1}{\theta^{\prime}}$, meaning that

$\displaystyle\eta_{1}+\eta_{2}$
$\displaystyle=\frac{(1-\frac{1}{\theta^{\prime}})(\alpha-\frac{1}{\tau})}{(\frac{2}{n}-\frac{1}{\theta^{\prime}})(1-\frac{1}{\tau}+\frac{2}{n})}+\frac{(\frac{2}{n}-\frac{1}{\tau}+\frac{1}{\theta^{\prime}})(\alpha-1)}{(\frac{2}{n}-\frac{1}{\theta^{\prime}})(1-\frac{1}{\tau}+\frac{2}{n})}<\frac{\alpha-\frac{1}{\tau}}{\frac{2}{n}-\frac{1}{\theta^{\prime}}}<1.$

Therefore, we can fix $\beta\in(\frac{1}{1-\eta_{2}},\frac{1}{\eta_{1}})$. It
is convenient to check the inequalities in (3.14) hold. ∎

Via standard $L^{p}$ testing procedure, our main idea is to control the cross-
diffusion contributions toward dissipation of $u$. Let us first prepare the
following spatio-temporal estimate by simple interpolation.

######  Lemma 3.2.

Let $\alpha>0$, $\mu>0$, $\delta>0$ and
$\frac{1}{\alpha}<k<\frac{np}{\alpha(n-2)_{+}}$ satisfy
$n\mu(\alpha-\frac{1}{k})\leq np-n+2$. There exists $C>0$ such that for all
$t\in(0,T_{max})$,

$\displaystyle\int_{0}^{t}e^{-\delta(t-s)}\|u^{\alpha}(\cdot,s)\|_{L^{k}(\Omega)}^{\mu}ds\leq
C\left(\int_{0}^{t}e^{-\delta(t-s)}\|\nabla
u^{\frac{p}{2}}(\cdot,s)\|_{L^{2}(\Omega)}^{2}ds\right)^{\frac{n\mu(\alpha-\frac{1}{k})}{np-n+2}}+C.$
(3.16)

###### Proof.

Let $d=\frac{\frac{np}{2}-\frac{np}{2\alpha k}}{1-\frac{n}{2}+\frac{np}{2}}$.
We can check that $d\in(0,1)$. The Gargliardo-Nirenberg inequality implies
that

$\displaystyle\|u^{\alpha}\|_{L^{k}(\Omega)}^{\mu}$
$\displaystyle=\|u^{\frac{p}{2}}\|_{L^{\frac{2\alpha
k}{p}}(\Omega)}^{\frac{2\alpha\mu}{p}}\leq c_{1}\|\nabla
u^{\frac{p}{2}}\|_{L^{2}(\Omega)}^{\frac{2\alpha\mu}{p}d}\|u^{\frac{p}{2}}\|_{L^{\frac{2}{p}}(\Omega)}^{\frac{2\alpha\mu}{p}(1-d)}+c_{1}\|u^{\frac{p}{2}}\|_{L^{\frac{2}{p}}(\Omega)}^{\frac{2\alpha\mu}{p}}$
$\displaystyle\leq c_{2}\|\nabla
u^{\frac{p}{2}}\|_{L^{2}(\Omega)}^{\frac{2\alpha\mu}{p}d}+c_{2}\qquad\mbox{for
all }t\in(0,T_{max}).$

Integrating over $(0,t)$, we obtain that

$\displaystyle\int_{0}^{t}e^{-\delta(t-s)}\|u^{\alpha}(\cdot,s)\|_{L^{k}(\Omega)}^{\mu}ds$
$\displaystyle\leq c_{2}\int_{0}^{t}e^{-\delta(t-s)}\|\nabla
u^{\frac{p}{2}}(\cdot,s)\|_{L^{2}(\Omega)}^{\frac{2\alpha\mu}{p}d}ds+\frac{c_{2}}{\delta}.$

If $n\mu(\alpha-\frac{1}{k})=np-n+2$, this is (3.16). If
$n\mu(\alpha-\frac{1}{k})<np-n+2$, meaning that $\frac{2\alpha\mu}{p}d<2$.
Applying Hölder’s inequality, we see that

$\displaystyle\int_{0}^{t}e^{-\delta(t-s)}\|\nabla
u^{\frac{p}{2}}(\cdot,s)\|_{L^{2}(\Omega)}^{\frac{2\alpha\mu}{p}d}ds$
$\displaystyle\leq\left(\int_{0}^{t}e^{-\delta(t-s)}\|\nabla
u^{\frac{p}{2}}(\cdot,s)\|_{L^{2}(\Omega)}^{2}ds\right)^{\frac{\alpha\mu}{p}d}\left(\int_{0}^{t}e^{-\delta(t-s)}ds\right)^{1-\frac{\alpha\mu}{p}d}$
$\displaystyle\leq\left(\frac{1}{\delta}\right)^{1-\frac{\alpha\mu}{p}d}\left(\int_{0}^{t}e^{-\delta(t-s)}\|\nabla
u^{\frac{p}{2}}(\cdot,s)\|_{L^{2}(\Omega)}^{2}ds\right)^{\frac{\alpha\mu}{p}d}.$
(3.17)

∎

Our next step to handle the cross-diffusion effect involves a priori estimate
of $w$. In particular, we focus on the superlinear transmission case, this
causes poor information on the source term in the third equation, hence no
regularity is known for $w$. However, we can still use interpolation and
control spatial-temporal integration against $u^{\alpha}$.

######  Lemma 3.3.

Let $\delta\in(0,\min\\{1,\frac{2}{2-\lambda},\frac{2}{2-\lambda}b\beta\\})$.
There exists $C>0$ such that for all $t\in(0,T_{max})$, we have

$\displaystyle~{}~{}~{}~{}\int_{0}^{t}e^{-\delta(t-s)}\|w(\cdot,s)\|_{L^{\theta^{\prime}}(\Omega)}^{\frac{2}{2-\lambda}}ds$
$\displaystyle\leq
C\left(\int_{0}^{t}e^{-\delta(t-s)}\|u^{\alpha}(\cdot,s)\|_{L^{\tau}(\Omega)}^{\frac{2}{2-\lambda}b\beta}ds\right)^{\frac{1}{\beta}}\left(\int_{0}^{t}e^{-\delta(t-s)}\|u^{\alpha}(\cdot,s)\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}(1-b)\beta^{\prime}}ds\right)^{\frac{1}{\beta^{\prime}}}$
$\displaystyle\quad+C\left(\int_{0}^{t}e^{-\delta(t-s)}\|u^{\alpha}(\cdot,s)\|_{L^{\tau}(\Omega)}^{\frac{2}{2-\lambda}b\beta}ds\right)^{\frac{1}{\beta}}+C\left(\int_{0}^{t}e^{-\delta(t-s)}\|u^{\alpha}(\cdot,s)\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}(1-b)\beta^{\prime}}ds\right)^{\frac{1}{\beta^{\prime}}}$
$\displaystyle\qquad+C\int_{0}^{t}e^{-\delta(t-s)}\|u^{\alpha}(\cdot,s)\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}}ds+C.$
(3.18)

###### Proof.

Invoking the Gargliardo-Nirenberg inequality, we obtain $c_{1}>0$ such that

$\displaystyle\|w(\cdot,s)\|_{L^{\theta^{\prime}}(\Omega)}\leq c_{1}\|\Delta
w(\cdot,s)\|_{L^{\tau}(\Omega)}^{b}\|w(\cdot,s)\|_{L^{1}(\Omega)}^{1-b}+c_{1}\|w(\cdot,s)\|_{L^{1}(\Omega)}\text{
for all }s\in(0,T_{max}).$

Multiplying $e^{-\delta(t-s)}$ and integrating over $(0,t)$, applying Young’s
inequality, we get

$\displaystyle~{}~{}~{}~{}\int_{0}^{t}e^{-\delta(t-s)}\|w(\cdot,s)\|_{L^{\theta^{\prime}}(\Omega)}^{\frac{2}{2-\lambda}}ds$
$\displaystyle\leq c_{1}\int_{0}^{t}e^{-\delta(t-s)}\|\Delta
w(\cdot,s)\|_{L^{\tau}(\Omega)}^{\frac{2}{2-\lambda}b}\|w(\cdot,s)\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}(1-b)}ds+c_{1}\int_{0}^{t}e^{-\delta(t-s)}\|w(\cdot,s)\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}}ds$
$\displaystyle\leq c_{1}\left(\int_{0}^{t}e^{-\delta(t-s)}\|\Delta
w(\cdot,s)\|_{L^{\tau}(\Omega)}^{\frac{2}{2-\lambda}b\beta}ds\right)^{\frac{1}{\beta}}\left(\int_{0}^{t}e^{-\delta(t-s)}\|w(\cdot,s)\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}(1-b)\beta^{\prime}}ds\right)^{\frac{1}{\beta^{\prime}}}$
$\displaystyle\quad\quad+c_{1}\int_{0}^{t}e^{-\delta(t-s)}\|w(\cdot,s)\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}}ds\qquad\mbox{for
all }t\in(0,T_{max}).$ (3.19)

Since $\frac{2}{2-\lambda}(1-b)\beta^{\prime}>1$ due to (3.11), employing
Lemmata 2.2 and 2.4, we find $c_{2}>0$ such that

$\displaystyle~{}~{}~{}~{}\int_{0}^{t}e^{-\delta(t-s)}\|w(\cdot,s)\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}(1-b)\beta^{\prime}}ds$
$\displaystyle\leq\int_{0}^{t}e^{-\delta(t-s)}\left({\int_{0}^{s}e^{-(s-\sigma)}\|u^{\alpha}(\cdot,\sigma)\|_{L^{1}(\Omega)}+\|w_{0}\|_{L^{1}(\Omega)}d\sigma}\right)^{\frac{2}{2-\lambda}(1-b)\beta^{\prime}}ds$
$\displaystyle\leq
c_{2}\int_{0}^{t}e^{-\delta(t-s)}\left(\int_{0}^{s}e^{-(s-\sigma)}\|u^{\alpha}(\cdot,\sigma)\|_{L^{1}(\Omega)}d\sigma\right)^{\frac{2}{2-\lambda}(1-b)\beta^{\prime}}ds+c_{2}\int_{0}^{t}e^{-\delta(t-s)}\|w_{0}\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}(1-b)\beta^{\prime}}ds$
$\displaystyle\leq\frac{c_{2}}{1-\delta}\int_{0}^{t}e^{-\delta(t-s)}\|u^{\alpha}(\cdot,s)\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}(1-b)\beta^{\prime}}ds+\frac{c_{2}}{\delta}\|w_{0}\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}(1-b)\beta^{\prime}}.$
(3.20)

Due to $(x+y)^{\frac{1}{\beta^{\prime}}}\leq
x^{\frac{1}{\beta^{\prime}}}+y^{\frac{1}{\beta^{\prime}}}$ for $x,y\geq 0$
with $\beta^{\prime}>1$, we have

$\displaystyle~{}~{}~{}~{}\left(\int_{0}^{t}e^{-\delta(t-s)}\|w(\cdot,s)\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}(1-b)\beta^{\prime}}ds\right)^{\frac{1}{\beta^{\prime}}}$
$\displaystyle\leq\left(\frac{c_{2}}{1-\delta}\right)^{\frac{1}{\beta^{\prime}}}\left(\int_{0}^{t}e^{-\delta(t-s)}\|u^{\alpha}(\cdot,s)\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}(1-b)\beta^{\prime}}ds\right)^{\frac{1}{\beta^{\prime}}}+\left(\frac{c_{2}}{\delta}\right)^{\frac{1}{\beta^{\prime}}}\|w_{0}\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}(1-b)}.$
(3.21)

Similar to (3), since $\frac{2}{2-\lambda}>1$, Lemma 2.4 is applicable to
yield that

$\displaystyle\int_{0}^{t}e^{-\delta(t-s)}\|w(\cdot,s)\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}}ds\leq\frac{c_{3}}{1-\delta}\int_{0}^{t}e^{-\delta(t-s)}\|u^{\alpha}(\cdot,s)\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}}ds+\frac{c_{3}}{\delta}\|w_{0}\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}}.$
(3.22)

An application of Lemma 2.3 implies $c_{4}>0$ such that

$\displaystyle\int_{0}^{t}e^{-\delta(t-s)}\|\Delta
w(\cdot,s)\|_{L^{\tau}(\Omega)}^{\frac{2}{2-\lambda}b\beta}ds\leq
c_{4}\int_{0}^{t}e^{-\delta(t-s)}\|u^{\alpha}(\cdot,s)\|_{L^{\tau}(\Omega)}^{\frac{2}{2-\lambda}b\beta}ds+c_{4}\|w_{0}\|_{W^{2,\infty}(\Omega)}^{\frac{2}{2-\lambda}b\beta}.$
(3.23)

Substituting (3),(3.22) and (3.23) into (3), the assertion follows. ∎

Combining Lemmata 3.3 and 3.2, we can readily control cross-diffusion by
dissipation of $u$.

######  Lemma 3.4.

Let $\delta\in(0,\min\\{1,\frac{2}{2-\lambda},\frac{2}{2-\lambda}b\beta\\})$.
For any $\varepsilon>0$, there is $C>0$ such that

$\displaystyle\int_{0}^{t}e^{-\delta(t-s)}\|\Delta
v(\cdot,s)\|_{L^{\theta^{\prime}}(\Omega)}^{\frac{2}{2-\lambda}}ds\leq\varepsilon\int_{0}^{t}e^{-\delta(t-s)}\|\nabla
u^{\frac{p}{2}}(\cdot,s)\|_{L^{2}(\Omega)}^{2}ds+C\qquad\mbox{for all
}t\in(0,T_{max}).$ (3.24)

###### Proof.

Thanks to (3.11), it holds that
$\frac{1}{\alpha}<\tau<\frac{np}{\alpha(n-2)_{+}}$ and
$\frac{1}{\alpha}<1<\frac{np}{\alpha(n-2)_{+}}$, also (3.14) ensures
$\frac{n(\frac{2}{2-\lambda}b\beta)(\alpha-\frac{1}{\tau})}{np-n+2}=\beta\eta_{1}<1$
and
$\frac{n\mu(\frac{2}{2-\lambda})(1-b)\beta^{\prime}(\alpha-1)}{np-n+2}=\beta^{\prime}\eta_{2}<1$,
together with the fact that $\eta_{3}<1$ from (3.13), we invoke Lemma 3.2 to
derive

$\displaystyle\int_{0}^{t}e^{-\delta(t-s)}\|u^{\alpha}(\cdot,s)\|_{L^{\tau}(\Omega)}^{\frac{2}{2-\lambda}b\beta}ds\leq
c_{1}\left(\int_{0}^{t}e^{-\delta(t-s)}\|\nabla
u^{\frac{p}{2}}(\cdot,s)\|_{L^{2}(\Omega)}^{2}ds\right)^{\beta\eta_{1}}+c_{1},$
$\displaystyle\int_{0}^{t}e^{-\delta(t-s)}\|u^{\alpha}(\cdot,s)\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}(1-b)\beta^{\prime}}ds\leq
c_{2}\left(\int_{0}^{t}e^{-\delta(t-s)}\|\nabla
u^{\frac{p}{2}}(\cdot,s)\|_{L^{2}(\Omega)}^{2}ds\right)^{\beta^{\prime}\eta_{2}}+c_{2},$
and
$\displaystyle\int_{0}^{t}e^{-\delta(t-s)}\|u^{\alpha}(\cdot,s)\|_{L^{1}(\Omega)}^{\frac{2}{2-\lambda}}{ds}\leq
c_{3}\left(\int_{0}^{t}e^{-\delta(t-s)}\|\nabla
u^{\frac{p}{2}}(\cdot,s)\|_{L^{2}(\Omega)}^{2}ds\right)^{\eta_{3}}+c_{3}.$

Applying Lemmata 2.3 and 3.3, we obtain $c_{4},c_{5}>0$ such that

$\displaystyle~{}~{}~{}~{}\int_{0}^{t}e^{-\delta(t-s)}\|\Delta
v(\cdot,s)\|_{L^{\theta^{\prime}}(\Omega)}^{\frac{2}{2-\lambda}}ds$
$\displaystyle\leq
c_{4}\int_{0}^{t}e^{-\delta(t-s)}\|w(\cdot,s)\|_{L^{\theta^{\prime}}(\Omega)}^{\frac{2}{2-\lambda}}ds+c_{4}\|v_{0}\|_{W^{2,\infty}(\Omega)}^{\frac{2}{2-\lambda}}$
$\displaystyle\leq c_{5}\left(\int_{0}^{t}e^{-\delta(t-s)}\|\nabla
u^{\frac{p}{2}}(\cdot,s)\|_{L^{2}(\Omega)}^{2}ds\right)^{\eta_{1}+\eta_{2}}+c_{5}\left(\int_{0}^{t}e^{-\delta(t-s)}\|\nabla
u^{\frac{p}{2}}(\cdot,s)\|_{L^{2}(\Omega)}^{2}ds\right)^{\eta_{1}}$
$\displaystyle\qquad+c_{5}\left(\int_{0}^{t}e^{-\delta(t-s)}\|\nabla
u^{\frac{p}{2}}(\cdot,s)\|_{L^{2}(\Omega)}^{2}ds\right)^{\eta_{2}}+c_{5}\left(\int_{0}^{t}e^{-\delta(t-s)}\|\nabla
u^{\frac{p}{2}}(\cdot,s)\|_{L^{2}(\Omega)}^{2}ds\right)^{\eta_{3}}+c_{5}.$
(3.25)

Since $0<\eta_{1}+\eta_{2}<1$ and $0<\eta_{i}<1$ for $i=1,2,3$ have been
confirmed in Lemma 3.1, we conclude the assertion by applying Young’s
inequality. ∎

We are now prepared to close the loop by employing Lemma 3.4 together with an
$L^{p}$ testing argument with sufficiently large $p$.

######  Lemma 3.5.

Let $n\leq 3$. Assume that $1<\alpha<\min\\{\frac{4}{n},1+\frac{2}{n}\\}$. For
any

$p>\max\left\\{1,\frac{\alpha(n-2)_{+}}{n},\frac{\alpha\tau(n-2)_{+}}{n},\frac{\frac{2}{n}-\frac{1}{\theta^{\prime}}}{1-b}+1-\frac{2}{n}\right\\},$

there is $C>0$ such that

$\displaystyle\int_{\Omega}u^{p}(\cdot,t)\leq C\qquad\mbox{for all
}t\in(0,T_{max}).$ (3.26)

###### Proof.

We test the first equation in (1.1) with $pu^{p-1}$, integrate by parts,

$\displaystyle\frac{d}{dt}\int_{\Omega}u^{p}+\frac{4(p-1)}{p}\int_{\Omega}|\nabla
u^{\frac{p}{2}}|^{2}=-(p-1)\int_{\Omega}u^{p}\Delta
v\leq(p-1)\left(\int_{\Omega}u^{p\theta}\right)^{\frac{1}{\theta}}\|\Delta
v\|_{L^{\theta^{\prime}}(\Omega)}\qquad\mbox{for all }t\in(0,T_{max}).$ (3.27)

The Gagliardo-Nirenberg inequality yields the existence of $c_{1}>0$ such that

$\displaystyle(p-1)\left(\int_{\Omega}u^{p\theta}\right)^{\frac{1}{\theta}}$
$\displaystyle=(p-1)\|u^{\frac{p}{2}}\|_{L^{2\theta}(\Omega)}^{2}\leq
c_{1}\|\nabla
u^{\frac{p}{2}}\|_{L^{2}(\Omega)}^{2a}\|u^{\frac{p}{2}}\|_{L^{\frac{2}{p}}(\Omega)}^{2(1-a)}+c_{1}\|u^{\frac{p}{2}}\|_{L^{\frac{2}{p}}(\Omega)}^{2}$
$\displaystyle\leq c_{2}\|\nabla
u^{\frac{p}{2}}\|_{L^{2}(\Omega)}^{\lambda}+c_{2}.$

Therefore, Young’s inequality implies existence of $c_{3}>0$ fulfilling

$\displaystyle(p-1)\left(\int_{\Omega}u^{p\theta}\right)^{\frac{1}{\theta}}\|\Delta
v\|_{L^{\theta^{\prime}}(\Omega)}\leq\frac{(p-1)}{p}\int_{\Omega}|\nabla
u^{\frac{p}{2}}|^{2}+c_{3}\|\Delta
v\|_{L^{\theta^{\prime}}(\Omega)}^{\frac{2}{2-\lambda}}+c_{3}.$ (3.28)

Fixing
$\delta\in(0,\min\\{1,\frac{2}{2-\lambda},\frac{2}{2-\lambda}b\beta\\})$, we
derive from the Gagliardo-Nirenberg inequality to see that

$\displaystyle\delta\int_{\Omega}u^{p}\leq\frac{(p-1)}{p}\int_{\Omega}|\nabla
u^{\frac{p}{2}}|^{2}+c_{4}.$ (3.29)

From (3.27)-(3.29), we obtain that

$\displaystyle\frac{d}{dt}\int_{\Omega}u^{p}+\delta\int_{\Omega}u^{p}+\frac{2(p-1)}{p}\int_{\Omega}|\nabla
u^{\frac{p}{2}}|^{2}\leq c_{3}\|\Delta
v\|_{L^{\theta^{\prime}}(\Omega)}^{\frac{2}{2-\lambda}}+c_{4}\qquad\mbox{for
all }t\in(0,T_{max}).$ (3.30)

By applying variation of constants formula and Lemma 2.3 with
$\varepsilon=\frac{p-1}{p}>0$, we derive that

$\displaystyle~{}~{}~{}~{}\int_{\Omega}u^{p}(\cdot,t)+\frac{2(p-1)}{p}\int_{0}^{t}e^{-\delta(t-s)}\|\nabla
u^{\frac{p}{2}}(\cdot,s)\|_{L^{2}(\Omega)}^{2}ds$ $\displaystyle\leq
c_{3}\int_{0}^{t}e^{-\delta(t-s)}\|\Delta
v(\cdot,s)\|_{L^{\theta^{\prime}}(\Omega)}^{\frac{2}{2-\lambda}}ds+\frac{c_{4}}{\delta}+e^{-\delta
t}\int_{\Omega}u_{0}^{p}\qquad\mbox{for all }t\in(0,T_{max}).$

Employing Lemma 3.4 with $\varepsilon=\frac{p-1}{p}\cdot\frac{1}{c_{3}}>0$
leads to the assertion. ∎

###### Proof of Theorem 1.1.

According to Lemma 3.5 and (2.5), if
$\alpha<\min\\{\frac{4}{n},1+\frac{2}{n}\\}$, then

$\displaystyle\sup_{t\in(0,T_{max})}\|f(u(\cdot,t))\|_{L^{1}(\Omega)}<\infty.$
(3.31)

This implies that
$\sup_{t\in(0,T_{max})}\|u(\cdot,t)\|_{L^{\infty}(\Omega)}<\infty$ by well-
established semigroup estimates. Applying Lemma 2.1, we conclude global
existence of the solution and the regularity therein. ∎

## Acknowledgment

The author acknowledges support of the Natural Science Foundation of Shanghai
(Project:23ZR1400100).

## References

  * [1] X. Cao and Y. Tao.  Boundedness and stabilization enforced by mild saturation of taxis in a producer-scrounger model.  Nonlinear Anal. Real World Appl., 57, 103189, 2021\.
  * [2] K. Fujie and T. Senba.  Application of an adams type inequality to a two-chemical substances chemotaxis system.  Journal of Differential Equations, 263(1):88–148, 2017.
  * [3] K.J. Painter and M. Winkler.  Phenotype switching in chemotaxis aggregation models controls the spontaneous emergence of large densities.  SIAM Journal on Applied Mathematics, 83(5), 2096-2117, (2023).
  * [4] E. F. Keller and L. A. Segel.  Model for chemotaxis.  Journal of theoretical biology, 30(2):225–234, 1971.
  * [5] D. Liu and Y. Tao.  Boundedness in a chemotaxis system with nonlinear signal production.  Appl. Math. J. Chinese Univ. Ser. B, 31(4):379–388, 2016.
  * [6] F. R. Macfarlane, T. Lorenzi, and K. J. Painter.  The impact of phenotypic heterogeneity on chemotactic self-organisation.  Bulletin of Mathematical Biology, 84(12):143, 2022.
  * [7] Y. Tao and H. Zhang.  Nonlinear transmission exponent for boundedness of solutions to a chemotaxis system with indirect signal production.  Appl. Math. Lett., 149, 108928, 2024.
  * [8] M. Winkler.  Finite-time blow-up in the higher-dimensional parabolic–parabolic Keller–Segel system.  J. Mathématiques Pures Appliquées, 100(5):748–767, 2013\.
  * [9] M. Winkler.  A critical blow-up exponent in a chemotaxis system with nonlinear signal production.  Nonlinearity, 31(5):2031–2056, 2018.