# Controllable Accented Text-to-Speech Synthesis

Rui Liu, , Berrak Sisman, , Guanglai Gao, Haizhou Li This paper was submitted
on 22 Spe 2022 for review. The research by Rui Liu was funded by the High-
level Talents Introduction Project of Inner Mongolia University (No.
10000-22311201/002) and the Young Scientists Fund of the National Natural
Science Foundation of China (No. 62206136). The work by Haizhou Li was partly
supported by the Agency of Science, Technology and Research, Singapore,
through the National Robotics Program under Grant No. 192 25 00054, and by
Programmatic Grant No. A18A2b0046 from the Singapore Government’s Research,
Innovation and Enterprise 2020 plan (Advanced Manufacturing and Engineering
domain). (Corresponding author: Guanglai Gao)Rui Liu is with the Department of
Computer Science, Inner Mongolia University, Hohhot 010021, China. (e-mail:
liurui_imu@163.com).Berrak Sisman is with the Department of Electrical and
Computer Engineering at University of Texas at Dallas, United States. Guanglai
Gao is with the Department of Computer Science, Inner Mongolia University,
Hohhot 010021, China. (e-mail: csggl@imu.edu.cn)Haizhou Li is with School of
Data Science, The Chinese University of Hong Kong, Shenzhen 518172, China. He
is also with University of Bremen, Faculty 3 Computer Science / Mathematics,
Enrique-Schmidt-Str. 5 Cartesium, 28359 Bremen, Germany (e-mail:
haizhouli@cuhk.edu.cn).

###### Abstract

Accented text-to-speech (TTS) synthesis seeks to generate speech with an
accent (L2) as a variant of the standard version (L1). Accented TTS synthesis
is challenging as L2 is different from L1 in both in terms of phonetic
rendering and prosody pattern. Furthermore, there is no easy solution to the
control of the accent intensity in an utterance. In this work, we propose a
neural TTS architecture, that allows us to control the accent and its
intensity during inference. This is achieved through three novel mechanisms,
1) an accent variance adaptor to model the complex accent variance with three
prosody controlling factors, namely pitch, energy and duration; 2) an accent
intensity modeling strategy to quantify the accent intensity; 3) a consistency
constraint module to encourage the TTS system to render the expected accent
intensity at a fine level. Experiments show that the proposed system attains
superior performance to the baseline models in terms of accent rendering and
intensity control. To our best knowledge, this is the first study of accented
TTS synthesis with explicit intensity control.

###### Index Terms:

Text-to-speech (TTS) synthesis, Controllable, Accent, Accent intensity

##  I Introduction

Accented text-to-speech (TTS) synthesis aims to synthesis speech with foreign
accent instead of native speech. In other words, it seeks to generate speech
with an accent (L2) as a variant of the standard version (L1). Recently, much
progress has been made in the quality of neural TTS synthesis for L1 speech
[1, 2], for example, WaveNet [3], Deep-Voice 1, 2, 3 [4, 5, 6], Char2Wav [7],
Tacotron 1, 2 [8, 9], Transformer-TTS [10] and FastSpeech1 [11]. Neural TTS
has outperformed conventional concatenative and statistical parametric
approaches, where neural vocoders play an important role [12, 13, 14, 15].
Accent is characterized by a distinctive manner of expression that is
influenced by the mother tongue, social group of speakers, or spoken in a
particular region [16]. Generally, people find it easier to speak with others
within their own accent group. Therefore, the wide adoption of speech
applications, such as chatbot and movie dubbing, calls for the study of
accented TTS synthesis.

Phoneme substitution and accent variance are two key aspects in accent
rendering [16, 17]. Phoneme substitution refers to the situation where L2
speakers pronounce L1 phonemes differently [18]. On the other hand, L2
speakers exhibit complex prosody in terms of pitch, energy, and duration [19],
that is referred to as accent variance. This paper is focused on the study of
accent variance.

An L2-accented TTS system is expected to produce human-like natural speech,
just like any TTS system, at the same time to generate speech with an intended
accent at an appropriate intensity [19]. The state-of-the-art neural TTS
systems typically are trained for a standard voice. It remains a challenge how
to effectively control the prosody of accent when rendering speech. We
identify two challenges when dealing with accent variance.

First, inadequate modeling of accent variance often leads to a flat L2 speech
without noticeable accent signature. Note that an L2 speech is highly
expressive particularly in prosody, such as pitch, energy and rhythm. In the
statistical approach to TTS, such as hidden Markov model (HMM), there have
been attempts to address this problem [20, 21]. In the neural TTS approach,
FastSpeech2 [1], FastPitch [22], Meta-StyleSpeech [23], and DAFT-EXPRT [24],
pitch and energy predictors are studied. It remains as a challenge how we
model the accent variance effectively with interpretable prosodic attributes
such as pitch, energy, and duration. To this end, we study a novel neural
approach to model the accent variance.

Second, it is difficult to control the intensity of accent because of the lack
of well-defined intensity descriptors. In the HMM-based TTS framework,
Wutiwiwatchai et al. [25] proposed an accent level adjustment mechanism for
bilingual TTS synthesis, where the accent level is adjusted by means of
interpolation between HMMs of native phones and HMMs of corresponding foreign
phones. In a recent multilingual TTS study [26], the accent level can be
controlled by varying the domain adversarial weight [26]. In another attempt
[27], accent is manipulated via tone or stress embedding input. Accent as
perceived in human speech is subtle and at a fine level. In the prior attempts
to control the accent intensity, there is no direct and measurable correlation
between the controlling factor and the natural accent intensity. The question
is how to characterize the accent intensity, and employ the intensity to
control the synthesis of L2 speech, which is another focus of this paper.

In this paper, we propose a neural TTS solution with controllable accent and
its intensity, that is referred to as the CAI-TTS system. We address the above
two challenges by studying three novel mechanisms. 1) An accent variation
adaptor that models the complex accent variance. The accent variance adaptor
seeks to project the L2 speaker identity (ID), L2 accent identity and its
intensity into the output speech at a phoneme level. 2) An accent intensity
modeling scheme that uses relative attribute [28] technique to automatically
learn the relationship between L1 and L2 speech, thus quantifying the accent
intensity for L2 speech. 3) A consistency constraint module that ensures the
synthesized L2 speech manifests the expected accent intensity precisely.

During run-time inference, the CAI-TTS system takes the L2 speaker ID, L2 ID
and its intensity along with the phoneme sequence as input, and outputs the L2
speech with the expected accent intensity, thereby regulating the accented
speech generation explicitly. The significant contributions of this work
include,

  * •

We introduce a novel accented TTS synthesis paradigm that explicitly controls
the accent and its intensity in output speech.

  * •

We successfully design and implement a neural architecture, CAI-TTS, with
three novel mechanisms.

  * •

We show that the proposed CAI-TTS framework outperforms the baseline models
and generates high-quality L2-accented speech.

The rest of this paper is organized as follows. In Section II, we discuss the
background to motivate our work. In Section III, we propose the CAI-TTS
framework. We report the experimental results in Section IV. Finally, Section
V concludes the study.

##  II Related Work

###  II-A Accent Variance Learning

Recently, much progress was made in neural TTS for L1 speech [8, 9, 10, 11,
29, 30]. However, the state-of-the-art models are not designed to deal with
accent variance. In speech analysis, there has been study on prosodic variance
of speech in terms of pitch and energy [31]. Pitch is a key feature that
characterizes speech variance, and has a great impact on perceptual quality of
speech [22]. Energy indicates frame-level magnitude of mel-spectrum and
directly affects the loss computed on mel-spectrum. There have been valuable
attempts towards learning the variance information. For example, in
FastSpeech2 [1], a variance adaptor is introduced, which has a pitch and an
energy predictor, to learn the speech variance at frame-level. However, L2
speech is far more complex in terms of speaking style and variation than L1
speech, that calls for further study of two research problems.

The first problem is how to represent an accent. The L1 variance adaptor is
not explicitly trained to encode accent information. The second problem is the
characterization of accent variance. It is believed that phoneme-level
acoustic features are more accent informing than frame-level features [32, 33,
22]. There have been studies to model the prosodic variance for phonemes, such
as FastPitch [22]. However, FastPitch doesn’t model the energy. Addressing the
above two problems, we propose a novel accent variance adaptor, which encodes
L2 speaker, L2 accent, and its intensity, and predicts phoneme-level pitch,
energy and duration.

Figure 1: The diagrams for the proposed CAI-TTS system. (a) shows the overall
model architecture; (b) shows the components of an accent variance adaptor,
which includes encoders to encode the controlling factors, and predictors to
generate accented prosodic embeddings and phoneme duration.

###  II-B Attribute Ranking

Attribute ranking learns the difference between two samples that are
significantly different in a particular attribute. It has been widely studied
in computer vision [28, 34, 35, 36].

Parikh et al. [28] first proposed the concept of “relative attributes” in
which pairs of visual entities can be compared, with respect to their relative
strength of any specific attribute. With a set of paired samples, they learn a
global ranking function for each attribute that can be used to compare a pair
of new samples respective to the target attribute. Bringing this idea forward,
we propose to compare the acoustic features from a pair, $<$L1, L2$>$, of
speech samples of the same speaker and determine the difference between them
to quantify the accent intensity.

In emotional TTS synthesis [37, 38], the idea of relative attributes has been
explored. However, accented TTS presents us a new challenge. In emotional TTS,
the $<$neutral, emotional$>$ paired speech is available for emotion strength
learning. However, the reference L1 speech sample is not available for accent
intensity modeling. Learning of ranking functions is not possible without
parallel data. Furthermore, the specific acoustic representations of emotion
and accent attributes are completely different, that calls for a new solution
to the problem.

In this work, we start with a L2 speech corpus, and construct a $<$L1, L2$>$
paired speech corpus by synthesizing high-quality L1 speech for each L2
speaker. We introduce a ranking function learning method to model the accent
intensity from the $<$L1, L2$>$ corpus.

###  II-C Controllable Expressive TTS

There are two general approaches to sythesize expressive speech. 1) Global
speaking style: The speaking style can be obtained from a reference utterance,
or a manual setting. Wang et al. [39] adopt a reference encoder to derive a
style embedding from an input utterance as a linear combination of some basis
style vectors. Others [40, 26, 41, 42] use the variational autoencoder (VAE)
to model latent representations for styles and prosody of speech. 2) Token-
wise fine-grained prosody embeddings: In FastSpeech2 [1], a variance adaptor
is studied to add the variance information (e.g., duration, pitch, energy,
etc.) to the phoneme hidden sequence in a way to generate expressive speech.
Furthermore, more variance information has been added in the variance adaptor,
such as emotion [43, 44, 45] and style [24, 46, 47] to control the fine-
grained expressiveness accordingly.

Strictly speaking, the above techniques don’t really control the accent
rendering. At run-time inference, the system generates the output speech by
conditioning on the style embedding or the variance model. In this way, the
speech is synthesized to follow a given style or prosody pattern. There is a
lack of direct control over the synthesized speech, and there is no measurable
correlation between the controlling factor and the intended accent intensity.

To the best of our knowledge, there has no intuitive method for controlling
the fine-grained accent in the literature. Motivated by the previous findings,
we propose a novel neural architecture that quantitatively controls the accent
and its intensity for the first time.

##  III CAI-TTS Neural Architecture

We propose a neural architecture, termed as CAI-TTS, as shown in Fig. 1 (a)
that consists of a text encoder, an accent variance adaptor, a mel-spectrum
decoder, an additional accent intensity predictor and an universal HiFi-GAN
vocoder. The text encoder encodes the input phoneme sequence into phoneme
embedding. The accent variance adaptor modulates the input phoneme embeddings
towards the target accent. The mel-spectrum decoder converts the modulated
phoneme embeddings into a mel-spectrum sequence. Note that the accent
intensity of the output L2 speech is estimated from the generated mel-spectrum
sequence by the additional accent intensity predictor during CAI-TTS training.
We impose a consistency constraint loss to minimize the difference between the
estimated accent intensity and the expected intensity. Finally, the universal
HiFi-GAN vocoder was used to synthesize high-quality L2 speech.

The text encoder and mel-spectrum decoder share a similar architecture with
FastSpeech2 [1]. We use the feed-forward Transformer block, which is a stack
of self-attention [48] layer and 1D-convolution, as the basic structure. We
denote the text encoder as $E_{t}(\cdot)$, which produces the phoneme hidden
representation $\mathcal{H}_{ph}=E_{t}(\mathcal{P}+P\\!E)$, where
$\mathcal{P}$ is the phoneme sequence, and $P\\!E$ is a triangle positional
embedding [48] to indicate the positional information.

In Sec. III-A, we will introduce the details of the accent variance adaptor
and the consistency constraint for accent intensity predictor. Note that the
most important input of accent variance adaptor, that is accent intensity, is
obtained through the pre-training step, termed as “accent intensity modeling”.
Therefore, in Sec. III-A, we will demonstrate the workflow of the accent
intensity modeling on the available L2 speech corpus. At last, the run-time
inference will be explained.

###  III-A Accent Variance Adaptor

The traditional variance adaptor in [1] just adds different variance
information such as duration, pitch and energy into the hidden sequence to
predict the mel-spectrogram features, that lacks a controlling mechanism. Next
we introduce an accent variance adaptor to provide accent information
according to the accent intensity. Specifically, we fuse the L2 speaker
identity, accent identity and its accent intensity before the prediction of
pitch, energy and duration. In other words, the accent variance adaptor learns
to project the desired accent and its intensity into the input phoneme
embedding $\mathcal{H}_{ph}$. This enables all variables, namely the pitch,
energy, and duration, to be regulated by the fine-grained accent intensity.
Note that the duration predictor is located at the end of the accented
variance adaptor instead of the beginning of the variance adaptor in the
original FastSpeech2 [1]. It was reported that it is more effective to render
accent at phoneme-level than frame-level [22].

As shown in Fig. 1(b), the adaptor consists of 1) a L2 speaker encoder, 2) a
L2 accent encoder, 3) an intensity encoder, 4) a phoneme pitch predictor, 5) a
phoneme energy predictor, and 6) a phoneme duration predictor.

  * •

L2 Speaker Encoder: The L2 speaker encoder is a learnable lookup table, that
encodes the L2 speaker identity $\mathbf{s}$ into a speaker code,
$\mathbf{e}_{\mathbf{s}}=E_{spk}(\mathbf{s})$ [5, 6], to represent a speaker.

  * •

L2 Accent Encoder: Similarly, the L2 accent encoder $E_{acc}(\cdot)$ is
another learnable lookup table that transforms an L2 accent category
$\mathbf{a}$ into an accent embedding,
$\mathbf{e}_{\mathbf{a}}=E_{acc}(\mathbf{a})$.

  * •

Intensity Encoder: The intensity encoder $E_{int}(\cdot)$ is a linear layer
that transforms a real-valued accent intensity scalar ${i}$ to an embedding
vector, $\mathbf{e}_{{i}}=E_{int}({i})$. The real-value scalar of accent
intensity is generated by a novel pre-training module, named “accent intensity
modeling”, which will be described in Sec. III-C.

As shown in Fig. 1(b), we concatenate [44] the accent embedding
$\mathbf{e}_{\mathbf{a}}$ and its intensity $\mathbf{e}_{{i}}$, which is then
summed with the speaker code $\mathbf{e}_{{s}}$ and the $\mathcal{H}_{ph}$ to
form an accented phoneme embedding $\mathcal{H}^{\prime}_{ph}$.

  * •

Variance Predictors: With the accented phoneme embeddings, we would like to
predict three key prosodic elements that characterize accents, i.e. pitch,
energy and duration of phonemes. The phoneme pitch and energy predictors take
$\mathcal{H}^{\prime}_{ph}$ as input and generate the pitch and energy
embedding $\mathbf{p}$, $\mathbf{e}$. The two predictors share a similar
architecture, that has a 2-layer 1D-convolutional network, a fully connected
(FC) layer and an extra 1D-convolutional network.

To train the predictors, we first extract the frame-level pitch from the
voiced speech frame, and set it to 0 for unvoiced frames. We compute the
L2-norm of the amplitude of each short-time Fourier transform (STFT) frame as
the energy. We obtain frame-level pitch and energy averages for each phoneme
using the ground-truth duration $\mathcal{D}$. The phoneme pitch and energy
scalars are normalized to zero mean and unit variance over all training data,
to serve as the supervision signals. We define mean square error loss
$\mathcal{L}_{p\\_pitch}$ and $\mathcal{L}_{p\\_energy}$ behind the FC layers
as the objective functions for the training of the pitch and energy
predictors.

Unlike in FastSpeech2 [1] where the phoneme duration predictor takes the
encoder output directly as input, our phoneme duration predictor takes the
accented phoneme embedding $\mathcal{H}^{{}^{\prime}}_{ph}$ as input and is
expected to output more accurate phoneme duration $\mathcal{D}$. At last, we
sum the accented phoneme, pitch and energy embeddings to form an accented
phoneme embedding
$\mathcal{H}^{{}^{\prime\prime}}_{ph}=\mathcal{H}^{{}^{\prime}}_{ph}+\mathbf{p}+\mathbf{e}$.
A length regulator (LR) is used to transform the
$\mathcal{H}^{{}^{\prime\prime}}_{ph}$ to frame-level embeddings
$\mathcal{H}_{fm}$ based on the predicted phoneme duration $\mathcal{D}$.
Finally, the mel-spectrum decoder converts the $\mathcal{H}_{fm}$, along with
a triangle positional embedding $P\\!E$ [48], into a mel-spectrum sequence.

###  III-B Consistency Constraint for Accent Intensity

The consistency constraint module ensures the synthesized L2-accented speech
manifests the expected accent intensity at a fine level, as shown in Fig.
1(a). The consistency constraint is applied via a consistency constraint loss
$\mathcal{L}_{cc}$ between the intended accent intensity $i$ and the accent
intensity $\hat{i}$ measured by the accent intensity predictor, i.e.,
$\mathcal{L}_{cc}={\rm MSE}(i,\hat{i})$.

The accent intensity predictor consists of a bi-directional recurrent neural
network layer with Gated Recurrent Unit (GRU), that is followed by an FC
layer. It is trained in a supervised manner together with the CAI-TTS model
subject to the total loss,
$\mathcal{L}_{final}=\mathcal{L}_{mel}+\mathcal{L}_{dur}+\mathcal{L}_{p\\_pitch}+\mathcal{L}_{p\\_energy}+\mathcal{L}_{cc}$,
where $\mathcal{L}_{mel}$ and $\mathcal{L}_{dur}$ are the MSE loss for mel-
spectrum and duration loss as in FastSpeech2 [1].

Figure 2: The workflow of the accent intensity modeling. The accented TTS
corpus includes $<$Text, L2 Speech$>$ pairs.

###  III-C Accent Intensity Modeling

The key to accent control at a fine level is to find a descriptor to quantify
the accent intensity. To this end, we propose a novel pre-training strategy,
named “accent intensity modeling” scheme, that seeks to quantify the accent
intensity for each sample of L2-accented speech before CAI-TTS training, as
illustrated in Fig. 2. The accent intensity modeling scheme consists of L1
speech generation and ranking function learning. The ranking function learning
includes accent feature extraction and ranking function training.

Note that L1 speech generation aims to develop a $<$L1, L2$>$ paired speech
dataset, that is used for the ranking function learning. Then the ranking
function learning will learn the difference between L1 and L2 speech pair to
quantify the accent intensity for L2 speech. We will introduce the workflow in
order.

####  III-C1 L1 Speech Generation

We first create a $<$L1, L2$>$ parallel speech database of the same speech
content, where an L1 speech sample is synthesized for each L2 utterance of the
same speaker by SC-GlowTTS [49], a state-of-the-art zero-shot TTS model as in
Fig. 3. The high-quality L1 speech samples serve as the reference for the
ranking function learning.

As in [49], the SC-GlowTTS model employs an external speaker extractor, that
encodes an utterance into a speaker embedding. We adopt the flow-based non-
autoregressive model to predict the mel-spectrum sequence. Finally, a HiFi-GAN
[14] vocoder converts the output mel-spectrum sequence to speech waveform.

In practice, for a text-speech pair in the accented TTS corpus, we first
obtain a phoneme-speech pair $\\{\mathcal{P},Y\\}$ where
$\mathcal{P}=\\{\mathcal{P}_{1},\mathcal{P}_{2},...,\mathcal{P}_{T}\\}$
represents a phoneme sequence of $T$ tokens, while
$Y=\\{y_{1},y_{2},...,y_{N}\\}$ represents the L2 speech of $N$ samples. The
text encoder takes the phoneme sequence $\mathcal{P}$ as input, while the
speaker extractor encodes a L2 speech utterance into a speaker embedding. In
this way, the system generates L1 speech in L2 speaker’s voice.

Figure 3: L1 speech generation using SC-GlowTTS.

####  III-C2 Ranking Function Learning

In ranking function learning, we first extract accent acoustic features
$<\\!\\!f^{L1},f^{L2}\\!\\!>$ from a pair of speech samples $<$L1, L2$>$, then
train the ranking function, following the attribute ranking idea as described
in Sec. II-B.

2.1) Accent Feature Extraction: Studies show that pitch and energy [50]
related features are the descriptors of accents. We use opensmile [31] to
extract 36-dimensional features $f^{L1}$ and $f^{L2}$ for each utterance, each
of which contains both pitch and energy features.

2.2) Ranking Function Training: Motivated by [28], we would like to learn a
ranking function $R(\cdot)$ from the paired feature sequence
$<\\!\\!f^{L1},f^{L2}\\!\\!>$ by using relative attribute ranking for the
accent attribute,

$i=R(f^{L2}_{m})=\textbf{w}f^{L2}_{m}$ (1)

where we weight $f^{L2}_{m}$ with a learnable weighting vector w and return a
weighted sum $i$ indicating the accent intensity of sample $m$.

To achieve this, we first build two sets, that are $O$ and $S$, which contains
ordered and similar paired samples respectively, according to the following
relative constraints:

$\left\\{\begin{matrix}\forall(m,h)\in
O:\textbf{w}\mathcal{F}_{m}>\textbf{w}\mathcal{F}_{h}\\\ \forall(m,h)\in
S:\textbf{w}\mathcal{F}_{m}=\textbf{w}\mathcal{F}_{h}\end{matrix}\right.$ (2)

where $\mathcal{F}$ represents a sample from the union of $f^{L2}$ and
$f^{L2}$: $\mathcal{F}\in\\{f^{L1}\cup f^{L2}\\}$.

Specifically, we pick up one sample $\mathcal{F}_{m}$ from $f^{L1}$ and
another sample $\mathcal{F}_{h}$ from $f^{L2}$ to build the ordered set $O$.
We expect that the accent intensity of the L2-accented sample
$\mathcal{F}_{h}$ is higher than that of the L1 sample $\mathcal{F}_{m}$. For
the similar set $S$, we pick up two samples $\mathcal{F}_{m}$ and
$\mathcal{F}_{h}$ from $f^{L1}$ (or $f^{L2}$). We assume that two samples from
the same domain (L1 or L2) have similar accent intensities.

Next, we can learn a support vector machines (SVM) [51] to estimate w by
solving the following optimization problem:

${\rm
minimize}\left(\frac{1}{2}||\textbf{w}||^{2}_{2}+C\left(\sum{\xi^{2}_{ij}}+\sum{\eta^{2}_{ij}}\right)\right)$
(3)

All parameters need to satisfy the following constraints:

$\left\\{\begin{matrix}w(\mathcal{F}_{m}-\mathcal{F}_{h})\geqslant
1-\xi_{mh};\forall(m,h)\in O\\\
|w(\mathcal{F}_{m}-\mathcal{F}_{h})|\leqslant\eta_{mh};\forall(m,h)\in S\\\
\xi_{mh}\geqslant 0;\eta_{mh}\geqslant 0\end{matrix}\right.$ (4)

where $C$ is to control the trade-off between the margin and the size of the
slack variables $\xi_{mh}$ and $\eta_{mh}$. The use of $\xi_{mh}$ and
$\eta_{mh}$ is to relax the constraints on classifying all of the ordered and
similar samples in $O$ and $S$, respectively. This primal problem can be
solved by the Newton’s method [52]. Note that the intensity scalar $i$ is
normalized to (0,1) with 1 as the highest intensity.

###  III-D Run-time Inference

At run-time, the CAI-TTS architecture takes Input text, L2 speaker ID, L2
accent ID and the custom accent intensity as input to generate the output L2
speech with the help of universal HiFi-GAN vocoder [14] in this paper.

Note that the predicted pitch contour and energy also can be modified during
inference to control certain perceived qualities of the generated speech like
FastPitch [22] etc. However, synthesized speech with a large pitch (or energy)
shift scale suffers from audio quality degradation, and speaker
characteristics deformation [53]. In our work, we just try to modify the L2
speaker ID, accent ID and accent intensity to achieve accented and intensity
controllable TTS synthesis.

##  IV Experiments

We evaluate CAI-TTS through accented text-to-speech synthesis experiments
111Audio samples: https://speechdemo.github.io/caitts.

###  IV-A Datasets

We train CAI-TTS on the publicly available L2-ARCTIC corpus [54], which
includes about 26 hours recordings of accented English from 24 non-native
speakers, whose are native in Hindi, Korean, Mandarin, Spanish, Arabic and
Vietnamese. Two male and two female speakers contributed in each language.

In L2-ARCTIC, scripts and their phoneme-level duration annotations are
provided. The scripts for each speaker consists of about 1,130 utterances,
resulting in about 27,120 utterances in total. Its phonetic transcription
follows the ARPAbet phoneme set 222http://www.speech.cs.cmu.edu/cgi-
bin/cmudict. The speech data are sampled at 44.10kHz and coded in 16 bits. For
each accented speaker, we partition the speech data into training, validation,
and test set at a ratio of 8:1:1.

###  IV-B Comparative Study

This work is one of the first attempts to control multiple accents and their
intensity at a fine level. As there is no reference system in the literature,
we choose some state-of-the-art TTS systems as the benchmark. 1) GT (oracle):
This is the natural L2 speech by the speakers; 2) GT mel + HiFi-GAN (oracle):
This is the synthesized speech with a HiFi-GAN vocoder using GT mel-spectrum;
3) Tacotron2: This is a multi-speaker Tacotron2 [9] model which learns a look-
up-table to map embeddings for different speaker identity; 4) Transformer-TTS:
This is a multi-speaker Transformer-TTS [55] model which learns a look-up-
table to map embeddings for different speaker identity; 5) FastSpeech2: This
is a multi-speaker FastSpeech2 [1] model which learns a look-up-table to map
embeddings for different speaker identity; 6) CAI-TTS (proposed): This is our
proposed model. We also develop 3 variants of CAI-TTS model for an ablation
study. 7) CAI-TTS w/o accent intensity: We replace the accent variance adaptor
with the variance adaptor of FastSpeech2 [1]; 8) CAI-TTS w/o phoneme pitch &
energy: We replace the phoneme level pitch and energy predictors with frame-
level pitch and energy predictors as in [1]; 9) CAI-TTS w/o consistency
constraint: We remove the consistency constraint module from the CAI-TTS
model.

###  IV-C Experimental Setup

The text encoder and mel-spectrum decoder use 6 Feed-Forward Transformer (FFT)
blocks. Note that the text encoder takes the 256 dimensional phoneme embedding
as input. The mel-spectrum decoder generates 80-channel mel-spectrum, which is
extracted with 12.5ms frame shift and 50ms frame length, as output. We
downsample all speech files to 22.05kHz and trimmed leading and trailing
silence.

The L2 speaker and accent encoders employ two lookup tables with 14$\times$256
and 6$\times$128 respectively. The intensity encoder consists of a linear
layer, which encodes the accent intensity scalar into a 128 dimensional
intensity embedding.

Following [22], we represent the pitch and energy scalar in linear scale. The
pitch and energy predictors employ a Conv1D with kernel size 3 and 384/256
channels, Conv1D with 256/256 channels, and an FC layer to project a
256-channel vector into a single pitch/energy scalar, and the last Conv1D
layer with kernel size 9 to upsamples pitch/energy scalar to pitch/energy
embeddings. We use Dropout rate of 0.5 in every Dropout layer. The accent
intensity predictor in consistency constraint module consists of a GRU layer
with hidden size 128 and an FC layer to output the intensity scalar.

We use the Adam optimizer [56] with $\beta_{1}$ = 0.9, $\beta_{2}$ = 0.98 and
follow the same learning rate schedule in [48]. All models are trained with
900k steps to ensure complete convergence. The codes are written in Python 3.6
using the Pytorch library 1.7.0. The GPU type is NVIDIA Quadro RTX 6000 with
24GB GPU memory. For a fair comparison, we adopt the same pretrained universal
HiFI-GAN [14] vocoder for all systems. As the universal HiFI-GAN vocoder is
trained on a collection of multiple dataset, that includes LJSpeech [57], VCTK
[58] and LibriTTS [59], it is known to produce high-quality voice for unseen
speakers [60, 61].

TABLE I: The comparison of the audio quality for different systems in terms of MCD in objective experiments, MOS and BWS in subjective experiments.  System | Audio Quality
---|---
MCD (dB) |  MOS | BWS Evaluation
|  |  Best (%) |  Worst (%)
GT |  NA |  4.62 $\pm$ 0.02 |  NA |  NA
GT (Mel + HiFi-GAN) |  NA |  4.59 $\pm$ 0.03 |  NA |  NA
Tacotron2 [9] (Mel + HiFi-GAN) |  6.57 |  4.33 $\pm$ 0.01 |  6 |  34
Transformer TTS [10] (Mel + HiFi-GAN) |  6.43 |  4.35 $\pm$ 0.02 |  8 |  22
FastSpeech2 [11] (Mel + HiFi-GAN) |  6.31 |  4.38 $\pm$ 0.03 |  5 |  18
CAI-TTS (proposed) (Mel + HiFi-GAN) |  6.08 |  4.57 $\pm$ 0.01 |  45 |  4
w/o phoneme pitch & energy |  6.11 |  4.52 $\pm$ 0.02 |  9 |  8
w/o accent intensity |  6.14 |  4.48 $\pm$ 0.03 |  13 |  6
w/o consistency constraint |  6.16 |  4.43 $\pm$ 0.01 |  14 |  8
TABLE II: The comparison of the accent variance information for different systems, including standard deviation ($\sigma$), skewness ($\gamma$), kurtosis ($\mathcal{K}$) and average dynamic time warping (DTW) distances ($\varrho$) for pitch, mean absolute error (MAE) for energy and the average of absolute boundary differences ($\Delta$) for duration.  System | Accent Variance
---|---
Pitch |  Energy |  Duration
$\sigma$ |  $\gamma$ |  $\mathcal{K}$ |  $\varrho$ |  MAE |  $\Delta$ (ms)
GT |  49.8 |  0.627 |  0.854 |  NA |  NA |  NA
GT (Mel + HiFi-GAN) |  49.5 |  0.629 |  0.852 |  15.27 |  0.192 |  NA
Tacotron2 [9] (Mel + HiFi-GAN) |  32.5 |  0.906 |  1.212 |  19.80 |  0.297 |  22.12
Transformer TTS [10] (Mel + HiFi-GAN) |  35.8 |  0.887 |  1.025 |  19.32 |  0.288 |  21.98
FastSpeech2 [11] (Mel + HiFi-GAN) |  43.2 |  0.752 |  0.973 |  18.01 |  0.274 |  20.08
CAI-TTS (proposed) (Mel + HiFi-GAN) |  48.9 |  0.613 |  0.882 |  16.78 |  0.235 |  18.54
w/o phoneme pitch & energy |  48.3 |  0.610 |  0.896 |  17.06 |  0.240 |  18.72
w/o accent intensity |  47.9 |  0.611 |  0.890 |  16.95 |  0.239 |  18.70
w/o consistency constraint |  47.8 |  0.594 |  0.889 |  17.13 |  0.245 |  18.85

###  IV-D Audio Quality of Generated L1 Speech

The accent intensity modeling cannot be done well without L1 speech with high
audio quality. Therefore, we first conduct a Mean Opinion Score (MOS) and a
Speaker Similarity MOS (SMOS) evaluation to validate the overall performance
of the L1 speech generated by zero-shot TTS in terms of audio quality and
speaker similarity.

Specifically, we conduct listening experiments with mean opinion score (MOS)
for audio quality and similarity MOS (SMOS) [62] for speaker similarity. Both
metrics are rated in 1-to-5 scale and reported with 95% confidence intervals
(CI).

We first conduct MOS listening experiments to compare the overall audio
quality between L2 speech and synthesized L1 speech. 100 $<$L1, L2$>$ pairwise
sentences were chosen as the test samples randomly. We invite 20 subjects to
rate the audio quality for each sample. Each listener listened to 200 speech
samples. The MOS values are calculated by taking the arithmetic average of all
scores provided by the subjects. To this end, the MOS score of L2 speech
reaches 4.57 $\pm$ 0.03, and the MOS score of synthesized L1 speech achieves
4.32 $\pm$ 0.02. Note that both MOS scores are higher than 4.0, indicating
that the quality of synthesized L1 speech is good enough to serve as the
reference speech for accent intensity modeling.

We also evaluate the speaker similarity between L1 and L2 speech. We follow
the MOS experiment settings, except that we randomly choose 100 samples as
reference samples which share same speaker identity with the test samples. 20
listeners are invited to listen to the reference samples and test samples, and
rate the speaker similarity for each test sample. After that, we calculate the
average SMOS according to all scores. Note that the SMOS scores of L1 and L2
speeches got 4.25 $\pm$ 0.03 and 4.49 $\pm$ 0.01 respectively, which are very
similar to the trending of MOS scores. We believe that the synthesized L1
speech generated by zero-shot TTS performs enough speaker similarity.

In a nutshell, the synthesized L1 speech achieves comparable scores with L2
speech on both experiments in terms of audio quality and speaker similarity,
which shows the effectiveness of the zero-shot TTS model for L1 speech
generation. In other words, we observe that the quality of the synthesized L1
speech is on par with the L2-accented speech in both experiments. Therefore,
we confirm that the $<$L1, L2$>$ paired speech samples are adequate for accent
intensive learning study.

###  IV-E Audio Quality of CAI-TTS

In this section, we conduct objective and subjection evaluations to validate
the overall audio quality of our CAI-TTS model in terms of accented
expression.

In the objective evaluation, we first employ Mel Cepstral Distortion (MCD) to
measure the spectral distance between the synthesized and reference Mel-
spectrum features. Since the sequences are not aligned, we perform Dynamic
Time Warping (DTW) algorithm [63] to align the sequences prior to comparison.
Lower MCD value indicates smaller distortion, thus better quality. We randomly
select 100 utterances from the test set as test samples and report the results
in the second column of Table I. We observe that the CAI-TTS outperforms all
other systems with the lowest MCD of 6.08 dB. In the ablation study, we
observe that the three ablation systems see a performance degradation to some
extent, which reaffirms the effectiveness of the individual prosody-
controlling modules.

In the subjective experiment, we first evaluate all systems in terms of MOS.
We choose 100 utterance as the test samples. 20 listeners were participated
and each listener listens to 100 speech samples. As shown in the third column
of Table I, it is observed that our CAI-TTS achieves a MOS of 4.57 $\pm$ 0.01,
that is significantly higher than others and very close to those of the GT
(Mel+HiFi-GAN) and GT.

We conduct the second listening experiment through Best Worst Scaling (BWS)
[64], which is an effective method to provide a ranking of a long list of
listening samples. BWS evaluation share same test samples with previous MOS
evaluation. For each utterance, seven speech samples produced by these seven
TTS systems (expect GT(Mel+HiFi-GAN) and GT)) form a group. 20 listeners were
participated and each listener picks the best and worst samples in terms of
naturalness for each group. We report the results in the last two columns of
Table I. We observe that CAI-TTS is selected for 45% of time as the best model
and 4% as the worst model. All baselines get lower best percentage and higher
worst percentage. The results suggest that the listeners have a clear
preference towards our proposed CAI-TTS system in terms of accent similarity.

###  IV-F Accent Variance Learning

To understand how the accent variance adaptor performs, we randomly select 100
utterances from the test set as the test samples.

Pitch: We follow [1] and compute the moments (including standard deviation
($\sigma$), skewness ($\gamma$) and kurtosis ($\mathcal{K}$)), and average DTW
distance ($\varrho$) of the pitch distribution between the synthesized
L2-accented speech and the ground truth reference. The results are summarized
in the second to fifth columns of Table II. It can be seen that the CAI-TTS
system is reported with the moments ($\sigma$, $\gamma$ and $\mathcal{K}$)
that are closer to those of the natural speech (GT) than Tacotron2,
Transformer TTS and FastSpeech2. In terms of the average DTW distance
$\varrho$ to the ground truth pitch, the CAI-TTS system outperforms all
baselines, with the lowest value of 16.78.

Energy: We also compute the accuracy by calculating the mean absolute error
(MAE) between the frame-level energy extracted from the synthesized and the
ground-truth L2-accented speech. The DTW algorithm is applied to align the
paired sequences. As shown in the sixth column of Table II, the CAI-TTS system
presents the lowest MAE among all benchmarking baselines.

Duration: We also evaluate the quality of phoneme duration by calculating the
distance [1] between the predicted duration and the ground-truth duration at a
phoneme level. Note that the accented TTS corpus have provided the phoneme
level duration. For speech generated by Tacotron2 and Transformer TTS, we
extract the phoneme duration from the trained attention alignment. For speech
generated by FastSpeech2 and CAI-TTS, we obtain the duration from duration
predictor output directly. The seventh column of Table II shows that CAI-TTS
generates more accurate phoneme duration than all benchmarking baselines.

Figure 4: Confusion matrices between perceived and intended accent intensity
categories of synthesized speech. (a) CAI-TTS; b) CAI-TTS w/o consistency
constraint. The X-axis and Y-axis of the figures represent the perceived and
intended category, namely slight, average, and strong.

Figure 5: Confusion matrices of synthesized speech from the a) CAI-TTS and b)
CAI-TTS w/o consistency constraint. The X-axis and Y-axis of subfigures
represent perceived and truth accent intensity scalar (ie. 0.1 to 0.9),
respectively.

###  IV-G Controllable Accent Intensity

We further evaluate the ability of CAI-TTS to adjust the L2 accent intensity
of the synthesized accented speech by comparing CAI-TTS with the CAI-TTS w/o
consistency constraint.

We first conduct an intensity classification experiment. At run-time, we
assign the L2 speaker ID, accent ID and its intensity from 0.1 to 0.9 to
synthesize the L2-accented speech with various accents. We consider the
intensity scalars from 0.1 to 0.3 as ‘slight’, 0.4 to 0.6 as ‘average’ and 0.7
to 0.9 as ‘strong’ in three categories. We select 100 utterances from the test
set, resulting in 100 samples for both systems. Accordingly, all listeners are
instructed to rate the accent intensity category, that are ‘slight’, ‘average’
or ‘strong’, for each sample. A listener can listen to the samples multiple
times when needed.

Fig. 4 presents the intensity confusion matrices. It is observed that the CAI-
TTS system shows a higher correlation between the perceived and intended
accent intensity categories, with a correlation of over 80%, that is
considered a competitive result against other intensity-controlled studies.
Furthermore, the CAI-TTS system clearly outperforms the contestant. The
experiments confirm the superiority of the proposed controllable intensity
mechanism.

We further evaluate the intensity-controlled speech at a fine level. For each
utterance, nine speech samples with intensity from 0.1 to 0.9 produced by CAI-
TTS and CAI-TTS w/o consistency constraint form two groups. We invite 20
listeners, each listening to all samples in an increasing order of accent
intensity. The confusion matrices are reported in Fig. 5. We observe that the
proposed CAI-TTS system provides a higher correlation between the perceived
and the intended fine categories than the contestant. All subjective
evaluations show consistently results that the CAI-TTS system provides an
effective fine level intensity control.

##  V Conclusion

We have studied a novel TTS model, named CAI-TTS, to control the L2 accent and
its intensity during the speech generation. We have conducted a series of
experiments on audio quality, accent variance and intensity control to
validate the effectiveness of the CAI-TTS model. The proposed CAI-TTS
consistently outperforms all baselines in terms of accent rendering and the
control of its intensity. This work marks an important step towards
controllable rendering of accented TTS synthesis. For future work, we plan to
extend CAI-TTS to support fine-grained (eg. phoneme level) accent control.

## References

  * [1] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fastspeech 2: Fast and high-quality end-to-end text to speech,” in _International Conference on Learning Representations_ , 2020.
  * [2] R. Liu, B. Sisman, G. Gao, and H. Li, “Expressive tts training with frame and style reconstruction loss,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 1806–1818, 2021.
  * [3] A. Van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” in _9th ISCA Speech Synthesis Workshop_ , 2016, pp. 125–125.
  * [4] S. Ö. Arık, M. Chrzanowski, A. Coates, G. Diamos, A. Gibiansky, Y. Kang, X. Li, J. Miller, A. Ng, J. Raiman _et al._ , “Deep voice: Real-time neural text-to-speech,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 195–204.
  * [5] A. Gibiansky, S. Arik, G. Diamos, J. Miller, K. Peng, W. Ping, J. Raiman, and Y. Zhou, “Deep voice 2: Multi-speaker neural text-to-speech,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [6] W. Ping, K. Peng, A. Gibiansky, S. O. Arik, A. Kannan, S. Narang, J. Raiman, and J. Miller, “Deep voice 3: Scaling text-to-speech with convolutional sequence learning,” in _International Conference on Learning Representations_ , 2018.
  * [7] J. Sotelo, S. Mehri, K. Kumar, J. F. Santos, K. Kastner, A. C. Courville, and Y. Bengio, “Char2wav: End-to-end speech synthesis,” in _ICLR (Workshop)_ , 2017.
  * [8] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio _et al._ , “Tacotron: A fully end-to-end text-to-speech synthesis model,” in _Proc. Interspeech 2017_ , 2017, pp. 4006–4010.
  * [9] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan _et al._ , “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,” in _ICASSP 2018 - 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 4779–4783.
  * [10] N. Li, S. Liu, Y. Liu, S. Zhao, and M. Liu, “Neural speech synthesis with transformer network,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, no. 01, 2019, pp. 6706–6713.
  * [11] Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fastspeech: fast, robust and controllable text to speech,” in _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , 2019, pp. 3171–3180.
  * [12] R. Yamamoto, E. Song, and J.-M. Kim, “Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 6199–6203.
  * [13] Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, “Diffwave: A versatile diffusion model for audio synthesis,” in _International Conference on Learning Representations_ , 2020.
  * [14] J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” _Advances in Neural Information Processing Systems_ , vol. 33, 2020.
  * [15] W. Ping, K. Peng, K. Zhao, and Z. Song, “Waveflow: A compact flow-based model for raw audio,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 7706–7716.
  * [16] L. Loots and T. Niesler, “Automatic conversion between pronunciations of different english accents,” _Speech Communication_ , vol. 53, no. 1, pp. 75–84, 2011.
  * [17] A. Deri and K. Knight, “Grapheme-to-phoneme models for (almost) any language,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2016, pp. 399–408.
  * [18] B. Kolluru, V. Wan, J. Latorre, K. Yanagisawa, and M. Gales, “Generating multiple-accent pronunciations for tts using joint sequence model interpolation,” in _Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH_ , 2014, pp. 1273–1277.
  * [19] L. M. Tomokiyo, A. W. Black, and K. A. Lenzo, “Foreign accents in synthetic speech: development and evaluation,” in _INTERSPEECH 2005 - Eurospeech, 9th European Conference on Speech Communication and Technology, Lisbon, Portugal, September 4-8, 2005_. ISCA, 2005, pp. 1469–1472.
  * [20] P. Malatji, M. Manamela, T. Sefara, T. Sefara, and P. Malatji, “Creating accented text-to-speech english voices to facilitate second language learning,” in _South Africa International Conference on Educational Technologies_ , 2016.
  * [21] T. J. Sefara, T. B. Mokgonyane, M. J. Manamela, and T. I. Modipa, “Hmm-based speech synthesis system incorporated with language identification for low-resourced languages,” in _2019 International Conference on Advances in Big Data, Computing and Data Communication Systems (icABCD)_. IEEE, 2019, pp. 1–6.
  * [22] A. Łańcucki, “Fastpitch: Parallel text-to-speech with pitch prediction,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6588–6592.
  * [23] D. Min, D. B. Lee, E. Yang, and S. J. Hwang, “Meta-stylespeech: Multi-speaker adaptive text-to-speech generation,” _arXiv preprint arXiv:2106.03153_ , 2021\.
  * [24] J. Zaïdi, H. Seuté, B. van Niekerk, and M.-A. Carbonneau, “Daft-exprt: Robust prosody transfer across speakers for expressive speech synthesis,” _arXiv preprint arXiv:2108.02271_ , 2021.
  * [25] C. Wutiwiwatchai, A. Thangthai, A. Chotimongkol, C. Hansakunbuntheung, and N. Thatphithakkul, “Accent level adjustment in bilingual thai-english text-to-speech synthesis,” in _2011 IEEE Workshop on Automatic Speech Recognition & Understanding_. IEEE, 2011, pp. 295–299.
  * [26] Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Z. Chen, R. Skerry-Ryan, Y. Jia, A. Rosenberg, and B. Ramabhadran, “Learning to speak fluently in a foreign language: Multilingual speech synthesis and cross-language voice cloning,” _Proc. Interspeech 2019_ , pp. 2080–2084, 2019.
  * [27] Z. Liu and B. Mak, “Multi-lingual multi-speaker text-to-speech synthesis for voice cloning with online speaker enrollment.” in _INTERSPEECH_ , 2020, pp. 2932–2936.
  * [28] D. Parikh and K. Grauman, “Relative attributes,” in _2011 International Conference on Computer Vision_. IEEE, 2011, pp. 503–510.
  * [29] N. Li, Y. Liu, Y. Wu, S. Liu, S. Zhao, and M. Liu, “Robutrans: A robust transformer-based text-to-speech model,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 05, 2020, pp. 8228–8235.
  * [30] S.-H. Lee, H.-W. Yoon, H.-R. Noh, J.-H. Kim, and S.-W. Lee, “Multi-spectrogan: High-diversity and high-fidelity spectrogram generation with adversarial style combination for speech synthesis,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 14, 2021, pp. 13 198–13 206.
  * [31] F. Eyben, M. Wöllmer, and B. Schuller, “Opensmile: the munich versatile and fast open-source audio feature extractor,” in _Proceedings of the 18th ACM international conference on Multimedia_ , 2010, pp. 1459–1462.
  * [32] P. Angkititrakul and J. H. Hansen, “Advances in phone-based modeling for automatic accent classification,” _IEEE transactions on audio, speech, and language processing_ , vol. 14, no. 2, pp. 634–646, 2006.
  * [33] C. Du and K. Yu, “Diverse and controllable speech synthesis with gmm-based phone-level prosody modelling,” in _Proc. Interspeech 2021_ , 2021, pp. 5–10.
  * [34] B. Siddiquie, R. S. Feris, and L. S. Davis, “Image ranking and retrieval based on multi-attribute queries,” in _CVPR 2011_. IEEE, 2011, pp. 801–808.
  * [35] Z. Meng, N. Adluru, H. J. Kim, G. Fung, and V. Singh, “Efficient relative attribute learning using graph neural networks,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 552–567.
  * [36] Y. Saquil, Q.-C. Xu, Y.-L. Yang, and P. Hall, “Rank3dgan: Semantic mesh generation using relative attributes,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 04, 2020, pp. 5586–5594.
  * [37] X. Zhu, S. Yang, G. Yang, and L. Xie, “Controlling emotion strength with relative attribute for end-to-end speech synthesis,” in _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , 2019, pp. 192–199.
  * [38] Y. Lei, S. Yang, and L. Xie, “Fine-grained emotion strength transfer, control and prediction for emotional speech synthesis,” in _2021 IEEE Spoken Language Technology Workshop (SLT)_. IEEE, 2021, pp. 423–430.
  * [39] Y. Wang, D. Stanton, Y. Zhang, R.-S. Ryan, E. Battenberg, J. Shor, Y. Xiao, Y. Jia, F. Ren, and R. A. Saurous, “Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 5180–5189.
  * [40] W.-N. Hsu, Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Y. Wang, Y. Cao, Y. Jia, Z. Chen, J. Shen _et al._ , “Hierarchical generative modeling for controllable speech synthesis,” in _International Conference on Learning Representations_ , 2018.
  * [41] G. Sun, Y. Zhang, R. J. Weiss, Y. Cao, H. Zen, and Y. Wu, “Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis,” in _ICASSP 2020-2020 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2020, pp. 6264–6268.
  * [42] Z. Liu, Q. Tian, C. Hu, X. Liu, M. Wu, Y. Wang, H. Zhao, and Y. Wang, “Controllable and lossless non-autoregressive end-to-end text-to-speech,” _arXiv preprint arXiv:2207.06088_ , 2022.
  * [43] S. Sivaprasad, S. Kosgi, and V. Gandhi, “Emotional prosody control for speech generation,” in _Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August - 3 September 2021_ , H. Hermansky, H. Cernocký, L. Burget, L. Lamel, O. Scharenborg, and P. Motlícek, Eds. ISCA, 2021, pp. 4653–4657.
  * [44] C.-B. Im, S.-H. Lee, S.-B. Kim, and S.-W. Lee, “Emoq-tts: Emotion intensity quantization for fine-grained controllable emotional text-to-speech,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 6317–6321.
  * [45] G. Zhang, Y. Qin, W. Zhang, J. Wu, M. Li, Y. Gai, F. Jiang, and T. Lee, “iemotts: Toward robust cross-speaker emotion transfer and control for speech synthesis based on disentanglement between prosody and timbre,” _arXiv preprint arXiv:2206.14866_ , 2022.
  * [46] K. Lee, K. Park, and D. Kim, “Styler: Style factor modeling with rapidity and robustness via speech decomposition for expressive and controllable neural text to speech,” in _Interspeech 2021_. ISCA, 2021.
  * [47] T. Raitio, J. Li, and S. Seshadri, “Hierarchical prosody modeling and control in non-autoregressive parallel neural tts,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 7587–7591.
  * [48] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [49] E. Casanova, C. Shulby, E. Gölge, N. M. Müller, F. S. de Oliveira, A. C. Junior, A. d. S. Soares, S. M. Aluisio, and M. A. Ponti, “Sc-glowtts: an efficient zero-shot multi-speaker text-to-speech model,” in _Proc. Interspeech 2021_ , 2021, pp. 1–5.
  * [50] D. L. Bolinger, “A theory of pitch accent in english,” _Word_ , vol. 14, no. 2-3, pp. 109–149, 1958.
  * [51] T. Joachims, “Optimizing search engines using clickthrough data,” in _Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2002, pp. 133–142.
  * [52] O. Chapelle, “Training a support vector machine in the primal,” _Neural computation_ , vol. 19, no. 5, pp. 1155–1178, 2007.
  * [53] T. Bak, J.-S. Bae, H. Bae, Y.-I. Kim, and H.-Y. Cho, “Fastpitchformant: Source-filter based decomposed modeling for speech synthesis,” _arXiv preprint arXiv:2106.15123_ , 2021.
  * [54] G. Zhao, S. Sonsaat, A. Silpachai, I. Lucic, E. Chukharev-Hudilainen, J. Levis, and R. Gutierrez-Osuna, “L2-arctic: A non-native english speech corpus,” _Proc. Interspeech 2018_ , pp. 2783–2787, 2018.
  * [55] M. Chen, X. Tan, Y. Ren, J. Xu, H. Sun, S. Zhao, and T. Qin, “Multispeech: Multi-speaker text to speech with transformer,” _Proc. Interspeech 2020_ , pp. 4024–4028, 2020.
  * [56] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [57] K. Ito and L. Johnson, “The lj speech dataset,” https://keithito.com/LJ-Speech-Dataset/, 2017.
  * [58] J. Yamagishi, C. Veaux, K. MacDonald _et al._ , “Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92),” 2019.
  * [59] H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu, “Libritts: A corpus derived from librispeech for text-to-speech,” _Proc. Interspeech 2019_ , pp. 1526–1530, 2019.
  * [60] J. Lorenzo-Trueba, T. Drugman, J. Latorre, T. Merritt, B. Putrycz, R. Barra-Chicote, A. Moinet, and V. Aggarwal, “Towards achieving robust universal neural vocoding,” _Proc. Interspeech 2019_ , pp. 181–185, 2019\.
  * [61] Y. Jiao, A. Gabryś, G. Tinchev, B. Putrycz, D. Korzekwa, and V. Klimkov, “Universal neural vocoding with parallel wavenet,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6044–6048.
  * [62] J. Lorenzo-Trueba, J. Yamagishi, T. Toda, D. Saito, F. Villavicencio, T. Kinnunen, and Z. Ling, “The voice conversion challenge 2018: Promoting development of parallel and nonparallel methods,” in _The Speaker and Language Recognition Workshop_. ISCA, 2018, pp. 195–202.
  * [63] M. Müller, “Dynamic time warping,” _Information retrieval for music and motion_ , pp. 69–84, 2007.
  * [64] J. A. Lee, G. Soutar, and J. Louviere, “The best–worst scaling approach: An alternative to schwartz’s values survey,” _Journal of personality assessment_ , vol. 90, no. 4, pp. 335–347, 2008.

|  Rui Liu is currently a Professor in National and Local Joint Engineering
Research Center of Mongolian Intelligent Information Processing, Inner
Mongolia University. Rui Liu received Ph.D degree from Inner Mongolia
University, China in 2020 and Bachelor degree in Taiyuan University of
Technology, ShanXi, China in 2014. From 2019 to 2020, he has been an exchange
PhD candidate at the Department of Electrical & Computer Engineering of
National University of Singapore (NUS), funded by China Scholarship Council
(CSC). From 2020 to 2022, he worked as a Research Fellow at the Department of
Electrical and Computer Engineering, National University of Singapore,
Singapore. He was the recipient of the “Best Paper Award” at the 2021
International Conference on Asian Language Processing (IALP). He has published
more than 20 papers in top-tier NLP/ML/AI conferences and journals, including
IEEE/ACM-TASLP, Neural Networks, ICASSP, COLING, INTERSPEECH, etc. He is a
member of IEEE, ISCA, CAAI and CCF, and serves as the reviewer for many major
referred journal and conference papers. His research interests broadly lie in
audio, speech and natural language processing, which include expressive Text-
to-Speech (TTS), expressive voice conversion, speech emotion recognition,
prosody structure prediction, grapheme-to-phoneme conversion (G2P), syntax
parsing et. al.
---|---
|  Berrak Sisman received the PhD degree in Electrical and Computer
Engineering from National University of Singapore in 2020, fully funded by
A*STAR Graduate Academy under Singapore International Graduate Award (SINGA).
She is currently working as a tenure-track Assistant Professor at the Erik
Jonsson School Department of Electrical and Computer Engineering at University
of Texas at Dallas, United States. Prior to joining UT Dallas, she was a
tenure-track faculty at Singapore University of Technology and Design
(2020-2022). She was a Postdoctoral Research Fellow at the National University
of Singapore (2019-2020), and a Visiting Researcher at Columbia University,
New York, United States (2020). She was an exchange PhD student at the
University of Edinburgh and a visiting scholar at The Centre for Speech
Technology Research (CSTR), University of Edinburgh in 2019. She was attached
to RIKEN Advanced Intelligence Project, Japan in 2018. Her research is focused
on machine learning, signal processing, emotion, speech synthesis and voice
conversion. She plays leadership roles in conference organizations and also
active in technical committees. She has served as the General Coordinator of
the Student Advisory Committee (SAC) of International Speech Communication
Association (ISCA). She has served as the Area Chair at INTERSPEECH 2021,
INTERSPEECH 2022, IEEE SLT 2022 and as the Publication Chair at ICASSP 2022.
She has been elected as a member of the IEEE Speech and Language Processing
Technical Committee (SLTC) in the area of Speech Synthesis for the term from
Jan. 2022 to Dec. 2024.
---|---
|  Guanglai Gao received the B.S. degree from Inner Mongolia University,
Hohhot, China, in 1985, and the M.S. degree from the National University of
Defense Technology, Changsha, China, in 1988. He was a Visiting Researcher at
University of Montreal, Canada. Currently, he is a Professor with the
Department of Computer Science, Inner Mongolia University. His research
interests include artificial intelligence and pattern recognition.
---|---
|  Haizhou Li Haizhou Li (M’91-SM’01-F’14) received the B.Sc., M.Sc., and Ph.D
degree in electrical and electronic engineering from South China University of
Technology, Guangzhou, China in 1984, 1987, and 1990 respectively. Dr Li is
currently a Professor at the School of Data Science, the Chinese University of
Hong Kong, Shenzhen, China, and the Department of Electrical and Computer
Engineering, National University of Singapore (NUS). His research interests
include automatic speech recognition, speaker and language recognition, and
natural language processing. Prior to joining NUS, he taught in the University
of Hong Kong (1988-1990) and South China University of Technology (1990-1994).
He was a Visiting Professor at CRIN in France (1994-1995), Research Manager at
the Apple-ISS Research Centre (1996-1998), Research Director in Lernout &
Hauspie Asia Pacific (1999-2001), Vice President in InfoTalk Corp. Ltd.
(2001-2003), and the Principal Scientist and Department Head of Human Language
Technology in the Institute for Infocomm Research, Singapore (2003-2016). Dr
Li served as the Editor-in-Chief of IEEE/ACM Transactions on Audio, Speech and
Language Processing (2015-2018), a Member of the Editorial Board of Computer
Speech and Language (2012-2018). He was an elected Member of IEEE Speech and
Language Processing Technical Committee (2013-2015), the President of the
International Speech Communication Association (2015-2017), the President of
Asia Pacific Signal and Information Processing Association (2015-2016), and
the President of Asian Federation of Natural Language Processing (2017-2018).
He was the General Chair of ACL 2012, INTERSPEECH 2014 and ASRU 2019. Dr Li is
a Fellow of the IEEE and the ISCA. He was a recipient of the National Infocomm
Award 2002 and the President’s Technology Award 2013 in Singapore. He was
named one of the two Nokia Visiting Professors in 2009 by the Nokia
Foundation, U Bremen Excellence Chair Professor in 2019, and Fellow of Academy
of Engineering Singapore in 2022.
---|---


current position: ]Institut Laue-Langevin, CS 20156, 38042 Grenoble Cedex 9,
France

current position: ]University of Wisconsin, Madison, Wisconsin, USA

current position: ]Department of Chemistry - TRIGA site, Johannes Gutenberg
University Mainz, 55128 Mainz, Germany

# New Limit on Axion-Like Dark Matter using Cold Neutrons

Ivo Schulthess<EMAIL_ADDRESS>Laboratory for High Energy
Physics and Albert Einstein Center for Fundamental Physics, University of
Bern, 3012 Bern, Switzerland  Estelle Chanel  [  Laboratory for High Energy
Physics and Albert Einstein Center for Fundamental Physics, University of
Bern, 3012 Bern, Switzerland  Anastasio Fratangelo  Laboratory for High Energy
Physics and Albert Einstein Center for Fundamental Physics, University of
Bern, 3012 Bern, Switzerland  Alexander Gottstein  Laboratory for High Energy
Physics and Albert Einstein Center for Fundamental Physics, University of
Bern, 3012 Bern, Switzerland  Andreas Gsponer  Laboratory for High Energy
Physics and Albert Einstein Center for Fundamental Physics, University of
Bern, 3012 Bern, Switzerland  Zachary Hodge  [  Laboratory for High Energy
Physics and Albert Einstein Center for Fundamental Physics, University of
Bern, 3012 Bern, Switzerland  Ciro Pistillo  Laboratory for High Energy
Physics and Albert Einstein Center for Fundamental Physics, University of
Bern, 3012 Bern, Switzerland  Dieter Ries  [  Laboratory for High Energy
Physics and Albert Einstein Center for Fundamental Physics, University of
Bern, 3012 Bern, Switzerland  Torsten Soldner  Institut Laue-Langevin, CS
20156, 38042 Grenoble Cedex 9, France  Jacob Thorne  Laboratory for High
Energy Physics and Albert Einstein Center for Fundamental Physics, University
of Bern, 3012 Bern, Switzerland  Florian M. Piegsa
<EMAIL_ADDRESS>Laboratory for High Energy Physics and Albert
Einstein Center for Fundamental Physics, University of Bern, 3012 Bern,
Switzerland

(August 27, 2024)

###### Abstract

We report on a search for dark matter axion-like particles (ALPs) using a
Ramsey-type apparatus for cold neutrons. A hypothetical ALP-gluon-coupling
would manifest in a neutron electric dipole moment signal oscillating in time.
Twenty-four hours of data have been analyzed in a frequency range from 23
$\mu$Hz to 1 kHz, and no significant oscillating signal has been found. The
usage of present dark-matter models allows to constrain the coupling of ALPs
to gluons in the mass range from $10^{-19}$ to $4\times 10^{-12}$ eV. The best
limit of $C_{G}$ / $f_{a}m_{a}=2.7\times 10^{13}$ GeV-2 (95% C.L.) is reached
in the mass range from $2\times 10^{-17}$ to $2\times 10^{-14}$ eV.

Dark matter makes up roughly 27% of our universe’s total mass-energy content
[1]. So far, no dark-matter model has been experimentally verified, but
promising candidates remain the axion and a more general class of axion-like
particles (ALPs) with relaxed property constraints. The axion was initially
suggested to solve the strong CP problem of quantum chromodynamics (QCD) [2,
3, 4, 5]. The solution is an additional U(1) symmetry to the Standard Model of
particle physics. The spontaneous and explicit breaking of this symmetry
results in a massive but ultra-light spin-0 particle, the axion. Since this
pseudo-scalar particle must satisfy the Klein-Gordon equation, it results in
an oscillating field that could explain the dark-matter content in our
universe. Most experiments, such as CAST, IAXO, or ADMX search for the axion
via its coupling to photons [6, 7]. Various models suggest interactions with
other particles such as standard model fermions (DFSZ models [8, 9]) or a new
exotic heavy quark (KSVZ models [10, 11]). A summary of recent axion theories
and experiments can be found in the PDG review [12]. The coupling of axions
and ALPs to gluons is a common feature in theoretical models [13, 14, 15, 16].
One consequence of this coupling is that an oscillating ALP field induces an
equally oscillating electric dipole moment (EDM) of the neutron [17]

$d_{n}^{a}(t)\approx+2.4\times
10^{-16}\,e\,\mathrm{cm}\cdot\frac{C_{G}}{f_{a}}a_{0}\cos{(m_{a}t)}\ ,$ (1)

where $C_{G}$ is a model-dependent parameter, $f_{a}$ the ALP decay constant,
$a_{0}$ its oscillation amplitude, $m_{a}$ its mass, and $e$ the elementary
charge. The parameter space of ALPs is defined by their mass and the coupling
$C_{G}/f_{a}$. It is restricted by various astrophysical and cosmological
constraints, as well as scrutinized in three recent laboratory searches. The
CASPEr experiment is dedicated to searching for an axion signal using nuclear
magnetic resonance techniques. The collaboration recently published an upper
limit in a narrow frequency band around 39 MHz [18, 19]. Two other experiments
search for a permanent EDM of the electron, using trapped molecular ions, and
the neutron, using ultra-cold neutrons in a storage experiment. Both
experiments analyzed their data for oscillating signals. However, no
significant signal from the nHz-region up to 0.4 Hz was found [20, 21]. Here,
we present the results of a complementary neutron EDM experiment, named Beam
EDM [22, 23]. It employs a continuous cold neutron beam with intrinsic sub-ms
time resolution. Thus, the accessible frequency range is extended to 1 kHz.
This allowed us to probe for ALP masses more than three orders of magnitude
heavier than tested by previous laboratory experiments. Since no significant
signal was found, a new constraint on the possible existence of such ultra-
light particles has been deduced.

We use Ramsey’s method of separated oscillatory fields applied to neutrons to
search for an axion-like dark-matter signal [24, 25]. In this technique,
neutrons act as a spin clock at their Larmor precession frequency in a
magnetic field $B_{0}$, which allows to precisely detect magnetic or pseudo-
magnetic field changes. The measured quantity is the phase that a neutron spin
acquires due to its coupling to a magnetic field change $\Delta B(t)$ and an
electric field $E$

$\varphi=\displaystyle\int_{0}^{T_{\text{int}}}\left(\gamma_{n}\Delta
B(t)+\frac{2d_{n}(t)}{\hbar}E\right)\,\mathrm{d}t$ (2)

where $\gamma_{n}$ is the gyromagnetic ratio of the neutron, $d_{n}(t)$ its
electric dipole moment, $\hbar$ the reduced Planck constant, and
$T_{\text{int}}$ the interaction time which depends on the neutron velocity.

Figure 1: (Color online) Schematic of the experimental setup where a polarized
neutron beam enters from the left. It shows the 6 m-long mu-metal shield
around the interaction region and the two 40 cm-long RF spin-flip coils for
the $\pi/2$-flips in green. The electrodes and the electric field direction
are shown in red and the magnetic field direction is indicated in blue. The
spin analyzer (purple) reflects one spin state and transmits the other. The
neutrons are detected using a 2D pixel detector with a sensitive area of
10$\times$10 cm2. The vacuum beam pipe surrounding the electrodes is not
shown.

Figure 1 shows a schematic of the experimental setup installed at the cold
neutron beam facility PF1b at the Institut Laue-Langevin in Grenoble, France
[26]. A beam of polarized cold neutrons with a Maxwell-Boltzmann-like velocity
distribution, peaking at about 1000 m/s, enters a constant and homogeneous
vertical magnetic field $B_{0}=220$ $\mu$T. The field is adjusted and
stabilized to the sub-nT level using fluxgate sensors and a 3D coil system. A
two-layer passive magnetic mu-metal shield surrounds the experimental setup.
Two radio frequency (RF) spin-flip coils, one before and one after the
interaction region, induce resonant $\pi$/2-flips of the neutron spin. The
interaction region with a length of 3 m is inside a vacuum beam pipe between
the spin-flip coils. It consists of three sets of one-meter-long electrode
stacks with a high-voltage electrode in the center and two ground electrodes
on top and bottom. The electrode separation is 1 cm. This setup allows for two
partial neutron beams passing between the electrodes, simultaneously sensing
the electric field direction parallel and anti-parallel to the magnetic field.
This double beam arrangement provides the possibility to compensate for global
field drifts and common-mode noise. Downstream of the setup, a neutron spin
analyzer spatially separates the two spin states of each partial beam before
they are counted in a 2D neutron pixel detector [27]. The neutron rate
integrated over the entire sensitive area of the detector was approximately
$10^{7}$ s-1. The statistical counting error of the detector was calibrated.
It was found that the Poisson error is overestimating the measured standard
deviation by approximately 10% for the given neutron rate and settings of data
acquisition due to event pile-up. The potential of the high-voltage electrode
was set to $\pm 35$ kV, and the resulting electric field was directly measured
with neutrons using the relativistic $\vec{v}\times\vec{E}$-effect [28]. The
measured electric field amplitude agrees with the nominal value within 4%. The
reason for the small deviation is the slight vertical displacement of the
central high-voltage electrode due to gravity. With this apparatus, the
oscillating neutron EDM caused by a hypothetical axion field would manifest in
an oscillation of the population in each spin state. This would lead to a
corresponding oscillation of the neutron asymmetry for each partial beam,
defined as

$A=\frac{N_{\uparrow}-N_{\downarrow}}{N_{\uparrow}+N_{\downarrow}}\ ,$ (3)

where $N_{\uparrow}$ and $N_{\downarrow}$ are the neutron counts in the spin
up and down state, respectively. To be most sensitive to changes in the
asymmetry, the frequency and relative phase of the RF spin-flip signals are
adjusted such that $A\approx 0$, i.e., $N_{\uparrow}\approx N_{\downarrow}$.
This corresponds to the point of steepest slope in a Ramsey resonance pattern.

To connect the signal amplitude of the neutron asymmetry of Eq. (3) to the
ALP-gluon-coupling in Eq. (1) multiple calibration measurements were
conducted. In these measurements, we created artificial signals by applying
homogeneous sinusoidally oscillating magnetic fields of various frequencies
and amplitudes $B_{a}$ parallel to $B_{0}$ through the entire setup using an
additional rectangular Helmholtz-type coil. Note, such a field can be
interpreted as a corresponding false EDM signal using Eq. (2)

$d_{n}=\frac{\hbar\gamma_{n}B_{a}}{2E}.$ (4)

First, we conducted an offline calibration measurement where we correlated the
magnetic field amplitude $B_{a}$ to the applied oscillating electric current
in the auxiliary coil. The field was determined at 47 positions over a
distance of 5.3 m along the neutron beam path with five fluxgates mounted in a
cross-shaped arrangement on a magnetic field mapper. The magnetic field was
recorded with a sampling rate of 10 kHz for two seconds at each position and a
sinusoidal function was fitted to the data. The amplitude was averaged over
the interaction region and all five fluxgates. The calibration parameter was
measured to $S_{B}=(12.13\pm 0.02)$ $\mu$T A-1.

A second calibration measurement was performed with neutrons to correlate the
amplitude of the oscillating neutron asymmetry in Eq. (3) to the same coil
currents applied in the first calibration measurement. Here, we acquired the
neutron asymmetry for 60 seconds at a sampling rate of 4 kHz and performed
again a sinusoidal fit to the data. This resulted in a value of
$S_{A}=(11.5\pm 0.5)$ A-1 for frequencies below 5 Hz. Together, the two
calibration measurements are used to translate the amplitude of an oscillating
neutron asymmetry into a corresponding (pseudo-)magnetic field amplitude via

$B_{a}=\frac{4.2}{3}\frac{S_{B}}{S_{A}}A\ .$ (5)

The factor of 4.2/3 comes from the fact that the magnetic and the electric
interaction length are different as shown in Fig. 1. The resulting calibration
curve as a function of frequency is presented in Fig. 2.

Figure 2: (Color online) Calibration factor $S_{B}/S_{A}$ as a function of
frequency. The measured data is shown as dots, whereas the red dashed line is
a least-squares fit of a Butterworth-filter function [29]. For instance,
typical neutron asymmetry signals of the order $10^{-5}$ correspond to a
pseudo-magnetic field of $14$ pT for frequencies smaller than 5 Hz using Eq.
(5).

The value is constant for low frequencies up to approximately 5 Hz. The
primary reason for its subsequent rise is the frequency-dependent RF shielding
of the aluminum parts of the setup, i.e., the construction frame, vacuum beam
pipe, and electrodes. Another reason is an effect that depends on the neutron
velocity: as shown in Eq. (2), the acquired neutron spin phase has to be
integrated over the interaction time. In the case of an oscillating field,
this integral becomes zero if the period of the oscillation matches the
interaction time. This effect is suppressed for a beam with a broad velocity
distribution but still results in a decrease in sensitivity at higher
frequencies. Calculations, simulations, and further test measurements that are
not included in this paper for brevity suggest that the actual decrease in
sensitivity would be smaller for real ALP signals. For instance, the RF
shielding of the aluminum does not apply to the ALP field. However, since
these effects cannot be simply decorrelated, we use the presented curve. This
results in a conservative upper limit at high frequencies if no ALPs were
found.

We performed several continuous measurements of the neutron EDM for the
dedicated ALP search with various duration and high-voltage polarities. The
presented analysis uses a total of 24 hours of data, taken with a sampling
rate of 4 kHz, i.e., we obtained a value for the neutron asymmetry and, hence,
the neutron EDM every 0.25 ms. The potential of the central high-voltage
electrode was set to +35 kV. Hence, the electric field used for the evaluation
is $E=2\times 35$ kV/cm as all the analysis is done for the difference of the
two beams. Data were taken on September 13/14, 2020 and are publicly available
[30]. A 5 second-long subset of the data is presented in Fig. 3a. The entire
data is split into two halves of 12 hours each. The first half acts as the
analysis set and the second as the control set. We performed the spectral
analysis on the neutron data using an adapted version of the generalized Lomb-
Scargle algorithm [31, 32, 33, 34]. The basic concept of the algorithm is to
perform a $\chi^{2}$ minimization of the fit function $f(t)=a\sin(\omega
t)+b\cos(\omega t)+c$, where $a$, $b$, and $c$ are the parameters to be
minimized for each frequency $\omega$. The signal amplitude
$\sqrt{a^{2}+b^{2}}$ is Rayleigh distributed assuming only white noise. An
oscillating signal is considered significant if it appears in the spectral
analysis of both data sets, at least on the 5-sigma level.

Figure 3: (Color online) The data for the top beam (blue $\blacktriangle$),
the bottom beam (yellow $\blacktriangledown$), and the difference between the
two beams (red $\bullet$) are shown for various stages in the data processing.
(a) Measured neutron asymmetry for a time window of 5 seconds. (b) Frequency
spectrum between 1 Hz and 60 Hz. The highly significant signal at 50 Hz from
the power line frequency are canceled out by analyzing the beam difference.
(c) Discrete peaks appear in the spectrum due to the data structure. (d) After
applying the calibration as shown in Fig. 2, the neutron asymmetry spectrum
translates into a pseudo-magnetic field spectrum. Error bars were omitted for
reasons of readability but are of order 3 pT in the most sensitive central
range. Note that the plots of (b), (c), and (d) are based on the whole 12 hour
data set and that only a fraction of the data points is shown for legibility
in all sub-figures.

We subtracted the signals of both neutron beams, i.e., opposite electric field
directions, from each other. Figure 3b shows how this eliminates the eminent
50 Hz signal coming from the power line frequency. The peak is visible in the
spectra of both beams separately but not in the spectrum of the difference of
the signals. The overall spectrum shows three groups of significant signals of
different origins that are not ALPs. The first group appears for frequencies
below 10 mHz. They can be explained by long-term magnetic gradient field
drifts due to temperature changes. They happen on the time scale of hours and
result in a rise in signal amplitude. The second group is located in the
frequency range between 10 mHz and 2 Hz. They are caused by the data structure
itself and a sub-range is presented in Fig. 3c. Our sequence of data taking is
divided into runs of 62.5 s duration. Each run consists of 57.5 s of
measurement time and 5 s of downtime to save the data. This time structure
leads to peaks at the inverse run time of 16 mHz and higher orders.
Additionally, the 5 s gap leads to an envelop hump structure with a period of
200 mHz. For frequencies higher than 2 Hz, these peaks are too small to be
detected. The third group of significant signals has a statistical origin.
Since the amplitudes of the signals follow a Rayleigh distribution, the
pull/significance is also distributed accordingly. We found $113\pm
11_{\text{stat}}\pm 9_{\text{sys}}$ and $132\pm 11_{\text{stat}}\pm
13_{\text{sys}}$ events above the 5-sigma threshold in the analysis and the
control data set, respectively. The frequencies of the events of both sets do
not coincide. The systematic error originates from the uncertainty of the
detector count-error calibration. These values are slightly below the 162
statistically expected events for a data set with 43.6 million analyzed
frequencies. With the use of Eq. (5), the neutron asymmetry amplitude shown in
Fig. 3b and Fig. 3c can be translated into the pseudo-magnetic field
amplitude. The full spectrum with a reduced spectral resolution is shown in
Fig. 3d. The most sensitive region of a few pT is in the central flat region.

Besides having a significant amplitude over the background noise level, an
actual oscillating EDM signal must disappear if no electric field is applied.
This way, noise signals or signals from external sources can be further
excluded. For this, we performed an additional measurement with no electric
field applied. Moreover, the amplitude of a real signal must be identical for
both electric field directions but must exhibit a phase-shift of $\pi$.
Overall, no significant signal was found at the same frequency in both partial
data sets. Thus, an upper limit on the ALP-gluon-coupling can be derived.
Using Eqs. (1) and (4) as well as the calibration shown in Fig. 2, the
coupling can be calculated with

$\frac{C_{G}}{f_{a}}=\frac{\gamma_{n}\hbar B_{a}}{a_{0}E\times 4.8\times
10^{-16}\,e\,\text{cm}}.$ (6)

The oscillation amplitude relates to the local dark-matter density via
$a_{0}=\sqrt{2\rho_{\text{\tiny{DM}}}}/m_{a}$, assuming all dark matter
consists of ALPs. Recently, Foster et al. published the possible effects of
the dark-matter substructure to the upper limits set by axion experiments
[35]. Depending on the ratio of the measurement time over the coherence time
of the dark-matter field $T/\tau_{c}$, the line shape of the field could
change. The coherence time of the dark-matter field is $10^{6}$ periods of the
oscillating signal [36] and our measurement time 12 hours. For $T\gg\tau_{c}$,
the field is deterministic and the local dark-matter density averages to
$\rho_{\text{\tiny{DM}}}=0.2-0.7$ GeV/cm3 [37]. To be consistent with the
measurement of the nEDM collaboration [20], we chose a value of
$\rho_{\text{\tiny{DM}}}=0.4$ GeV/cm3 [38]111This corresponds to a value of
$\rho_{\text{\tiny{DM}}}=3.1\times 10^{-42}$ GeV4 in natural units. . On the
other hand, if $T\ll\tau_{c}$ the field is stochastic and the amplitude
follows a Rayleigh distribution with scale parameter
$a_{0}^{\text{\tiny{det}}}/\sqrt{2}$ where $a_{0}^{\text{\tiny{det}}}$ is the
deterministic dark-matter amplitude. Since our measurement time and frequency
range cover both cases, we display both limits in the exclusion plot of Fig.
4. The upper limit at a given frequency is calculated by integrating the
normalized distribution of the coupling $C_{G}/f_{a}$ up to the confidence
limit of 95%. Hence, the upper integration constant corresponds to the upper
limit of the ALP-gluon-coupling. In the case of deterministic dark matter, the
coupling follows a Rayleigh distribution. However, in the case of stochastic
dark matter, the coupling corresponds to the ratio of two Rayleigh
distributions. In this case, the overall distribution has a much longer tail,
resulting in a higher upper limit. We determined a scaling factor of $3.2\pm
0.3$ compared to the deterministic limit. This is in agreement with similar
calculations by the CASPEr collaboration [40].

Figure 4: (Color online) Limits on the ALP-gluon-coupling are shown as a
function of the mass or frequency. The shaded areas are exclusion regions from
cosmology and astrophysical observations (blue: Galaxies [41], BBN [42, 43],
SN1987A [44, 45]) and laboratory experiments (orange: nEDM [20], HfH [21]).
The black outlines with the pink area mark the exclusion region of this
publication (labeled Beam EDM). The solid and dotted lines correspond to the
deterministic and stochastic dark-matter models, respectively. The green line
shows the canonical QCD axion.

Figure 4 shows our exclusion region of the ALP-gluon-coupling as a function of
mass or frequency. The most stringent constraint of
$C_{G}/f_{a}m_{a}=2.7\times 10^{13}$ GeV-2 (95% C.L.) for deterministic dark
matter was set in the frequency range between 5 mHz and 5 Hz 222As an example:
an oscillating magnetic field amplitude of $B_{a}=10$ pT which is typical for
this frequency range (see Fig. 3d) translates with Eq. (4) and $E=2\times 35$
kV/cm into an EDM amplitude of $d_{n}=8.6\times 10^{-24}\,e\,\text{cm}$. With
the use of Eq. (6) this can be translated into a ALP-gluon-coupling of
$C_{G}/f_{a}=1.4\times 10^{-11}$ GeV-1 for an ALP mass of $m_{a}=10^{-15}$ eV.
It follows $C_{G}/f_{a}m_{a}=1.4\times 10^{13}$ GeV-2. . For frequencies below
5 mHz, the upper limit increases due to magnetic gradient field drifts. For
frequencies above 5 Hz, the upper limit increases due to a decrease in
sensitivity of the apparatus, as shown in Fig. 3d. For reasons of legibility,
we smoothed the limits in Fig. 4 with a Savitzky-Golay filter [47]. To provide
context, the constraints on 95% C.L. from the other laboratory experiments are
also presented 333Note, other experimental constraints are sometimes presented
using the following relation: $g_{a}=(3.7\pm 1.5)\times
10^{-3}\left(\frac{C_{G}}{f_{a}}\right)\frac{1}{\mathrm{GeV}}$ [12, 49].. In
addition, astrophysical and cosmological constraints arise from galaxy
luminosity functions at high red-shifts, big bang nucleosynthesis (BBN)
models, and the SN1978A cooling. The QCD-axion line shows the region where an
axion would simultaneously solve the strong CP problem and explain all dark
matter.

In conclusion, we performed a dedicated search for axion-like particles but
did not find a significant oscillating signal. With only 24 hours of data, we
could constrain an ALP-gluon-coupling in a mass region covering almost eight
orders of magnitude. Together with the results of two other laboratory
experiments, a large region of the ALP-dark-matter parameter space could be
excluded, and future EDM searches may extend this even further.

###### Acknowledgements.

We gratefully acknowledge the excellent technical support by R. Hänni, J.
Christen, L. Meier, and D. Berruyer. We thank O. Zimmer for lending us the 35
kV bipolar power supply, which was essential for data-taking. This work was
supported via the European Research Council under the ERC Grant Agreement no.
715031 (BEAM-EDM) and via the Swiss National Science Foundation under grants
no. PP00P2-163663 and 200021-181996.

## References

  * Planck Collaboration [2020] Planck Collaboration, Planck 2018 results: I. Overview and the cosmological legacy of Planck, Astronomy & Astrophysics 641, A1 (2020).
  * Peccei and Quinn [1977a] R. D. Peccei and H. R. Quinn, CP Conservation in the Presence of Pseudoparticles, Physical Review Letters 38, 1440–1443 (1977a).
  * Peccei and Quinn [1977b] R. D. Peccei and H. R. Quinn, Constraints imposed by CP conservation in the presence of pseudoparticles, Physical Review D 16, 1791–1797 (1977b).
  * Weinberg [1978] S. Weinberg, A New Light Boson?, Physical Review Letters 40, 223–226 (1978).
  * Wilczek [1978] F. Wilczek, Problem of Strong P and T Invariance in the Presence of Instantons, Physical Review Letters 40, 279–282 (1978).
  * Dafni and Iguaz on behalf of the CAST and IAXO collaboration [2015] T. Dafni and F. J. Iguaz on behalf of the CAST and IAXO collaboration, Axion helioscopes update: the status of CAST and IAXO, PoS TIPP2014, 130 (2015).
  * ADMX Collaboration [2020] ADMX Collaboration, Extended Search for the Invisible Axion with the Axion Dark Matter Experiment, Physical Review Letters 124, 101303 (2020).
  * Dine  _et al._ [1981] M. Dine, W. Fischler, and M. Srednicki, A simple solution to the strong CP problem with a harmless axion, Physics Letters B 104, 199–202 (1981).
  * Zhitnitskii [1980] A. P. Zhitnitskii, Possible suppression of axion-hadron interactions, Sov. J. Nucl. Phys. (Engl. Transl.); (United States) 31:2, 260–263 (1980).
  * Kim [1979] J. E. Kim, Weak-Interaction Singlet and Strong CP Invariance, Physical Review Letters 43, 103–107 (1979).
  * Shifman  _et al._ [1980] M. A. Shifman, A. I. Vainshtein, and V. I. Zakharov, Can confinement ensure natural CP invariance of strong interactions?, Nuclear Physics B 166, 493–506 (1980).
  * Zyla  _et al._ [2020] P.A. Zyla _et al._ (Particle Data Group), Review of Particle Physics, PTEP 2020, 083C01 (2020), and 2021 update.
  * G. G. Raffelt [1996] G. G. Raffelt,  _Stars as Laboratories for Fundamental Physics_ (University of Chicago Press, 1996).
  * Hook [2018] A. Hook, Solving the Hierarchy Problem Discretely, Physical Review Letters 120, 261802 (2018).
  * Di Luzio  _et al._ [2021a] L. Di Luzio, B. Gavela, P. Quilez, and A. Ringwald, Dark matter from an even lighter QCD axion: trapped misalignment, Journal of Cosmology and Astroparticle Physics 2021 (10), 001.
  * Di Luzio  _et al._ [2021b] L. Di Luzio, B. Gavela, P. Quilez, and A. Ringwald, An even lighter QCD axion, Journal of High Energy Physics 2021, 184 (2021b).
  * Pospelov and Ritz [2000] M. Pospelov and A. Ritz, Theta vacua, QCD sum rules, and the neutron electric dipole moment, Nuclear Physics B 573, 177–200 (2000).
  * Aybas  _et al._ [2021] D. Aybas, J. Adam, E. Blumenthal, A. V. Gramolin, D. Johnson, A. Kleyheeg, _et al._ , Search for axion-like dark matter using solid-state nuclear magnetic resonance, Physical Review Letters 126, 141802 (2021).
  * Budker  _et al._ [2014] D. Budker, P. W. Graham, M. Ledbetter, S. Rajendran, and A. O. Sushkov, Proposal for a Cosmic Axion Spin Precession Experiment (CASPEr), Physical Review X 4, 021030 (2014).
  * Abel  _et al._ [2017] C. Abel, N. J. Ayres, G. Ban, G. Bison, K. Bodek, V. Bondar, _et al._ , Search for axion-like dark matter through nuclear spin precession in electric and magnetic fields, Physical Review X 7, 041034 (2017).
  * Roussy  _et al._ [2021] T. S. Roussy, D. A. Palken, W. B. Cairncross, B. M. Brubaker, D. N. Gresh, M. Grau, _et al._ , Experimental Constraint on Axionlike Particles over Seven Orders of Magnitude in Mass, Physical Review Letters 126, 171301 (2021).
  * Piegsa [2013] F. M. Piegsa, New Concept for a Neutron Electric Dipole Moment Search using a Pulsed Beam, Physical Review C 88, 045502 (2013).
  * Chanel  _et al._ [2019] E. Chanel, Z. Hodge, D. Ries, I. Schulthess, M. Solar, T. Soldner, O. Stalder, J. Thorne, and F. M. Piegsa, The pulsed neutron Beam EDM experiment, EPJ Web of Conferences 219, 02004 (2019).
  * Ramsey [1949] N. F. Ramsey, A New Molecular Beam Resonance Method, Physical Review 76, 996–996 (1949).
  * Ramsey [1950] N. F. Ramsey, A Molecular Beam Resonance Method with Separated Oscillating Fields, Physical Review 78, 695–699 (1950).
  * Abele  _et al._ [2006] H. Abele _et al._ , Characterization of a ballistic supermirror neutron guide, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 562, 407–417 (2006).
  * Klein and Schmidt [2011] M. Klein and C. J. Schmidt, CASCADE, neutron detectors for highest count rates in combination with ASIC/FPGA based readout electronics, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 628, 9–18 (2011).
  * Dress  _et al._ [1977] W. B. Dress, P. D. Miller, J. M. Pendlebury, P. Perrin, and N. F. Ramsey, Search for an electric dipole moment of the neutron, Physical Review D 15, 9–21 (1977).
  * S. Butterworth [1930] S. Butterworth, On the Theory of Filter Amplifiers, Experimental Wireless and the Wireless Engineer 7, 536–541 (1930).
  * Piegsa  _et al._ [2020] F. M. Piegsa _et al._ , New Neutron Electric Dipole Moment Search using a Pulsed Beam, Institut Laue-Langevin (ILL) doi:10.5291/ILL-DATA.3-07-393 (2020).
  * Lomb [1976] N. R. Lomb, Least-squares frequency analysis of unequally spaced data, Astrophysics and Space Science 39, 447–462 (1976).
  * Zechmeister and Kürster [2009] M. Zechmeister and M. Kürster, The generalised Lomb-Scargle periodogram: A new formalism for the floating-mean and Keplerian periodograms, Astronomy & Astrophysics 496, 577–584 (2009).
  * Press and Rybicki [1989] W. H. Press and G. B. Rybicki, Fast algorithm for spectral analysis of unevenly sampled data, The Astrophysical Journal 338, 277 (1989).
  * VanderPlas [2018] J. T. VanderPlas, Understanding the Lomb-Scargle Periodogram, The Astrophysical Journal Supplement Series 236, 16 (2018).
  * Foster  _et al._ [2018] J. W. Foster, N. L. Rodd, and B. R. Safdi, Revealing the Dark Matter Halo with Axion Direct Detection, Physical Review D 97, 123006 (2018).
  * Centers  _et al._ [2021] G. P. Centers _et al._ , Stochastic fluctuations of bosonic dark matter, Nature Communications 12, 7321 (2021).
  * Weber and de Boer [2010] M. Weber and W. de Boer, Determination of the local dark matter density in our Galaxy, Astronomy & Astrophysics 509, A25 (2010).
  * Catena and Ullio [2010] R. Catena and P. Ullio, A novel determination of the local dark matter density, Journal of Cosmology and Astroparticle Physics 2010 (08), 004–004.
  * Note [1] This corresponds to a value of $\rho_{\text{{DM}}}=3.1\times 10^{-42}$ GeV4 in natural units.
  * Garcon  _et al._ [2019] A. Garcon _et al._ , Constraints on bosonic dark matter from ultralow-field nuclear magnetic resonance, Science Advances 5, eaax4539 (2019).
  * Corasaniti  _et al._ [2017] P. S. Corasaniti, S. Agarwal, D. J. E. Marsh, and S. Das, Constraints on dark matter scenarios from measurements of the galaxy luminosity function at high redshifts, Physical Review D 95, 083512 (2017).
  * Blum  _et al._ [2014] K. Blum, R. T. D’Agnolo, M. Lisanti, and B. R. Safdi, Constraining axion dark matter with Big Bang Nucleosynthesis, Physics Letters B 737, 30–33 (2014).
  * Stadnik and Flambaum [2015] Y. V. Stadnik and V. V. Flambaum, Can Dark Matter Induce Cosmological Evolution of the Fundamental Constants of Nature?, Physical Review Letters 115, 201301 (2015).
  * Raffelt [1990] G. G. Raffelt, Astrophysical methods to constrain axions and other novel particle phenomena, Physics Reports 198, 1–113 (1990).
  * Graham and Rajendran [2013] P. W. Graham and S. Rajendran, New Observables for Direct Detection of Axion Dark Matter, Physical Review D 88, 035023 (2013).
  * Note [2] As an example: an oscillating magnetic field amplitude of $B_{a}=10$ pT which is typical for this frequency range (see Fig. 3d) translates with Eq. (4) and $E=2\times 35$ kV/cm into an EDM amplitude of $d_{n}=8.6\times 10^{-24}\,e\,\text{cm}$. With the use of Eq. (6) this can be translated into a ALP-gluon-coupling of $C_{G}/f_{a}=1.4\times 10^{-11}$ GeV-1 for an ALP mass of $m_{a}=10^{-15}$ eV. It follows $C_{G}/f_{a}m_{a}=1.4\times 10^{13}$ GeV-2.
  * Savitzky and Golay [1964] A. Savitzky and M. J. E. Golay, Smoothing and Differentiation of Data by Simplified Least Squares Procedures., Analytical Chemistry 36, 1627–1639 (1964).
  * Note [3] Note, other experimental constraints are sometimes presented using the following relation: $g_{a}=(3.7\pm 1.5)\times 10^{-3}\left(\frac{C_{G}}{f_{a}}\right)\frac{1}{\mathrm{GeV}}$ [12, 49].
  * O’Hare [2020] Ciaran O’Hare, cajohare/AxionLimits: AxionLimits (2020), version Number: v1.0.