11institutetext:  Max-Planck-Institut für Sonnensystemforschung, Justus-von-
Liebig-Weg 3, 37077 Göttingen, Germany
11email<EMAIL_ADDRESS>22institutetext: Instituto de Astrofísica de
Andalucía (IAA-CSIC), Apartado de Correos 3004, E-18080 Granada, Spain
33institutetext: Univ. Paris-Sud, Institut d’Astrophysique Spatiale, UMR 8617,
CNRS, Bâtiment 121, 91405 Orsay Cedex, France 44institutetext: Instituto
Nacional de Técnica Aeroespacial, Carretera de Ajalvir, km 4, E-28850 Torrejón
de Ardoz, Spain 55institutetext: Universitat de València, Catedrático José
Beltrán 2, E-46980 Paterna-Valencia, Spain 66institutetext: Leibniz-Institut
für Sonnenphysik, Schöneckstr. 6, 79104 Freiburg, Germany 77institutetext:
Institut für Datentechnik und Kommunikationsnetze der TU Braunschweig, Hans-
Sommer-Str. 66, 38106 Braunschweig, Germany 88institutetext: University of
Barcelona, Department of Electronics, Carrer de Martí i Franquès, 1 - 11,
08028 Barcelona, Spain 99institutetext: Instituto Universitario ”Ignacio da
Riva”, Universidad Politécnica de Madrid, IDR/UPM, Plaza Cardenal Cisneros 3,
E-28040 Madrid, Spain 1010institutetext: Institut für Astrophysik, Georg-
August-Universität Göttingen, Friedrich-Hund-Platz 1, 37077 Göttingen, Germany
1111institutetext: Fraunhofer-Institut für Kurzzeitdynamik, Ernst-Mach-
Institut, EMI Ernst-Zermelo-Str. 4, 79104 Freiburg, Germany

# Reconstruction of total solar irradiance variability as simultaneously
apparent from Solar Orbiter and Solar Dynamics Observatory

K.L. Yeo Corresponding author: K.L. Yeo11<EMAIL_ADDRESS>N.A. Krivova 11 S.K.
Solanki 11 J. Hirzberger 11 D. Orozco Suárez 22 K. Albert 11 N. Albelo Jorge
11 T. Appourchaux 33 A. Alvarez-Herrero 44 J. Blanco Rodríguez 55 A. Gandorfer
11 P. Gutierrez-Marques 11 F. Kahil 11 M. Kolleck 11 J.C. del Toro Iniesta R.
Volkmer J. Woch 226611 B. Fiethe 77 I. Pérez-Grande 99 E. Sanchis Kilders 55
M. Balaguer Jiménez 22 L.R. Bellot Rubio 22 D. Calchetti 11 M. Carmona 88 A.
Feller 11 G. Fernandez-Rico A. Fernández-Medina 119944 P. García Parejo 44
J.L. Gasent Blesa 55 L. Gizon B. Grauf 11101011 K. Heerlein 11 A. Korpi-Lagg
11 T. Maue 661111 R. Meller 11 A. Moreno Vacas 22 R. Müller 11 E. Nakai 66 W.
Schmidt 66 J. Schou 11 J. Sinjan 11 J. Staub 11 H. Strecker 22 I. Torralbo 99
G. Valori 11

Solar irradiance variability has been monitored almost exclusively from the
Earth’s perspective. We present a method to combine the unprecedented
observations of the photospheric magnetic field and continuum intensity from
outside the Sun-Earth line, which is being recorded by the Polarimetric and
Helioseismic Imager on board the Solar Orbiter mission (SO/PHI), with solar
observations recorded from the Earth’s perspective to examine the solar
irradiance variability from both perspectives simultaneously. Taking SO/PHI
magnetograms and continuum intensity images from the cruise phase of the Solar
Orbiter mission and concurrent observations from the Helioseismic and Magnetic
Imager onboard the Solar Dynamics Observatory (SDO/HMI) as input into the
SATIRE-S model, we successfully reconstructed the total solar irradiance
variability as apparent from both perspectives. In later stages of the SO
mission, the orbital plane will tilt in such a way as to bring the spacecraft
away from the ecliptic to heliographic latitudes of up to $33^{\circ}$. The
current study sets the template for the reconstruction of solar irradiance
variability as seen from outside the ecliptic from data that SO/PHI is
expected to collect from such positions. Such a reconstruction will be
beneficial to factoring inclination into how the brightness variations of the
Sun compare to those of other cool stars, whose rotation axes are randomly
inclined.

###### Key Words.:

Sun:activity – Sun: faculae, plages – Sun: magnetic fields – sunspots

##  1 Introduction

Solar irradiance variability is a key natural driver of climate change (Gray
et al., 2010; Solanki et al., 2013) and serves as a template for understanding
the brightness variability of other cool stars (Foukal, 1993; Radick et al.,
2018). The variability at timescales of days to the solar cycle is established
to be mainly driven by photospheric magnetism (Shapiro et al., 2017; Yeo et
al., 2017).

The photospheric magnetic field is partly confined in kilogauss-strength
concentrations, which are manifested as bright faculae and network
(hereinafter referred to collectively as faculae) and as dark sunspots (Spruit
& Roberts, 1983). Solar irradiance fluctuates as the time evolution of the
photospheric magnetic field and solar rotation change the prevalence and
spatial distribution of bright faculae and dark sunspots on the solar disc.
The intensity excess produced by an individual facular feature varies with
time, as its magnetic field emerges, decays, coalesces, or is cancelled out by
nearby magnetic flux (time evolution). Likewise, the intensity deficit
produced by a given sunspot is modulated by the time evolution of its magnetic
field. At the same time, the intensity excess or deficit produced by the
feature is modulated by the combination of solar rotation moving it across the
solar disc and the variation in its intensity contrast and projected area with
distance from disc centre (rotational modulation).

The photospheric magnetic field is spatially heterogeneous such that the
magnetism on the solar disc, and consequently solar irradiance variability,
can appear to be different to observers looking at the Sun from different
directions. How solar irradiance variability might change with the
heliographic latitude of the observer is of particular interest as it is
relevant to how the brightness variations of the Sun compare to those of other
cool stars, which come in all inclinations. Unfortunately, solar irradiance
has so far only been monitored from Earth-orbiting satellites (Ermolli et al.,
2013; Kopp, 2014) and from the Mars-orbiting MAVEN satellite (Jakosky et al.,
2015). Although models have been developed to reconstruct solar irradiance
variability from observations related to solar magnetism such as activity
indices and full solar disc imagery (see reviews by Domingo et al., 2009; Yeo
et al., 2014a; Chatzistergos et al., 2022), before the Solar Orbiter mission
(Müller et al., 2020), no suitable data had been recorded from outside the
Sun-Earth line. Consequently, attempts to determine how solar variability
might appear from outside the ecliptic have no alternatives but to extrapolate
this information from models of the Sun or solar observations recorded from
the ecliptic (Schatten, 1993; Knaack et al., 2001; Vieira et al., 2012;
Shapiro et al., 2014; Nèmec et al., 2020b, a).

The Solar Orbiter mission, launched in February 2020, is set to change this
state of affairs. The spacecraft is orbiting the Sun with a period of about
180 days such that, over the course of each orbit, it assumes a range of
positions outside the Sun-Earth line. At present, the orbital plane is close
to the ecliptic. In the later stages of the Solar Orbiter mission, the orbital
plane will tilt to bring the spacecraft away from the ecliptic to heliographic
latitudes of up to $33^{\circ}$. While there is no solar irradiance monitor on
board, the payload includes the Polarimetric and Helioseismic Imager (SO/PHI,
Solanki et al., 2020), which returns, amongst other data products, full-disc
magnetograms and continuum images suitable for reconstructing solar irradiance
variability. The out-of-ecliptic observations SO/PHI is expected to return
will present the unique opportunity to reconstruct solar irradiance
variability as apparent from outside the ecliptic from actual observations
recorded from such positions. Such a reconstruction is missing from existing
attempts to factor inclination into how the brightness variations of the Sun
compare to those of other cool stars (Schatten, 1993; Knaack et al., 2001;
Vieira et al., 2012; Shapiro et al., 2014; Nèmec et al., 2020b, a).

We present a method to augment the unprecedented observations SO/PHI is
recording from outside the Sun-Earth line with solar observations recorded
from the Earth’s perspective in order to examine solar irradiance variability
from both perspectives. This is a preliminary study aimed at setting the
template for the reconstruction of solar irradiance variability from the out-
of-ecliptic observations expected from SO/PHI in the later stages of the Solar
Orbiter mission. Taking SO/PHI full-disc magnetograms and continuum images
recorded during the cruise phase of the mission and concurrent observations
from the Helioseismic and Magnetic Imager onboard the Earth-orbiting Solar
Dynamics Observatory (SDO/HMI, Scherrer et al., 2012), we reconstruct total
solar irradiance (TSI) variability as apparent simultaneously from both
perspectives. For this purpose, we made use of the Spectral And Total
Irradiance REconstructions for the Satellite era model (SATIRE-S, Fligge et
al., 2000; Krivova et al., 2003; Yeo et al., 2014b, a, 2015), an established
model of solar irradiance variability that has been successfully applied to
such data from various solar telescopes, including SDO/HMI, to reconstruct
this quantity. In the following, we describe the data set (Sect. 2.1) and the
SATIRE-S model (Sect. 2.2) and we describe how it is adapted here to the
purposes of the current study (Sect. 2.3), and the results (Sect. 3). Finally,
we give our concluding remarks (Sect. 4).

##  2 Model

The SATIRE-S model computes the corresponding total and spectral solar
irradiance from a given full-disc line-of-sight magnetogram and the
simultaneous continuum intensity image, hereinafter referred to as an image
pair. For this preliminary study, we confine the scope to total solar
irradiance. We input concurrent SO/PHI and SDO/HMI image pairs into the model
to recover TSI variability as seen from both perspectives.

###  2.1 Data selection and reduction

Figure 1: Flight path of SO/PHI (blue lines) relative to the Sun (black cross)
and SDO/HMI (red cross) over the observation periods of P1, P2, and P3 as seen
from above the north pole of the Sun. From this perspective, SO/PHI is moving
anti-clockwise about the Sun towards the Sun-SDO/HMI line (black line) and the
angle subtended by SO/PHI and SDO/HMI from the Sun represents the separation
in heliographic longitude. The solid grey lines mark the annotated angular
distances and the dashed grey lines the annotated distances from the Sun. The
separation in heliographic latitude, hidden in this perspective, is relatively
minute, ranging from $3.7^{\circ}$ to $7.6^{\circ}$. Simultaneous SO/PHI and
SDO/HMI magnetograms from 05.09.2021 and 09.09.2021 (blue plus symbols) are
depicted in Fig. 4.

The SO/PHI instrument images the full solar disc with one of its two
telescopes, the Full Disc Telescope (FDT). The FDT records polarised full-disc
images in bandpasses centred on six wavelength positions, five within the
magnetically-sensitive Fe I 6173 Å line, and one in the nearby continuum, on a
$2048\times 2048$ pixel CMOS sensor. The pixel scale is 3.57 arcsec. From the
filtergram observations, various data products are generated, including vector
and line-of-sight magnetograms, images in the continuum near the Fe I 6173 Å
line, and Dopplergrams (Albert et al., 2020).

Over the periods of 01.09.2021 to 10.09.2021, 26.09.2021 to 04.10.2021, and
08.10.2021 to 12.10.2021, denoted P1, P2, and P3, the FDT returned an image
pair (i.e. simultaneous full-disc line-of-sight magnetogram and continuum
intensity image) every two hours, with interruptions at certain times. The
Sun-Solar Orbiter distance ranged from about 0.6 au to 0.7 au (Fig. 1), such
that each image pixel corresponded to an area of about $1600\ {\rm km}\times
1600\ {\rm km}$ to $1800\ {\rm km}\times 1800\ {\rm km}$ on the solar disc. At
this distance, the entire solar disc laid within a $1024\times 1024$ pixel
window in the middle of the field-of-view (FoV). For this reason, the
instrument returned this window alone, producing $1024\times 1024$ pixel
images. The image pairs with clear instrumental artefacts, such as missing
data, are excluded, leaving 173 image pairs, about seven per day on average.
At the time of study, the data reduction remains preliminary.

Similar to the FDT on SO/PHI, SDO/HMI records polarised full-disc images in
six filters within the Fe I 6173 Å line, from which similar data products are
generated at 45-s and 720-s cadences. The CCD/image size is $4096\times 4096$
pixel and the pixel scale is 0.5 arcsec, such that each image pixel
corresponds to an area of about $360\ {\rm km}\times 360\ {\rm km}$ on the
solar disc.

We refer, by the term SO/PHI, to the FDT on this instrument. We isolate the
SDO/HMI 720-s line-of-sight magnetogram and continuum image (hmi.M_720s and
hmi.Ic_720s data products) concurrent to each SO/PHI image pair taking into
account that the light travel time from the Sun to SO/PHI is shorter than that
to SDO/HMI. The average difference in the time of observation between the
SDO/HMI and SO/PHI image pairs is about four minutes. To match the SDO/HMI
data set to the SO/PHI data set, taking each SDO/HMI image, the solar disc is
binned down to the size of the solar disc in the corresponding SO/PHI image.
Then, we resampled the SDO/HMI image in such a way so as to account for solar
rotation in the time interval between the two images. From P1 to P3, SO/PHI
approached the Sun-SDO/HMI line such that the angular distance between SO/PHI
and SDO/HMI from the Sun declined steadily from $67.3^{\circ}$ to
$12.2^{\circ}$ (see Fig. 1).

###  2.2 SATIRE-S model

Figure 2: TSI excess produced by a given facular feature in SATIRE-S (blue
solid line) and the facular intensity contrast adopted by the model (blue
dashed line), from Unruh et al. (1999), as a function of distance from disc
centre, $\frac{r}{r_{\odot}}$. The red solid and dashed lines represent the
same for sunspots. The black dashed line follows the projected area of a given
solar surface feature (i.e., foreshortening). Each quantity is normalised to
the disc centre value (black solid line). The model treats sunspot umbra and
sunspot penumbra separately, and assumes an umbral to penumbral area ratio of
1:4. For each quantity depicted, the sunspot profile is given by the mean of
the umbral and penumbral profiles, weighted by this ratio.

The SATIRE-S model describes the variation in solar irradiance due to sunspot
darkening and facular and network brightening. The model does not distinguish
between faculae and network, which are collectively termed ’faculae’. The
model has two components. The first component is the solar disc coverage by
sunspot umbra, sunspot penumbra, and faculae at a given time, derived by
identifying these features in the image pair (line-of-sight magnetogram and
continuum intensity image) recorded then. The solar disc outside of sunspots
and faculae is classified as internetwork. The second component is the
intensity spectra of sunspot umbra, sunspot penumbra, faculae, and the
internetwork at various distances from disc centre, calculated from models of
their atmospheric structure with a radiative transfer code (Unruh et al.,
1999). Integrating each intensity spectrum over the full wavelength range, 115
nm to 170,000 nm, yields the corresponding bolometric intensity.

Hereinafter, we will refer to the difference between facular and internetwork
bolometric intensity, depicted in Fig. 2 (blue dashed line), as the facular
intensity contrast. Likewise, we will refer to the difference between sunspot
and internetwork bolometric intensity (red dashed line) as the sunspot
intensity contrast. Going from disc centre to limb, the facular intensity
contrast increases steadily from the disc centre level (black solid line)
while the sunspot intensity contrast decreases. To recover TSI, the model
assigns to each point on the solar disc the appropriate bolometric intensity
depending on whether it is inside the umbra or penumbra of a sunspot, faculae,
or the internetwork and the distance from disc centre. The summation of the
result over the solar disc yields TSI.

Sunspots are distinguishable in continuum images by the negative intensity
contrast and faculae are distinguishable in magnetograms from noise by the
elevated magnetogram signals. Let $B_{\rm LOS}$ denote the line-of-sight
magnetogram signal, which indicates the line-of-sight component of the mean
magnetic flux density over each resolution element. For a given image pair,
the points on the solar disc where the intensity contrast is below a certain
threshold are classified as sunspots and the points where $\left|B_{\rm
LOS}\right|$ is above a certain threshold and not already counted as sunspots
are taken to correspond to faculae.

The magnetic concentrations that comprise faculae and network remain largely
unresolved at the spatial resolution of available full-disc magnetograms,
including SO/PHI and SDO/HMI. In SATIRE-S, the filling factor of facular
pixels, that is, the effective proportion of each resolution element occupied
by faculae, is estimated by means of an empirical relationship. Let $\mu$
denote the cosine of the heliocentric angle. Taking into consideration that
magnetic flux tubes tend towards a surface normal orientation due to magnetic
buoyancy, the quantity $B_{\rm LOS}/\mu$ approximates the mean magnetic flux
density over each resolution element. The faculae filling factor is taken to
scale linearly with $B_{\rm LOS}/\mu$ from zero at 0 G to unity at what is
denoted $\left(B_{\rm LOS}/\mu\right)_{\rm sat}$, above which it saturates.
The $\left(B_{\rm LOS}/\mu\right)_{\rm sat}$ parameter modulates the facular
area and, thus, the amplitude of the facular brightening as well.

The TSI excess or deficit produced by a given facular or sunspot feature
depends on the projected area and intensity contrast. The projected area
changes with its time evolution and its passage across the solar disc with
solar rotation (i.e. foreshortening) and the intensity contrast with the
latter due to the centre-to-limb variation (CLV) of this quantity. The
SATIRE-S model, by establishing the disc-coverage by faculae and sunspots and
taking the CLV of facular, sunspot, and internetwork intensity into account
(Fig. 2), captures these processes.

###  2.3 TSI reconstruction

Figure 3: Total solar irradiance (TSI) over P1, P2 and P3. The blue and red
lines correspond to the SATIRE-S model reconstruction of TSI at 2h cadence
based on the SO/PHI and SDO/HMI data sets, respectively. The black line traces
the concurrent measurements from TSIS-1/TIM: the 6h cadence record is
interpolated to the times of the model reconstruction.

We apply the SATIRE-S model to the SDO/HMI and SO/PHI data sets to reconstruct
TSI variability as apparent from both perspectives. In the model, the
intensity contrast threshold determines the disc coverage by sunspots while
the $\left|B_{\rm LOS}\right|$ threshold and $\left(B_{\rm
LOS}/\mu\right)_{\rm sat}$ determine the disc coverage by faculae. While
SO/PHI and SDO/HMI survey the same spectral line, they do differ in instrument
design and data reduction, most notably, the significant difference in spatial
resolution (Sect. 2.1). The result is systematic differences in the apparent
sunspot intensity contrast, magnetogram noise, and magnetogram signal scale
between the two data sets. The intensity contrast and $\left|B_{\rm
LOS}\right|$ thresholds and $\left(B_{\rm LOS}/\mu\right)_{\rm sat}$ have to
be adapted to each data set to account for these differences. In earlier
implementations of the SATIRE-S model (Krivova et al., 2003; Yeo et al.,
2014b), the intensity contrast threshold is set at the level that brings the
resultant sunspot area into agreement with independent sunspot area
measurements, and the $\left|B_{\rm LOS}\right|$ threshold is set at the
3$\sigma$ magnetogram noise level. The $\left(B_{\rm LOS}/\mu\right)_{\rm
sat}$ parameter is set at the level that matches the amplitude of TSI
variability in the model output to that in TSI measurements, making it a free
parameter of the model. However, available sunspot area and TSI measurements
are all recorded from the Earth’s perspective, meaning we cannot apply the
above procedure as is to the SO/PHI data set. We modified the procedure of
Krivova et al. (2003) and Yeo et al. (2014b) in such a way so we can set the
various parameters at the appropriate level for each data set, while
circumventing the issue that there are no sunspot area and TSI measurements
from PHI’s perspective. The objective is to ensure that the disc coverage by
faculae and sunspots is derived from the SDO/HMI and SO/PHI data sets
consistently; in turn, this ensures that the model output based on the two are
consistent. In the following analysis, unless specified, we make use of all
available data, namely, P1, P2, and P3, in order to maximise the statistics.

The continuum images were normalised by their limb-darkening profile,
determined following Neckel & Labs (1994). Taking each SDO/HMI (SO/PHI)
continuum image, we classified the points on the solar disc where the
normalised intensity lies below 0.91 (0.94) as sunspots. For SDO/HMI, the
threshold is set at 0.91 by finding the level where the resultant disc-
integrated sunspot area complies with the independent record by Mandal et al.
(2020). The fact that SO/PHI is outside the Sun-Earth line excludes a similar
comparison to available sunspot area measurements. Instead, the threshold was
set at 0.94 by finding the level where the foreshortening-corrected sunspot
area measured by SO/PHI matches that by SDO/HMI over the part of the solar
surface imaged by both instruments. The sunspot pixels within each image are
ordered by the normalised intensity and the points below (above) the first
quintile are taken to correspond to sunspot umbra (penumbra). This sets the
umbral to penumbral area ratio at 1:4, in compliance with independent sunspot
observations (Solanki, 2003).

Taking each SDO/HMI (SO/PHI) magnetogram, the points on the solar disc where
$\left|B_{\rm LOS}\right|$ is above 45 G (31 G) and not already identified as
sunspots are taken to correspond to faculae. Standalone facular pixels are
reclassified as internetwork to minimise the wrongful inclusion of magnetogram
noise as faculae. We determined the appropriate $B_{\rm LOS}$ threshold to
separate faculae from magnetogram noise by the following procedure.

For SO/PHI, we determined the magnetogram noise level as a function of
position in the FOV following the procedure of Ortiz et al. (2002) and Yeo et
al. (2013). This revealed the 1$\sigma$ noise level to vary within the range
of 5.0 G and 10.2 G. Conservatively, we set the $B_{\rm LOS}$ threshold for
SO/PHI at 31 G, three times the upper limit of this range. Compared to the
SO/PHI magnetograms, the SDO/HMI magnetograms are less noisy, due mainly to
the longer integration time and the fact that we had resampled them so that
the size of the solar disc matches that in the former (Sect. 2.1).
Consequently, there are weak facular features that are distinguishable from
noise in the SDO/HMI magnetograms but obscured by noise in the SO/PHI
magnetograms. Therefore, instead of setting the $B_{\rm LOS}$ threshold for
SDO/HMI at its 3$\sigma$ noise level like we did for SO/PHI, we set it at the
level that ensures we are isolating only the facular features that SO/PHI can
also distinguish from noise. Confining ourselves to P3, we rotated the solar
disc in the SDO/HMI magnetograms to SO/PHI’s perspective. We let $\mu_{\rm
HMI}$ and $\mu_{\rm PHI}$ denote, for a given point on the solar surface, the
value of $\mu$ in SDO/HMI’s and in SO/PHI’s perspective, respectively. We
scaled each point on the solar disc in the rotated SDO/HMI magnetogram by
$\frac{\mu_{\rm PHI}}{\mu_{\rm HMI}}$. We refer to the result (an estimate of
what the SDO/HMI instrument would have recorded if it was in SO/PHI’s
position) as the projected SDO/HMI magnetograms. This step is more reliable
when the two perspectives are closer, which is why we confined this analysis
to P3. We set the $B_{\rm LOS}$ threshold for SDO/HMI at 45 G by finding the
value where the number of facular pixels in the projected SDO/HMI magnetograms
matches that in the SO/PHI magnetograms over the part of the solar surface
imaged by both instruments.

The free parameter in the model, $\left(B_{\rm LOS}/\mu\right)_{\rm sat}$ is
set at 94 G for the SO/PHI-based reconstruction and at 146 G for the SDO/HMI-
based reconstruction. Recall, $\left(B_{\rm LOS}/\mu\right)_{\rm sat}$
modulates the amplitude of facular brightening (Sect. 2.2). For the SDO/HMI-
based reconstruction, $\left(B_{\rm LOS}/\mu\right)_{\rm sat}$ is constrained
by matching the amplitude of TSI variability in the model output to that in
observed TSI. For this purpose, we make use of the TSI record from the Total
Irradiance Monitor (TIM) on the Total and Spectral Solar Irradiance Sensor
(TSIS-1) suite onboard the International Space Station111Version 3, available
at lasp.colorado.edu/home/tsis/data/. We emphasise that by this step, the
SDO/HMI-based reconstruction (red, Fig. 3) merely adopts the amplitude of TSI
variability in the TSIS-1/TIM record (black). No adjustment of the free
parameter can change the variation in TSI with time in the model output to
match that in the TSIS-1/TIM record. As illustrated, the SDO/HMI-based
reconstruction closely reproduces the TSIS-1/TIM record: the Pearson’s
correlation coefficient, $R^{2}$ is 0.981, and the root-mean-square
difference, RMSD, is $0.0361\ {\rm Wm^{-2}}$. The fact that SO/PHI is outside
the Sun-Earth line excludes a similar comparison to available TSI
measurements. To circumvent this issue, again we compared the SO/PHI
magnetograms from P3 to the result of projecting the corresponding SDO/HMI
magnetograms to SO/PHI’s perspective. For the SO/PHI-based reconstruction,
$\left(B_{\rm LOS}/\mu\right)_{\rm sat}$ is constrained by matching the disc-
integrated faculae filling factor in the SO/PHI magnetograms to that in the
projected SDO/HMI magnetograms over the part of the solar surface imaged by
both instruments. The SO/PHI-based reconstruction is shown in Fig. 3 (blue)
along with the SDO/HMI-based reconstruction (red) and the TSIS-1/TIM record
(black). Of course, due to the difference in perspective, the SO/PHI time
series is not expected to match the other two time series.

##  3 Results

Figure 4: Simultaneous SO/PHI (left) and SDO/HMI line-of-sight magnetogram
(right) recorded during P1 on 05.09.2021 (top), and 09.09.2021 (bottom). The
position of SO/PHI relative to the Sun-SDO/HMI line at these times is
indicated in Fig. 1 (blue plus symbols). The part of the solar surface within
view of both instruments is encircled in red. The five active regions (ARs)
that emerged within P1 and were imaged by both instruments are encircled in
blue. The NOAA AR number is indicated in the top-left panel. The magnetic
features marked ”A” and ”B” correspond to remnants of ARs which dissipated
before P1, and ”C” and ”D” to ARs which do not meet the aforementioned
criteria, discussed in Appendix A. The greyscale is saturated for the SO/PHI
and SDO/HMI magnetograms at $\pm 31\ {\rm G}$ and $\pm 45\ {\rm G}$,
respectively. This corresponds to the magnetogram signal threshold applied to
separate faculae from magnetogram noise (Sect. 2.3). Figure 5: Properties of
the five active regions that emerged during P1 and imaged by both SO/PHI and
SDO/HMI (outlined by blue contours in Fig. 4), as a function of time: a) the
distance from disc centre, $\frac{r}{r_{\odot}}$, b) foreshortening-corrected
facular area, $A_{\rm F}$ in millionths of the solar hemisphere, $\mu{\rm
Hem}$, c) the change in TSI due to faculae, $\rm FB$. Rows d) and e) represent
the same as rows b) and c), except for sunspots. In row f), we chart the total
change in TSI (i.e. both faculae brightening and sunspot darkening) produced
by each active region to an observer that follows it such that it is always at
disc centre, ${\rm\Delta{}TSI_{DC}}$. The blue (red) solid lines follow the
values from the SO/PHI (SDO/HMI) data set and the corresponding TSI
reconstruction. See Sect. 3 for details.

As discussed in the previous section, we constrained the SO/PHI-based model by
optimising the agreement between the disc coverage by faculae and sunspots
derived from the SO/PHI and SDO/HMI data sets (Sect. 2.3). Of course, even
with such a step, there will be residual discrepancies between the disc
coverage by faculae and sunspots derived from the two data sets. In this
section, we examine the extent to which faculae and sunspots are mapped
consistently between the two data sets, and the uncertainty in the model
output TSI produced by the inevitable uncertainty in the disc coverage by
faculae and sunspots.

To this end, we identify the active regions (ARs) that were imaged by both
instruments and examine the area of the enclosed faculae and sunspots and
their contribution to the TSI variability reproduced by the model. We confine
this analysis to P1, where the two perspectives are furthest apart. In Sect.
2.3, we made use of the data from P3, where the two perspectives are closest
to one another, to constrain the $B_{\rm LOS}$ threshold and $\left(B_{\rm
LOS}/\mu\right)_{\rm sat}$, setting them at the levels that brought the disc
coverage by faculae in the two data sets into agreement over this period.
There is a need to verify that faculae continue to be mapped consistently
between the two data sets outside of this period, especially when the two
perspectives are far apart.

Taking the SO/PHI and SDO/HMI data sets, we identified and established the
boundary of ARs that had emerged during P1 and were imaged by both instruments
(Appendix A), isolating five such ARs (blue contours, Fig. 4). We examine each
AR over the period where it contains sunspots and lies entirely within the
solar disc from a given perspective. We let $\frac{r}{r_{\odot}}$ denote the
distance of the magnetic centre-of-mass of a given AR from disc centre,
normalised to the solar radius. The trajectory of the various ARs across the
solar disc as seen from both perspectives, in terms of $\frac{r}{r_{\odot}}$,
is given in Fig. 5a. Compared to P1, periods P2 and P3 are shorter and solar
activity was weaker such that there are only two relatively small and weak
emerging ARs imaged by both instruments and even then only for a limited
period of time. This is another reason the current analysis is confined to P1.

From the disc-coverage by faculae and sunspots determined from the SO/PHI and
SDO/HMI data sets, we derive the foreshortening-corrected facular ($A_{\rm
F}$, Fig. 5b) and sunspot area within each AR ($A_{\rm S}$, Fig. 5d).
Comparing the $A_{\rm F}$ values from the SO/PHI (blue) and SDO/HMI data sets
(red), the $R^{2}$ is 0.956 and the RMSD is 88.9 $\mu$Hem. In the case of
$A_{\rm S}$, the $R^{2}$ is 0.939 and the RMSD is 52.3 $\mu$Hem. The values
from the two data sets are fairly similar, verifying the steps in Sect. 2.3 to
ensure we map faculae and sunspots consistently between the two data sets.

For each AR and from each perspective, we compute the contribution to TSI
variability by the enclosed faculae (facular brightening, $\rm FB$, Fig. 5c)
and by the enclosed sunspots (sunspot darkening, $\rm SD$, Fig. 5e). To derive
the $\rm FB$ of a given AR, we repeat the TSI reconstruction, treating the
faculae within this AR as internetwork and subtracting the result from the
original reconstruction, likewise for $\rm SD$. The discrepancies between the
$\rm FB$ and $\rm SD$ values from the SO/PHI-based (blue) and SDO/HMI-based
reconstructions (red) is due to the uncertainty in the disc coverage by
faculae and sunspots, just discussed, and rotational modulation. The
trajectory of the AR across the solar disc (Fig. 5a), and therefore the effect
of rotational modulation on $\rm FB$ and $\rm SD$, differs between the two
perspectives.

Next, we examine the discrepancy between the SO/PHI-based and SDO/HMI-based
reconstructions due to the uncertainty in the disc coverage by faculae and
sunspots. First, we have to account for the differences between the two due to
rotational modulation. Let $\rm FB_{\rm DC}$ and $\rm SD_{\rm DC}$ denote, for
a given AR, the $\rm FB$ and $\rm SD$ to an observer that follows it such that
it is always at disc centre, namely, no rotational modulation. In Fig. 2, we
chart the CLV of facular intensity contrast in SATIRE-S (blue dashed line),
based on Unruh et al. (1999), and of the projected area of a given solar
surface feature (i.e. foreshortening, black dashed line), normalised to the
disc centre level (black solid line). The product of the two normalised CLV
profiles yields the normalised CLV of the $\rm FB$ produced by a given facular
feature in the model (blue solid line). In the same manner, we derive the
normalised CLV of the $\rm SD$ produced by a given sunspot (red solid line).
For faculae, the two effects roughly cancel out, such that the CLV of $\rm FB$
is weak, explaining why the $\rm FB$ produced by the various ARs is similar
from both perspectives, as illustrated in Fig. 5c. In contrast, for sunspots,
the CLV of the two effects are quantitatively similar such that $\rm SD$, in
absolute terms, exhibits a monotonic and increasingly steep decline towards
the limb. This, together with the uncertainty in $A_{\rm S}$, resulted in the
discrepancy, between the two perspectives, of the $\rm SD$ produced by the
various ARs (Fig. 5e). We use these results to estimate $\rm FB_{\rm DC}$ and
$\rm SD_{\rm DC}$ from $\rm FB$ and $\rm SD$. Finally, for each AR, we took
the sum of $\rm FB_{\rm DC}$ and $\rm SD_{\rm DC}$, yielding the TSI
variability produced by the AR to an observer that follows it such that it is
always at disc centre, ${\rm\Delta{}TSI_{DC}}$ (Fig. 5f). Comparing the
${\rm\Delta{}TSI_{DC}}$ values from the SO/PHI-based (blue) and SDO/HMI-based
reconstructions (red), the $R^{2}$ is 0.864 and the RMSD is $0.0638\ {\rm
Wm^{-2}}$.

Between the SO/PHI and SDO/HMI data sets, the disc coverages by faculae and
sunspots, at least within ARs, agree to about 4% and 6%, respectively, and
this corresponds to an agreement of about 86% between the SO/PHI-based and
SDO/HMI-based reconstructions of TSI variability.

##  4 Conclusion

In this study, we presented a method to combine the unprecedented observations
of the photospheric magnetic field and continuum intensity that SO/PHI is
recording from outside the Sun-Earth line with solar observations recorded
from the Earth’s perspective to examine solar irradiance variability from both
perspectives simultaneously. Taking SO/PHI observations from the cruise phase
of the Solar Orbiter mission and concurrent observations from SDO/HMI as input
into the SATIRE-S model, we successfully reconstructed total solar irradiance
(TSI) variability as apparent from both perspectives (Sects. 2 and 3).

In later stages of the Solar Orbiter mission, the orbital plane will tilt in
such a way so as to bring the spacecraft away from the ecliptic to
heliographic latitudes of up to $33^{\circ}$. The current study sets the
template for the reconstruction of solar irradiance variability as seen from
outside the ecliptic from the data SO/PHI is expected to collect from such
positions. In the absence of out-of-ecliptic observations of the Sun, such a
reconstruction is missing from existing attempts to factor inclination into
how the brightness variations of the Sun compares to that of other cool stars,
which come in all inclinations (Schatten, 1993; Knaack et al., 2001; Vieira et
al., 2012; Shapiro et al., 2014; Nèmec et al., 2020b, a).

This study is based on a relative small body of SO/PHI and SDO/HMI
observations and the reduction of the SO/PHI data is preliminary. The results
obtained here will have to be confirmed, in future efforts, with a larger set
of SO/PHI and SDO/HMI observations and the finalised reduction of the former.
Future efforts will also aim to reconstruct solar brightness variability in
the passbands employed in stellar surveys which monitor the photometry of
their program stars, for example, CoRot (Auvergne et al., 2009), Kepler
(Borucki et al., 2010), TESS (Ricker et al., 2015), and PLATO (Rauer et al.,
2014).

###### Acknowledgements.

The authors would like to thank the referee, Greg Kopp for his invaluable
comments on the manuscript. KLY and NAK received funding from BMBF (project
01LG1909C), and KLY and SKS from ERC through the Horizon 2020 research and
innovation program of the EU (grant 695075). The Solar Orbiter mission is an
international cooperation between ESA and NASA operated by ESA. We are
grateful to the ESA SOC and MOC teams for their support. The German
contribution to SO/PHI is funded by BMWK through DLR (grants 50 OT
1001/1201/1901 and 50 OT 0801/1003/1203/1703) and by MPG through its central
funds. The Spanish contribution is funded by AEI/MCIN/10.13039/501100011033
(RTI2018-096886-C5, PID2021-125325OB-C5, PCI2022-135009-2), ’ERDF A way of
making Europe´, the Severo Ochoa Center of Excellence accreditation awarded to
IAA-CSIC (SEV2017-0709, CEX2021-001131-S) and a Ramón y Cajal fellowship
awarded to DOS. The French contribution is funded by CNES. The HMI data are
courtesy of NASA/SDO and the HMI science team.

## References

  * Albert et al. (2020) Albert, K., Hirzberger, J., Kolleck, M., et al. 2020, J. Astron. Telesc. Instrum. Syst., 6, 048004
  * Auvergne et al. (2009) Auvergne, M., Bodin, P., Boisnard, L., et al. 2009, A&A, 506, 411
  * Borucki et al. (2010) Borucki, W. J., Koch, D., Basri, G., et al. 2010, Science, 327, 977
  * Chatzistergos et al. (2022) Chatzistergos, T., Krivova, N. A., & Ermolli, I. 2022, Front. Astron. Space Sci., 9, 1038949
  * Domingo et al. (2009) Domingo, V., Ermolli, I., Fox, P., et al. 2009, Space Sci. Rev., 145, 337
  * Ermolli et al. (2013) Ermolli, I., Matthes, K., Dudok de Wit, T., et al. 2013, Atmos. Chem. Phys., 13, 3945
  * Fligge et al. (2000) Fligge, M., Solanki, S. K., & Unruh, Y. C. 2000, A&A, 353, 380
  * Foukal (1993) Foukal, P. 1993, Sol. Phys., 148, 219
  * Gray et al. (2010) Gray, L. J., Beer, J., Geller, M., et al. 2010, Rev. Geophys., 48, RG4001
  * Jakosky et al. (2015) Jakosky, B. M., Lin, R. P., Grebowsky, J. M., et al. 2015, Space Sci. Rev., 195, 3
  * Knaack et al. (2001) Knaack, R., Fligge, M., Solanki, S. K., & Unruh, Y. C. 2001, A&A, 376, 1080
  * Kopp (2014) Kopp, G. 2014, J. Space Weather Space Clim., 4, A14
  * Krivova et al. (2003) Krivova, N. A., Solanki, S. K., Fligge, M., & Unruh, Y. C. 2003, A&A, 399, L1
  * Mandal et al. (2020) Mandal, S., Krivova, N. A., Solanki, S. K., Sinha, N., & Banerjee, D. 2020, A&A, 640, A78
  * Müller et al. (2020) Müller, D., St. Cyr, O. C., Zouganelis, I., et al. 2020, A&A, 642, A1
  * Neckel & Labs (1994) Neckel, H. & Labs, D. 1994, Sol. Phys., 153, 91
  * Nèmec et al. (2020a) Nèmec, N. E., Işık, E., Shapiro, A. I., et al. 2020a, A&A, 638, A56
  * Nèmec et al. (2020b) Nèmec, N. E., Shapiro, A. I., Krivova, N. A., et al. 2020b, A&A, 636, A43
  * Ortiz et al. (2002) Ortiz, A., Solanki, S. K., Domingo, V., Fligge, M., & Sanahuja, B. 2002, A&A, 388, 1036
  * Radick et al. (2018) Radick, R. R., Lockwood, G. W., Henry, G. W., Hall, J. C., & Pevtsov, A. A. 2018, ApJ, 855, 75
  * Rauer et al. (2014) Rauer, H., Catala, C., Aerts, C., et al. 2014, Exp. Astron., 38, 249
  * Ricker et al. (2015) Ricker, G. R., Winn, J. N., Vanderspek, R., et al. 2015, J. Astron. Telesc. Instrum. Syst., 1, 014003
  * Schatten (1993) Schatten, K. H. 1993, J. Geophys. Res., 98, 18907
  * Scherrer et al. (2012) Scherrer, P. H., Schou, J., Bush, R. I., et al. 2012, Sol. Phys., 275, 207
  * Shapiro et al. (2017) Shapiro, A. I., Solanki, S. K., Krivova, N. A., et al. 2017, Nat. Astron., 1, 612
  * Shapiro et al. (2014) Shapiro, A. I., Solanki, S. K., Krivova, N. A., et al. 2014, A&A, 569, A38
  * Solanki (2003) Solanki, S. K. 2003, A&A Rev., 11, 153
  * Solanki et al. (2020) Solanki, S. K., del Toro Iniesta, J. C., Woch, J., et al. 2020, A&A, 642, A11
  * Solanki et al. (2013) Solanki, S. K., Krivova, N. A., & Haigh, J. D. 2013, ARA&A, 51, 311
  * Spruit & Roberts (1983) Spruit, H. C. & Roberts, B. 1983, Nature, 304, 401
  * Unruh et al. (1999) Unruh, Y. C., Solanki, S. K., & Fligge, M. 1999, A&A, 345, 635
  * Vieira et al. (2012) Vieira, L. E. A., Norton, A., Dudok de Wit, T., et al. 2012, Geochim. Res. Lett., 39, L16104
  * Yeo et al. (2015) Yeo, K. L., Ball, W. T., Krivova, N. A., et al. 2015, J. Geophys. Res., 120, 6055
  * Yeo et al. (2014a) Yeo, K. L., Krivova, N. A., & Solanki, S. K. 2014a, Space Sci. Rev., 186, 137
  * Yeo et al. (2014b) Yeo, K. L., Krivova, N. A., Solanki, S. K., & Glassmeier, K. H. 2014b, A&A, 570, A85
  * Yeo et al. (2013) Yeo, K. L., Solanki, S. K., & Krivova, N. A. 2013, A&A, 550, A95
  * Yeo et al. (2017) Yeo, K. L., Solanki, S. K., Norris, C. M., et al. 2017, Phys. Rev. Lett., 119, 9.1102

##  Appendix A Active region mapping

We identified the ARs that emerged within P1 and were imaged by both SO/PHI
and SDO/HMI. We established their boundaries as per the following procedure.

We have 96 concurrent SO/PHI and SDO/HMI image pairs from this 10 day period
(01.09.2021 to 10.09.2021). Taking each SO/PHI image pair, we created a
$1024\times 1024$ Boolean map and set to true the points that were classified
as sunspots or faculae (Sect. 2.3), that is, harbouring magnetic activity. Let
us term each contiguous cluster of magnetic points as a magnetic island. We
discard the magnetic islands that do not enclose sunspots, leaving the
sunspot-bearing magnetic islands (SBMIs), which we take to be the cores of
active regions. To allow us to compare the Boolean maps from all the image
pairs, we project each of them to a heliographic latitude versus Carrington
longitude grid, namely, heliocentric coordinates. With this projection, a SBMI
that is present in multiple image pairs will appear in a similar location in
the corresponding Boolean maps. The Boolean maps from the 96 image pairs are
combined into a single Boolean map using the logical OR operator. The result
is a map that indicates the maximum extent of each SBMI. This is repeated for
the SDO/HMI image pairs, producing a separate map of the SBMIs that were
imaged by this instrument.

We combined the SBMI maps from the SO/PHI and SDO/HMI data sets using the
logical AND operator. The result is a map of the SBMIs that appeared to both
instruments, of which there are five. Taking each SBMI as the core of an
active region, we expanded the boundary in every direction by 60 Mm and took
the result as the final boundary of the active region. The five active regions
are marked in Fig. 4, along with their NOAA active region number. We
confirmed, from visual inspection, that the boundary established here encloses
each AR in its entirety. Each of these ARs emerged within P1, as evident from
the facular and sunspot area time series beginning near zero (Figs. 5b and
5e). Meeting our selection criteria, these five ARs were retained for the
analysis presented in Sect. 3.

Combining the SO/PHI and SDO/HMI maps using the logical AND operator excludes
SBMIs that are in one map but not the other. The discarded SBMIs correspond to
NOAA active region 12860, 12865, and 12870. They do not meet our selection
criteria. NOAA active region 12865, marked ”C” in Fig. 4, contained sunspots
at the beginning of P1, where it was visible to SO/PHI but not to SDO/HMI. By
the time it rotated onto the solar disc from SDO/HMI’s perspective later in
P1, the embedded sunspots had dissipated. Even though this active region was
imaged by both instruments, it had emerged so much before P1 that its sunspots
never appeared to SDO/HMI. As for NOAA active regions 12860 and 12870, the
former was imaged only by SDO/HMI and the latter, marked ’D’ in Fig. 4, only
by SO/PHI.


# KazakhTTS2: Extending the Open-Source Kazakh TTS Corpus
With More Data, Speakers, and Topics

###### Abstract

We present an expanded version of our previously released Kazakh text-to-
speech (KazakhTTS) synthesis corpus. In the new KazakhTTS2 corpus, the overall
size has increased from 93 hours to 271 hours, the number of speakers has
risen from two to five (three females and two males), and the topic coverage
has been diversified with the help of new sources, including a book and
Wikipedia articles. This corpus is necessary for building high-quality TTS
systems for Kazakh, a Central Asian agglutinative language from the Turkic
family, which presents several linguistic challenges. We describe the corpus
construction process and provide the details of the training and evaluation
procedures for the TTS system. Our experimental results indicate that the
constructed corpus is sufficient to build robust TTS models for real-world
applications, with a subjective mean opinion score ranging from 3.6 to 4.2 for
all the five speakers. We believe that our corpus will facilitate speech and
language research for Kazakh and other Turkic languages, which are widely
considered to be low-resource due to the limited availability of free
linguistic data. The constructed corpus, code, and pretrained models are
publicly available in our GitHub repository.

Keywords: text-to-speech, TTS, speech synthesis, speech corpus, open-source,
Kazakh, Turkic, agglutinative

languageresourceLanguage Resources is file/.style=edge
path'/.expanded=([xshift=folder indent]!u.parent anchor) |- (.child anchor),
inner sep=0pt, this folder size/.style=edge path'/.expanded=([xshift=folder
indent]!u.parent anchor) |- (.child anchor) pic[solid]folder=#1, inner
xsep=0.75*#1, folder tree indent/.style=before computing xy=l=#1, folder
icons/.style=folder, this folder size=#1, folder tree indent=3*#1, folder
icons/.default=12pt,

KazakhTTS2: Extending the Open-Source Kazakh TTS Corpus
With More Data, Speakers, and Topics

Saida Mussakhojayeva, Yerbolat Khassanov, Huseyin Atakan Varol
---
Institute of Smart Systems and Artificial Intelligence (ISSAI),
Nazarbayev University, Nur-Sultan, Kazakhstan
{saida.mussakhojayeva, yerbolat.khassanov<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Text-to-speech (TTS), also known as speech synthesis, is the automatic process
of converting written text into speech [Taylor, 2009], which has wide
application potential and a substantial social impact, including digital
assistants, improved accessibility for people with reading disabilities,
speech and vision impairments, to name a few. For visually impaired people, in
particular, it enables voice-controlled access to Internet-of-things devices,
on-demand access to books and websites, and access to other vocalized
assistive technologies. In turn, these enhance the overall quality of life,
consumption of information, and access to knowledge. In addition, TTS can
complement other important language and vision technologies, such as speech
recognition [Tjandra et al., 2017], speech-to-speech translation [Wahlster,
2013], face-to-face translation [Prajwal et al., 2019], and visual-to-sound
[Zhou et al., 2018]. Considering the aforementioned benefits, TTS is
undoubtedly an essential speech processing technology for any language.

In recent years, TTS research has progressed remarkably thanks to neural
network-based architectures [Tan et al., 2021], regularly organized challenges
[Black and Tokuda, 2005, Dunbar et al., 2019], and open-source datasets [Ito
and Johnson, 2017, Zen et al., 2019, Shi et al., 2020]. Especially, impressive
results have been achieved for commercially viable languages, such as English
and Mandarin. However, there is still a lack of research into the development
of TTS technologies for low-resource languages. To address this problem in
regard to Kazakh, ?) have recently developed the first open-source Kazakh
text-to-speech (KazakhTTS) corpus, which contains 93 hours of manually
transcribed audio from two professional speakers (one female and one male)
reading news articles. The developed corpus has generated substantial interest
and has been downloaded over 200 times in less than a year by academia and
industry, both from local and global organizations. This demonstrates high
demand for open-source and high-quality transcribed speech data in the Kazakh
language.

Motivated by this, in this paper, we present a new version of the KazakhTTS
corpus called KazakhTTS2, which adds more data, speakers, and topics to our
corpus. Specifically, we have increased the data size from 93 hours to 271
hours. We have added three new professional speakers (two females and one
male), with over 25 hours of transcribed data for each speaker. In addition to
news, we have diversified the topic coverage with a book and Wikipedia
articles. Like the first version, KazakhTTS2 is freely available to both
academic researchers and industry practitioners in our GitHub
repository111https://github.com/IS2AI/Kazakh_TTS.

To validate the KazakhTTS2 corpus, we built a state-of-the-art TTS system
based on the Tacotron 2 [Shen et al., 2018] architecture. The constructed TTS
system was evaluated using the subjective mean opinion score (MOS) measure.
The obtained MOSs for all the speakers ranged from 3.6 to 4.2, which indicates
the utility of the KazakhTTS2 corpus for building robust TTS systems suitable
for real-world applications. We believe that our corpus will further
facilitate the rapid development of TTS systems in the Kazakh language and
thus serve as an enabler for the wide range of applications mentioned above.
We also believe that this work will encourage subsequent efforts in this area
to address some of the practical issues that arise when training TTS systems
for the Kazakh language. Additionally, our corpus can be employed to bootstrap
speech technologies for other similar languages from the Turkic family, for
example, by means of cross-lingual transfer learning [Chen et al., 2019] and
self-supervised pretraining [Baevski et al., 2020].

To sum up, our main contributions are:

  * •

We developed a text-to-speech synthesis corpus for the Kazakh language
containing five speakers (three females and two males) comprising 271 hours of
carefully transcribed data from various sources (news, book, and Wikipedia).

  * •

We validated the efficacy of the corpus, by training state-of-the-art neural
TTS models, which achieved a sufficient subjective MOS for most practical
applications.

  * •

The KazakhTTS2 corpus, code, and pretrained models were made publicly
available1 for both commercial and academic use.

The rest of this paper is organized as follows: Section 2 reviews the work on
Kazakh language corpus creation. In Section 3, we briefly summarize the
previous release of the corpus and explain the changes made in KazakhTTS2,
including the corpus structure and statistics. The experimental setup and
evaluation results are described in Section 4. Section 5 discusses the
challenges of Kazakh speech synthesis and future research directions. Section
6 concludes this work.

## 2\. Related Work

Despite its under-resourced status, Kazakh language research is emerging as an
evolving field with an increasing number of recently released open-source
corpora. For example, ?) developed the first large-scale publicly available
corpus for automatic speech recognition (ASR). The corpus was collected by
means of crowdsourcing, with over 2,000 people contributing around 330 hours
of audio recordings. Similarly, ?) developed an open-source Kazakh named
entity recognition dataset consisting of over 100,000 sentences annotated for
25 entity classes. Linguistic corpora development has also been observed in
neighboring countries with languages similar to Kazakh, such as Uzbek [Musaev
et al., 2021]. Additionally, there are other large-scale projects aimed at
collecting open-source corpora for various languages, including Kazakh, such
as Common Voice [Ardila et al., 2020]. However, all these datasets are
unsuitable for building robust Kazakh TTS systems, which require a large
number of high-quality audio recordings of a single speaker.

The first attempt to collect a large-scale open-source TTS dataset for the
Kazakh language was made by ?). The collected dataset was called KazakhTTS and
consisted of 93 hours of carefully transcribed audio from two professional
speakers. Specifically, the speakers were assigned to read local news
articles. The recorded articles were manually segmented into sentences and
then aligned with the corresponding text with the help of native Kazakh
transcribers. The TTS systems developed using KazakhTTS achieved an MOS of
above 4.0, demonstrating the high quality of the collected data. This work
further extends the KazakhTTS corpus, as described in the following sections.

The other existing corpora dedicated to Kazakh TTS are either proprietary or
have been collected by leveraging unsupervised and semi-supervised approaches.
For example, ?) extracted readings of the Bible in hundreds of languages,
including Kazakh. The extracted recordings were automatically segmented and
aligned with the corresponding text. Although a TTS system built using this
corpus is sufficient to deploy in some use-cases, its overall quality is
unsatisfactory for most real-world applications. Specifically, in the
evaluation experiments, the Kazakh TTS system achieved a mel-cepstral
distortion score of more than 6, which is considered low
quality222http://festvox.org/cmu_wilderness/index.html. In another work, ?)
developed a Kazakh TTS system using a female voice. However, the authors did
not provide any information on their corpus, such as its size, how the
recordings were acquired, and how to download it. Additionally, the authors
did not describe the evaluation procedures performed to assess the developed
TTS system.

## 3\. KazakhTTS2 Corpus

In this section, we describe the curation procedures for the KazakhTTS2
corpus. The KazakhTTS2 corpus collection was approved by the Institutional
Research Ethics Committee of Nazarbayev University. We first briefly summarize
the previous version of the corpus (i.e., KazakhTTS) and then systematically
explain the changes made to extend it.

### 3.1. KazakhTTS

KazakhTTS is the first version of our corpus, which contains around 93 hours
of transcribed audio consisting of over 42,000 sentences. The audio was
recorded by two professional speakers, both of whom had had over ten years of
narration experience in local television and radio stations. The speakers were
assigned to read news articles covering various topics, such as sports,
business, politics, and so on. The recorded audio was manually segmented into
sentences, with defective segments (e.g., mispronunciation and external noise)
filtered out. The correspondence between audio and text was verified by native
Kazakh transcribers. The statistics for the first and second versions of the
corpus are provided in Table 1.

Category | KazakhTTS | KazakhTTS2
---|---|---
# Speakers | 2 | 5
# Segments | 42,082 | 136,196
# Tokens | 565.6k | 1.7M
# Unique tokens | 54.9k | 107.9k
Duration | 93.2 h | 271.7 h
Table 1: The comparison of statistics for the KazakhTTS and KazakhTTS2 corpora

### 3.2. Text Collection

We began by collecting additional news articles from four local news websites.
To further broaden the topic coverage, we added a book from the public domain
and Wikipedia articles. From Wikipedia, we extracted articles on science,
computer technology, countries, and history. All the articles were manually
extracted to eliminate defects peculiar to web crawlers and saved in the DOC
format for the professional speakers' convenience (i.e., font size, line
spacing, and typeface could be adjusted to the preferences of the speakers).
In total, over 2,500 additional news articles, one book, and 159 Wikipedia
articles were extracted.

### 3.3. Recording Process

To narrate the collected text, we auditioned several candidates and, as a
result, hired three professional speakers (two females and one male). Each
speaker participated voluntarily and was informed of the protocols for data
collection and use through an informed consent form. All the hired speakers
were tasked to read news articles only. In addition, we rehired the male
speaker (speaker M1) from the previous corpus creation process, because of his
extensive experience in narrating documentaries. He was subsequently tasked to
read the book and Wikipedia articles. The speaker specifications, including
gender, age, professional experience as a narrator, and recording device
information, are provided in Table 2. Speakers F1 and M1 were part of
KazakhTTS, whereas F2, F3, and M2 are newly hired speakers.

Due to the COVID-19 pandemic, we could not invite the speakers to our
laboratory for data collection. Therefore, the speakers were allowed to record
audio in their makeshift studios that they had set up to work from home. The
speakers were instructed to read the texts in a quiet indoor environment with
neutral tone and pace. They were also asked to follow orthoepic rules, to
maintain a constant distance between the microphone and lips, to pause at
commas, and to intonate sentences ending with a question mark appropriately.
In total, each newly hired speaker read around 1,400 news articles, and
Speaker M1 read one book entitled Abai Zholy (The Path of Abai) and 159
Wikipedia articles.

Speaker ID | Gender | Age |  | Work
---
experience
| Recording
---
device
F1∗ | Female | 44 | 14 years | AKG P120
F2 | Female | 39 | 15 years | RØDE
F3 | Female | 52 | 25 years | Behringer C-1
M1∗ | Male | 46 | 12 years | Tascam DR-40
M2 | Male | 33 | 11 years | Mi

Note. Speakers F1 and M1 are from KazakhTTS.

Table 2: The KazakhTTS2 speaker information

### 3.4. Segmentation and Alignment

For audio segmentation and audio-to-text alignment, we employed the same
approach as in the KazakhTTS corpus construction. We hired five native Kazakh
transcribers with different backgrounds and thorough knowledge of Kazakh
grammar rules. The transcribers manually segmented the recordings into
sentence-level chunks and aligned them with the corresponding text using the
Praat toolkit [Boersma, 2001]. All the texts were represented using a Cyrillic
script consisting of 42 letters333Note that at the time of writing, the
Cyrillic alphabet is the official alphabet used for the Kazakh language,
though the transition process to the Latin alphabet has already begun. and
other punctuation marks, such as period, comma, hyphen, question mark, and
exclamation mark. The transcribers were instructed to remove segments with
mispronunciation and background noise, to trim long pauses at the beginning
and end of segments, and to convert numbers and special characters (e.g., `%',
`$', `+', etc) into the written form. To ensure the uniform quality of work
among the transcribers, we assigned a linguist to randomly check the completed
tasks and to organize regular ``go through errors" sessions.

To ensure the correctness of the audio-to-text alignment process, the
segmented recordings were inspected using our internal ASR system trained on
the KSC dataset [Khassanov et al., 2021]. Specifically, the ASR system was
used to generate segment transcriptions, which were then compared to the
corresponding manually annotated transcripts. Segments with a high character
error rate (CER) were regarded as incorrectly transcribed, and therefore
rechecked by the linguist.

### 3.5. Corpus Structure and Statistics

The file structure of the KazakhTTS2 corpus is shown in Figure 1. Collections
of audio recordings and the corresponding transcriptions are stored in a
separate folder for each speaker. Additionally, for Speaker M1, we split the
data from different sources into separate folders (i.e., News, Wiki, and
Book). All audio recordings were downsampled to 22.05 kHz and stored at 16
bits per sample in the WAV format. All transcripts are stored as TXT files in
the UTF-8 encoding. The audio and the corresponding transcript filenames are
identical except for the extension. The name of each file consists of the
source name, document ID, and utterance ID (i.e., source_docID_uttID). Speaker
information, including gender, age, professional experience, and recording
device, is provided in the speaker_metadata.txt file.

for tree= font=, grow'=0, inner sep=1pt, folder indent=0.0em, folder icons,
edge=densely dotted  [KazakhTTS2, this folder size=-30pt [F1, this folder
size=8pt [Transcripts, this folder size=8pt [source_docID_uttID.txt, is file]]
[Audio, this folder size=8pt [source_docID_uttID.wav, is file]]] [F2, this
folder size=8pt [Transcripts, this folder size=8pt [source_docID_uttID.txt, is
file]] [Audio, this folder size=8pt [source_docID_uttID.wav, is file]]] [F3,
this folder size=8pt [Transcripts, this folder size=8pt
[source_docID_uttID.txt, is file]] [Audio, this folder size=8pt
[source_docID_uttID.wav, is file]]] [M1, this folder size=8pt [News, this
folder size=8pt [Transcripts, this folder size=8pt [source_docID_uttID.txt, is
file]] [Audio, this folder size=8pt [source_docID_uttID.wav, is file]]] [Wiki,
this folder size=8pt [Transcripts, this folder size=8pt
[source_docID_uttID.txt, is file]] [Audio, this folder size=8pt
[source_docID_uttID.wav, is file]]] [Book, this folder size=8pt [Transcripts,
this folder size=8pt [source_docID_uttID.txt, is file]] [Audio, this folder
size=8pt [source_docID_uttID.wav, is file]]]] [M2, this folder size=8pt
[Transcripts, this folder size=8pt [source_docID_uttID.txt, is file]] [Audio,
this folder size=8pt [source_docID_uttID.wav, is file]]]
[speaker_metadata.txt, is file] ]

Figure 1: The file structure of KazakhTTS2

The statistics for the KazakhTTS2 corpus are given in Table 3. The overall
corpus size is around 271 hours, with each speaker having at least 25 hours of
transcribed audio. The total number of sentences is around 136 thousand, and
the total number of tokens is over 1.7 million, with unique token types per
speaker ranging from 28.5 thousand to 80.7 thousand. Figure 2 presents the
histograms of the distributions of sentence duration and length (in words) for
each speaker in KazakhTTS2. For all speakers, the majority of sentence
durations are between 3 and 6 seconds. The majority of sentence lengths are
between 11 and 15 words for female speakers, and between 6 and 10 words for
male speakers.

Speaker ID | Source | # Segments | Segment duration | Tokens
---|---|---|---|---
Total | Mean | Min | Max | Total | Mean | Min | Max | Unique
F1 |  News∗ | 17,426 | 36.1 h | 7.5 s | 1.0 s | 24.2 s | 245.4k | 14.1 | 2 | 42 | 34.3k
F2 | News | 12,921 | 25.7 h | 7.2 s | 0.8 s | 22.0 s | 177.9k | 13.8 | 1 | 42 | 28.5k
F3 | News | 23,696 | 48.5 h | 7.4 s | 0.7 s | 21.4 s | 331.3k | 14.0 | 1 | 48 | 43.0k
M1 |  News∗ | 24,608 | 57.0 h | 8.3 s | 0.8 s | 55.9 s | 319.6k | 13.0 | 1 | 75 | 42.5k
Wiki | 13,189 | 29.7 h | 8.1 s | 0.9 s | 29.7 s | 166.8k | 12.6 | 1 | 47 | 33.6k
Book | 11,453 | 16.7 h | 5.3 s | 0.8 s | 21.1 s | 107.6k | 9.4 | 2 | 40 | 23.5k
All | 49,250 | 103.5 h | 7.6 s | 0.8 s | 55.9 s | 594.1k | 12.1 | 1 | 75 | 80.7k
M2 | News | 32,903 | 57.9 h | 6.3 s | 0.7 s | 28.3 s | 393.3k | 12.0 | 1 | 60 | 53.3k

Note. The news source data of speakers F1 and M1 are from KazakhTTS.

Table 3: The KazakhTTS2 dataset specifications

(a) F1

(b) F2

(c) F3

(d) M1

(e) M2

(f) F1

(g) F2

(h) F3

(i) M1

(j) M2

Figure 2: Segment duration (a, b, c, d, e) and length (f, g, h, i, j)
distributions for each speaker of KazakhTTS2

## 4\. Speech Synthesis Experiments

In this section, we describe the experiments conducted to validate the utility
of the KazakhTTS2 corpus. We first describe the experimental setup, followed
by our evaluation procedures and results.

### 4.1. Experimental Setup

We used the ESPnet-TTS toolkit [Hayashi et al., 2020] to build end-to-end TTS
models based on the Tacotron 2 [Shen et al., 2018] architecture. Specifically,
we followed the training recipe of LJ Speech [Ito and Johnson, 2017]. All TTS
models were trained using Tesla V100 GPUs running on NVIDIA DGX 2 machines.
The input for each model is a sequence of characters consisting of 42 letters
and 5 symbols (‘.’, ‘,’, ‘-’,‘?’, ‘!’), and the output is a sequence of
acoustic features (80 dimensional log Mel-filter bank features). To transform
these acoustic features into the time-domain waveform samples, we employed
WaveGAN [Yamamoto et al., 2020] vocoders.

In the Tacotron 2 model, the encoder module was modeled as a single
bidirectional LSTM layer with 512 units (256 units in each direction), and the
decoder module was modeled as a stack of two unidirectional LSTM layers with
1,024 units. The parameters were optimized using the Adam algorithm [Kingma
and Ba, 2015] with an initial learning rate of $10^{-3}$ for 200 epochs. To
mitigate overfitting, we applied a dropout of 0.5. A separate Tacotron 2 model
was trained for each speaker (i.e., a single speaker model). More details on
the model specifications and training procedures are provided in our GitHub
repository1.

### 4.2. Experimental Evaluation

To assess the quality of the synthesized recordings, we performed a subjective
evaluation using the MOS measure. We evaluated only the voices developed using
the newly collected data444For Speaker M1, we trained two separate models from
scratch using Wikipedia and Book data. (i.e., F2, F3, M1 Wikipedia and Book,
and M2), as the other data had already been evaluated in the previous work
[Mussakhojayeva et al., 2021a]. The evaluation procedure was similar to that
of our previous work, except for the number of sentences selected as a test
set. Specifically, in this work, we selected 25 sentences of varying lengths
from each speaker, whereas, in the previous work, 50 sentences per speaker
were selected. The reason for selecting a smaller number of sentences is based
on our observation that raters become exhausted or bored after around 25
sentences and quit the evaluation session. The evaluation sentences were not
used to train the models.

The speakers were evaluated in separate sessions, and in each session we
compared the ground truth (i.e., natural speech) recordings against the
Tacotron 2 synthesized recordings. The ground truth sentences were manually
checked to ensure that the speaker read them well (i.e., without disfluencies,
mispronunciations, or background noise).

Evaluation sessions were conducted using the instant messaging platform
Telegram [Telegram Messenger Inc., 2013], as it is difficult to find native
Kazakh raters on other well-known platforms, such as Amazon Mechanical Turk
[Amazon.com Inc., 2005]. We developed a separate evaluation Telegram bot for
each speaker. The bots first presented a welcome message with instructions and
then started the evaluation process. During the evaluation, the bots sent a
sentence recording555Note that in Telegram, to send audio recordings, we had
to convert them into MP3 format. with the associated transcript to a rater and
received the corresponding evaluation score. Recordings were rated using a
five-point Likert scale: 5 for excellent, 4 for good, 3 for fair, 2 for poor,
and 1 for bad.

The raters were instructed to assess the overall quality through headphones in
a quiet environment666Due to the crowdsourced nature of the evaluation
process, we cannot guarantee that all raters used headphones and sat in a
quiet environment.. They were allowed to listen to the recordings several
times, but they were not allowed to alter the ratings once submitted.
Additionally, the Telegram bots kept track of the raters' ID, to prevent them
from participating in the evaluation session more than once.

The evaluation recordings were presented in the same order and one at a time.
However, at each time step, the bots randomly decided which version of a
recording to select (i.e., ground truth or synthesized). As a result, each
rater heard only one of the versions of a recording, and both systems (i.e.,
ground truth and Tacotron 2) were presented to all the raters. Each recording
was rated at least 24 times for all the three speakers. The numbers of raters
were 57, 61, 116, 89, and 53 for speakers F2, F3, M1 Wikipedia, M1 Book, and
M2, respectively777In fact, the number of raters was higher, but we excluded
the ratings of those who did not go through the session to the end, or whose
ratings were suspicious (e.g., all scores are “excellent” or all scores are
“bad”.).

At the end of the evaluation, the bots thanked the raters and invited them to
fill in an optional questionnaire about their age, region (where a rater grew
up and learned the Kazakh language), and gender. The questionnaire results
showed that the raters varied in gender and region, but not in age (most of
them were under 20). Specifically, the majority of raters were from the south
and west of Kazakhstan, and females outnumbered males by a factor of 1.5.

### 4.3. Experiment Results

The subjective evaluation results are given in Table 4. As expected, the
ground truth recordings received higher MOS scores than the Tacotron 2
synthesized ones. Nevertheless, all synthesized recordings except M1 Wikipedia
scored above 4.0 on the MOS measure and were close to the ground truth (i.e.,
8.7%, 3.1%, 18.1%, 11.1% and 5.2% relative MOS reductions for speakers F2, F3,
M1 Wikipedia, M1 Book, and M2, respectively). These results demonstrate the
utility of our KazakhTTS2 dataset for TTS applications.

Overall, the highest MOS score among the synthesized recordings was achieved
by Speaker F1, and the lowest score was achieved by M1 Wikipedia. Presumably,
the reason for the poor performance of M1 Wikipedia is the wide variety of
topics and the abundance of rare scientific terms (from chemistry, biology,
information technology, etc.). We believe that the performance of M1 Wikipedia
can be improved by exploiting other data from Speaker M1. For example, by pre-
training a model on M1 News and Book data, followed by fine-tuning using M1
Wikipedia.

In addition, we conducted an objective evaluation in which we manually
analyzed the synthesized evaluation set recordings. Specifically, we counted
the various error types made by the Tacotron 2 systems built using the newly
collected data. The objective evaluation results are given in Table 5, which
are consistent with the subjective evaluation, with Speaker M2 having the
lowest number of errors, followed by F2 and M1 Book, and then F3 and M1
Wikipedia. The most common error types among all speakers are
mispronunciation, incomplete words, and word skipping. This analysis indicates
that there is still room for improvement and future work should focus on
eliminating these errors.

Speaker ID | Source | Ground truth | Tacotron 2
---|---|---|---
F1 |  News∗ |  4.726 $\pm$ 0.037 |  4.535 $\pm$ 0.049
F2 | News |  4.449 $\pm$ 0.059 |  4.061 $\pm$ 0.070
F3 | News |  4.178 $\pm$ 0.073 |  4.049 $\pm$ 0.076
M1 |  News∗ |  4.360 $\pm$ 0.050 |  4.144 $\pm$ 0.063
Wiki |  4.483 $\pm$ 0.040 |  3.673 $\pm$ 0.070
Book |  4.564 $\pm$ 0.045 |  4.057 $\pm$ 0.068
M2 | News |  4.431 $\pm$ 0.062 |  4.200 $\pm$ 0.073

Note. The scores of speakers F1 and M1 News are from the previous work
[Mussakhojayeva et al., 2021a].

Table 4: Mean opinion score (MOS) results with 95% confidence intervals Error types | Speaker ID
---|---
F2 | F3 | M1 | M2
Wiki | Book
# repeated words | 0 | 0 | 0 | 0 | 0
# skipped words | 5 | 6 | 0 | 1 | 1
# mispronounced words | 2 | 1 | 11 | 5 | 4
# incomplete words | 2 | 7 | 0 | 3 | 4
# long pauses | 0 | 0 | 4 | 1 | 0
# nonverbal sounds | 1 | 0 | 0 | 0 | 0
Total | 10 | 14 | 15 | 10 | 9
Table 5: Manual analysis of error types made by Tacotron 2

## 5\. Challenges and Future Work

The Kazakh language presents several challenges to the speech synthesis task.
The first one is code-switching, as the majority of Kazakh speakers are
bilingual in Kazakh and Russian. While the languages are not mixed in most
formal situations (e.g., news, books, law, etc.), intrasentential code-
switching often occurs in informal conversations. Moreover, intra-word code-
switching is also possible (e.g., Kazakh stem words with Russian suffixes or
vice versa), which may further deteriorate TTS quality.

Additionally, Kazakh has a large number of loanwords from Russian, and these
words usually retain the orthographic and phonological properties of the
source language. This has especially important consequences for TTS
applications, as Russian differs from Kazakh in many aspects. For example, in
most Kazakh words, the stress is fixed on the final syllable, while in
Russian, the stress can be on any syllable of a word [Jouravlev and Lupker,
2014]. Furthermore, the spelling of Kazakh words closely matches their
pronunciation, which is not the case with Russian words; for example, the
letter ``o'' is sometimes pronounced as /a/. It is important to mention that
due to globalization, the number of loanwords from other languages, especially
English, is also increasing, which is likely to pose an additional challenge
in the near future [Mussakhojayeva et al., 2021b].

Another challenge is that Kazakh is an agglutinative language, with a very
large vocabulary and many characters per word. It is also susceptible to
morphophonemic changes arising during word formation. One of the solutions
would be to increase the size of the Kazakh speech corpus to cover more word
formation variants. We believe that overcoming these challenges for the Kazakh
language will be an interesting direction for future research.

## 6\. Conclusion

We have presented KazakhTTS2, a large-scale open-source Kazakh text-to-speech
corpus, which further extends the previous work with more data, voices, and
topics. The corpus consists of five voices (three female and two male), with
over 270 hours of high-quality transcribed data. The corpus is publicly
available, which permits both academic and commercial use. We validated the
corpus by means of crowdsourced subjective evaluation, where all voices
synthesized using the Tacotron 2 model achieved an MOS of above 3.6, making it
suitable for practical deployment. To enable experiment reproducibility and
facilitate future research, we shared our training recipes and pretrained
models in our GitHub repository1. Although the corpus was designed with TTS
application in mind, it can be used to complement other speech processing
applications, such as speech recognition and speech translation. We hope the
TTS corpus construction and evaluation procedures described in this paper will
contribute to the burgeoning field of Kazakh speech and language research and
help advance the state-of-the-art for other low-resource languages of the
Turkic family.

## 7\. Acknowledgements

The authors would like to thank Aigerim Borambayeva, Almas Mirzakhmetov, Dias
Bakhtiyarov, and Rustem Yeshpanov for their help in data collection, voice
evaluation, and paper revision. The authors would also like to thank the
speakers for their recordings and the anonymous raters for their evaluations.

## 8\. Bibliographical References

## References

  * Amazon.com Inc., 2005 Amazon.com Inc.  (2005).  Amazon Mechanical Turk (MTurk).
  * Ardila et al., 2020 Ardila, R., Branson, M., Davis, K., Kohler, M., Meyer, J., Henretty, M., Morais, R., Saunders, L., Tyers, F. M., and Weber, G.  (2020).  Common Voice: A massively-multilingual speech corpus.  In LREC, pages 4218–4222. ELRA.
  * Baevski et al., 2020 Baevski, A., Zhou, Y., Mohamed, A., and Auli, M.  (2020).  wav2vec 2.0: A framework for self-supervised learning of speech representations.  In Advances in Neural Information Processing Systems (NIPS), volume 33, pages 12449–12460.
  * Black and Tokuda, 2005 Black, A. W. and Tokuda, K.  (2005).  The Blizzard Challenge - 2005: Evaluating corpus-based speech synthesis on common datasets.  In European Conference on Speech Communication and Technology (Interspeech), pages 77–80. ISCA.
  * Black, 2019 Black, A. W.  (2019).  CMU wilderness multilingual speech dataset.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5971–5975.
  * Boersma, 2001 Boersma, P.  (2001).  Praat, a system for doing phonetics by computer.  Glot International, 5(9):341–345.
  * Chen et al., 2019 Chen, Y., Tu, T., Yeh, C., and Lee, H.  (2019).  End-to-end text-to-speech for low-resource languages by cross-lingual transfer learning.  In Interspeech, pages 2075–2079. ISCA.
  * Dunbar et al., 2019 Dunbar, E., Algayres, R., Karadayi, J., Bernard, M., Benjumea, J., Cao, X., Miskic, L., Dugrain, C., Ondel, L., Black, A. W., Besacier, L., Sakti, S., and Dupoux, E.  (2019).  The zero resource speech challenge 2019: TTS without T.  In Interspeech, pages 1088–1092. ISCA.
  * Hayashi et al., 2020 Hayashi, T., Yamamoto, R., Inoue, K., Yoshimura, T., Watanabe, S., Toda, T., Takeda, K., Zhang, Y., and Tan, X.  (2020).  ESPnet-TTS: Unified, reproducible, and integratable open source end-to-end text-to-speech toolkit.  In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7654–7658.
  * Ito and Johnson, 2017 Ito, K. and Johnson, L.  (2017).  The LJ speech dataset.  https://keithito.com/LJ-Speech-Dataset/.
  * Jouravlev and Lupker, 2014 Jouravlev, O. and Lupker, S. J.  (2014).  Stress consistency and stress regularity effects in russian.  Language, Cognition and Neuroscience, 29(5):605–619.
  * Khassanov et al., 2021 Khassanov, Y., Mussakhojayeva, S., Mirzakhmetov, A., Adiyev, A., Nurpeiissov, M., and Varol, H. A.  (2021).  A crowdsourced open-source Kazakh speech corpus and initial speech recognition baseline.  In European Chapter of the Association for Computational Linguistics (EACL), pages 697–706, Online, April. Association for Computational Linguistics.
  * Khomitsevich et al., 2015 Khomitsevich, O., Mendelev, V., Tomashenko, N. A., Rybin, S., Medennikov, I., and Kudubayeva, S.  (2015).  A bilingual kazakh-russian system for automatic speech recognition and synthesis.  In Speech and Computer (SPECOM), volume 9319 of Lecture Notes in Computer Science, pages 25–33. Springer.
  * Kingma and Ba, 2015 Kingma, D. P. and Ba, J.  (2015).  Adam: A method for stochastic optimization.  In Proc. International Conference on Learning Representations (ICLR).
  * Musaev et al., 2021 Musaev, M., Mussakhojayeva, S., Khujayorov, I., Khassanov, Y., Ochilov, M., and Varol, H. A.  (2021).  USC: an open-source uzbek speech corpus and initial speech recognition experiments.  In Speech and Computer (SPECOM), volume 12997 of Lecture Notes in Computer Science, pages 437–447. Springer.
  * Mussakhojayeva et al., 2021a Mussakhojayeva, S., Janaliyeva, A., Mirzakhmetov, A., Khassanov, Y., and Varol, H. A.  (2021a).  KazakhTTS: An Open-Source Kazakh Text-to-Speech Synthesis Dataset.  In Proc. Interspeech 2021, pages 2786–2790.
  * Mussakhojayeva et al., 2021b Mussakhojayeva, S., Khassanov, Y., and Varol, H. A.  (2021b).  A study of multilingual end-to-end speech recognition for Kazakh, Russian, and English.  In Speech and Computer (SPECOM), volume 12997 of Lecture Notes in Computer Science, pages 448–459. Springer.
  * Prajwal et al., 2019 Prajwal, K. R., Mukhopadhyay, R., Philip, J., Jha, A., Namboodiri, V., and Jawahar, C. V.  (2019).  Towards automatic face-to-face translation.  In Proceedings of the International Conference on Multimedia (MM), pages 1428–1436. ACM.
  * Shen et al., 2018 Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Ryan, R., Saurous, R. A., Agiomyrgiannakis, Y., and Wu, Y.  (2018).  Natural TTS synthesis by conditioning Wavenet on MEL spectrogram predictions.  In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4779–4783.
  * Shi et al., 2020 Shi, Y., Bu, H., Xu, X., Zhang, S., and Li, M.  (2020).  AISHELL-3: A multi-speaker Mandarin TTS corpus and the baselines.  CoRR, abs/2010.11567.
  * Tan et al., 2021 Tan, X., Qin, T., Soong, F., and Liu, T.-Y.  (2021).  A survey on neural speech synthesis.  arXiv preprint arXiv:2106.15561.
  * Taylor, 2009 Taylor, P.  (2009).  Text-to-speech synthesis.  Cambridge university press.
  * Telegram Messenger Inc., 2013 Telegram Messenger Inc.  (2013).  Telegram.
  * Tjandra et al., 2017 Tjandra, A., Sakti, S., and Nakamura, S.  (2017).  Listening while speaking: Speech chain by deep learning.  In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 301–308.
  * Wahlster, 2013 Wahlster, W.  (2013).  Verbmobil: foundations of speech-to-speech translation.  Springer Science & Business Media.
  * Yamamoto et al., 2020 Yamamoto, R., Song, E., and Kim, J.  (2020).  Parallel Wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram.  In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6199–6203.
  * Yeshpanov et al., 2021 Yeshpanov, R., Khassanov, Y., and Varol, H. A.  (2021).  KazNERD: Kazakh named entity recognition dataset.  arXiv preprint arXiv:2111.13419.
  * Zen et al., 2019 Zen, H., Dang, V., Clark, R., Zhang, Y., Weiss, R. J., Jia, Y., Chen, Z., and Wu, Y.  (2019).  LibriTTS: A corpus derived from LibriSpeech for text-to-speech.  In Interspeech, pages 1526–1530. ISCA.
  * Zhou et al., 2018 Zhou, Y., Wang, Z., Fang, C., Bui, T., and Berg, T. L.  (2018).  Visual to sound: Generating natural sound for videos in the wild.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).