11institutetext: Anton Pannekoek Institute for Astronomy, University of
Amsterdam, Science Park 904, 1098 XH Amsterdam, The Netherlands
11email<EMAIL_ADDRESS>22institutetext: Astronomical Institute, Graduate
School of Science, Tohoku University, 6-3 Aramaki, Aoba-ku, Sendai 980-8578,
Japan

# Revealing magnetic field structure at the surfaces of protoplanetary disks
via near-infrared circular polarization

I. de Langen 11 R. Tazaki 1122

###### Abstract

Context. Magnetic fields play a fundamental role in the dynamical evolution of
protoplanetary disks, in particular via magnetically induced disk winds. The
magnetic field structure at the disk surface is crucial for driving the disk
winds; however, it is still poorly understood observationally.

Aims. We explore a new method to probe the magnetic field structure at the
disk surface using near-infrared (NIR) circular polarization. Near-infrared
circular polarization arises when unpolarized stellar light is scattered by
magnetically aligned grains at the disk surface. In this study, we aim to
clarify to what extent the observed circular polarization pattern can be used
to diagnose the magnetic field structure.

Methods. We first calculated light scattering properties of aligned spheroids,
and the results were then used to create expected observational images of the
degree of circular polarization at a NIR wavelength.

Results. Magnetically aligned grains can produce circular polarization,
particularly when the field configuration deviates from a purely toroidal
field. We find that disk azimuthal dependence of the degree of circular
polarization tends to exhibit a double peaked profile when the field structure
is favorable for driving disk winds by centrifugal force. We also find that
even if the disk is spatially unresolved, a net circular polarization can
possibly be nonzero. We also show that the amplitude of circular polarization
is strongly dependent on grain composition and axis ratio.

Conclusions. Our results suggest that circular polarization observations would
be useful to study the magnetic field structure and dust properties at the
disk surface.

###### Key Words.:

Protoplanetary disks – Magnetic fields – Polarization

##  1 Introduction

Magnetic fields are a crucial factor in determining how protoplanetary disks
form, evolve, and disperse. Recent studies suggest that magnetohydrodynamical
(MHD) disk winds play an important role in disk evolution. The occurrence of
MHD disk winds is strongly dependent on the magnetic field structure at the
disk surface. A poloidal magnetic field structure with a large angle with
respect to the disk normal would give rise to disk winds via centrifugal force
(Blandford & Payne, 1982), while a more toroidal field would drive winds
through a magnetic pressure gradient (e.g., Bai, 2017). Therefore, to study
how MHD disk winds are launched, it is essential to understand the magnetic
field geometry at the disk surface.

Observing the magnetic field in protoplanetary disks is a notoriously
difficult task. Polarized thermal emission from magnetically aligned grains
has been the most successful technique in the interstellar medium (Lazarian,
2007; Andersson et al., 2015). However, this technique is not always reliable
in protoplanetary disks, because larger grains that are responsible for
thermal emission are not necessarily aligned with the magnetic field (Tazaki
et al., 2017; Yang, 2021). Alternatively, molecular line polarization might
contain information about both $B$-field orientation and strength, but
requires exquisite sensitivity, which can only marginally be reached with
current telescopes (Teague et al., 2021; Harrison et al., 2021; Stephens et
al., 2020). Recently, Yang & Li (2022) explored a new technique, finding that
linear polarization angles of near-infrared (NIR) disk scattered light carry
information on the magnetic field, as long as micron-sized grains at the disk
surfaces are magnetically aligned. One of the great advantages of this
technique is its sensitivity to the disk-surface magnetic field, which is
crucial for understanding MHD disk winds.

Although Yang & Li (2022) mainly focused on linearly polarized scattered light
arising from magnetically aligned grains, we can also expect circular
polarization (CP) of scattered light (e.g., Bandermann & Kemp, 1973; Fukushima
et al., 2020). However, there have been no quantitative studies on how disk-
scattered light is circularly polarized at near-IR and to what extent it can
be used for diagnosing disk magnetic fields.

In this study, we aim to trace the magnetic field structure in protoplanetary
disks using circular polarized scattered light. To achieve this goal, we will
create synthetic maps showing the degree of circular polarization throughout a
protoplanetary disk and study how and to what extent the $B$-field is
reflected in the circular polarization pattern.

This paper is organized as follows. We introduce our method and the dust model
in Sect. 2. In Sect. 3, we present the CP patterns. We discuss the
implications for future observations in Sect. 4. Finally, we conclude in Sect.
5.

##  2 Methods and models

To calculate the CP of disk-scattered light in the NIR, we follow a similar
method to Yang & Li (2022), where the authors calculated linearly polarized
scattered light by aligned grains at the disk surface. In this study, we
assume single scattering; the stellar radiation will be scattered by a grain
at the disk surface only once before reaching the observer.

###  2.1 Circular polarization calculation

At the disk surface, dust particles scatter unpolarized stellar light to the
observer. The scattered light is in general linearly and circularly polarized.
The polarization of scattered light is described by the Stokes parameter
$(I_{s},Q_{s},U_{s},V_{s})$, which can be calculated from the Stokes parameter
of incident light ($I_{i},Q_{i},U_{i},V_{i}$) and the scattering matrix
(Bohren & Huffman, 1983):

$\pmatrix{I}_{s}\\\ Q_{s}\\\ U_{s}\\\
V_{s}\propto\pmatrix{S}_{11}&S_{12}S_{13}S_{14}\\\ S_{21}S_{22}S_{23}S_{24}\\\
S_{31}S_{32}S_{33}S_{34}\\\ S_{41}S_{42}S_{43}S_{44}\pmatrix{I}_{i}\\\
Q_{i}\\\ U_{i}\\\ V_{i}.$ (1)

The scattering matrix elements $S_{ij}$ describe how the polarization state of
light changes upon scattering by a grain. Considering unpolarized incident
light from the star, the degree of circular polarization in scattered light
$V_{s}/I_{s}$ is equal to $S_{41}$/$S_{11}$. The scattering matrix elements
are given by (Bohren & Huffman, 1983)

$\displaystyle
S_{11}=\frac{1}{2}\,(|S_{1}|^{2}+|S_{2}|^{2}+|S_{3}|^{2}+|S_{4}|^{2}),$ (2)
$\displaystyle S_{41}=\textrm{Im}\\{S_{2}^{\ast}S_{4}+S_{3}^{\ast}S_{1}\\},$
(3)

where $S_{i}\,(i=1,2,3,4)$ represent the scattering amplitude. To calculate
Eqs. (2) and (3), we assume the dipole approximation (Rayleigh limit), where a
grain has to be small compared to the wavelength. We also assume oblate dust
grains with an axis ratio of 1.5. With these assumptions, we can find
analytical expressions for the scattering amplitude $S_{i}$ (Bohren & Huffman,
1983). In the Rayleigh limit, $S_{41}/S_{11}$ does not depend on the size
parameter, which is determined by grain size and wavelength, but depends
solely on dust composition (refractive index) and the axis ratio of the oblate
grain. The validity of the dipole approximation is discussed in Sect. 4.2.

We assume a dust composition model proposed by Birnstiel et al. (2018), where
each grain consists of a mixture of water ice (Warren & Brandt, 2008),
carbonaceous material, troilite (Henning & Stognienko, 1996), and astronomical
silicate (Draine, 2003). For carbonaceous material, we consider two
possibilities: one is refractory organics (Henning & Stognienko, 1996), as
with Birnstiel et al. (2018), and the other is amorphous carbon (Zubko et al.,
1996). Although dust composition including refractory organics has been
commonly used in the literature (Pollack et al., 1996; D’Alessio et al., 2001;
Kataoka et al., 2014; Birnstiel et al., 2018), a composition including
amorphous carbon has also been used to model disk observational quantities
(e.g., DIANA opacity: Woitke et al., 2016). The detailed effect of dust
composition on CP is discussed in Sect. 3.4. Using `optool` (Dominik et al.,
2021), we derived an effective refractive index at $\lambda=1.6\,\rm{\mu m}$
using the Bruggeman mixing rule (Bruggeman, 1935). As a result, we find a
refractive index of $m=1.5625+i0.0204$ and $m=2.0217+i0.4308$ for a mixture
comprising refractory organics and amorphous carbon, respectively.

###  2.2 Disk model and magnetic field geometries

To model the disk-scattering surface at NIR wavelengths, we assumed an
axisymmetric flared disk. We determined the disk flaring geometry motivated by
NIR observations (Avenhaus et al., 2018). By fitting ring structures seen in
scattered light images, Avenhaus et al. (2018) determined the heights of disk
scattering surfaces. As a result, they obtained an empirical fit to the ratio
of the height of the scattering surface $h$ and the midplane distance from the
star $r$ as follows:

$\frac{h}{r}=0.1617\cdot\left(\frac{r}{100\leavevmode\nobreak\
\mathrm{au}}\right)^{0.219}.$ (4)

At the disk surface, the oblate grains are aligned with the magnetic field.
For the sake of simplicity, we assume that grains are perfectly aligned with
their short axis parallel to the local magnetic field direction. We introduce
two parameters ($\theta_{B},\phi_{B}$) to specify the local magnetic field
direction, as done in Yang & Li (2022) (see Fig. 1). For example, when
$\theta_{B}=0^{\circ}$, the magnetic field is vertical. When
$\theta_{B}=90^{\circ}$ and $\phi_{B}=0^{\circ}$, the field is radial. When
$\theta_{B}=90^{\circ}$ and $\phi_{B}=90^{\circ}$, the field is azimuthal
(toroidal). By considering the range of [$0^{\circ},90^{\circ}$] for both
$\theta_{B}$ and $\phi_{B}$, we can mimic various magnetic field structures at
the disk surface.

Given the disk geometry, the orientation of aligned oblate grains, and the
inclination angle, we can calculate scattering angles at each location of the
disk surface (see e.g., Stolker et al., 2016, for the case of spherical
grains). By applying these scattering angles to the CP calculation method
described in Sect. 2.1, we can make a projected disk image of the degree of
CP. We also calculated the linear polarization of scattered light and
confirmed that our code can successfully reproduce the results presented in
Yang & Li (2022).

Figure 1: Schematic view of the disk model and magnetic field geometry. The
location of a scattering grain at the disk surface is defined by $r$ and $h$.
The magnetic field orientation is specified by $\phi_{B}$ and $\theta_{B}$,
measured in the cylindrical coordinate frame
$(\hat{e}_{r},\hat{e}_{\phi},\hat{e}_{z})$.

##  3 Results

We calculated the degree of circular polarization produced by aligned dust
grains at the disk surface for a set of assumed magnetic field structures and
analyzed the imprint of the $B$-field on the CP pattern.

###  3.1 Integrated CP values

We show two synthetic maps of the degree of circular polarization projected at
the surface of the disk model in Fig. 2. For both images, we assumed an
inclination of $i=55^{\circ}$, and polar magnetic field angle
$\theta_{B}=45^{\circ}$, but we assumed different values of $\phi_{B}$. The
upper panel represents a disk with $\phi_{B}=0^{\circ}$, which corresponds to
a purely poloidal field. The CP pattern that arises from this disk surface is
mirror symmetric (except for the sign) with respect to the disk minor axis. In
the lower panel, we added a toroidal component to the magnetic field by
increasing $\phi_{B}$ to $60^{\circ}$. In this situation, the mirror symmetry
is broken, which demonstrates the effect of a different magnetic field
structure on the CP pattern.

Figure 2: Synthetic maps of the CP produced at the protoplanetary disk
surface, assuming a dust composition with amorphous carbon, $i=55^{\circ}$ and
$\theta_{B}=45^{\circ}$. The upper panel shows the result for a poloidal field
($\phi_{B}=0^{\circ}$), resulting in a symmetric CP pattern. The bottom panel
shows the result for a magnetic field with a poloidal and toroidal component
($\phi_{B}=60^{\circ}$), where the symmetry is broken.

To further analyze the disk CP patterns, we calculated the azimuthally
averaged degree of circular polarization, given by

$\left<P_{\textrm{circular}}\right>_{\phi}=\frac{1}{2\pi}\int_{0}^{2\pi}P_{\textrm{circular}}(\phi)d\phi,$
(5)

where $\phi$ is the disk azimuthal angle. We examined the integrated CP for a
$50\times 50$ grid of $\phi_{B}$ and $\theta_{B}$ values, assuming
$i=55^{\circ}$ and the dust composition with amorphous carbon. Figure 3 shows
the azimuthally averaged CP values for various magnetic field configurations.
The majority of possible magnetic field angles produce an integrated negative
degree of CP, although this could also be positive if we assume negative
$\phi_{B}$ values. The averaged CP amplitude tends to become larger for larger
$\theta_{B}$ at intermediate $\phi_{B}$ values. This suggests that even if the
disk is spatially unresolved, CP signals can remain nonzero, particularly when
the magnetic field makes a large angle with the $z$-axis.

Figure 3: Degree of circular polarization azimuthally averaged over a disk
surface ring at $h/r=0.2$. We assumed $i=55^{\circ}$ and the mixture with
amorphous carbon as dust composition. Each symbol represents a different
category of azimuthal profile shape. The blue dashed line indicates the disk
wind boundary: disk winds may occur at field configurations at the right side
of the line.

The averaged CP percentage is zero in some situations: when the magnetic field
is vertical ($\theta_{B}=0^{\circ}$), poloidal ($\phi_{B}=0^{\circ}$), or
toroidal ($\theta_{B}=90^{\circ}$, $\phi_{B}=90^{\circ}$). This can be caused
by either an averaging effect or no intrinsic CP production at all. To
distinguish which of these two causes is responsible, Fig. 4 shows the
nonaveraged maximum absolute value of CP at a disk surface ring of $h/r=0.2$.
The nonzero maximum values for $\phi_{B}=0^{\circ}$ or $\theta_{B}=0^{\circ}$
reveal that oblates aligned with these magnetic fields produce a symmetric CP
pattern, where one side of the disk has negative CP values and the other side
has positive (see Fig. 2, upper panel). As a result, CP signals cancel out
when they are averaged. When the field is purely toroidal
($\theta_{B}=90^{\circ};\phi_{B}=90^{\circ}$), the maximum CP is zero.
Alignment with this magnetic field structure results in the symmetry axis of
the oblate being perpendicular to the incident light, which results in no
intrinsic circular polarization (Bandermann & Kemp, 1973). Because of this, we
conclude that the toroidal magnetic field seems to be better probed by the
linear polarization technique instead of the CP-based technique (see Sect.
4.1). In Fig. 4, we also find that there is another case where the intrinsic
CP value becomes zero, that is, when $\theta_{B}=78^{\circ}$ and
$\phi_{B}=0^{\circ}$. In this case, the incident light is parallel to the
symmetry axis of the oblate grain. As a result, the $S_{41}$ component
vanishes, and therefore no circular polarization is produced (Bandermann &
Kemp, 1973).

Figure 4: Absolute value of the maximum degree of circular polarization of a
disk surface ring at $h/r=0.2$. We assumed $i=55^{\circ}$ and the mixture with
amorphous carbon as dust composition. Each symbol represents a different
category of azimuthal profile shape. The blue dashed line indicates the disk
wind boundary: disk winds may occur at field configurations at the right side
of the line.

###  3.2 Azimuthal profiles

To study the effect of the $B$-field on the CP pattern in more detail, we
created azimuthal profiles. These profiles show the CP of a disk surface ring
at $h/r=0.2$, measured counterclockwise from the near side of the disk. After
evaluating all possible magnetic field angles, we distinguished three
different shapes of azimuthal CP dependence: a dip followed by a peak
(“regular”), a peak followed by a dip (“reversed”), and two dips (“double
peaked”). We present examples for these three profile shapes in Fig. 5, where
we assume the dust composition with amorphous carbon and an inclination of
$i=55^{\circ}$. Because we find that the reversed profile occurs in only a
relatively small part of the parameter space, we mainly focus on the regular
and doubled-peaked profiles hereafter.

We find that the regular and double-peaked azimuthal CP profiles can place
additional constraints on the magnetic field configuration, as shown in Figs.
3 and 4. In those plots, we overlaid the distribution of the three azimuthal
profile shapes. In addition, we indicated the boundary of magnetic field
structures at which MHD disk winds by centrifugal force might occur with a
blue dashed line. In order to launch a wind along the magnetic field lines,
the centrifugal force needs to exceed the stellar gravity. This requirement is
met when $\sin\theta_{B}\geq 1/\sqrt{3\cos^{2}\phi_{B}+1}$. When $\phi_{B}=0$,
we obtain $\theta_{B}=30^{\circ}$, which agrees with the result of Blandford &
Payne (1982). Magnetic field configurations at the right side of the line have
field lines that make a sufficiently large angle with the $z$-axis to give
rise to MHD disk winds by centrifugal force. Almost all double-peaked CP
profiles are situated inside the disk-wind boundary, which suggests that these
types of profiles are produced by magnetic field lines that make a large angle
with the $z$-axis.

Figure 5: Azimuthal profiles of the degree of circular polarization produced
by aligned grains at a disk surface ring with $i=55^{\circ}$ and $h/r=0.2$.
The azimuthal angle is measured counterclockwise from the near side of the
disk. The three shown $B$-field configurations each produce a different CP
profile: ‘regular’ (black), ‘reversed’ (orange), and ‘double peaked’ (purple).

###  3.3 Effect of disk inclination

In order to study the impact of disk inclination angle, we also compare CP at
$i=35^{\circ}$ and $75^{\circ}$ in Fig. 6. The effect of disk inclination can
be seen in the differences between the boundaries separating the regular and
double-peaked profiles. For the case of $i=35^{\circ}$, the double-peaked
profile occurs even if $\theta_{B}$ is as small as $40^{\circ}$. For the case
of $i=75^{\circ}$, the double-peaked profiles only occur at even higher
$\theta_{B}$ values than in a lower inclination case. Therefore, as long as
the disk is moderately or highly inclined, the occurrence of the double-peaked
profile indicates a magnetic field configuration with a high $\theta_{B}$
value that is favorable for launching winds by centrifugal force.

Figure 6: Same as Fig. 3, but we assumed different inclinations:
$i=35^{\circ}$ for the upper panel and $i=75^{\circ}$ for the lower panel. The
boundary between the regular and double-peaked profiles shifts to the
left/right for lower/higher inclinations.

###  3.4 Effect of other dust-grain parameters

We examine how different dust models would affect our results by evaluating
the CP patterns for another dust composition and other axis ratios. In the
alternative dust composition, we replaced amorphous carbon with refractory
organics (Birnstiel et al., 2018). We find that this less absorbing material
results in less circular polarized light, with a maximum value of
$\sim\mkern-4.0mu0.4\%$ compared to $\sim\mkern-4.0mu6\%$ for the mixture with
amorphous carbon. These lower CP values can be physically explained as
follows. Circular polarization is produced when there is a phase lag between
dipole radiation along the major and minor axis of the grain. The amount of
the phase lag is related to the absorption efficiency and becomes zero when
grains are nonabsorbing (Bandermann & Kemp, 1973). Consequently, less
absorbing material results in a smaller phase lag and therefore weaker CP
production. Lower axis ratios also result in less CP because circular
polarization arises from the difference in polarizability along each spheroid
axis, which is smaller for more spherical particles. Fortunately, although
both grain composition and axis ratio affect the absolute value of CP, the
shape of the azimuthal profiles does not change. This demonstrates that the
occurrence of double-peaked profiles is independent of the dust composition
and grain-axis ratio, and is therefore a reliable probe of the magnetic field
structure, as long as the Rayleigh approximation holds. Therefore, if we
detect CP, its amplitude would be a useful diagnostic of grain properties.

As long as we work on the Rayleigh limit, the effects of axis ratio and dust
composition on the CP amplitude are degenerate. To better constrain dust
properties, a combined analysis with the degree of CP and disk-scattered flux
would be useful. This is because the imaginary part of the refractive index is
not only important for CP, but also for the single scattering albedo, which in
turn strongly affects disk-scattered flux.

##  4 Discussion

###  4.1 Role of linear and circular polarization in probing disk magnetic
fields

Our study, together with previous studies, demonstrates that polarization
signals in scattered optical and NIR light contain valuable information about
the $B$-field in protoplanetary disks. Yang & Li (2022) explored how linearly
polarized scattered light can be used to diagnose the magnetic field. Here, we
discuss how two types of polarization observations (linear and circular) can
complement each other and advance our understanding of disk magnetic fields.

It would be very useful to combine the linear and circular polarization
analysis because this allows us to exclude polarization signals that are
produced via pathways other than magnetically aligned grains, such as multiple
scattering (Canovas et al., 2015) or instrumental errors. Observing both
linear and circular polarization provides an independent check as to whether
or not the polarization signals are truly produced by aligned grains.

In addition, linear and circular polarization in disk-scattered NIR light are
sensitive to different magnetic field components. Linear polarization angles
seem more suitable to indicate the presence of the toroidal component, whereas
circular polarization profiles better probe the poloidal component with a
large angle to the $z$-axis. The former $B$-field configuration drives the
wind by a magnetic pressure gradient (e.g., Bai, 2017), while the latter does
this via centrifugal force (Blandford & Payne, 1982). Therefore, the
combination of these could narrow down the possible magnetic field structures
and eventually reveal a launching mechanism of MHD winds.

To study the linear and circular polarization signals as described in the
techniques of our study and of Yang & Li (2022), the disk needs to be
resolved. However, the azimuthally averaged CP values can be measured without
resolving the disk. We find that an integrated nonzero degree of circular
polarization already points towards the presence of aligned grains at the disk
surface, which is an essential requirement to study the magnetic field
structure using linear and circular polarization. The integrated CP value can
reveal disks that meet this requirement, which helps to search for suitable
targets.

###  4.2 Circular polarization from large aligned grains

In this study, we adopt the Rayleigh approximation, which is only applicable
for grains smaller than the wavelength. At a wavelength of
$\lambda=1.6\leavevmode\nobreak\ \mu$m, the Rayleigh approximation is
applicable when the grain radius is smaller than $\lambda/2\pi\sim
0.25\leavevmode\nobreak\ \mu$m. Here, we discuss how the presence of larger
grains would affect the CP and whether or not the Rayleigh limit is a valid
assumption.

Fukushima et al. (2020) calculated circularly polarized scattered light by
aligned grains in interstellar slab-like clouds. These authors found that
increasing grain radius results in increasing the degree of CP. They also
found that the CP pattern outside the Rayleigh limit would become less simple
than in the Rayleigh limit (see their Fig. 4). Therefore, the presence of
micron-sized grains at the disk surface would make much more complicated CP
patterns than those presented in this study.

However, one caveat of Fukushima et al. (2020) is that the authors only
considered compact micron-sized grains. At the disk surface, micron-sized dust
particles at the disk surface would be aggregates (e.g., Dominik & Tielens,
1997; Blum & Wurm, 2008), which are clusters of small single particles
(monomers) glued together by intermolecular forces, such as the van der Waals
force. Although these studies suggest that aggregates may have sizes beyond
the Rayleigh limit, the degree of polarization of scattered light by large
aggregates rather resembles that of individual monomers (e.g., West, 1991;
Kozasa et al., 1993; Volten et al., 2007; Tazaki & Dominik, 2022). Tazaki &
Dominik (2022) estimated the upper limit on the monomer radius to be
$0.4\,\rm{\mu m}$ for a set of protoplanetary disks based on polarimetric
observations. Therefore, as long as such dust aggregates are the dominant
population of the disk surface, and monomers in the aggregate are aligned, our
argument would remain approximately valid. Another way to circumvent possible
deviations from the Rayleigh solution would be to observe the disk at longer
wavelengths, such as $2.2\,\mu$m ($K$-band).

###  4.3 Do grains align with magnetic fields?

In the above sections, we assume that grains at the disk surface perfectly
align with magnetic fields. Here, we discuss to what extent this assumption is
reasonable.

Tazaki et al. (2017) and Yang & Li (2022) suggest that grain alignment with
the magnetic field, also known as $B$-RAT, is feasible at the disk surfaces
when grains contain a sufficient amount of superparamagnetic inclusions.
However, $B$-RAT is not the only known mechanism to align grains. Dust grains
at the disk surface can also become aligned with the stellar radiation, which
is called $k$-RAT (Lazarian & Hoang, 2007). $k$-RAT alignment predicts that
the grain’s minor axis becomes parallel to the incoming radiation direction.
In this case, we expect no circular polarization as long as the grain
polarization property is approximated by the Rayleigh limit. This is because
no CP is produced when the particle symmetry axis is either in the scattering
plane ($k$-RAT for oblate grains) or perpendicular to the incident light
direction ($k$-RAT for prolate grains) (Bandermann & Kemp, 1973).

Therefore, if CP observations end up with a nondetection, this could be
attributed to $k$-RAT alignment, although other reasons still exist, such as
very low alignment efficiency, $B$-RAT alignment with a symmetric $B$-field
structure, and so on. Conversely, if we detect a CP signal, this is likely
arising from $B$-RAT alignment. In this way, CP observations will shed light
on grain-alignment processes at the disk surface.

It is also worth mentioning that CP can be produced by multiple scattering
from nonaligned or spherical grains. However, such a CP signal will
efficiently vanish when it is integrated over the disk as long as we can
assume an axisymmetric disk structure. In contrast, as we discuss in Sect.
3.1, an integrated CP signal can be nonzero if it is due to aligned grains.
Therefore, if we detect a net nonzero integrated CP from an axisymmetric disk,
it is likely attributable to grains aligned with the $B$-field rather than the
multiple-scattering origin.

###  4.4 Future prospects

The next step in this study would be to compare our results with direct
observations of the degree of circular polarization in protoplanetary disks.
No direct circular polarization measurements of protoplanetary disks have been
published yet, but promising instruments and techniques are emerging. The
Subaru Coronagraphic Extreme Adaptive Optics (SCExAO) instrument on the Subaru
telescope will allow us to obtain circular polarization images of resolved
protoplanetary disks at NIR wavelengths. In addition, although not originally
designed for circular polarization measurements, VLT/SPHERE could also be used
to observe CP, as proposed by van Holstein et al. (2020).

One caveat of our model is that we assume single scattering to simplify the
disk radiative transfer processes. However, protoplanetary disks are optically
thick, and in general, multiple scattering plays a role, which can affect the
degree of polarization (e.g., Ma & Schmid, 2022). In addition, aligned grains
also produce nonazimuthal linear polarization (Yang & Li, 2022) which can be
converted into circular polarization via multiple scattering. Further
clarification of these effects is left for our future study.

##  5 Summary

In this study, we explored a new method to study the magnetic field structure
at the surface of protoplanetary disks using NIR circular polarization of
scattered light. Assuming a disk model and various magnetic field geometries,
we created synthetic maps of the degree of circular polarization produced by
scattering oblate spheroids at the disk surface. By analyzing the azimuthal
dependence and the integrated value, we studied to what extent the
configuration of the $B$-field is reflected in the CP patterns. We also
investigated the effects of disk inclination and dust grain parameters on our
results. We examined what we can learn from the CP disk images about the
launching mechanism of MHD disk winds, and how this method ties in with linear
polarization techniques. Our primary findings are as follows.

  * •

It is shown that magnetically aligned grains at the disk surface can produce
circular polarization signals. Also, circular polarization signals remain
nonzero even if the disk is not spatially resolved.

  * •

It is found that a disk exhibiting a CP pattern with a double-peaked azimuthal
profile likely has a poloidal magnetic field with a large angle with respect
to the $z$-axis. In contrast, circular polarization becomes zero for a
toroidal configuration.

  * •

The degree of CP strongly depends on dust composition. A more absorbing
composition, such as when grains are comprised of amorphous carbon, produces a
higher degree of CP. The composition does not change the CP azimuthal
profiles, as long as the Rayleigh approximation holds.

  * •

We find that circular polarization is complementary to the linear polarization
technique to probe the disk $B$-field because the combined analysis allows us
to break degeneracy, and the polarization signals trace different magnetic
field components.

###### Acknowledgements.

We thank an anonymous referee for useful comments. In addition, we thank C.
Dominik and J. Hashimoto for a fruitful discussion. R.T. acknowledges the JSPS
overseas research fellowship.

## References

  * Andersson et al. (2015) Andersson, B. G., Lazarian, A., & Vaillancourt, J. E. 2015, ARA&A, 53, 501
  * Avenhaus et al. (2018) Avenhaus, H., Quanz, S. P., Garufi, A., et al. 2018, ApJ, 863, 44
  * Bai (2017) Bai, X.-N. 2017, ApJ, 845, 75
  * Bandermann & Kemp (1973) Bandermann, L. W. & Kemp, J. C. 1973, Monthly Notices of the Royal Astronomical Society, 162, 367
  * Birnstiel et al. (2018) Birnstiel, T., Dullemond, C. P., Zhu, Z., et al. 2018
  * Blandford & Payne (1982) Blandford, R. D. & Payne, D. G. 1982, Monthly Notices of the Royal Astronomical Society, 199, 883
  * Blum & Wurm (2008) Blum, J. & Wurm, G. 2008, ARA&A, 46, 21
  * Bohren & Huffman (1983) Bohren, C. F. & Huffman, D. R. 1983, Absorption and scattering of light by small particles (Wiley), 530
  * Bruggeman (1935) Bruggeman, D. A. G. 1935, Annalen der Physik, 416, 636
  * Canovas et al. (2015) Canovas, H., Ménard, F., de Boer, J., et al. 2015, A&A, 582, L7
  * D’Alessio et al. (2001) D’Alessio, P., Calvet, N., & Hartmann, L. 2001, ApJ, 553, 321
  * Dominik et al. (2021) Dominik, C., Min, M., & Tazaki, R. 2021, OpTool: Command-line driven tool for creating complex dust opacities, Astrophysics Source Code Library, record ascl:2104.010
  * Dominik & Tielens (1997) Dominik, C. & Tielens, A. G. G. M. 1997, ApJ, 480, 647
  * Draine (2003) Draine, B. T. 2003, ARA&A, 41, 241
  * Fukushima et al. (2020) Fukushima, H., Yajima, H., & Umemura, M. 2020, Monthly Notices of the Royal Astronomical Society, 496, 2762
  * Harrison et al. (2021) Harrison, R. E., Looney, L. W., Stephens, I. W., et al. 2021, ApJ, 908, 141
  * Henning & Stognienko (1996) Henning, T. & Stognienko, R. 1996, A&A, 311, 291
  * Kataoka et al. (2014) Kataoka, A., Okuzumi, S., Tanaka, H., & Nomura, H. 2014, A&A, 568, A42
  * Kozasa et al. (1993) Kozasa, T., Blum, J., Okamoto, H., & Mukai, T. 1993, A&A, 276, 278
  * Lazarian (2007) Lazarian, A. 2007, J. Quant. Spec. Radiat. Transf., 106, 225
  * Lazarian & Hoang (2007) Lazarian, A. & Hoang, T. 2007, MNRAS, 378, 910
  * Ma & Schmid (2022) Ma, J. & Schmid, H. M. 2022, A&A, 663, A110
  * Pollack et al. (1996) Pollack, J. B., Hubickyj, O., Bodenheimer, P., et al. 1996, Icarus, 124, 62
  * Stephens et al. (2020) Stephens, I. W., Fernández-López, M., Li, Z.-Y., Looney, L. W., & Teague, R. 2020, ApJ, 901, 71
  * Stolker et al. (2016) Stolker, T., Dominik, C., Min, M., et al. 2016, A&A, 596, A70
  * Tazaki & Dominik (2022) Tazaki, R. & Dominik, C. 2022
  * Tazaki et al. (2017) Tazaki, R., Lazarian, A., & Nomura, H. 2017
  * Teague et al. (2021) Teague, R., Hull, C. L. H., Guilloteau, S., et al. 2021, ApJ, 922, 139
  * van Holstein et al. (2020) van Holstein, R. G., Girard, J. H., de Boer, J., et al. 2020, A&A, 633, A64
  * Volten et al. (2007) Volten, H., Muñoz, O., Hovenier, J. W., et al. 2007, A&A, 470, 377
  * Warren & Brandt (2008) Warren, S. G. & Brandt, R. E. 2008, Journal of Geophysical Research: Atmospheres, 113
  * West (1991) West, R. A. 1991, Appl. Opt., 30, 5316
  * Woitke et al. (2016) Woitke, P., Min, M., Pinte, C., et al. 2016, A&A, 586, A103
  * Yang (2021) Yang, H. 2021, ApJ, 911, 125
  * Yang & Li (2022) Yang, H. & Li, Z.-Y. 2022, arXiv e-prints, arXiv:2203.03213
  * Zubko et al. (1996) Zubko, V. G., Mennella, V., Colangeli, L., & Bussoletti, E. 1996, MNRAS, 282, 1321


# UML: A Universal Monolingual Output Layer for Multilingual ASR

###### Abstract

Word-piece models (WPMs) are commonly used subword units in state-of-the-art
end-to-end automatic speech recognition (ASR) systems. For multilingual ASR,
due to the differences in written scripts across languages, multilingual WPMs
bring the challenges of having overly large output layers and scaling to more
languages. In this work, we propose a universal monolingual output layer (UML)
to address such problems. Instead of one output node for only one WPM, UML re-
associates each output node with multiple WPMs, one for each language, and
results in a smaller monolingual output layer shared across languages.
Consequently, the UML enables to switch in the interpretation of each output
node depending on the language of the input speech. Experimental results on an
11-language voice search task demonstrated the feasibility of using UML for
high-quality and high-efficiency multilingual streaming ASR.

Index Terms—  Multilingual, ASR, word-piece, UTF-8 byte

##  1 Introduction

Automatic speech recognition (ASR) is used by a massive amount of users.
Although more than 7,100 languages are actively spoken in the world [1], only
about a hundred most common ones have commercial ASR products, restricting the
benefits of such useful artificial intelligence technology for people. To
extend ASR to cover more languages and users [2], it is better to serve many
languages with a single multilingual ASR system instead of with many
monolingual ASR systems, which not only enables code-switch and resource
sharing across languages but also reduces maintenance cost.

The choice of subword units is critical for multilingual ASR. Phonemes are
often used in the modularised ASR framework [3, 4, 5, 6], which requires
complex text processing to convert pronunciations to words. Text units are
often used to resolve the issue. Graphemes, as a collection of characters, can
result in an uneven distribution of subword units across languages [7, 8, 9,
10]. As an alternative, bytes can be used as the common subword units shared
across languages by decomposing each character into multiple bytes [11]. Word-
piece models (WPMs) [12, 13] and similarly sentence-piece models [14],
obtained by segmenting words or sentences into pieces, are superior in
performance to graphemes and bytes [15, 16] and are therefore the de facto
choices in monolingual end-to-end ASR and natural language processing.
However, it is inevitable to have a large number of multilingual WPMs when
multiple writing systems are involved [17, 18, 19, 20]. A solution is to use
separate monolingual output layers and decoders [21, 22], which considerably
increases the storage space and requires the management of more concurrent
beam search procedures when scaling up to more languages.

In this paper, we propose a universal monolingual output layer (UML) to
resolve the aforementioned issues. Compared to using separate monolingual
output layers or decoders, the UML reuses the same output layer for different
languages, simply by re-associating each output node with different subword
units for different languages. This can be achieved with only a tokeniser-
level change, which allows a multilingual decoder to keep almost the same
structure and size as the monolingual decoder. Compared to graphemes and
bytes, better performance is expected as monolingual WPMs with controllable
sizes can be used in UML. Streaming ASR experiments on a massive multilingual
voice search setup [17] with 11 languages and known language identifiers
(LIDs) showed that compared with the baseline with 8K output nodes, UML with
four thousand (4K) output nodes and a rule-based language grouping achieved
the same word error rate (WER) by reducing about 40% of parameters in
decoders. A further reduction of the decoder size was achieved by replacing
WPMs with bytes for logogram-coded languages (e.g. Chinese and Japanese),
which resulted in better WERs than a byte-only baseline with an output layer
size of 512.

In the rest of the paper: Sec. 2 reviews related work. Sec. 3 presents the
proposed UML method. Sec. 4 and 5 are the experimental setup and results. We
conclude in Sec. 6.

##  2 Related Work

Phonetic units, including shared and language-dependent phonemes, decision-
tree clustered context-dependent phonemes [3, 5, 4, 6], and articulatory
features [23], are commonly used in modularised ASR. Although it is natural to
model the pronunciation variations across languages with phonetic units, the
need for lexicons and language models made it less suitable for on-device
applications.

Context-dependent graphemes were first developed for modularised ASR. Decision
tree clustering is often used to cover unseen graphemes and complex segmental
writing scripts, whose question sets can be derived based on the Unicode
character descriptions [24]. Context-independent graphemes are prevalent in
end-to-end ASR, such as recurrent neural network transducer (RNN-T). For
multilingual ASR, a union of monolingual graphemes is used [7, 8, 9, 10]. By
segmenting each grapheme into one to four Unicode bytes based on its UTF-8
encoding, the same bytes can be used to build ASR to transcribe any writing
scripts [11]. Having a small number of common “sub-character” units, byte ASR
models require more steps to decode than graphemes and the partial outputs are
not human-readable.

By incrementally combining graphemes into frequent pieces of words, the
vocabulary size can be controlled using WPMs [12]. Since fewer decoding steps
can be achieved with more WPMs, with a sufficient number of WPMs, RNN-T models
were found not only to produce better WERs than with graphemes [15], but also
to produce similar WERs using a full history decoder and a limited history
decoder [25]. As graphemes can be considered as the minimum WPMs, in practice,
WPMs and graphemes are often mixed to use in multilingual ASR. For instance,
similar numbers of English WPMs and Chinese characters are often used together
in English-Chinese code-switch studies [26, 27], to avoid the 26 English
characters being overwhelmed by thousands of Chinese characters. As an
extension of WPMs, the sentence-piece model allows cross-word subword
segmentation based on different algorithms [14].

Although WPMs are more commonly adopted over graphemes for monolingual ASR, an
output layer with multilingual WPMs generated by pooling all monolingual data
together can often be overly large when many languages and writing scripts are
integrated [28, 29]. Separate monolingual output layers can be used as a
solution, which can be dated back to the previous works with phonemes and
graphemes [30, 31]. In RNN-T, other parts of the decoder and even a part of
the encoder can be monolingual as well [21, 22]. The UML method proposed in
this paper differs from all these works by tying all monolingual output layers
together, which enables the monolingual ASR decoder structure to be used for
multilingual ASR. It is worth noting although UML is introduced for WPMs, it
is a generic method applicable to other kinds of subword units as well.

##  3 Proposed UML Method

###  3.1 A Universal Monolingual Output Layer

UML is a monolingual output layer shared by all languages. Specifically, let
$L$ be the number of languages, $V_{l}$ be the number of WPMs for the $l$-th
language, in UML, each output node $o$ is mapped to $L$ different monolingual
WPMs $W_{1,o},\ldots,W_{L,o}$ for $L$ different languages, whereas $o$ is
mapped to only one WPM in a conventional output layer. Let $H$ be the input
dimension of the output layer, compared to alternative methods, UML enables
the use of more WPMs with fewer parameters:

  * •

UML uses only one $H\times\max(V_{1},\ldots,V_{L})$-dimensional (-dim) output
layer to model the $\sum\nolimits_{l=1}^{L}V_{l}$ WPMs.

  * •

The method using a conventional output layer for all multilingual WPMs [32,
17, 18, 19, 20] requires to use a $H\times(\sum\nolimits_{l=1}^{L}V_{l})$-dim
layer for the $\sum\nolimits_{l=1}^{L}V_{l}$ WPMs.

  * •

The methods in [21, 22] use $L$ separate monolingual output layers whose
dimensions are $H\times V_{1},\ldots,H\times V_{L}$. It requires
$H\times(\sum\nolimits_{l=1}^{L}V_{l})$ parameters to model the
$\sum\nolimits_{l=1}^{L}V_{l}$ WPMs.

In UML, since each WPM is determined jointly by the LID and the output node
index, LIDs need to be taken into account in inference. At test-time, let
$\mathbf{x}$, $\mathbf{y}$, and $\mathbf{z}$ be the input, output, and LID
prediction sequences of an utterance, $\mathbf{y}^{*}$ is the decoding result,
the maximum a posteriori decoding rule of ASR can be modified to marginalise
the LID predictions as

$\displaystyle\mathbf{y}^{*}=\arg\max
P(\mathbf{y}|\mathbf{x})=\sum\nolimits_{\mathbf{z}}P(\mathbf{z}|\mathbf{x})P(\mathbf{y}|\mathbf{x},\mathbf{z}),$
(1)

where $P(\mathbf{y}|\mathbf{x},\mathbf{z})$ are the output distributions of a
UML-based LID-aware multilingual decoder and a LID predictor,
$P(\mathbf{z}|\mathbf{x})$ are the LID predictions. Eqn. (1) can also be
applied to the training, which has $P(\mathbf{y}|\mathbf{x})$ in the loss
function. This allows LID prediction and code-switch to be handled jointly and
explicitly.

There are a few key advantages to using the UML:

  * •

First, the UML allows multilingual ASR to scale gracefully to any number of
languages without increasing the output layer size [28, 29]. This is smaller
in size than the conventional multilingual output layer and improves the
computation efficiency in both RNN-T training and decoding. It also reduces
the difficulty to adapt an existing system to new languages by reusing the
same output layer.

  * •

Second, the UML provides us with fine-grained control of WPM tying across
languages. Languages with similar writing systems (e.g. English and French)
often have duplicated WPMs. To avoid this, such languages can be grouped to
derive a combined set of WPMs. In this setting, the UML is no longer fully
“monolingual” by tying the same WPMs within each group. Meanwhile, the
flexibility is reserved to untie some WPMs with the same written form but
dissimilar pronunciations and contexts, such as the same characters in Chinese
and Japanese, by setting them into different groups.

  * •

Third, the UML enables us to handle contextual biasing in a per-language way.
Due to a page limit, contextual biasing is not included in this paper, but it
is necessary for real-world ASR applications that can require us to
incorporate thousands of biasing words (such as contact names) into ASR. In
multilingual ASR, each UML output distribution is “monolingual” enough to keep
the relevant biasing word list monolingual, without the need to combine them
into an overly large list.

###  3.2 Applying UML to Multilingual RNN-T

Fig. 1: A sketch map of an RNN-T decoder with WPM-based UML for English and
Chinese. LID to the prediction network is optional.

In this section, the application of UML in an RNN-T is considered. As shown in
Fig. 1, the output layer of an RNN-T decoder produces its current output
distribution based on the joint representation derived from the joint network.
The standard joint network is a single hidden layer in RNN-T, which fuses the
acoustic representation derived from the encoder based on the input speech,
with the text representation derived from the prediction network based on the
outputs from the previous time steps. Not only the output layer but also the
prediction network needs to be considered when using UML.

Since LID is missing from the prediction network, the input to the prediction
network, which is the output node index of the previous step, is ambiguous as
it can not determine a WPM alone in UML. To disambiguate, the LID information
can be leveraged in the prediction network, by either augmenting each output
node index with a LID or using language-specific parameters. As shown in Eqn.
(1), this requires expanding the search space in decoding, by re-weighting
each path in the beam with the relevant LID probability. Alternatively, if LID
is ignored in the prediction network, the LID predictor, such as the one
proposed in [33], can be decoupled from the beam search that simplifies UML
implementation.

In the applications where per-utterance oracle LIDs are available,
$P(\mathbf{z}|\mathbf{x})$ are 0-1 hard probabilities and Eqn. (1) becomes
$\mathbf{y}^{*}=P(\mathbf{y}|\mathbf{x},\mathbf{z})$. Using UML in this case
does not require any change to model training and testing. Only the output
node indexes are interpreted according to the language, which can be achieved
simply by switching among a set of monolingual tokenisers controlled by the
LIDs. Here a tokeniser converts a sequence of output node indexes into a
sequence of WPMs and joins them into word-level transcriptions, or vice versa.
As a result, a UML-based multilingual ASR system can have the same decoder
structure and size as a monolingual ASR system, which results in lower
storage, computation, and energy costs. In this paper, we focus on UML with
oracle LIDs.

In UML, having a different number of WPMs for different languages requires us
to apply partial matrix multiplications and softmax functions with different
sizes. For simplicity, we unify all sets of WPMs to have the same size $V$.
The UML in this case can be viewed as folding the multilingual decoder output
layer $L$ times. Furthermore, to reduce the output layer size to be smaller
than the number of Chinese characters, bytes can be used as the alternative
subword units for Chinese and Japanese, which have a different vocabulary size
than those of the WPMs used for other languages.

##  4 Experimental Setup

###  4.1 Data

Our dataset consists of 11 languages: Arabic, Chinese, German, English,
Spanish, French, Hindi, Italian, Japanese, Portuguese and Russian. For each
language, the training data is collected from two application domains: Voice
Search and YouTube, with complex noise and acoustic conditions. All data are
de-identified and human transcribed. All data collection and handling abide by
Google AI Principles [34]. Table 1 shows the distribution in more detail. Each
test set utterance is less than 5.5 seconds in length and is sampled from the
traffic of Google’s Voice Search product. All test sets are similarly de-
identified and human transcribed for evaluation. All data are normalised to
have zero mean and unit variance based on the statistics collected from the
whole training set.

LID | Language | #Utterance | Duration | #Utterance
---|---|---|---|---
Train (M) | Train (K) | Test (K)
ar | Arabic | 3.8 | 6.4 | 4.1
de | German | 3.8 | 3.8 | 7.6
en | English | 18.1 | 15.7 | 9.8
es | Spanish | 45.5 | 52.8 | 17.5
fr | French | 10.8 | 14.7 | 7.1
hi | Hindi | 14.2 | 29.8 | 6.5
it | Italian | 13.0 | 21.3 | 14.8
ja | Japanese | 10.9 | 11.5 | 11.2
pt | Portuguese | 13.4 | 20.7 | 12.3
ru | Russian | 5.3 | 12.4 | 11.1
zh | Chinese | 0.9 | 5.1 | 6.1
Total |  | 135.0 | 182.6 | 108.1
Table 1: Data statistics. The number of utterances (#Utterance) is in millions
(M) and the duration is in thousands (K) of hours.

###  4.2 Model

The Conformer-based cascaded encoder RNN-T model is used in this study [35].
The acoustic features are 128-dim log Mel filter banks computed on a 32ms
window with a 10ms shift. SpecAugment is used to improve model robustness.
Every 4 contiguous frames with 3 frames on the left are stacked to form a
512-dim input representation, which is further sub-sampled with one frame
overlap to achieve a 30ms frame rate. Oracle LID vectors are one-hot-coded and
appended to the acoustic features. The causal encoder has 46 million (M)
parameters and consists of 3 convolution layers followed by seven Conformer
blocks. Causal convolution and left-context attention layers are used in the
Conformer block to strictly exclude any right context frame for streaming
purposes. The right-context encoder with 100M parameters uses a 640-dim linear
projection to transform the input features, followed by ten 640-dim Conformer
blocks and a final linear normalisation layer. More details about the encoder
structures and training configurations are in [36].

Though separate decoders with identical structures are used in all
experiments, only WERs from the 2nd-pass decoder are reported. Each decoder
has an embedding prediction network [25] operating on 2 non-blank model
predictions with 640-dim embeddings, which is termed as an embedding decoder.
For the output layer size $O$, there are two 640$\times O$-dim embedding
layers. The two 640-dim embeddings for the previous two non-blank predictions
are concatenated and projected down to a 640-dim text representation,
resulting in a 1280$\times$640-dim linear projection. Two projections are used
to further transform the encoder and prediction network outputs separately.
The joint network fuses these transformed representations into a 640-dim
vector and then passes it to the final 640$\times O$-dim softmax output layer.
The total number of parameters for each decoder is 18.1M when $O$ is 8K. A
decoder structure with an LSTM prediction network is evaluated for comparison,
which has 26.5M parameters. The LSTM prediction network consists of two
projected LSTM layers, whose memory cell and hidden representations are
1280-dim and 640-dim. A bilinear pooling (BP) joint network, referring to the
Eqns. (11) and (12) in [36], can be used to improve the decoder performance,
which increases the joint network size by 1.1M parameters. The LID information
is not used in the prediction network of the UML systems unless claimed
explicitly.

##  5 Results and Discussions

Table 2: Comparisons on test set %WERs and numbers of decoder parameters
(Decoder #Params) in million (M). The “B” and “U” systems are the baseline and
UML systems. “512”, “1k”, “2k”, “4k”, “6k” and “8k” are the output layer
sizes. The “G”s are followed by a number of distinct languages or groups used
in UML, and “G5Mix” replaces the WPMs of Chinese and Japanese with bytes in G5
in UML. “LSTM”, “BP”, and “LID” refer to the use of the LSTM prediction
network, the BP joint network, and the LIDs in the prediction network
respectively.

System |  Decoder #Params |  Mean %WER | Per-language %WERs
---|---|---|---
ar | de | en | es | fr | hi | it | ja | pt | ru | zh
B0${}^{\text{8kG1}}$ | 18.1M | 11.3 | 11.7 | 13.2 | 7.8 | 6.5 | 9.8 | 19.9 | 9.3 | 14.2 | 7.7 | 12.3 | 11.4
B1${}^{\text{8kG1}}_{\text{LSTM}}$ | 26.5M | 10.9 | 10.8 | 12.9 | 7.6 | 6.1 | 9.6 | 19.8 | 9.0 | 13.3 | 7.5 | 11.7 | 11.9
B2${}^{\text{Bytes}}$ | 3.1M | 12.6 | 12.3 | 14.1 | 8.8 | 7.2 | 11.1 | 22.4 | 10.5 | 17.1 | 8.9 | 14.3 | 11.4
U0${}^{\text{6kG11}}$ | 14.2M | 12.0 | 12.4 | 14.0 | 8.1 | 7.2 | 10.6 | 20.2 | 10.5 | 16.0 | 8.4 | 13.1 | 11.9
U1${}^{\text{6kG7}}$ | 14.2M | 11.7 | 11.7 | 13.6 | 8.0 | 6.9 | 10.1 | 20.0 | 9.8 | 15.8 | 8.0 | 12.7 | 11.7
U2${}^{\text{6kG5}}$ | 14.2M | 11.7 | 11.5 | 13.2 | 7.9 | 6.5 | 10.0 | 20.0 | 9.7 | 16.6 | 8.1 | 12.8 | 12.0
U3${}^{\text{6kG6}}$ | 14.2M | 11.5 | 11.6 | 13.2 | 7.9 | 6.5 | 9.9 | 19.7 | 9.6 | 15.3 | 8.0 | 12.8 | 11.8
U4${}^{\text{6kG6}}_{\text{BP}}$ | 15.3M | 10.9 | 11.1 | 12.3 | 7.1 | 6.2 | 9.4 | 19.5 | 9.0 | 13.9 | 7.5 | 12.0 | 11.9
U5${}^{\text{4kG6}}$ | 10.2M | 11.7 | 11.7 | 13.5 | 7.8 | 6.6 | 10.0 | 19.8 | 9.8 | 15.6 | 8.0 | 12.9 | 13.1
U6${}^{\text{4kG6}}_{\text{BP}}$ | 11.3M | 11.3 | 10.9 | 12.1 | 7.3 | 6.1 | 9.4 | 19.4 | 9.0 | 17.2 | 7.4 | 11.8 | 13.5
U7${}^{\text{4kG6}}_{\text{LID}}$ | 10.2M | 11.7 | 11.5 | 13.3 | 7.8 | 6.6 | 10.1 | 19.8 | 9.7 | 15.6 | 7.9 | 12.9 | 13.3
U8${}^{\text{2kG6}}$ | 6.3M | 12.5 | 11.3 | 13.5 | 7.9 | 6.5 | 10.0 | 19.9 | 9.9 | 16.8 | 7.9 | 12.9 | 21.2
U9${}^{\text{1kG5Mix}}_{\text{BP}}$ | 4.7M | 11.9 | 11.9 | 13.8 | 8.4 | 6.9 | 10.4 | 20.2 | 10.2 | 16.0 | 8.4 | 13.7 | 11.3
U10${}^{\text{512G5Mix}}_{\text{BP}}$ | 3.7M | 12.1 | 11.8 | 14.3 | 8.3 | 6.9 | 10.6 | 20.4 | 10.3 | 16.4 | 8.7 | 14.0 | 11.5

Full ASR results are presented in Table 2 and discussed as follows.

Baseline models (B0–B2). B0${}^{\text{8kG1}}$ is a baseline system using a
single conventional output layer with 8K multilingual WPMs. The multilingual
WPMs were generated by pooling the training data of all languages into one
group denoted as G1. We use the natural distribution of each language in WPM
generation as well as the mini-batch sampling in ASR training. For on-device
applications, the smaller decoder is always preferred for better efficiency
and latency. In our 8K vocabulary embedding decoder, most parameters are in
the output layer and embedding look-up tables. The 8K WPM set is the smallest
we can build without bringing many out-of-vocabulary (OOV) tokens due to the
reduction of Chinese characters. B1${}^{\text{8kG1}}_{\text{LSTM}}$ is another
baseline with an LSTM prediction network. Our third baseline
B2${}^{\text{Bytes}}$ has 384 output nodes including 256 for bytes and 128 for
special tokens. Due to the small decoder size, B2${}^{\text{Bytes}}$ has much
worse WER than B0${}^{\text{8kG1}}$ and B1${}^{\text{8kG1}}_{\text{LSTM}}$,
apart from Chinese.

UML feasibility and language grouping (U0-U4). To test the feasibility of the
UML approach, we start with systems U0–U4, whose structures are similar to B1
but have a 6K UML instead of an 8K conventional output layer. 6K is the least
WPMs for Chinese and Japanese on our data without causing OOVs.
U$0^{\text{6kG11}}$ has 11 distinct sets of monolingual WPMs without any
grouping. G7 groups the Germanic (de and en) and the Romance (es, fr, it, and
pt) languages separately, and hence U$1^{\text{6kG7}}$ has 7 distinct
languages or groups. To combine all languages with similar writing scripts, G5
further merges the Germanic and Romance languages into one group, and Chinese
and Japanese into another group, leading to U$2^{\text{6kG5}}$ with 5 distinct
language groups. From the results, both U$1^{\text{6kG7}}$ and
U$2^{\text{6kG5}}$ have lower WERs than U$0^{\text{6kG11}}$, which are still
worse than B0${}^{\text{8kG1}}$. This proves that building a UML-based system
without any grouping is difficult. U$2^{\text{6kG5}}$ were worse than
U$1^{\text{6kG7}}$ in Chinese and Japanese, but outperformed in other
languages, indicating that even the same characters in Chinese and Japanese
can have very different pronunciations and contexts that are not suitable to
share the same nodes. Therefore, we built another group G6, by splitting
Chinese and Japanese from G5, and the resulting system U$3^{\text{6kG6}}$
indeed had improved WERs in these two languages. Such fine-grained control is
a benefit of using UML. Further replacing the joint network in
U$3^{\text{6kG6}}$ with BP with a small parameter quantity increase, resulted
in our best-performing UML system U$4^{\text{6kG6}}_{\text{BP}}$, which has
the same WER as our best LSTM-based baseline
B1${}^{\text{8kG1}}_{\text{LSTM}}$ with much fewer parameters.
U$4^{\text{6kG6}}_{\text{BP}}$ outperformed B0${}^{\text{8kG1}}$ by relatively
3.5% lower WER and reduced 15% of the parameters.

Reducing WPM-based UML size to the limit (U5-U8). Next, we further reduced the
size of all G6 WPM sets to 4K and 2K. U5${}^{\text{4kG6}}$ have a 0.2%
increase in the mean WER compared with U3${}^{\text{6kG6}}$, while most of the
increase was caused by the OOVs in Chinese. By further using the BP joint
network, U6${}^{\text{4kG6}}_{\text{BP}}$ has another 3.5% relative WER
reduction, and the result is the same mean WER as B0${}^{\text{8kG1}}$ while
saving about 40% of the decoder parameters. Meanwhile, we tested the
importance of including the LID information in the prediction network. Two
additional $11\times 640$-dim projections are used to transform the LIDs and
their outputs are added to the 640-dim embeddings of the previous two steps
accordingly. The resulting system U7${}^{\text{4kG6}}_{\text{LID}}$ had the
same mean WER as U5${}^{\text{4kG6}}$, meaning it is not necessary to
disambiguate the output node indexes with the LID information in this case.
Furthermore, a 2K WPM-based UML system, U8${}^{\text{2kG6}}$ is built, with
even more OOVs in Chinese and Japanese than U5${}^{\text{4kG6}}$. The mean WER
is much worse than U5${}^{\text{4kG6}}$ as expected, but the WER increases are
mostly in Chinese and Japanese. This inspires us to further improve the method
by using bytes for these two languages.

Using UML to mix WPMs and Bytes (U9, U10). To avoid the degradation of WERs
due to OOV issues, we propose to use bytes for the logogram-coded languages,
WPMs for other alphabet-coded languages and mix such different types of
subword units using UML. The resulting 1K output-node and 512 output-node
systems with BP joint network, U9${}^{\text{1kG5Mix}}_{\text{BP}}$ and
U10${}^{\text{512kG5Mix}}_{\text{BP}}$, both achieved better WERs with fewer
parameters than U8${}^{\text{2kG6}}$. In addition,
U10${}^{\text{512kG5Mix}}_{\text{BP}}$ had a 0.5% mean absolute WER reduction
compared to B2${}^{\text{Bytes}}$, a byte baseline with a similar amount of
decoder parameters. This again reveals the importance of the flexibility
provided by the UML.

##  6 Conclusions

In this work, we develop the method of UML for multilingual ASR, which reuses
each output node for a different subword unit for each language, making the
output layer monolingual and language universal. A simple tokeniser-level
change addresses many challenges in building massive-scale multilingual ASR
such as the significant increase in output layer size when integrating more
languages with one system. Production scale experiments were conducted on 11
languages that justified the feasibility of building UML-based multilingual
ASR systems with high accuracy. The potential of reducing the multilingual
output size was also explored. When using a 4K-dim UML, the same averaged WER
can be achieved compared to the baseline with a conventional 8K-dim WPM output
layer. This reduces about 40% of parameters in the decoder and considerably
improves RNN-T training and test speed. Further reductions in size can be
achieved by mixing WPMs with bytes using UML.

## References

  * [1] SIL International,  “Ethnologue: Languages of the world,” https://www.ethnologue.com/, 2022.
  * [2] M. Slaney, R.F. Lyon, R. Garcia, B. Kemler, C. Gnegy, K. Wilson, D. Kanevsky, S. Savla, and V.G. Cerf,  “Auditory measures for the next billion users,”  Ear and Hearing, vol. 41 Suppl 1, pp. 131S–139S, 2020.
  * [3] K.M. Knill, M.J.F. Gales, S.P. Rath, P.C. Woodland, C. Zhang, and S.-X. Zhang,  “Investigation of multilingual deep neural networks for spoken term detection,”  in Proc. ASRU, 2013.
  * [4] C. Zhang and P.C. Woodland,  “Standalone training of context-dependent deep neural network acoustic models,”  in Proc. ICASSP, Florence, 2014.
  * [5] N.T. Vu, D. Imseng, D. Povey, P. Motlicek, T. Schultz, and H. Bourlard,  “Multilingual deep neural network based acoustic modeling for rapid language adaptation,”  in Proc. ICASSP, Florence, 2014.
  * [6] X. Li, S. Dalmia, J. Li, M. Lee, P. Littell, J. Yao, A. Anastasopoulos, D.R. Mortensen, G. Neubig, A.W. Black, and F. Metze,  “Universal phone recognition with a multilingual allophone system,”  in Proc. ICASSP, Barcelona, 2020.
  * [7] S. Watanabe, T. Hori, and J. Hershey,  “Language independent end-to-end architecture for joint language identification and speech recognition,”  in Proc. ASRU, Okinawa, 2017.
  * [8] S. Kim and M.L. Seltzer,  “Towards language-universal end-to-end speech recognition,”  in Proc. ICASSP, Calgary, 2018.
  * [9] W. Hou, Y. Dong, B. Zhuang, L. Yang, J. Shi, , and T. Shinozaki,  “Large-scale end-to-end multilingual speech recognition and language identification with multi-task learning,”  in Proc. Interspeech, Shanghai, 2020.
  * [10] S. Toshniwal, T. Sainath, R. Weiss, B. Li, P. Moreno, E. Weinstein, and K. Rao,  “Multilingual speech recognition with a single end-to-end model,”  in Proc. ICASSP, Calgary, 2018.
  * [11] B. Li, Y. Zhang, T. Sainath, Y. Wu, and W. Chan,  “Bytes is all you need,”  in Proc. ICASSP, Brighton, 2019.
  * [12] M. Schuster and K. Nakajima,  “Japanese and Korean voice search,”  in Proc. ICASSP, Kyoto, 2012.
  * [13] R. Sennrich, B. Haddow, and A. Birch,  “Neural machine translation of rare words with subword units,”  in Proc. ACL, 2016.
  * [14] T. Kudo,  “Subword regularization: Improving neural network translation models with multiple subword candidates,”  in Proc. ACL, Melbourne, 2018.
  * [15] K. Rao, R. Prabhavalkar, and H. Sak,  “Exploring architectures, data and units for streaming end-to-end speech recognition with RNN-Transducer,”  in Proc. ASRU, Okinawa, 2017.
  * [16] A. Zeyer, K. Irie, R. Schlüter, and H. Ney,  “Improved training of end-to-end attention models for speech recognition,”  in Proc. Interspeech, Hyderabad, 2018.
  * [17] B. Li, R. Pang, and T. Sainath et al.,  “Scaling end-to-end model for large-scale multilingual ASR,”  arXiv:2104.14830, 2021.
  * [18] L. Zhou, J. Li, E. Sun, and S. Liu,  “A configurable multilingual model is all you need to recognize all languages,”  in Proc. ICASSP, Singapore, 2022.
  * [19] V. Pratap, A. Sriram, P. Tomasello, A. Hannun, V. Liptchinsky, G. Synnaeve, and R. Collobert,  “Massively multilingual ASR: 50 languages, 1 model, 1 billion parameters,”  in Proc. Interspeech, Shanghai, 2020.
  * [20] N.-Q. Pham, A. Waibel, and J. Niehues,  “Adaptive multilingual speech recognition with pretrained models,”  in Proc. Interspeech, Incheon, 2022.
  * [21] V. Joshi, A. Das, E. Sun, R.R. Mehta, J. Li, and Y. Gong,  “Multiple softmax architecture for streaming multilingual end-to-end ASR systems,”  in Proc. Interspeech, Brno, 2021.
  * [22] S. Mavandadi, B. Li, C. Zhang, B. Farris, T. Sainath, and T. strohman,  “A truly multilingual first pass and monolingual second pass streaming on-device ASR system,”  in Proc. SLT, Doha, 2022.
  * [23] C. Zhang, Y. Liu, and C.-H. Lee,  “Detection-based accented speech recognition using articulatory features,”  in Proc. ASRU, Hawaii, 2011.
  * [24] M.J.F. Gales, K.M. Knill, and A. Ragni,  “Unicode based graphemic systems for limited resource languages,”  in Proc. ICASSP, South Brisbane, 2015.
  * [25] R. Prabhavalkar, Y. He, D. Rybach, S. Campbell, A. Narayanan, T. Strohman, and T.N. Sainath,  “Less is more: Improved RNN-T decoding using limited label context and path merging,”  in Proc. ICASSP, Toronto, 2021.
  * [26] Z. Zeng, Y. Khassanov, V.T. Pham, H. Xu, E.S. Chng, and H. Li,  “On the end-to-end solution to Mandarin-English code-switching speech recognition,”  in Proc. Interspeech, 2019.
  * [27] S. Dalmia, Y. Liu, S. Ronanki, and K. Kirchhoff,  “Transformer-transducers for code-switched speech recognition,”  in Proc. ICASSP, Toronto, 2021.
  * [28] O. Adams, M. Wiesner, S. Watanabe, and D. Yarowsky,  “Massively multilingual adversarial speech recognition,”  in Proc. NAACL-HLT, Minneapolis, 2019.
  * [29] B. Li, R. Pang, Y. Zhang, T.N. Sainath, T. Strohman, P. Haghani, and et al.,  “Massively multilingual ASR: A lifelong learning solution,”  in Proc. ICASSP, Singapore, 2022.
  * [30] G. Heigold, V. Vanhoucke, A. Senior, P. Nguyen, M. Ranzato, M. Devin, and J. Dean,  “Multilingual acoustic models using distributed deep neural networks,”  in Proc. ICASSP, 2013.
  * [31] J.-T. Huang, J. Li, D. Yu, L. Deng, and Y. Gong,  “Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers,”  in Proc. ICASSP, 2013.
  * [32] S. Punjabi, H. Arsikere, and Z. Raeesy et al.,  “Joint ASR and language identification using RNN-T: An efficient approach to dynamic language switching,”  in Proc. ICASSP, 2021.
  * [33] C. Zhang, B. Li, T.N. Sainath, T. Strohman, S. Mavandadi, S. Chang, and P. Haghani,  “Streaming end-to-end multilingual speech recognition with joint language identification,”  in Proc. Interspeech, Incheon, 2022.
  * [34] Google,  “Artificial Intelligence at Google: Our Principles,” https://ai.google/principles/.
  * [35] A. Narayanan, T.N. Sainath, R. Pang, J. Yu, C.-C. Chiu, R. Prabhavalkar, E. Variani, and T. Strohman,  “Cascaded encoders for unifying streaming and non-streaming ASR,”  in Proc. ICASSP, Toronto, 2021.
  * [36] C. Zhang, B. Li, Z. Lu, T.N. Sainath, and S. Chang,  “Improving the fusion of acoustic and text representations in RNN-T,”  in Proc. ICASSP, Singapore, 2022.