# Leveraging Timestamp Information for Serialized Joint Streaming Recognition
and Translation

###### Abstract

The growing need for instant spoken language transcription and translation is
driven by increased global communication and cross-lingual interactions. This
has made offering translations in multiple languages essential for user
applications. Traditional approaches to automatic speech recognition (ASR) and
speech translation (ST) have often relied on separate systems, leading to
inefficiencies in computational resources, and increased synchronization
complexity in real time. In this paper, we propose a streaming Transformer-
Transducer (T-T) model able to jointly produce many-to-one and one-to-many
transcription and translation using a single decoder. We introduce a novel
method for joint token-level serialized output training based on timestamp
information to effectively produce ASR and ST outputs in the streaming
setting. Experiments on {it,es,de}$\leftrightarrow$en prove the effectiveness
of our approach, enabling the generation of one-to-many joint outputs with a
single decoder for the first time.

Index Terms—  speech recognition, speech translation, streaming, joint,
timestamp

##  1 Introduction

With the expansion of global communication and cross-lingual interactions, the
demand for real-time spoken language transcription and translation in multiple
languages is rapidly increasing [1]. Conventionally, this task is addressed by
separate automatic speech recognition (ASR) and speech translation (ST)
models, leading to the necessity of running several models in parallel to
obtain the required outputs. This leads to a huge demand for computational
resources, in contrast with Green AI [2], and also increases the complexity of
coordinating several systems in real-time. Moreover, in some applications like
news reports, maintaining consistency between on-screen transcriptions and
translations is crucial to deliver to the user similar content [3, 4, 5].

Previous works [6, 7] have shown improvements in quality and consistency when
the model is trained to jointly generate both ASR and ST outputs. This
approach was later adapted to the streaming scenario by Weller et al. [8]
using an attention-based encoder-decoder architecture [9] with re-translation
[10]. More recently, Papi et al. [11] proposed the adoption of a Transformer-
Transducer (T-T) architecture [12, 13], which is more suitable for the
simultaneous scenario [14], with the joint token-level serialized output
training (joint t-SOT). This method employs an off-the-shelf textual aligner
to determine how to effectively produce transcription and translation words in
real time. However, their study only focused on the many-to-one language
setting and can be applied to one translation direction at a time, since
finding the alignment between one source language and multiple target
languages is a very complex task [15, 16, 17].

To overcome this limitation, in this paper, we propose a streaming T-T model
that is able to jointly produce both many-to-one and one-to-many outputs using
a single decoder. We introduce a novel interleaving method based on timestamp
information that enables the model to learn how to produce multiple target
languages while maintaining a low latency. Comparative experiments on
{it,es,de}$\leftrightarrow$en with separate and multilingual state-of-the-art
T-T architectures show the effectiveness of our interleaving approach,
yielding significant improvements in terms of transcription quality while
being competitive when producing multiple translation languages.

##  2 Method

ms$0$$100$$200$$300$$400$$500$$600$$700$$800$$900$$1000$$1100$$1200$ASR:Iamhappy.ST1:Estoyfeliz.ST2:Ichbinfroh.$t<500$$500\leq
t<1000$$1000<t\leq 1500$INTER TIME:  #ASR# I #ES# Estoy #ASR# am #DE# Ich
#ASR# happy. #DE# bin #ES# feliz. #DE# froh.\+ time step grouping:  #ASR# I am
#ES# Estoy #DE# Ich bin #ASR# happy. #ES# feliz. #DE# froh. Fig. 1: Timeline
visualization of the word sequence for the example in Section 2.2 with the
corresponding INTER TIME and INTER TIME + time step grouping joint t-SOT
references. English reference ($\mathbf{r_{asr}}$) is ASR, Spanish reference
($\mathbf{r_{st_{1}}}$) is ST1, and German reference ($\mathbf{r_{st_{2}}}$)
is ST2. Time step grouping with time step $T=500ms$ is represented with dashed
rectangles.

###  2.1 Joint t-SOT: Review

The joint t-SOT [11] was proposed to achieve simultaneous ASR and ST by one
streaming T-T model [18], inspired by t-SOT [19] proposed for multi-talker
ASR. Given we have reference transcription
$\mathbf{r_{asr}}=[r_{asr_{1}},...,r_{asr_{M}}]$ and reference translation
$\mathbf{r_{st}}=[r_{st_{1}},...,r_{st_{N}}]$ for a training audio, where $M$
and $N$ are the number of transcription tokens and that of translation tokens,
respectively. In the joint t-SOT framework, the T-T model is trained to
generate a single sequence of tokens including both ASR tokens and ST tokens.
In order to distinguish the tokens for ASR and ST, two special tokens,
$\langle asr\rangle$ and $\langle st\rangle$, are inserted in the token
sequence like $[\langle asr\rangle,r_{asr_{1}},r_{asr_{2}},\langle
st\rangle,r_{st_{1}},r_{st_{2}},\langle
asr\rangle,r_{asr_{3}},...,r_{st_{N}}]$.

Two methods for creating such a serialized token sequence were investigated in
[11]. The first method was called INTER $\gamma$ [8], where $\gamma$ can be a
real value between 0 to 1. In the INTER $\gamma$ method, the token sequence is
constructed by repeatedly adding either transcription tokens or translation
tokens starting from $asr_{1}$ and $st_{1}$. In each step, the transcription
token is inserted if $(1-\gamma)\cdot(1+asr_{i})>\gamma\cdot(1+st_{j})$,
otherwise, the translation token is inserted. The special tokens $\langle
asr\rangle$ and $\langle st\rangle$ are inserted between adjacent
transcription tokens and translation tokens. In case of $\gamma=0.0$ (INTER
0.0), all transcription tokens will be first generated before the translation
tokens. Contrary, if $\gamma=1.0$ (INTER 1.0), all translation tokens will be
first generated before the transcription tokens. In case of $\gamma=0.5$
(INTER 0.5), the transcription and translation tokens are interleaved in the
serialized token sequence. Among these three variants, only INTER 0.5 is
appropriate for simultaneous ASR and ST because INTER 0.0 and INTER 1.0 cause
a high latency on either ST or ASR.

The second method proposed in [11] was called INTER ALIGN, where the reference
transcription $\mathbf{r_{asr}}$ and the reference translation
$\mathbf{r_{st}}$ are first aligned by a source-target text alignment tool
developed for machine translation. The alignment forms a bipartite graph
between transcription tokens and translation tokens (see Fig. 2 of [11]). We
then find disjoint sub-graphs, each of which represents a pair of
transcription tokens and translation tokens. Finally, the serialized token
sequence is created based on the transcription and translation pairs in the
disjoint sub-graphs. In [11], the INTER ALIGN method achieved a better trade-
off between accuracy and latency compared to the INTER $\gamma$ method.

###  2.2 Timestamp-based joint t-SOT

While INTER $\gamma$ and INTER ALIGN methods achieved simultaneous ASR and ST,
it is not trivial to extend these methods into the multilingual target
scenario. It is especially challenging to extend the INTER ALIGN method for
the multilingual scenario because of the difficulty of finding a one-to-many
alignment.

To overcome this limitation, in this work, we propose INTER TIME, a novel
interleaving method based on word-level timestamps. The INTER TIME method not
only can build more effective joint t-SOT outputs but also enables the one-to-
many multilingual scenario by interleaving more than one translation language
at a time. Note that the usage of the timestamps has been proposed in the
t-SOT based multi-talker ASR [19] where a phone-based forced aligner [20] was
used to estimate the timestamps. However, it is not possible to follow the
same procedure for the ST task where the input audio and output translation
are not monotonically aligned. Therefore, we propose to use model-based
emission timestamp. Specifically, for each reference word, we compute the
corresponding emission timestamp by applying the Viterbi algorithm with
pretrained streaming models. This process is executed for each modality (ASR
and ST) and language direction.

In the one-to-many setting, let $\mathbf{r_{st_{1}}},...,\mathbf{r_{st_{L}}}$
be the corresponding translations in $L$ different languages, and $\langle
st_{1}\rangle$,…, $\langle st_{L}\rangle$ be the special tokens indicating the
language.111During training, $\langle asr\rangle$, $\langle st_{1}\rangle$,…,
$\langle st_{L}\rangle$ are added to the regular vocabulary and considered in
the loss computation as same with all other tokens. Each element of
$\mathbf{r_{asr}}$, $\mathbf{r_{st_{1}}},...,\mathbf{r_{st_{L}}}$ is composed
of three elements $(time,tag,word)$, where $time$ is the timestamp (integer
number, in milliseconds), $tag$ is the corresponding special token (either
$\langle asr\rangle$ or $\langle st_{l}\rangle$), and $word$ is the word that
has been emitted with timestamp $time$. For instance, if the word “I” has been
uttered at 200ms, the word “am” at 400ms, and the word “happy.” at 700ms (see
ASR in Figure 1), the corresponding $\mathbf{r_{asr}}$ extracted from the ASR
model is:

$\displaystyle\mathbf{r_{asr}}=[(200,\langle
asr\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{I}}\textquotedblright\ltxml@oqmark@close{}}),(400,\langle
asr\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{am}}\textquotedblright\ltxml@oqmark@close{}})$
$\displaystyle(700,\langle
asr\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{happy.}}\textquotedblright\ltxml@oqmark@close{}})].$

If the Spanish translation is “Estoy feliz.” with emission timestamps
$[300,900]$ (see ST1 in Figure 1) and the German translation is “Ich bin
froh.” with timestamps $[500,800,1100]$ (see ST2 in Figure 1), the
corresponding $\mathbf{r_{st_{1}}}$ and $\mathbf{r_{st_{2}}}$ extracted from
the ST models are:

$\displaystyle\mathbf{r_{st_{1}}}=[(300,\langle
st_{1}\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{Estoy}}\textquotedblright\ltxml@oqmark@close{}}),(900,\langle
st_{1}\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{feliz.}}\textquotedblright\ltxml@oqmark@close{}})],$
$\displaystyle\mathbf{r_{st_{2}}}=[(500,\langle
st_{2}\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{Ich}}\textquotedblright\ltxml@oqmark@close{}}),(800,\langle
st_{2}\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{bin}}\textquotedblright\ltxml@oqmark@close{}})$
$\displaystyle(1100,\langle
st_{2}\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{froh.}}\textquotedblright\ltxml@oqmark@close{}})].$

Algorithm 1 INTER TIME

$\mathbf{r_{asr}}$, $\mathbf{r_{st_{1}}},...,\mathbf{r_{st_{n}}}$
$\triangleright$ ASR and multi ST references

$\mathbf{w}\leftarrow[\quad]$

for $r_{i}$ in $[\mathbf{r_{asr}}$,
$\mathbf{r_{st_{1}}},...,\mathbf{r_{st_{n}}}]$ do

$\mathbf{w}\leftarrow\mathbf{w}+w_{i}$ $\triangleright$ Concatenate all the
reference words

end for

$\mathbf{w}\leftarrow\text{{sort}}_{time}\big{(}\mathbf{w}(time,tag,word)\big{)}$
$\triangleright$ Sort by timestamp

$\mathbf{r_{\text{t-SOT}}}\leftarrow[\quad]$

$prev\\_tag\leftarrow\text{None}$

for $(time,tag,word)$ in $\mathbf{w}$ do

if $tag\neq prev\\_tag$ then $\triangleright$ Language switch

$\mathbf{r_{\text{t-SOT}}}\leftarrow\mathbf{r_{\text{t-SOT}}}+tag$

$prev\\_tag\leftarrow tag$

end if

$\mathbf{r_{\text{t-SOT}}}\leftarrow\mathbf{r_{\text{t-SOT}}}+word$

end for

The INTER TIME output is built by applying Algorithm 1 to $\mathbf{r_{asr}}$,
$\mathbf{r_{st_{1}}}$, $\mathbf{r_{st_{2}}}$ to obtain the final
$\mathbf{r_{\text{t-SOT}}}$. In particular, the reference words for each
modality and language are concatenated, sorted by timestamp (increasing order)
and then interleaved following the temporal order. The special tokens are
inserted only if the previous interleaved word was of a different language or
domain (ASR or ST). Following the previous example, the output is:

$\displaystyle\mathbf{r_{\text{t-SOT}}}=[$ $\displaystyle(200,\langle
asr\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{I}}\textquotedblright\ltxml@oqmark@close{}}),(300,\langle
st_{1}\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{Estoy}}\textquotedblright\ltxml@oqmark@close{}}),$
$\displaystyle(400,\langle
asr\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{am}}\textquotedblright\ltxml@oqmark@close{}}),(500,\langle
st_{2}\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{Ich}}\textquotedblright\ltxml@oqmark@close{}}),$
$\displaystyle(700,\langle
asr\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{happy.}}\textquotedblright\ltxml@oqmark@close{}}),(800,\langle
st_{2}\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{bin}}\textquotedblright\ltxml@oqmark@close{}}),$
$\displaystyle(900,\langle
st_{1}\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{feliz.}}\textquotedblright\ltxml@oqmark@close{}}),(1100,\langle
st_{2}\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{froh.}}\textquotedblright\ltxml@oqmark@close{}})],$

with $\langle
asr\rangle=\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt\\#ASR\\#\textquotedblright\ltxml@oqmark@close{}},\langle
st_{1}\rangle=\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt\\#ES\\#\textquotedblright\ltxml@oqmark@close{}},\langle
st_{2}\rangle=\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt\\#DE\\#\textquotedblright\ltxml@oqmark@close{}}$. The corresponding
textual output used during training is shown in Figure 1 (INTER TIME). Note
that Algorithm 1 can be easily applied to the many-to-one scenario by using
different ASR models to obtain multilingual $\mathbf{r_{asr}}$ and a unique
$\mathbf{r_{st}}$.

Table 1: WER$\downarrow$ and BLEU$\uparrow$ on CoVoST 2 for the Many-to-English setting with their ASR latency LASR$\downarrow$ and ST latency LST$\downarrow$. Bold represents overall best result, underline represents best result balancing both quality and latency (there can be multiple combinations for each language).  Model |  # inf. steps | it-en |  | es-en |  | de-en
---|---|---|---|---|---|---
WER |  LASR | BLEU |  LST |  | WER |  LASR | BLEU |  LST |  | WER |  LASR | BLEU |  LST
separate ASR & ST [11] | 6 | 25.83 | 1191 | 16.41 | 1844 |  | 22.69 | 1149 | 19.24 | 1682 |  | 23.11 | 1071 | 19.11 | 1613
multilingual ASR & ST [11] | 23.48 | 1181 | 21.06 | 1663 |  | 22.84 | 1147 | 22.76 | 1622 |  | 21.82 | 1133 | 21.51 | 1642
joint t-SOT INTER 0.5 [11] | 3 | 22.35 | 1110 | 20.22 | 1515 |  | 21.19 | 1126 | 22.25 | 1468 |  | 21.35 | 1051 | 20.19 | 1547
joint t-SOT INTER ALIGN [11] | 21.74 | 1092 | 21.80 | 1355 |  | 21.04 | 1094 | 23.42 | 1341 |  | 22.07 | 1043 | 21.36 | 1335
joint t-SOT INTER TIME | 3 | 21.11 | 1141 | 21.70 | 1442 |  | 19.79 | 1143 | 23.38 | 1452 |  | 21.16 | 1112 | 19.96 | 1791
\+ 500ms step grouping | 21.22 | 1142 | 22.05 | 1493 |  | 19.74 | 1139 | 24.09 | 1489 |  | 21.17 | 1103 | 20.81 | 1664
\+ 1000ms step grouping | 21.64 | 1115 | 21.75 | 1457 |  | 20.26 | 1052 | 23.75 | 1467 |  | 21.49 | 1076 | 20.58 | 1651

###  2.3 Time Step Grouping

With the aim of limiting the frequency of the switch between languages, we
propose the adoption of a grouping mechanism in the data construction process.
The grouping mechanism is guided by the size of the time step $T$ (e.g.,
500ms, 1000ms, …). It groups the $(time,tag,word)$ tuple of each reference
word $r_{i}$ of the sorted reference
$\text{{sort}}_{time}\big{(}\mathbf{w}(time,tag,word)\big{)}$ in Algorithm 1
by looking at the $time$ attribute. Then, the words that belong to the same
current time step group $t_{s}$, i.e. $t_{s}-T\leq time<t_{s}$, are
interleaved together. The final sequence can be obtained by substituting the
$time$ attribute of each word with its corresponding $t_{s}$ in Algorithm 1.

For instance, if we look at the example in Section 2.2 and set the step size
$T$ to 500ms, we have three groups $[time<500,500\leq time<1000,1000\leq
time<1500]$, which are visualized as dotted rectangles in Figure 1. If we
substitute $time$ with the corresponding $t_{s}$ in the sorted $\mathbf{w}$,
we obtain:

$\displaystyle\mathbf{r_{\text{t-SOT}}}=[$ $\displaystyle(500,\langle
asr\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{I}}\textquotedblright\ltxml@oqmark@close{}}),(500,\langle
st_{1}\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{Estoy}}\textquotedblright\ltxml@oqmark@close{}}),$
$\displaystyle(500,\langle
asr\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{am}}\textquotedblright\ltxml@oqmark@close{}}),(1000,\langle
st_{2}\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{Ich}}\textquotedblright\ltxml@oqmark@close{}}),$
$\displaystyle(1000,\langle
asr\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{happy.}}\textquotedblright\ltxml@oqmark@close{}}),(1000,\langle
st_{2}\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{bin}}\textquotedblright\ltxml@oqmark@close{}}),$
$\displaystyle(1000,\langle
st_{1}\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{feliz.}}\textquotedblright\ltxml@oqmark@close{}}),(1500,\langle
st_{2}\rangle,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002pt{{froh.}}\textquotedblright\ltxml@oqmark@close{}})]$

that corresponds to the output INTER TIME + time step grouping shown in Figure
1. The overall language switch reduction ratio depends on the time step $T$.
In general, the larger $T$ will result in the serialized transcription with
less special tokens. On our training data, the reduction ratio is estimated as
34% for 500ms, and 54% for 1000ms.

##  3 Experimental Settings

For all our experiments, we use a streaming T-T architecture [18] with 24
Transformer layers for the encoder with 8 attention heads, 6 LSTM layers for
the predictor and 2 feed-forward layers for the joiner. The embedding
dimension of the encoder is 512 and the feed-forward units are 4096. We use a
chunk size of 1 second with 18 left chunks. The LSTM predictor and feed-
forward layers of the joiner have 1024 hidden units. We use 80-dimensional
log-mel filterbanks (fbanks) as features, sampled every 10 milliseconds.
Before feeding them to the Transformer encoders, we apply 2 layers of CNN with
stride 2 and a kernel size of (3, 3), with an overall input compression of 4.
The total number of parameters is 188.5M.

Our Many-to-English experiments follow the settings of previous work [11]: all
models are trained for 6.4M steps on 1k hours of proprietary data for each
source language (Italian (it), Spanish (es), German (de)) and tested on the
CoVoST2 dataset [21]. 8k-sized SentencePiece vocabulary [22] was trained with
coverage 1.0 and shared between languages.

For the English-to-Many experiments, we used 1k hours of English audio with
the corresponding translation into Italian, Spanish, and German. The models
are tested on the FLEURS dataset [23]. The multitask multilingual ASR & ST
model is realized by pre-pending the language ID (LID) tag [24], i.e. by
replacing the <SOS> with <LID> in the target sequence. Pre-pended LID is used
also to train the single-translation version of the joint t-SOT. All but
separate models are trained for 6.4M steps starting from the multitask
multilingual ASR & ST model weights pretrained for 3.2M steps, including the
multitask multilingual model itself. Timestamps for INTER TIME are estimated
using monolingual ASR and ST models trained on the same data. Time step
grouping is applied at 500ms and 1000ms since preliminary experiments with
higher values (e.g., 2000ms) showed quality degradation.

AdamW [25] is used as optimizer with the RNN-T loss [26]. Checkpoints are
saved every 320k steps. The learning rate is set to 3e-4 with Noam scheduler,
800k warm-up steps and linear decay. We use 16 NVIDIA V100 GPUs with 32GB of
RAM for all the training and a batch size of 350k. We select the last
checkpoint for inference, which is then converted to open neural network
exchange (ONNX) format and compressed. The beam size of the beam search is set
to 7.

We report WER for the ASR quality and BLEU222sacreBLEU [27] version 2.3.1 for
the ST quality. Latency is measured in milliseconds (ms) with the length-
adaptive average lagging (LAAL) [28].

Table 2: WER$\downarrow$ and BLEU$\uparrow$ on FLEURS for the English-to-Many setting with their ASR latency LASR$\downarrow$ and ST latency LST$\downarrow$. Bold represents overall best result, underline represents best result balancing both quality and latency (there can be multiple combinations for each language). ASR results of joint t-SOT INTER TIME with single translation are averaged among the three languages. Model |  # inf. steps | en |  | en-it |  | es-en |  | de-en
---|---|---|---|---|---|---|---|---
WER |  LASR |  | BLEU |  LST |  | BLEU |  LST |  | BLEU |  LST
separate ASR & ST | 4 | 29.02 | 1089 |  | 8.76 | 1932 |  | 9.50 | 1853 |  | 9.87 | 2156
\+ multilingual ST | 29.02 | 1089 |  | 11.17 | 1612 |  | 11.34 | 1618 |  | 13.14 | 1799
multitask multilingual ASR & ST | 27.53 | 917 |  | 11.56 | 1607 |  | 11.38 | 1608 |  | 13.11 | 1844
joint t-SOT INTER TIME multi | 1 | 36.23 | 1544 |  | 7.52 | 2313 |  | 8.28 | 2331 |  | 8.54 | 2497
\+ 500ms step grouping | 31.51 | 1118 |  | 9.68 | 1668 |  | 9.86 | 1852 |  | 11.30 | 1993
\+ 1000ms step grouping | 29.34 | 913 |  | 10.85 | 1395 |  | 10.90 | 1509 |  | 12.74 | 1918
[1pt/2pt][0pt/1pt] joint t-SOT INTER TIME single | 3 | 26.33 | 959 |  | 10.38 | 1564 |  | 11.24 | 1520 |  | 12.39 | 1733
\+ 500ms step grouping | 27.00 | 918 |  | 11.45 | 1580 |  | 11.89 | 1610 |  | 12.79 | 1830
\+ 1000ms step grouping | 26.81 | 892 |  | 11.25 | 1776 |  | 11.52 | 1797 |  | 12.85 | 1999

##  4 Results

###  4.1 Many-to-English

Table 1 shows the results for the {it,es,de}-en language directions. For
comparison, we report the results for the joint t-SOT INTER 0.5 and INTER
ALIGN approaches. We do not include the results for INTER 0.0 and 1.0 since
they are not streaming for one of the two modalities (either ASR or ST).

We first observe that the proposed INTER TIME method without time step
grouping achieves higher or similar BLEU scores, except for de-es, and obtains
the best WER on all the source languages (it, es, and de). The INTER TIME
method also achieves comparable ASR and ST latencies with the INTER ALIGN
method, which shows the lowest latency among all methods. We then observe that
the time step grouping improves overall translation quality while achieving
almost the same ASR accuracy and latency scores. With 500ms time step, the
INTER TIME method achieves an average of 0.54 points of BLEU improvement
compared to the multilingual ASR & ST. It also achieves 0.12 and 0.64 BLEU
improvement compared to INTER ALIGN and INTER TIME without time step grouping,
respectively.

All in all, the best quality-latency trade-off is achieved by INTER TIME with
time step grouping of 500ms, yielding the best results in most languages and
modalities.

###  4.2 English-to-Many

Table 2 shows a comparison of various ASR & ST models for English-to-Many
setting. In rows 1-3, we show the results by using English ASR model and three
ST models (row 1), one English ASR model and one multi-lingual ASR model (row
2), and one multitask multilingual ASR & ST model. Note that, all models in
rows 1 to row 3 can output only one ASR result or ST result at one inference
step, so we need to execute 4 times of inference steps to obtain the listed
result. In this evaluation, we observe that the multitask multilingual ASR &
ST model achieves the best results, with an improved WER and similar or better
BLEU scores compared to using separate models for modalities (ASR and ST) and
languages.

We then evaluate the proposed joint t-SOT INTER TIME model using multiple
translation languages (rows 4-6), or only one translation language (rows 7-9)
in the target. The latter model produces the transcription and the
corresponding translation in a single language at one inference step, similar
to the experiment in Section 4.1.

For the joint t-SOT INTER TIME model trained on multiple translation languages
(rows 4-6), we notice that the time step grouping helps with the performance,
both in terms of quality and latency. It yields 6.89 points of WER improvement
and an average of 3.38 points of BLEU improvement with 983ms latency reduction
when the 1000ms of time step grouping is applied. Compared with the strongest
system (i.e. multitask multilingual ASR & ST at row 3), our model shows a
marginal degradation of ASR and ST accuracy, with 1.81 point WER degradation
and 0.52 point BLEU degradation. The latency is slightly improved. It is
noteworthy that, while marginal degradation of ASR and ST accuracy is
observed, the joint t-SOT INTER TIME model requires only a single inference
step. It is significantly efficient compared to the multitask multilingual ASR
& ST model that requires 4 times of inference steps to obtain all the results.

If we constrain the joint t-SOT INTER TIME strategy to deal with only one
translation language (rows 7-9), we observe significant improvements compared
to its multiple translation languages counterpart (rows 4-6), especially with
the time step grouping. WER improvements range from 2.34 to 3.01 and average
BLEU improvements are up to 0.55 compared to the multiple translation model
with 1000ms of time step grouping (row 6). As was the case with the many-to-
English experiment in Section 4.1, the 500ms time step grouping is the best-
performing model and, compared with the multitask multilingual ASR & ST model,
it yields 0.53 points of WER improvement while maintaining comparable or
slightly better BLEU score and latency.

To conclude, we show the effectiveness of the joint t-SOT INTER TIME,
especially when time step grouping is applied. Results on both ASR and ST
tasks show that our method achieves the best overall results compared to the
strongest multitask multilingual ASR & ST model. When it is extended to deal
with multiple translation languages all at once (rows 4-6), our proposed
method maintains comparable results while significantly reducing the inference
step to only one step.

##  5 Conclusions

In this paper, we proposed a streaming T-T that is able to simultaneously
produce many-to-one and one-to-many transcriptions and translations. To
effectively train the model to maximize ASR and ST quality while minimizing
latency, we proposed INTER TIME, a novel method for the joint t-SOT framework
where the tokens are sorted based on the model-based emission timestamp
information. We also proposed a variant of this method based on grouping the
timestamp according to a fixed time step. Comparative studies on {it,es,de}-en
and en-{it,es,de} prove the effectiveness of our approach, especially when the
time step grouping is adopted. It achieved the best ASR and ST accuracy in
many-to-English ASR and ST scenario while keeping the ability of low latency
inference. In English-to-many ASR and ST scenarios, the proposed method
achieved comparable ASR and ST accuracy to the baseline model while
significantly reducing the inference cost.

## References

  * [1] E. Steigerwald, V. Ramírez-Castañeda, D. Y. C. Brandt, A. Báldi, J. T. Shapiro, L. Bowker, and R. D. Tarvin,  “Overcoming language barriers in academia: Machine translation tools and a vision for a multilingual future,”  Bioscience, vol. 72, pp. 988 – 998, 2022.
  * [2] R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni,  “Green ai,”  Commun. ACM, vol. 63, no. 12, pp. 54–63, 2020.
  * [3] C. Fügen,  A System for Simultaneous Translation of Lectures and Speeches,  Ph.D. thesis, 2009.
  * [4] A. Karakanta, M. Gaido, M. Negri, and M. Turchi,  “Between flexibility and consistency: Joint generation of captions and subtitles,”  in Proc. 18th IWSLT, 2021, pp. 215–225.
  * [5] J. Xu, F. Buet, J. Crego, E. Bertin-Lemée, and F. Yvon,  “Joint generation of captions and subtitles with dual decoding,”  in Proc. 19th IWSLT, 2022, pp. 74–82.
  * [6] M. Sperber, H. Setiawan, C. Gollan, U. Nallasamy, and M. Paulik,  “Consistent transcription and translation of speech,”  TACL, vol. 8, pp. 695–709, 2020.
  * [7] H. Le, J. Pino, C. Wang, J. Gu, D. Schwab, and L. Besacier,  “Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation,”  in Proc. 28th COLING, Dec. 2020, pp. 3520–3533.
  * [8] O. Weller, M. Sperber, C. Gollan, and J. Kluivers,  “Streaming models for joint speech recognition and translation,”  in Proc. 16th EACL, 2021, pp. 2533–2539.
  * [9] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin,  “Attention is all you need,”  in Proc. 31st NeurIPS, 2017, p. 6000–6010.
  * [10] J. Niehues, N. Pham, T. Ha, M. Sperber, and A. Waibel,  “Low-Latency Neural Speech Translation,”  in Proc. Interspeech, 2018, pp. 1293–1297.
  * [11] S. Papi, P. Wang, J. Chen, J. Xue, J. Li, and Y. Gaur,  “Token-level serialized output training for joint streaming asr and st leveraging textual alignments,”  2023\.
  * [12] C. Yeh, J. Mahadeokar, K. Kalgaonkar, Y. Wang, D. Le, M. Jain, K. Schubert, C. Fuegen, and M. L Seltzer,  “Transformer-transducer: End-to-end speech recognition with self-attention,” 2019.
  * [13] Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar,  “Transformer transducer: A streamable speech recognition model with transformer encoders and RNN-T loss,”  in Proc. ICASSP, 2020, pp. 7829–7833.
  * [14] J. Li,  “Recent advances in end-to-end automatic speech recognition,”  APSIPA Transactions on Signal and Information Processing, vol. 11, no. 1.
  * [15] E. Grave, A. Joulin, and Q. Berthet,  “Unsupervised alignment of embeddings with wasserstein procrustes,”  in Proc. 22nd AISTATS, Apr. 2019, pp. 1880–1890.
  * [16] A. Kalinowski and Y. An,  “A survey of embedding space alignment methods for language and knowledge graphs,”  arXiv preprint arXiv:2010.13688, 2020.
  * [17] A. Imani, L. K. Senel, M. Jalili Sabet, F. Yvon, and H. Schuetze,  “Graph neural networks for multiparallel word alignment,”  in Findings of ACL 2022, Dublin, Ireland, May 2022, pp. 1384–1396.
  * [18] X. Chen, Y. Wu, Z. Wang, S. Liu, and J. Li,  “Developing real-time streaming transformer transducer for speech recognition on large-scale dataset,”  in Proc. ICASSP, 2021, pp. 5904–5908.
  * [19] N. Kanda, J. Wu, Y. Wu, X. Xiao, Z. Meng, X. Wang, Y. Gaur, Z. Chen, J. Li, and T. Yoshioka,  “Streaming multi-talker ASR with token-level serialized output training,”  in Proc. Interspeech, 2022, pp. 3774–3778.
  * [20] Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger,  “Montreal forced aligner: Trainable text-speech alignment using kaldi.,”  in Proc. Interspeech, 2017, pp. 498–502.
  * [21] C. Wang, A. Wu, J. Gu, and J. Pino,  “CoVoST 2 and Massively Multilingual Speech Translation,”  in Proc. Interspeech 2021, 2021, pp. 2247–2251.
  * [22] T. Kudo and J. Richardson,  “SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,”  in Proc. EMNLP: System Demonstrations, 2018, pp. 66–71.
  * [23] A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna,  “Fleurs: Few-shot learning evaluation of universal representations of speech,”  in Proc. SLT, 2023, pp. 798–805.
  * [24] Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean,  “Google’s multilingual neural machine translation system: Enabling zero-shot translation,”  Transactions of the Association for Computational Linguistics, vol. 5, pp. 339–351, 2017.
  * [25] I. Loshchilov and F. Hutter,  “Decoupled weight decay regularization,”  in Proc. ICLR, 2019.
  * [26] A. Graves,  “Sequence transduction with recurrent neural networks,”  2012\.
  * [27] M. Post,  “A call for clarity in reporting BLEU scores,”  in Proc. 3rd WMT, 2018, pp. 186–191.
  * [28] S. Papi, M. Gaido, M. Negri, and M. Turchi,  “Over-generation cannot be rewarded: Length-adaptive average lagging for simultaneous speech translation,”  in Proc. 3rd AutoSimTrans, 2022, pp. 12–17.