# A Template-guided Hybrid Pointer Network for Knowledge-based Task-oriented
Dialogue Systems

Dingmin Wang1, Ziyao Chen2, Wanwei He3, Li Zhong4, Yunzhe Tao5, Min Yang3
1Department of Computer Science, University of Oxford, UK
2Tencent, Guangzhou, China
3Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences,
China
4ByteDance, Shenzhen, China, 5Amazon Web Services, Seattle, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Most existing neural network based task-oriented dialogue systems follow
encoder-decoder paradigm, where the decoder purely depends on the source texts
to generate a sequence of words, usually suffering from instability and poor
readability. Inspired by the traditional template-based generation approaches,
we propose a template-guided hybrid pointer network for the knowledge-based
task-oriented dialogue system, which retrieves several potentially relevant
answers from a pre-constructed domain-specific conversational repository as
guidance answers, and incorporates the guidance answers into both the encoding
and decoding processes. Specifically, we design a memory pointer network model
with a gating mechanism to fully exploit the semantic correlation between the
retrieved answers and the ground-truth response. We evaluate our model on four
widely used task-oriented datasets, including one simulated and three manually
created datasets. The experimental results demonstrate that the proposed model
achieves significantly better performance than the state-of-the-art methods
over different automatic evaluation metrics 111https://github.com/wdimmy/THPN.

Table 1: Two example conversations from real dialogues about navigation and weather.  Navigation | Weather
---|---
User |  please give me directions to 5677_spring_street |  User |  what is the temperature of carson on tuesday
Retrieve |  q1: direct me to stanford children’s health |  Retrieve |  q1: the temperature of new_york on wednesday
a1: no problem, I will be navigating you to stanford children’s health right now |  a1: the temperature in new_york on wednesday will be low_of_80f and high_of_90f
KB |  |  KB |  carson: tuesday low_of_20f
|  carson: tuesday high_of_40f
Gold |  no problem, I will be navigating you to 5677_spring_street right now |  Gold |  the temperature in carson on tuesday will be low_of_20f and high_of_40f

##  1 Introduction

Task oriented dialogue systems have attracted increasing attention recently
due to broad applications such as reserving restaurants and booking flights.
Conventional task-oriented dialogue systems are mainly implemented by rule-
based methods Lemon et al. (2006); Wang and Lemon (2013), which rely heavily
on the hand-crafted features, establishing significant barriers for adapting
the dialogue systems to new domains. Motivated by the great success of deep
learning in various NLP tasks, the neural network based methods Bordes et al.
(2017); Eric and Manning (2017); Madotto et al. (2018) have dominated the
study since these methods can be trained in an end-to-end manner and scaled to
different domains.

Despite the remarkable progress of previous studies, the performance of task-
oriented dialogue systems is still far from satisfactory. On one hand, due to
the exposure bias problem Ranzato et al. (2016), the neural network based
models, e.g., the sequence to sequence models (seq2seq), tend to accumulate
errors with increasing length of the generation. Concretely, the first several
generated words can be reasonable, while the quality of the generated sequence
deteriorates quickly once the decoder produces a “bad” word. On the other
hand, as shown in previous works Cao et al. (2018); Madotto et al. (2018), the
Seq2Seq models are likely to generate non-committal or similar responses that
often involve high-frequency words or phrases. These responses are usually of
low informativeness or readability. This may be because that arbitrary-length
sequences can be generated, and it is not enough for the decoder to be purely
based on the source input sentence to generate informative and fluent
responses.

We demonstrate empirically that in task-oriented dialogue systems, the
responses for the requests with similar types often follow the same sentence
structure except that different named entities are used according to the
specific dialogue context. Table 1 shows two conversations from real task-
oriented dialogues about navigation and weather. From the navigation case, we
can observe that although the two requests are for different destinations, the
corresponding responses are similar in sentence structure, replacing
“children’s health” with “5677_springer_street”. For the weather example, it
requires the model to first detect the entity “carson” and then query the
corresponding information from the knowledge base (KB). After obtaining the
returned KB entries, we generate the response by replacing the corresponding
entities in the retrieved candidate answer. Therefore, we argue that the
golden responses of the requests with similar types can provide a reference
point to guide the response generation process and enable to generate high-
quality responses for the given requests.

In this paper, we propose a template-guided hybrid pointer network (THPN to
generate the response given a user-issued query, in which the domain specific
knowledge base (KB) and potentially relevant answers are leveraged as extra
input to enrich the input representations of the decoder. Here, knowledge base
refers to the database to store the relevant and necessary information for
supporting the model in accomplishing the given tasks. We follow previous
works and use a triple (subject, relation, object) representation. For
example, the triple (Starbucks, address, 792 Bedoin St) is an example in KB
representing the information related to the Starbucks. Specifically, given a
query, we first retrieve top-$n$ answer candidates from a pre-constructed
conversational repository with question-answer pairs using BERT Devlin et al.
(2018). Then, we extend memory networks Sukhbaatar et al. (2015) to
incorporate the commonsense knowledge from KB to learn the knowledge-enhanced
representations of the dialogue history. Finally, we introduce a gating
mechanism to effectively utilize candidate answers and improve the decoding
process. The main contributions of this paper can be summarized as follows:

  * •

We propose a hybrid pointer network consisting of entity pointer network (EPN)
and pattern pointer network (PPN) to generate informative and relevant
responses. EPN copies entity words from dialogue history, and PPN extracts
pattern words from retrieved answers.

  * •

We introduce a gating mechanism to learn the semantic correlations between the
user-issued query and the retrieved candidate answers, which reduces the
“noise” brought by the retrieved answers.

  * •

We evaluate the effectiveness of our model on four benchmark task-oriented
dialogue datasets from different domains. Experimental results demonstrate the
superiority of our proposed model.

##  2 Related Work

Task-oriented dialogue systems are mainly studied via two different
approaches: pipeline based and end-to-end. Pipeline based models Williams and
Young (2007); Young et al. (2013) achieve good stability but need domain-
specific knowledge and handcrafted labels. End-to-end methods have shown
promising results recently and attracted more attention since they are easily
adapted to a new domain.

Neural network based dialogue systems can avoid the laborious feature
engineering since the neural networks have great ability to learn the latent
representations of the input text. However, as revealed by previous studies
Koehn and Knowles (2017); Cao et al. (2018); He et al. (2019), the performance
of the sequence to sequence model deteriorates quickly with the increase of
the length of generation. Therefore, how to improve the stability and
readability of the neural network models has attracted increasing attention.
Eric et al. (2017) proposed a copy augmented Seq2Seq model by copying relevant
information directly from the KB information. Madotto et al. (2018) proposed a
generative model by employing the multi-hop attention over memories with the
idea of pointer network. Wu et al. (2019) proposes a global-to-locally pointer
mechanism to effectively utilize the knowledge base information, which
improves the quality of the generated response.

Previous proposed neural approaches have shown the importance of external
knowledge in the sequence generation Chen et al. (2017); Zhu et al. (2018);
Yang et al. (2019); Zhang et al. (2019); Ding et al. (2019), especially in the
task-oriented dialogue systems where an appropriate response usually requires
correctly extracting knowledge from the domain-specific or commonsense
knowledge base Madotto et al. (2018); Zhu et al. (2018); Qin et al. (2019).
However, it is still under great exploration with regard with the inclusion of
external knowledge into the model. Yan et al. (2016); Song et al. (2018) argue
that retrieval and generative methods have their own demerits and merits, and
they have achieved good performance in the chit-chat response generation by
incorporating the retrieved results in the Seq2Seq based models. Zhu et al.
(2018) proposed an adversarial training approach, which is enhanced by
retrieving some related candidate answers in the neural response generation,
and Ghazvininejad et al. (2018) also applies a similar method in the neural
conversation model. In addition, in task-oriented dialogue tasks, the copy
mechanism Gulcehre et al. (2016) has also been widely utilized Eric and
Manning (2017); Madotto et al. (2018), which shows the superiority of
generation based methods with copy strategy.

##  3 Methodology

We build our model based on a seq2seq dialogue generation mode, and the
overall architecture is exhibited in Figure 1. Each module will be elaborated
in the following subsections.

Figure 1: The overall structure of our model. During test time, given a user
query $q$, we retrieve at most $3$ similar questions to $q$ using BERT from QA
Paris repository, and the corresponding answers are used as our answer
templates. The retrieved answers as well as the dialogue history and KB
information are then utilized for the response generation. Especially, we
utilize the gating mechanism to filter out noise from unrelated retrieval
results. Finally, words are generated either from the vocabulary or directly
copying from the multi-source information using a hybrid pointer network.

###  3.1 Encoder Module

By checking if a word is in the given KB, we divide words into two types:
entity words (EW) and non-entity words (NEW). Taking “what is the temperature
of carson on tuesday” as an example, all words are NEW except for “carson” and
“tuesday”.

We represent a multi-turn dialogue as $D=\\{(u_{i},s_{i})\\}_{i=1}^{T}$, where
$T$ is the number of turns in the dialogue, and $u_{i}$ and $s_{i}$ denote the
utterances of the user and the system at the $i^{th}$ turn, respectively. KB
information is represented as $KB=\\{k_{1},k_{2},\cdots,k_{l}\\}$, where
$k_{i}$ is a tuple and $l$ is the size of KB. Following Madotto et al. (2018),
we concatenate the previous dialogue and KB as input. At first turn, input to
the decoder is $[u_{1};KB]$, the concatenation of first user request and KB.
For $i>1$, previous history dialog information is included, namely, input is
supposed to be $[u_{1},s_{1},\cdots,u_{i};KB]$. We define words in the
concatenated input as a sequence of tokens $W=\\{w_{1},w_{2},\cdots,w_{n}\\}$,
where $w_{j}\in\\{u_{1},s_{1},\cdots,u_{i},KB\\}$ , $n$ is the number of
tokens.

In this paper, we use the memory network (MemNN) proposed in Sukhbaatar et al.
(2015) as the encoder module. The memories of MemNN are represented by a set
of trainable embedding matrices $M=\\{M^{1},M^{2},\cdots,M^{K}\\}$, where $K$
represents the number of hops and each $M^{k}$ maps the input into vectors.
Different from Sukhbaatar et al. (2015); Madotto et al. (2018), we initialize
each $M^{k}$ with the pre-trained embeddings222https://s3-us-
west-1.amazonaws.com/fasttext-vectors/wiki.en.vec., whose weights are set to
be trainable. At hop $k$, $W$ is mapped to a set of memory vectors,
$\\{m_{1}^{k},m_{2}^{k},\cdots,m_{n}^{k}\\}$, where the memory vectors
$m_{i}^{k}$ of dimension $d$ from $M^{k}$ is computed by embedding each word
in a continuous space, in the simplest case, using an embedding matrix $A$. A
query vector $q$ is used as a reading head, which will loop over $K$ hops and
compute the attention weights at hop $k$ for each memory by taking the inner
product followed by a softmax function,

$p_{i}^{k}=\mathit{softmax}\left(\left(q^{k}\right)^{T}m_{i}^{k}\right)$ (1)

where $p_{i}^{k}$ is a soft memory selector that decides the memory relevance
with respect to the query vector $q$. The model then gets the memory $c^{k}$
by the weighted sum over $m^{k+1}$,

$c^{k}=\sum_{i}p_{i}^{k}m_{i}^{k+1}\ $ (2)

In addition, the query vector is updated for the next hop by
$q^{k+1}=q^{k}+c^{k}$. In total, we can achieve K hidden states encoded from
MemNN, represented as $C=\\{c^{1},c^{2},\cdots,c^{K}\\}$.

#### Masking NEW in the history dialogue

We observe that the ratio of non-entity words in both the history dialogue and
the expected response is extremely low. Therefore, to prevent the model from
copying non-entity words from the history dialogue, we introduce an array
$R_{h}$333The length of $R_{h}$ equals to that of $W$. whose elements are
zeros and ones, where $0$ denotes NEW and $1$ for EW. When $w_{i}$ is pointed
to, and if $i$ is the sentinel location or $R_{h}[i]=0$, then $w_{i}$ will not
be copied.

###  3.2 Retrieval Module

For each dataset, we use the corresponding training data to pre-construct a
question-answer repository. In particular, we treat each post-response
($u_{i}$ and $s_{i}$) in a dialogue as a pair of question-answer. To
effectively retrieve potentially relevant answers, we adopt a sentence
matching based approach, in which each sentence is represented as a dense
vector, and the cosine similarity serves as the selection metrics. We have
explored several unsupervised text matching methods, such as BM25 Robertson et
al. (2009), Word2Vec Mikolov et al. (2013b), and BERT Devlin et al. (2018),
and revealed that BERT could achieve the best performance. In addition, based
on our preliminary experiments, we observed that the number of retrieved
answer candidates have an impact on the model performance, so we define a
threshold $\theta$ for controlling the number of retrieval answer candidates.

Specifically, for each question in the pre-constructed database, we pre-
compute the corresponding sentence embedding using BERT. Then, for each new
user-issued query $u_{q}$, we embed $u_{q}$ into $u_{q}^{e}$, and search in
the pre-constructed database for the most similar requests based on cosine
similarity. The corresponding answers are selected and serve as our answer
candidates.

#### Masking EW in the retrieved answers

In real dialogue scenes, the reply’s sentence structure might be similar but
the involved entities are usually different. To prevent the model from copying
these entities, we introduce another array $R_{r}$ similar to $R_{h}$
mentioned before. Finally, the retrieved candidate answers are encoded into
low-dimension distributed representations, denoted as
$AN=\\{a_{1},a_{2},\cdots,a_{m}\\}$, where $m$ is the total number of the
words. Moreover, by an interaction between $c^{K}$ and
$AN=\\{a_{1},a_{2},\cdots,a_{m}\\}$, we obtain a dense vector $h_{a}$ as the
representation of the retrieved answers,

$h_{a}=W_{2}tanh\left(\sum_{i=1}^{m}\left(W_{1}\left[c^{K};a_{i}\right]\right)\right)$
(3)

###  3.3 Decoder Module

We first apply Gated Recurrent Unit (GRU) Chung et al. (2014) to obtain the
hidden state $h_{t}$,

$h_{t}=\text{GRU}\left(\phi^{emb}(y_{t-1}),h_{t-1}^{*}\right)$ (4)

where $\phi^{emb}(\cdot)$ is an embedding function that maps each token to a
fixed-dimensional vector. At the first time step, we use the special symbol
“SOS” as $y_{0}$ and the initial hidden state $h_{0}^{*}=h_{a}$. $h_{t-1}^{*}$
consists of three parts, namely, the last hidden state $h_{t-1}$, the
attention over $C=\\{c^{1},c^{2},\cdots,c^{K}\\}$ from the encoder module,
denoted as $H^{c}$, and $H^{g}$, which is calculated by linearly transforming
last state $h_{t-1}$ and $h_{a}$ with a multi-layer perceptron network. We
formulate $H^{c}$ and $H^{g}$ as follows:

#### Attention over $\mathbf{C=\\{c^{1},c^{2},\cdots,c^{K}\\}}$

Since MemNN consists of multiple hops, we believe that different hops are
relatively independent and have their own semantic meanings over the history
dialog. At different time steps, we need to use different semantic information
to generate different tokens, so our aim is to get a context-aware
representation. We can achieve it by applying attention mechanism to the
hidden states achieved at different hops,

$H^{c}=\sum_{i=1}^{K}\alpha_{i,t}c^{i},\quad\alpha_{i,t}=\frac{e^{\eta(h_{t-1},c^{i})}}{\sum_{i=1}^{K}e^{\eta(h_{t-1},c^{i})}}$
(5)

where $\eta$ is the function that represents the correspondence for attention,
usually approximated by a multi-layer neural network.

#### Template-guided gating mechanism

As reported in Song et al. (2018), the top-ranked retrieved reply is not
always the one that best match the query, and multiple retrieved replies may
provide different reference information to guide the response generation.
However, using multiple retrieved replies may increase the probability of
introducing “noisy” information, which adversely reduces the quality of the
response generation. To tackle this issue, we add a gating mechanism to the
hidden state of candidate answers, aiming at extracting valuable “information”
at different time steps. Mathematically,

$H^{g}=\mathit{sigmoid}(h_{a}\odot h_{t-1})\odot h_{a}$ (6)

We use element-wise multiplication to model the interaction between candidate
answers ($h_{a}$) and last hidden state of GRU. $h_{t-1}^{*}$ is obtained by
concatenating $h_{t-1}$, $H^{c}$, and $H^{g}$.

#### Hybrid pointer networks

We use another MemNN with three hops for the response generation, where
$h_{t}$ of GRU serves as the initial reading head, as shown in Figure 1. The
output of MemNN is denoted as $O=\\{o^{1},o^{2},o^{3}\\}$ and attention
weights are $P_{o}=\\{p_{o}^{1},p_{o}^{2},p_{o}^{3}\\}$.

Other than a candidate softmax $P_{v}$ used for generating a word from the
vocabulary, we adopt the idea of Pointer Softmax in Gulcehre et al. (2016),
and introduce an Entity Pointer Networks (EPN) and a Pattern Pointer Networks
(PPN), where EPN is trained to learn to copy entity words from dialogue
history (or KB), and PPN is responsible for extracting pattern words from
retrieved answers. For EPN, we use a location softmax $P_{h}$, which is a
pointer network where each of the output dimension corresponds to the location
of a word in the context sequence. Likewise, we introduce a location softmax
$P_{r}$ for PPN. $P_{v}$ is generated by concatenating the first hop attention
read out and the current query vector,

$P_{v}=\textit{softmax}(W_{v}[o^{1};h_{t}])$ (7)

For $P_{r}$ and $P_{h}$, we take the attention weights at the second MemNN hop
and the third hop of the decoder, respectively: $P_{r}=p_{o}^{2}$ and
$P_{h}=p_{o}^{3}$. The output dimensions of $P_{h}$ and $P_{v}$ vary according
to the length of the corresponding target sequence.

With the three distributions, the key issue is how to decide which
distribution should be chosen to generate a word $w_{i}$ for the current time
step. Intuitively, entity words are relatively important, so we set the
selection priority order as $P_{r}>P_{h}>P_{v}$. Instead of using a gate
function for selection Gulcehre et al. (2016), we adopt the sentinel mechanism
proposed in Madotto et al. (2018). If the expected word is not appearing in
the memories, then $P_{h}$ and $P_{r}$ are trained to produce a sentinel
token444We add a special symbol to the end of each sentence. For example,
“good morning” is converted to “good morning $$$”. Therefore, if the model
predicts the location of “$$$”, it means that the expected word is not
appearing in the context sequence.. When both $P_{h}$ and $P_{r}$ choose the
sentinel token or the masked position, our model will generate the token from
$P_{v}$. Otherwise, it takes the memory content using $P_{v}$ or $P_{r}$.

##  4 Experimental Settings

###  4.1 Datasets

We use four public multi-turn task-oriented dialog datasets to evaluate our
model, including bAbI Weston et al. (2015), In-Car Assistant Eric and Manning
(2017) , DSTC2 Henderson et al. (2014) and CamRest Wen et al. (2016). bAbI is
automatically generated and the other three datasets are collected from real
human dialogs.

#### bAbI

We use tasks 1-5 from bAbI dialog corpus for restaurant reservation to verify
the effectiveness of our model. For each task, there are 1000 dialogs for
training, 1000 for development, and 1000 for testing. Tasks 1-2 verify dialog
management to check if the model can track the dialog state implicitly. Tasks
3-4 verify if the model can leverage the KB tuples for the task-oriented
dialog system. Tasks 5 combines Tasks 1-4 to produce full dialogs.

#### In-Car Assistant

This dataset consists of 3,031 multi-turn dialogs in three distinct domains:
calendar sheduling, weather information retrieval, and point-of-interest
navigation. This dataset has an average of 2.6 conversation turns and the KB
information is complicated. Following the data processing in Madotto et al.
(2018), we obtain 2,425/302/304 dialogs for training/validation/testing
respectively.

#### DSTC2

The dialogs were extracted from the Dialogue State Tracking Challenge 2 for
restaurant reservation. Following Bordes et al. (2017), we use merely the raw
text of the dialogs and ignore the dialog state labels. In total, there are
1618 dialogs for training, 500 dialogs for validation, and 1117 dialogs for
testing. Each dialog is composed of user and system utterances, and API calls
to the domain-specific KB for the user’s queries.

#### CamRest

This dataset consists of 676 human-to-human dialogs in the restaurant
reservation domain. This dataset has much more conversation turns with 5.1
turns on average. Following the data processing in Wen et al. (2017), we
divide the dataset into training/validation/testing sets with 406/135/135
dialogs respectively.

###  4.2 Implementation Detail

We use the 300-dimensional word2vec vectors to initialize the word embeddings.
The size of the GRU hidden units is set to 256. The recurrent weight
parameters are initialized as orthogonal matrices. We initialize the other
weight parameters with the normal distribution $N(0,0.01)$ and set the bias
terms as zero. We train our model with Adam optimizer Kingma and Ba (2015)
with an initial learning rate of $1e-4$. By tuning the hyper-parameters with
the grid search over the validation sets, we find the other best settings in
our model as follows. The number of hops for the memory network is set to 3,
and gradients are clipped with a threshold of 10 to avoid explosion. In
addition, we apply the dropout Hinton et al. (2012) as a regularizer to the
input and output of GRU, where the dropout rate is set to be 0.4.

###  4.3 Baseline Models

We compare our model with several existing end-to-end task-oriented dialogue
systems555Part of experimental results of baseline models are directly
extracted from corresponding published papers.:

  * •

Retrieval method: This approach directly uses the retrieved result as the
answer of the given utterance. Specifically, we use BERT-Base as a feature
extractor for the sentences, and we use the cosine distance of the features as
our retrieve scores, and then select the one with the highest score.

  * •

Attn: Vanilla sequence-to-sequence model with attention Luong et al. (2015).

  * •

MemNN: An extended Seq2Seq model where the recurrence read from a external
memory multiple times before outputting the target word Sukhbaatar et al.
(2015).

  * •

PtrUnk: An augmented sequence-to-sequence model with attention based copy
mechanism to copy unknown words during generation Gulcehre et al. (2016).

  * •

CASeq2Seq: This is a copy-augmented Seq2Seq model that learns attention
weights to dialogue history with copy mechanism Eric and Manning (2017).

  * •

Mem2Seq: A memory network based approach with multi-hop attention for
attending over dialogue history and KB tuples Madotto et al. (2018).

  * •

BossNet: A bag-of-sequences memory architecture is proposed for disentangling
language model from KB incorporation in task-oriented dialogues Raghu et al.
(2019).

  * •

WMM2Seq: This method adopts a working memory to interact with two separated
memory networks for dialogue history and KB entities Chen et al. (2019).

  * •

GLMP: This is an augmented memory based model with a global memory pointer and
a local memory pointer to strengthen the model’s copy ability Wu et al.
(2019).

###  4.4 Automatic Evaluation Metrics

In bAbI dataset, we adopt a common metric per-response accuracy Bordes et al.
(2017) to evaluate the model performance. Following previous works Madotto et
al. (2018), for three real human dialog datasets, we employ bilingual
evaluation understudy (BLEU) Papineni et al. (2002) and Entity F1 scores to
evaluate the model’s ability to generate relevant entities from knowledge base
and to capture the semantics of the user-initiated dialogue flow Eric and
Manning (2017).

#### BLEU

We use BLEU to measure the n-gram (i.e., 4-gram) matching between the
generated responses and the reference responses. The higher BLEU score
indicates a better performance of the conversation system. Formally, we
compute the 4-gram precision for the generated response $Y$ as:

$P(Y,\hat{Y})=\frac{\sum_{\tilde{Y}}\text{min}(\eta(\tilde{Y},Y),\eta(\tilde{Y},\hat{Y}))}{\sum_{\tilde{Y}}\eta(\tilde{Y},Y)}$
(8)

where $\tilde{Y}$ traverses all candidate 4-grams, $Y$ and $\hat{Y}$ are the
ground-truth and predicted responses, $\eta(\tilde{Y},Y)$ indicates the number
of 4-grams in $Y$. After achieving the precision, the BLEU score is then
calculated as:

$BLEU=\nu(Y,\hat{Y})\exp(\sum_{n=1}^{4}\beta_{n}\log P(Y,\hat{Y}))$ (9)

where $\beta_{n}=1/4$ is a weight score. $\nu(Y,\hat{Y})$ is a brevity penalty
that penalizes short sentences. The higher BLEU score indicates better
performance of the conversation system.

#### Per-response Accuracy

We adopt the per-response accuracy metric to evaluate the dialog system’s
capability of generating an exact, correct responses. A generated response is
considered right only if each word of the system output matches the
corresponding word in the gold response. The final per-response accuracy score
is calculated as the percentage of responses that are exactly the same as the
corresponding gold dialogues. Per-response accuracy is a strict evaluation
measure, which may only be suitable for the simulated dialog datasets.

#### Entity F1

Entity F1 metric is used measure the system’s capability of generating
relevant entities from the provided task-oriented knowledge base. Each
utterance in the test set has a set of gold entities. An entity F1 is computed
by micro-averaging over all the generated responses.

Method | BLEU | Ent.F1 | Sch.F1 | Wea.F1 | Nav.F1
---|---|---|---|---|---
$R_{h}^{+}$ & $R_{r}^{+}$ | 12.8 | 37.8 | 50.0 | 37.9 | 27.5
$R_{h}^{+}$ & $R_{r}^{-}$ | 12.5 | 36.1 | 49 | 34.6 | 26.7
$R_{h}^{-}$ & $R_{r}^{+}$ | 12.3 | 36.8 | 49.8 | 36.6 | 26.1
$R_{h}^{-}$ & $R_{r}^{-}$ | 11.6 | 34.8 | 48.3 | 31.8 | 26.5
Table 2: Masking comparison experiment on In-Car Assistant. $+$ means with
masking and $-$ denotes without. $R_{h}^{+}$ & $R_{r}^{+}$ means that we
simultaneously mask NEW and EW in the history dialogue and retrieved answers.

##  5 Experimental Results

###  5.1 Automatic Evaluation on Four Datasets

$\theta$ | # of RA | BLEU
---|---|---
0.3 | 2.48 | 56.1
0.4 | 2.16 | 56.2
0.5 | 1.90 | 59.8
0.6 | 1.75 | 56.6
1.0 | 1.00 | 56.5
Table 3: Experimental results in terms of BLEU on DSTC2 by using different
$\theta$. # of RA denotes the average number of retrieved answers.

#### bAbI

The dataset is automatically generated based on some rules, thus many requests
and their corresponding replies are quite similar in terms of the syntactic
structure and the wording usage. According to the results shown in Table 5, we
can see that our model achieves the best per-response scores in all the five
tasks. It is also believed that the retrieved results can contribute to
guiding the response generation in this case, which can be inferred from the
high threshold value ($\theta=0.8$).

Dataset | BM25 | word2vec | BERT
---|---|---|---
Task1 | 68.7 | 63.1 | 74.8
Task2 | 80.6 | 83.2 | 93.7
Task3 | 83.4 | 77.3 | 80.3
Task4 | 87.5 | 87.5 | 87.5
Task5 | 82.9 | 66.6 | 83.8
DSTC2 | 45.3 | 37.3 | 47.1
CAMREST | 27.7 | 29.0 | 30.9
KVR | 33.5 | 33.7 | 35.3
Table 4: Comparison of different matching methods. Task | Retrieval | Attn | MemNN | PtrUnk | Mem2Seq | BossNet | GLMP | WMM2Seq | THPN
---|---|---|---|---|---|---|---|---|---
Task1 | 74.8 | 100 | 99.9 | 100 | 100 | 100 | 100 | 100 | 100
Task2 | 93.7 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100
Task3 | 80.3 | 74.8 | 74.9 | 85.1 | 94.5 | 95.2 | 96.3 | 94.9 | 95.8
Task4 | 87.5 | 57.2 | 59.5 | 100 | 100 | 100 | 100 | 100 | 100
Task5 | 83.8 | 98.4 | 96.1 | 99.4 | 98.2 | 97.3 | 99.2 | 97.9 | 99.6
Table 5: Per-response scores on the five tasks of the bAbI dataset with
$\theta=0.8$.

#### In-Car Assistant Dataset

As shown in Table 6, our model achieves all best metrics (BLEU, Ent.F1,
Sch.F1, Wea.F1 and Nav.F1) over other reported models. The possible reason is
that the retrieved answers with high relevance to the gold answers provide
valid sentence pattern information. By using this sentence pattern
information, our model can better control the generation of responses.
Additionally, our model improves the success rate of generation correct
entities which appeared in the dialogue history.

Method | BLEU | Ent.F1 | Sch.F1 | Wea.F1 | Nav.F1
---|---|---|---|---|---
Retrieval | 15.3 | 20.1 | 24.9 | 26.3 | 9.4
Attn | 9.3 | 19.9 | 23.4 | 25.6 | 10.8
CASeq2Seq | 8.7 | 13.3 | 13.4 | 15.6 | 11.0
MemNN | 8.3 | 22.7 | 26.9 | 26.7 | 14.9
PtrUnk | 8.3 | 22.7 | 26.9 | 26.7 | 14.9
Mem2Seq | 12.6 | 33.4 | 49.3 | 32.8 | 20.0
BossNet | 8.3 | 35.9 | 50.2 | 34.5 | 21.6
THPN | 12.8 | 37.8 | 50.0 | 37.9 | 27.5
Table 6: Evaluation results on the In-Car Assistant dataset with $\theta=0.3$. Task | Task1 | Task2 | Task3 | Task4 | Task5 | DSTC2 | DSTC2 | DSTC2
---|---|---|---|---|---|---|---|---
(BLEU) | (BLEU) | (BLEU) | (BLEU) | (BLEU) | (BLEU) | (F1) | (Per-Res)
THPN | 100 | 100 | 98.9 | 100 | 99.9 | 59.8 | 76.8 | 47.7
W/O IR | 100 | 100 | 96.5 | 100 | 99.2 | 57.8 | 73.2 | 45.9
W/O Ptr | 100 | 100 | 97.7 | 89.9 | 98.5 | 58.1 | 72.6 | 46.1
W/O Gate | 100 | 100 | 95.9 | 94.4 | 99.2 | 57.7 | 74.1 | 45.8
Table 7: Ablation test results of our THPN model on bAbI and DSTC2 datasets.

#### DSTC2 and CamRest Datasets

We also present the evaluation on DSTC2 and CamRest datasets in Table 8 and
Table 9, respectively. By comparing the results, we can notice that our model
performs better than the compared methods. On the DSTC2, our model achieves
the state-of-the-art performance in terms of both Entity F1 score and BLEU
metrics, and has a comparable per-response accuracy with compared methods. On
the CamRest, our model obtains the best Entity F1 score but has a drop in BLEU
in comparison to Mem2Seq model.

Method | Ent.F1 | BLEU
---|---|---
Retrieval | 21.1 | 47.1
Attn | 67.1 | 56.6
KV Net | 71.6 | 55.4
Mem2Seq | 75.3 | 55.3
GLMP | 67.4 | 58.1
THPN | 76.8 | 59.8
Table 8: Evaluation on DSTC2($\theta=0.5$). Method | Ent.F1 | BLEU
---|---|---
Retrieval | 7.9 | 21.2
Attn | 21.4 | 5.9
PtrUnk | 16.4 | 2.1
KV Net | 9.1 | 4.3
Mem2Seq | 27.7 | 12.6
THPN | 30.9 | 12.9
Table 9: Evaluation on CamRest($\theta=0.4$).

###  5.2 Ablation Study

An ablation study typically refers to removing some components or parts of the
model, and seeing how that affects performance. To measure the influence of
the individual components, we evaluate the proposed THPN model with each of
them removed separately, and then measure the degradation of the overall
performance. Table 7 reports ablation study results of THPN on bAbI and DSTC2
datasets by removing retrieved answers (w/o IR), removing EPN and PPN in
decoding (w/o Ptr), removing answer-guided gating mechanism (w/o Gate),
respectively. For example, “w/o Gate” means we do not use the answer-guided
gating mechanism while keeping other components intact.

If the retrieved answer is not used, the performance reduces dramatically,
which can be interpreted that without the guiding information from the
retrieved answer, the decoder may deteriorate quickly once it produce a “bad”
word since it solely relies on the input query.

If no copy mechanism is used, we can see that Entity F1 score is the lowest,
which indicates that many entities are not generated since these entity words
may not be included in the vocabulary. Therefore, the best way to generate
some unseen words is to directly copy from the input query, which is
consistent with the findings of previous work Eric et al. (2017); Madotto et
al. (2018).

If the gate is excluded, we can see around $2$% drop for DSTC2. A possible
reason is that some useless retrieved answers introduce “noise” to the system,
which deteriorates the response generation.

###  5.3 Effect of Masking Operation

To validate the effectiveness of the masking operation, we carry out a
comparison experiment on In-Car Assistant, and present the results in Table 2.
From Table 2, we can see that $R_{h}^{+}$ & $R_{r}^{+}$ achieves the best
performance while $R_{h}^{-}$ & $R_{r}^{-}$ has the lowest scores. By diving
into the experimental results, we find that if we do not mask EW in the
retrieved answers, the model copies many incorrect entities from the retrieved
answers, which reduces the Entity F1 scores. If we do not mask NEW in the
history dialogue, the percentage of NEW copied from the history dialogue is
high, most of which are unrelated to the gold answer, thus bringing down the
BLEU score.

###  5.4 Analysis on Retrieved Results

#### Comparison of Different Retrieval Methods

According to our preliminary experimental results, we observed that better
retrieved candidate answers could further improve the overall model
performance in response generation. Therefore, we also conduct experiments to
evaluate the effectiveness of three popular text matching methods, including
BM25 Robertson et al. (2009), word2vec Mikolov et al. (2013a) and BERT Devlin
et al. (2018). Here, BLEU is utilized as our evaluation criterion. From the
experimental results shown in Table 4, we can see that using BERT Devlin et
al. (2018), a transformer-based pre-trained language model, achieves the
highest BLEU scores. A possible reason is that the size of each training
dataset is limited, the word co-occurrence based algorithms (e.g., BM25) may
not capture the semantic information, thus result in poor retrieving
performance.

#### One vs. Multiple Retrieved Answers

Cosine similarity is not an absolute criterion and there is no guarantee that
a candidate with higher cosine value will always provide more reference
information to the response generation. Therefore, we conduct an experiment to
investigate the effect of the number of retrieved answers. By setting
different cosine threshold values $\theta$, we retrieve different numbers of
answer candidates. In particular, if no answer candidate satisfies the given
threshold, we choose one with the highest cosine value. To limit the number of
retrieved answers, we only select the top-$3$ results if there are more than
three answer candidates that have higher consine values than the given
threshold $\theta$.

Table 3 gives the experimental results of DSTC2 dataset under different
threshold $\theta$ values. When $\theta$ is set to be $1.0$, it is considered
as a special case where only one answer is retrieved. We can observe that
using multiple answer candidates obtains higher performance than only using
one result. It is intuitive that the model will be misguided if the retrieved
single answer has no relation to the given request, and using multiple
candidate answers can ameliorate this issue.

#### Setting of $\theta$

Although using more retrieved answers might improve the chance of including
the relevant information, it may also bring more “noise” and adversely affect
the quality of retrieved answers. From Table 3, we can see that with the
reduced value of $\theta$, the average number of retrieved candidate answers
increase, but the model performance does not improve accordingly. Experimental
results on the other datasets demonstrate that the $\theta$ is not fixed and
needs to be adjusted according to the experimental data.

##  6 Conclusion

In task-oriented dialog systems, the words and sentence structures are
relatively limited and fixed, thus it is intuitive that the retrieved results
can provide valuable information in guiding the response generation. In this
paper, we retrieve several potentially relevant answers from a pre-constructed
domain-specific conversation repository as guidance answers, and incorporate
the guidance answers into both the encoding and decoding processes. We copy
the words from the previous context and the retrieved answers directly, and
generate words from the vocabulary. Experimental results over four datasets
have demonstrated the effectiveness of our model in generating informative
responses. In the future, we plan to leverage the dialogue context information
to retrieve candidate answers turn by turn in multi-turn scenarios.

## References

  * Bordes et al. (2017) Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017.  Learning End-to-end Goal-oriented Dialog.  _International Conference on Learning Representations_.
  * Cao et al. (2018) Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018.  Retrieve, Rerank and Rewrite: Soft Template Based Neural Summarization.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , volume 1, pages 152–161.
  * Chen et al. (2017) Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Diana Inkpen, and Si Wei. 2017.  Neural natural language inference models enhanced with external knowledge.  _arXiv preprint arXiv:1711.04289_.
  * Chen et al. (2019) Xiuyi Chen, Jiaming Xu, and Bo Xu. 2019.  A working memory model for task-oriented dialog response generation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2687–2693.
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling.  _arXiv preprint arXiv:1412.3555_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _North American Chapter of the Association for Computational Linguistics_.
  * Ding et al. (2019) Xiao Ding, Kuo Liao, Ting Liu, Zhongyang Li, and Junwen Duan. 2019.  Event representation learning enhanced with external commonsense knowledge.  _arXiv preprint arXiv:1909.05190_.
  * Eric et al. (2017) Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D Manning. 2017\.  Key-value retrieval networks for task-oriented dialogue.  In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 37–49.
  * Eric and Manning (2017) Mihail Eric and Christopher D Manning. 2017.  A Copy-augmented Sequence-to-sequence Architecture Gives Good Performance on Task-oriented Dialogue.  _European Association of Computational Linguistics_ , page 468.
  * Ghazvininejad et al. (2018) Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018.  A Knowledge-grounded Neural Conversation Model.  In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Gulcehre et al. (2016) Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016\.  Pointing the unknown words.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , volume 1, pages 140–149.
  * He et al. (2019) Tianxing He, Jingzhao Zhang, Zhiming Zhou, and James Glass. 2019.  Quantifying exposure bias for neural language generation.  _arXiv preprint arXiv:1905.10617_.
  * Henderson et al. (2014) Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014.  The Second Dialog State Tracking Challenge.  In _Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 263–272.
  * Hinton et al. (2012) Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012.  Improving Neural Networks by Preventing Co-adaptation of Feature Detectors.  _arXiv preprint arXiv:1207.0580_.
  * Kingma and Ba (2015) Diederik P Kingma and Lei Ba. 2015.  ADAM: A Method for Stochastic Optimization.  In _Proceedings of the 3rd International Conference on Learning Representations_.
  * Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles. 2017.  Six Challenges for Neural Machine Translation.  _arXiv preprint arXiv:1706.03872_.
  * Lemon et al. (2006) Oliver Lemon, Kallirroi Georgila, James Henderson, and Matthew Stuttle. 2006.  An isu dialogue system exhibiting reinforcement learning of dialogue policies: generic slot-filling in the talk in-car system.  In _Proceedings of the Eleventh Conference of the European Chapter of the Association for Computational Linguistics: Posters & Demonstrations_, pages 119–122. Association for Computational Linguistics.
  * Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D Manning. 2015.  Effective Approaches to Attention-based Neural Machine Translation.  _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1412–1421.
  * Madotto et al. (2018) Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2018.  Mem2seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog systems.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1468–1478.
  * Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b.  Distributed representations of words and phrases and their compositionality.  In _Advances in neural information processing systems_ , pages 3111–3119.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  BLEU: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting on association for computational linguistics_ , pages 311–318.
  * Qin et al. (2019) Libo Qin, Yijia Liu, Wanxiang Che, Haoyang Wen, Yangming Li, and Ting Liu. 2019\.  Entity-consistent end-to-end task-oriented dialogue system with kb retriever.  _arXiv preprint arXiv:1909.06762_.
  * Raghu et al. (2019) Dinesh Raghu, Nikhil Gupta, et al. 2019.  Disentangling language and knowledge in task-oriented dialogs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1239–1255.
  * Ranzato et al. (2016) Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016.  Sequence level training with recurrent neural networks.  _International Conference on Learning Representations_.
  * Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Foundations and Trends® in Information Retrieval_ , 3(4):333–389.
  * Song et al. (2018) Yiping Song, Cheng-Te Li, Jian-Yun Nie, Ming Zhang, Dongyan Zhao, and Rui Yan. 2018.  An ensemble of retrieval-based and generation-based human-computer conversation systems.  In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden_ , pages 4382–4388. ijcai.org.
  * Sukhbaatar et al. (2015) Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015.  End-to-end memory networks.  In _Advances in neural information processing systems_ , pages 2440–2448.
  * Wang and Lemon (2013) Zhuoran Wang and Oliver Lemon. 2013.  A simple and generic belief tracking mechanism for the dialog state tracking challenge: On the believability of observed information.  In _Proceedings of the SIGDIAL 2013 Conference_ , pages 423–432.
  * Wen et al. (2016) Tsung-Hsien Wen, Milica Gasic, Nikola Mrkšić, Lina M Rojas Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve Young. 2016.  Conditional generation and snapshot learning in neural dialogue systems.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2153–2162.
  * Wen et al. (2017) Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina Maria Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve J. Young. 2017.  A network-based end-to-end trainable task-oriented dialogue system.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 1: Long Papers_ , pages 438–449. Association for Computational Linguistics.
  * Weston et al. (2015) Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015.  Towards ai-complete question answering: A set of prerequisite toy tasks.  _arXiv preprint arXiv:1502.05698_.
  * Williams and Young (2007) Jason D Williams and Steve Young. 2007.  Partially observable markov decision processes for spoken dialog systems.  _Computer Speech & Language_, 21(2):393–422.
  * Wu et al. (2019) Chien-Sheng Wu, Richard Socher, and Caiming Xiong. 2019.  Global-to-local memory pointer networks for task-oriented dialogue.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Yan et al. (2016) Rui Yan, Yiping Song, and Hua Wu. 2016.  Learning to Respond with Deep Neural Networks for Retrieval-based Human-computer Conversation System.  In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_ , pages 55–64.
  * Yang et al. (2019) Pengcheng Yang, Lei Li, Fuli Luo, Tianyu Liu, and Xu Sun. 2019.  Enhancing topic-to-essay generation with external commonsense knowledge.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2002–2012.
  * Young et al. (2013) Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013\.  Pomdp-based statistical spoken dialog systems: A review.  _Proceedings of the IEEE_ , 101(5):1160–1179.
  * Zhang et al. (2019) Yue Zhang, Rui Wang, and Luo Si. 2019.  Syntax-enhanced self-attention-based semantic role labeling.  _arXiv preprint arXiv:1910.11204_.
  * Zhu et al. (2018) Qingfu Zhu, Lei Cui, Weinan Zhang, Furu Wei, Yining Chen, and Ting Liu. 2018.  Retrieval-Enhanced Adversarial Training for Neural Response Generation.  _arXiv preprint arXiv:1809.04276_.


# AppTek’s Submission to the IWSLT 2022
Isometric Spoken Language Translation Task

Patrick Wilken
AppTek
Aachen, Germany
<EMAIL_ADDRESS>
&Evgeny Matusov
AppTek
Aachen, Germany
<EMAIL_ADDRESS>

###### Abstract

To participate in the Isometric Spoken Language Translation Task of the IWSLT
2022 evaluation, constrained condition, AppTek developed neural Transformer-
based systems for English-to-German with various mechanisms of length control,
ranging from source-side and target-side pseudo-tokens to encoding of
remaining length in characters that replaces positional encoding. We further
increased translation length compliance by sentence-level selection of length-
compliant hypotheses from different system variants, as well as rescoring of
N-best candidates from a single system. Length-compliant back-translated and
forward-translated synthetic data, as well as other parallel data variants
derived from the original MuST-C training corpus were important for a good
quality/desired length trade-off. Our experimental results show that length
compliance levels above 90% can be reached while minimizing losses in MT
quality as measured in BERT and BLEU scores.

##  1 Introduction

In this paper, we describe AppTek’s submission to the IWSLT 2022 Isometric
Spoken Language Translation evaluation (Anastasopoulos et al., 2022). Our goal
was to create a system that produces translations which are within 10% of the
source sentence length, but have similar levels of quality as a baseline
system translations without length control. AppTek participated in the
constrained condition with an English-to-German neural machine translation
(NMT) system that we describe in Section 2. The system was extended with 5
different length control methods, which we explain in detail in Section 3. We
also created synthetic data with back-translation, forward-translation, as
well as a novel data augmentation method of synonym replacement. All three
methods are described in Section 4. Our experimental results on the MuST-C
tst-COMMON test set and the official evaluation test set are presented in
Section 5, including ablation studies that prove the effectiveness of
synthetic data and noisy length encoding for a better trade-off between length
compliance and MT quality. We summarize our findings in Section 6.

##  2 Baseline system

###  2.1 Data

We follow the constrained condition of the IWSLT Isometric SLT task and use
only English-to-German TED-talk data from the MuST-C corpus (Di Gangi et al.,
2019). The corpus contains 251K sentence pairs with 4.7M and 4.3M English and
German words, respectively.

We apply minimal text pre-processing, mainly consisting of normalization of
quotes and dashes. 2K sentences that have mismatching digits or parentheses in
source and target were filtered out.

We use a joint English and German SentencePiece model Kudo and Richardson
(2018), trained on the whole corpus using a vocabulary size of 20K, to split
the data into subwords.

###  2.2 Neural NMT model

In preliminary experiments we tried several Transformer model configurations,
including base and big from the original paper (Vaswani et al., 2017), a 12
encoder and decoder layer variant of base, and a "deep" 20 encoder layer
version with halved feed-forward layer dimension in the encoder and only 4
attention heads. These attempts to optimize the model architecture for the
given, rather low resource task did not yield a better architecture than
Transformer big, which we end up using in all our experiments.

We however find an increased dropout rate of 0.3 and an increased label
smoothing of 0.2 to be crucial. We further optimize the model by sharing the
parameters of the source and target embeddings as well as the softmax
projection matrix.

In all experiments we use two translation factors (García-Martínez et al.,
2016) on both the source and target side to represent the casing of the
subwords and the binary decision whether a subword is attached to the previous
subword Wilken and Matusov (2019). This allows for explicit sharing of
information between closely related variants of a subword and reduces the
model vocabulary size.

All models are trained on a single GPU for 162 to 198 epochs of 100K sentence
pairs each in less than two days. We use batches of 1700 subwords and
accumulate gradients over 8 subsequent batches. The global learning rate of
the Adam optimizer is increased linearly from $3\text{\times}{10}^{-5}$ to
$3\text{\times}{10}^{-4}$ in the first 10 epochs and then decreased
dynamically by factor 0.9 each time perplexity on the MuST-C dev set increases
during 4 epochs. For decoding we use beam search with a beam size of 12.

We train the Transformer models using RETURNN (Doetsch et al., 2017; Zeyer et
al., 2018), which is a flexible neural network toolkit based on Tensorflow
(Abadi et al., 2015). Automation of the data processing, training and
evaluation pipelines is implemented with Sisyphus (Peter et al., 2018).

##  3 Length control methods

In this work we perform an extensive evaluation of different ways to control
the length of the translations generated by the NMT model, all applied to the
same baseline Transformer big model.

###  3.1 N-best rescoring

A simple method to achieve length compliant translation is to generate N-best
lists and select translation hypotheses from the lists that adhere to the
desired length constraints. Saboo and Baumann (2019) and Lakew et al. (2021)
compute a linear combination of the original MT model score and a length-
related score to reorder the N-best list. In this work, we simply extract the
translation from the N-best list with the best MT score that has a character
count within a 10% margin of the source character count and fall back to the
first best hypothesis if there is no such translation. This approach is
tailored towards the evaluation condition of the IWSLT Isometric SLT task
where length compliance within a 10% margin is a binary decision and the
absolute length difference is not considered.

While N-best rescoring has the advantage of being applicable to any NMT model
that uses beam search, it is outperformed by learned length control methods
because in many cases there is no length compliant translation in the N-best
list, and also because learned methods are able to shorten the translation in
a more semantically meaningful way. However, we use N-best rescoring on top of
other methods to further improve length compliance, as done by Lakew et al.
(2021).

###  3.2 Length class token

Lakew et al. (2019) introduce a special token at the start of the source
sentence to control translation length. For this, the training data is
classified into difference length classes based on the target-to-source ratio
measured in number of characters. In this work we use two variants of length
classes:

  1. 1.

3 length bins representing "too short", "length compliant" and "too long".
Length compliant here means the number of characters in source and target
differs by less than 10%;

  2. 2.

7 length bins from "extra short" to "extra long", such that an approximately
equal number of training sentence pairs falls into each bin.

The first option is focused on isometric MT, i.e. equal source and target
length, while the second option offers a more fine-grained length control.

In addition, we analyze the difference of adding the token to the source
versus the target side. Adding the token on the target side has the advantage
of offering the option to not enforce a length class at inference time and
instead let the model perform an unbiased translation. This is especially
important in a commercial setting where costs can be saved by deploying a
single model for general and isometric MT.

####  3.2.1 Length ROVER

A system that takes a length class as input can produce multiple different
translations of a given source sentence. To maximize the chance for length
compliant translations, we produce translations of the whole test set for each
of the length bins and then, for each sentence, select the hypothesis which
adheres to the length constraint. We refer to this as length ROVER, in analogy
to the automatic speech recognition system combination technique called ROVER
Fiscus (1997). If multiple length bins produce a length compliant translation,
precedence is determined by the corpus-level translation quality scores for
the different length bins. If no bin produces a length compliant translation
the bin with the best corpus-level translation quality is used as fallback.

As we use a target-side length token, we can let the model predict the length
token instead of forcing one. This usually leads to the best corpus-level
translation quality. We include this freely decoded translation in the length
ROVER.

When applying the length ROVER to the 7-bin model, we exclude the bins
corresponding to the longest and shortest translations as those rarely lead to
length compliant translations but generally to degraded translation quality.
The same is true for the "too short" and "too long" bins in the 3-bin model,
which is why we do not use the length ROVER for this model.

###  3.3 Length encoding

We adopt length-difference positional encoding (LDPE) from Takase and Okazaki
(2019). It replaces the positional encoding in the transformer decoder, which
usually encodes the absolute target position, with a version that "counts
down" from a desired output length $L_{\mathrm{forced}}$ to zero. At each
decoding step the available remaining length is an input to the decoder and
thus the model learns to stop at the right position. In training,
$L_{\mathrm{forced}}$ is usually set to the reference target length
$L_{\mathrm{target}}$, while at inference time it can be set as desired. For
isometric MT, setting it to the source length
$L_{\mathrm{forced}}=L_{\mathrm{source}}$ is the natural choice.

The original work of Takase and Okazaki (2019) uses a character-level decoder,
which means that the number of decoding steps equals the translation length,
assuming the latter is measured in number of characters. Using subwords
Sennrich et al. (2016) as the output unit of the decoder is more common in
state-of-the-art systems (Akhbardeh et al., 2021). In this case, one can
either encode the target length in terms of number of subword tokens (Liu et
al., 2020; Niehues, 2020; Buet and Yvon, 2021), or keep the character-level
encoding which however requires subtracting the number of characters in the
predicted subword token in each decoding step (Lakew et al., 2019). The former
has the disadvantage that the number of subword tokens is a less direct
measure of translation length, especially for the case of the IWSLT Isometric
SLT task where length compliance is measured in terms of number of characters.
The second option is more exact but arguably a bit more complex to implement.
In this work we compare results for both methods.

In contrast to (Lakew et al., 2019) we do not combine standard token-level
positional encoding and character-level length encoding, instead we only use
the latter.

####  3.3.1 Length perturbation

For both the token-level and character-level version we add random noise to
the encoded translation length $L_{\mathrm{forced}}$ during training (Oka et
al., 2020). We find that this is necessary to make the model robust to the
mismatch between training, where the target length is taken from a natural
translation, and inference, where the enforced target length is a free
parameter. Especially in the case of character-length encoding one cannot
expect that a high-quality translation with a given exact character count
exists. As opposed to Oka et al. (2020), who add a random integer to the
token-level target length sampled from a fixed interval, e.g. $[-4,4]$, we
chose a relative +/-10% interval:

$L_{\mathrm{forced}}\sim U\left(\lfloor 0.9\cdot
L_{\mathrm{target}}\rceil,\lfloor 1.1\cdot L_{\mathrm{target}}\rceil\right)$
(1)

Here, $U(n,m)$ denotes the discrete uniform distribution in the interval
$[n,m]$, and $\lfloor\cdot\rceil$ denotes rounding to the nearest integer.
This is in line with the +/-10% length compliance condition used in the
evaluation. The length difference subtracted in each decoder step is left
unaltered, which means counting down will stop at a value that in general is
different from zero.

####  3.3.2 Second-pass length correction

Length encoding as described above does not result in a length compliant
translation in all cases. The reasons for this are: 1. general model
imperfections, intensified by the small size of the training data in the
constrained track; 2. the noise added to the target length in training
(although it is within the "allowed" 10% range); 3. for the case of token-
level length encoding, an equal number of source and target tokens does not
necessarily mean an equal number of characters.

We therefore perform a second decoding pass for those sentences where the
first pass does not generate a length compliant translation. In this second
pass, instead of attempting to enforce
$L_{\mathrm{forced}}=L_{\mathrm{source}}$, we make a correction by multiplying
by the source-to-target ratio observed in the first pass (measured in tokens
or characters, depending on the unit used for length encoding):

$L^{\mathrm{2\text{-}pass}}_{\mathrm{forced}}=\left\lfloor
L_{\mathrm{source}}\cdot\frac{L_{\mathrm{source}}}{L^{\mathrm{1\text{-}pass}}_{\mathrm{target}}}\right\rceil$
(2)

$L^{\mathrm{1\text{-}pass}}_{\mathrm{target}}$ is the first pass translation
length, $\lfloor\cdot\rceil$ denotes rounding. That way, an over-translation
of factor $r$ in the first pass will be counteracted by "aiming" at a
translation length of $1/r$ of the source length in the second pass.

This procedure could be applied iteratively, one could even run a grid search
of many different values for $L_{\mathrm{forced}}$ until a length compliant
translation is generated. We refrain from doing so as we find it to be
impracticable in real-world applications.

##  4 Synthetic data

We expand the original MuST-C data with synthetic data of different types, all
derived from the given MuST-C corpus.

First, we include a copy of the data111Including, if applicable, the synthetic
data described below. in which two consecutive sentences from the same TED
talk are concatenated into one. Since many segments in the original data are
short, this helps to learn more in-context translations. Then, we also include
a copy of the data where the English side is preprocessed by lowercasing,
removing punctuation marks and replacing digits, monetary amounts and other
entities with their spoken forms. This helps to adjust to the spoken style of
TED talks and imperfections in the (manual) transcriptions of the training and
evaluation data.

We also use 82K bilingual phrase pairs extracted from word-aligned MuST-C
data, as described below, as training instances.

###  4.1 Word synonym replacement

To enrich the training data with more examples of length-compliant
translations, we experiment with a novel technique of replacing a few randomly
selected source (English) words in a given sentence pair with their synonyms
which are shorter/longer in the number of characters, so that the resulting
modified synthetic sentence is closer to being length compliant. Whereas in an
unconstrained conditions the synonyms can come from WordNet or other sources,
in the constrained track we rely on synonyms extracted from a bilingual
lexicon. The replacement of a source word with a synonym in a given sentence
pair happens only if it is aligned to a target word, for which another word
translation exists in the bilingual lexicon.

The word alignment and bilingual word lexicon extraction is performed on the
lowercased MuST-C corpus itself using FastAlign Dyer et al. (2013). The
bilingual lexicon is filtered to contain entries with the costs (negative log
of the word-level translation probability) of 50 or lower.

We apply the synonym replacements only to sentence pairs for which the target
sentence is not length-compliant with the source. We first generate multiple
versions of modified source sentences for these data, which all differ in the
choice of randomly selected words that are to be replaced with synonyms and in
the actual synonyms selected for replacement (also at random). Each word in a
sentence has a 0.5 chance of being considered for replacement (regardless of
whether it has synonyms as defined above or not), and the replacement is done
with (at most) one of 3 synonym candidates with the highest lexicon
probability which have fewer or more characters than the word being replaced,
depending on whether the length of the original sentence was too long or too
short.

From the resulting data (ca. 1M sentences), we keep only those modified source
sentences for which the BERT F1 score (Zhang et al., 2020) with respect to the
original (unmodified) source sentence is 0.94 or higher. In this way we try to
make sure that the meaning of the modified source sentence stays very close to
the original meaning. This way, only 192K sentences are kept, which are then
paired with the original target (German) sentences to form a synthetic synonym
replacement parallel corpus.

###  4.2 Back-translated data

We train the reverse, German-to-English system with 7 length bins and source
length token as described in Section 3 using the same architecture and
settings as for the English-to-German system. We then use this system to
translate the MuST-C corpus from German to English, generating 7 translations
of each sentence for each of the 7 bins. From these data, we keep all back-
translations which make the corresponding German sentence length-compliant.
This resulted in a back-translated corpus of 172K sentence pairs.

|  | tst-COMMON v2 | blind test
---|---|---|---
# |  | BLEU | BERT | LC | BLEU | BERT | LC
0 |  baseline (no length control) | 32.0 | 84.00 | 44.03 | 19.2 | 77.94 | 45.50
1 | source-side token, 3 bins | 31.3 | 83.94 | 51.59 | 20.6 | 78.40 | 62.50
2 |  \+ N-best rescoring | 30.5 | 83.60 | 78.41 | 20.1 | 77.78 | 81.50
3 | target-side token, 3 bins | 31.4 | 83.88 | 50.12 | 19.7 | 78.37 | 53.50
4 |  \+ N-best rescoring | 30.7 | 83.58 | 77.40 | 18.3 | 77.43 | 82.50
| target-side token, 7 bins |  |  |  |  |  |
5 | predicted token (no length control) | 32.0 | 84.00 | 45.23 | 18.3 | 77.55 | 46.50
6 |  \+ N-best rescoring | 31.1 | 83.75 | 71.20 | 18.9 | 77.38 | 72.50
7 | M token | 31.7 | 83.99 | 49.19 | 19.1 | 78.24 | 56.00
8 |  \+ N-best rescoring | 31.0 | 83.74 | 76.39 | 18.6 | 77.68 | 81.00
9 | S token | 30.5 | 83.73 | 62.95 | 18.9 | 78.05 | 59.00
10 |  \+ N-best rescoring | 29.8 | 83.38 | 87.64 | 18.9 | 77.52 | 85.50
11 | XS token | 28.1 | 83.09 | 72.13 | 18.2 | 77.81 | 68.00
12 |  \+ N-best rescoring | 27.8 | 82.91 | 92.21 | 17.8 | 77.32 | 90.00
13 | ROVER over XS to XL | 29.0 | 83.35 | 80.66 | 17.5 | 77.59 | 76.50
14 |  \+ N-best rescoring | 28.0 | 82.94 | 94.19 | 17.6 | 77.09 | 93.00
15 | ROVER over S to L | 31.1 | 83.83 | 66.90 | 18.2 | 77.76 | 65.50
16 |  \+ N-best rescoring | 30.0 | 83.38 | 88.57 | 18.7 | 77.32 | 86.50
17 | length encoding (tokens) | 31.5 | 83.91 | 48.57 | 19.6 | 77.45 | 55.50
18 |  \+ 2-pass length correction | 30.0 | 83.42 | 68.14 | 19.5 | 77.75 | 75.50
19 |  \+ N-best rescoring | 30.9 | 83.66 | 72.36 | 19.3 | 77.47 | 80.50
20 |  \+ 2-pass length correction | 29.5 | 83.12 | 88.41 | 19.0 | 76.95 | 92.00
21 | length encoding (characters) | 30.7 | 83.57 | 63.64 | 20.1 | 78.27 | 73.00
22 |  \+ 2-pass length correction | 29.3 | 82.89 | 89.50 | 19.2 | 77.55 | 90.50
23 |  \+ N-best rescoring | 30.0 | 83.24 | 88.10 | 19.2 | 77.22 | 95.50
24 |  \+ 2-pass length correction | 29.2 | 82.76 | 98.14 | 18.8 | 76.80 | 98.00
Table 1: English$\rightarrow$German translation results for MuST-C tst-COMMON
and the IWSLT 2022 Isomtetric SLT blind test. All values in %. LC = length
compliance within 10% in number of characters. All systems are based on the
same Transformer big model. Length bins of the 7-bin system are referred to as
XXS, XS, S, M, L, XL and XXL from short to long. For explanation of N-best
rescoring, ROVER, and 2-pass length correction refer to Section 3.

###  4.3 Forward-translated data

In addition to back-translated data, we also augmented our training corpus
with forward-translated data. For this, we generated translations using our
English-to-German system with 7 length bins and a source length token for each
of the length classes. Then, we kept only those translations which turned out
to be length-compliant with the corresponding source sentence. The resulting
synthetic corpus has 213K sentence pairs.

##  5 Experimental results

Table 1 presents results for all length control methods explored in this work.
We evaluate on MuST-C tst-COMMON v2222The official evaluation uses tst-COMMON
v1. Differences in metric scores are minor though. and the blind test set
provided by the shared task organizers using the official scoring
script333Blind test set and scoring script are published under
https://github.com/amazon-research/isometric-slt.. As a measure of MT quality
it computes BLEU (Papineni et al., 2002; Post, 2018) and BERT F1 score (Zhang
et al., 2020). Length compliance (LC) is calculated as the proportion of
translations that have a character count which differs by 10% or less from the
number of characters in the source sentence. For this, spaces are not counted
and sentences with less than 10 characters are ignored. References for the
blind test set were made available only after development of the systems. Line
0 in Table 1 corresponds to a system trained without any of the length control
methods from Section 3. All systems use all synthetic data as described in
Section 4 if not stated otherwise.

###  5.1 Length token systems

Rows 1 to 4 of Table 1 show results for the 3-bin length token systems. The
"length compliant" bin is used for all translations. (When used on the target
side it is enforced as the first decoding step.) Overall, we observe no major
differences between a source-side and target-side length token in both LC and
MT quality scores. Synthetic data and selection of the length bin alone leads
to length compliant translations in about 50% of cases (rows 1 and 3). This
shows that the model has to compromise between translation quality and length
and that a length token is not a strong enough signal to enforce the
corresponding length class in all cases. N-best rescoring, i.e. selection of a
length compliant translation from the beam search output of size 12, can
improve LC to 78% on tst-COMMON but comes at the cost of a loss in translation
quality by 0.8% BLEU and 0.3% BERTScore absolute.

The 7-bin system shown in rows 5 to 16 offers a greater variety of trade-off
points. We refer to the 7 length bins with size labels from "XXS" to "XXL".
The target-to-source ratio boundaries for equally sized bins in terms of
training examples are computed to be 0.90, 0.98, 1.02, 1.06, 1.10, and 1.23.
This means the desired 1.0 ratio for isometric MT falls into the "S" bin.

Row 5 shows the scores achieved when not forcing any length token. This
configuration leads to the same quality on tst-COMMON as the baseline system,
namely 32.0% BLEU and 84.0% BERTScore. This indicates that the model is able
to predict the right length class corresponding to an unbiased translation.
Setting the length token to either "M", "S" or "XS" offers different trade-
offs between translation quality and length compliance. Interestingly, the
"XS" class has a higher LC than the class "S" which should represent
translations with a target-to-source ratio closer to 1. Again, this shows that
the effect of length tokens is in conflict with general translation quality,
which is optimal when not skipping any information present in the source. A
more extreme length class has to be chosen to achieve the desired amount of
compression. In all cases N-best rescoring has the same effect as observed for
the 3-bin systems, namely a higher LC at the cost of worse translation
quality. All length classes not shown in the table lead to either clearly
worse LC or quality scores.

The outputs for different length tokens, possibly after N-best rescoring, can
be combined with the length ROVER. As mentioned in Section 3.2.1, we exclude
the extreme length classes. We consider two variants: excluding the bins with
shortest and longest translations, or excluding the two shortest and longest.
As expected, both variants lead to more length compliant translations in the
combined output. However, they provide different trade-offs: while the first
variant (rows 13, 14) can achieve 94% length compliance on tst-COMMON,
translation quality drops to similarly low values as observed for the "XS"
length class. The second variant is more conservative and achieves only 89%
length compliance, but preserves higher BLEU and BERT scores.

###  5.2 Length encoding systems

Rows 17 to 24 of Table 1 show the results of systems trained with length
encoding as described in Section 3.3. They are also trained using 3 length
bins and a "length compliant" token is forced on the target side, we however
observe no significant differences to not using the token.

Using the source length as input to the decoder
($L_{\mathrm{forced}}=L_{\mathrm{source}}$), the token-level length encoding
model (row 17) does not achieve a higher LC value than the length token
systems (49%), while the model with character-level length encoding (row 21)
is able to produce compliant translations in 64% of the cases. Doing a length-
corrected second decoding pass is very effective for both systems. This shows
that the decoder input $L_{\mathrm{forced}}$ has a strong impact on the model
output, however has to be adjusted to get the desired output length. In
Section 3.3.1 we give explanations for such imperfections. In addition,
similar to the case of length tokens, we attribute this to the fact that in
training the desired length is always conform with the reference translation,
while at inference time the model often has to compress its output to fulfill
the length constraints, which might require a more extreme value for the
targeted length $L_{\mathrm{forced}}$.

N-best rescoring can be applied on top to achieve a further large increase in
length compliance444First-best translation length of first pass is used for
length correction, N-best rescoring only applied in the second pass.. This
indicates that there is length variety in the N-best list that at least in
part can be attributed to the noise added through length perturbation (Section
3.3.1). The resulting character-level length encoding system in row 24
achieves the overall best length compliance value of 98.14%.

$82.8$$83$$83.2$$83.4$$83.6$$83.8$$84$$40$$50$$60$$70$$80$$90$$100$123456789101112131415161718192021222324BERTLC
Figure 1: Visualization of length compliance (LC) vs. BERTScore trade-offs on
MuST-C tst-COMMON for systems taken from Table 1. Data point labels are the
row numbers (#) from Table 1. Submitted systems are labeled in bold blue.

###  5.3 System selection

To select systems for our submission, in Figure 1 we visualize the inherent
trade-off between length compliance and translation quality for the systems
from Table 1. We look at BERT scores as they were announced to be the main MT
quality metric for the evaluation. We chose system 16, the 7-bin length token
system using the length ROVER, as our primary submission. As contrastive
submissions we include systems 2 (3 length bins using source-side token), 14
(ROVER variation of the primary submission) and 24 (character-level length
encoding with second-pass length correction). All submissions use N-best
rescoring. As it can be seen, the different length control methods are all
able to provide useful trade-off points. While only length encoding can
achieve a near perfect length compliance, length token-based methods can offer
a good compromise that preserves more of the baseline MT performance.

###  5.4 Ablation study

For a selected subset of the systems we show the contribution of the most
important types of synthetic data used in our systems (Section 4), as well as
the effect of length perturbation (Section 3.3.1).

|  | tst-COMMON v2 | blind test
---|---|---|---
# |  | BLEU | BERT | LC | BLEU | BERT | LC
| target-side token, 7 bins |  |  |  |  |  |
1 | Row 16, Table 1 | 30.0 | 83.38 | 88.57 | 18.7 | 77.32 | 86.50
2 |  \+ no synonym replacement | 29.6 | 83.41 | 88.41 | 20.0 | 77.58 | 88.50
3 |  \+ no back-/forward-translation | 29.5 | 83.20 | 87.48 | 19.5 | 77.49 | 87.50
| length encoding (tokens) |  |  |  |  |  |
4 | Row 19, Table 1 | 30.9 | 83.66 | 72.36 | 19.3 | 77.47 | 80.50
5 |  \+ no length perturbation | 28.6 | 82.32 | 76.12 | 18.3 | 74.51 | 81.00
| length encoding (characters) |  |  |  |  |  |
6 | Row 21, Table 1 | 30.7 | 83.57 | 63.64 | 20.1 | 78.27 | 73.00
7 |  \+ no length perturbation | 26.6 | 81.66 | 98.26 | 18.4 | 76.07 | 99.00
|  \+ no synonyms replacement, |  |  |  |  |  |
8 |  + no back-/forward-translation | 30.0 | 83.37 | 61.94 | 19.8 | 77.86 | 75.50
Table 2: Ablation study results. All values in %.

####  5.4.1 Effect of synthetic data

Comparison of the first two rows of Table 2 shows that taking away synthetic
data created using word synonym replacement (Section 4.1) from the 7-bin
length token system causes a slight degradation of the BLEU score and no
significant change of BERT and length compliance score on tst-COMMON. We
consistently observe the same tendencies when taking other configurations of
the 7-bin system from Table 1 as baseline (not shown here). This indicates
that synonym replacement has some positive effect on MT quality as a data
augmentation method, but fails to lead to the desired effect of improved
length compliance. This could also in part be explained by the fact that in
our experiment setting, removing synonym data resulted in the increased
relative proportion of length-compliant back- and forward-translated data.

Removing also the back- and forward-translated data from training leads to a
consistent drop in all quality metrics on tst-COMMON. In particular, length
compliance becomes worse, even in the considered case that uses the length
ROVER and N-best rescoring. When training the length-unbiased system of row 5,
Table 1 without synthetic data LC even drops from 45.27 to 30.70 (not shown in
Table 2). This shows that length-compliant back- and forward-translated data
clearly has the desired effect of learning isometric translation and it is
still noticeable when combined with other length control methods. Also for the
length encoding model (row 8) we observe a similar positive effect of the
synthetic data, despite the translation length being predominantly determined
by the length value fed into the decoder.

On the blind test set we observe contradicting results. For this we can
provide no better explanation than referring to statistical randomness. In
Table 1 one can see that ranking of independently trained neural models (e.g.
rows 1, 3, 5, 17 and 21) disagrees on the two test sets, which we attribute to
the small size of 200 lines of the blind test set. In fact, according to
paired bootstrap resampling computed with SacreBLEU Post (2018), the large
difference of 1.3 BLEU between row 1 and 2 of Table 2 is not statistically
significant with $p<0.05$, and the 95% confidence interval of row 1 is 2.8
BLEU.

####  5.4.2 Effect of length perturbation

Without length perturbation the character-level length encoding model is able
to produce length compliant translations in almost all cases, as can be seen
in Row 7 of Table 2, without the need for subsequent steps like N-best
rescoring or second-pass length correction. This however comes at the cost of
a severe drop in translation quality as measured in both BLEU and BERTScore.
When comparing to row 24 of Table 1 it is apparent that the system trained
with length perturbation and using the above-mentioned methods can achieve a
similar high level of length compliance while offering a better translation
quality by 2.6% BLEU and 1.1% BERT F1 score absolute.

A similar drop in translation quality due to lack of length perturbation can
be observed for the case of token-level length encoding comparing rows 4 and 5
of Table 2. The gain in LC from training without noise is outperformed by the
combination of N-best rescoring and second-pass length correction applied to
the baseline system (row 20, Table 1). Notably, even without noise in training
token-level length encoding does not surpass a length compliance value of 80%.
This shows that the number of subwords is not accurate enough as a measure of
length when targeting a precise character count.

##  6 Conclusion

In this paper, we described AppTek’s neural MT system with length control that
we submitted to the IWSLT 2022 Isometric Spoken Translation Evaluation. We
showed that by using length-compliant synthetic data, as well as encoding the
desired translation length in various ways, we can significantly increase the
length compliance score, while at the same time limiting the loss of
information as reflected in only slightly lower BERT scores. As one of the
best methods for real-time production settings not involving system
combination, N-best list rescoring or 2-pass search, the modified positional
encoding that counts the desired length in characters achieves the best
quality/length compliance trade-off in our experiments. We attribute this to
more fine-grained length control capabilities of this system as compared to
systems that use source-side or target-side length pseudo-tokens.

## References

  * Abadi et al. (2015) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015.  TensorFlow: Large-scale machine learning on heterogeneous systems.  Software available from tensorflow.org.
  * Akhbardeh et al. (2021) Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondřej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. 2021.  Findings of the 2021 conference on machine translation (WMT21).  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 1–88, Online. Association for Computational Linguistics.
  * Anastasopoulos et al. (2022) Antonios Anastasopoulos, Luisa Bentivogli, Marcely Z. Boito, Ondřej Bojar, Roldano Cattoni, Anna Currey, Georgiana Dinu, Kevin Duh, Maha Elbayad, Marcello Federico, Christian Federmann, Hongyu Gong, Roman Grundkiewicz, Barry Haddow, Benjamin Hsu, Dávid Javorský, Věra Kloudová, Surafel M. Lakew, Xutai Ma, Prashant Mathur, Paul McNamee, Kenton Murray, Maria Nădejde, Satoshi Nakamura, Matteo Negri, Jan Niehues, Xing Niu, Juan Pino, Elizabeth Salesky, Jiatong Shi, Sebastian Stüker, Katsuhito Sudoh, Marco Turchi, Yogesh Virkar, Alex Waibel, Changhan Wang, and Shinji Watanabe. 2022\.  FINDINGS OF THE IWSLT 2022 EVALUATION CAMPAIGN.  In _Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)_ , Dublin, Ireland. Association for Computational Linguistics.
  * Buet and Yvon (2021) François Buet and François Yvon. 2021.  Toward genre adapted closed captioning.  In _Interspeech 2021_ , pages 4403–4407. ISCA.
  * Di Gangi et al. (2019) Mattia A Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019.  Must-c: a multilingual speech translation corpus.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2012–2017.
  * Doetsch et al. (2017) Patrick Doetsch, Albert Zeyer, Paul Voigtlaender, Ilia Kulikov, Ralf Schlüter, and Hermann Ney. 2017.  Returnn: The rwth extensible training framework for universal recurrent neural networks.  In _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5345–5349. IEEE.
  * Dyer et al. (2013) Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013.  A simple, fast, and effective reparameterization of IBM model 2.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 644–648, Atlanta, Georgia. Association for Computational Linguistics.
  * Fiscus (1997) J.G. Fiscus. 1997.  A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (rover).  In _1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings_ , pages 347–354.
  * García-Martínez et al. (2016) Mercedes García-Martínez, Loïc Barrault, and Fethi Bougares. 2016\.  Factored neural machine translation architectures.  In _Proceedings of the 13th International Conference on Spoken Language Translation_ , Seattle, Washington D.C. International Workshop on Spoken Language Translation.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Lakew et al. (2021) Surafel M Lakew, Marcello Federico, Yue Wang, Cuong Hoang, Yogesh Virkar, Roberto Barra-Chicote, and Robert Enyedi. 2021.  Machine translation verbosity control for automatic dubbing.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7538–7542. IEEE.
  * Lakew et al. (2019) Surafel Melaku Lakew, Mattia Di Gangi, and Marcello Federico. 2019.  Controlling the output length of neural machine translation.  In _Proceedings of the 16th International Conference on Spoken Language Translation_ , Hong Kong. Association for Computational Linguistics.
  * Liu et al. (2020) Danni Liu, Jan Niehues, and Gerasimos Spanakis. 2020.  Adapting end-to-end speech recognition for readable subtitles.  In _Proceedings of the 17th International Conference on Spoken Language Translation_ , pages 247–256, Online. Association for Computational Linguistics.
  * Niehues (2020) Jan Niehues. 2020.  Machine translation with unsupervised length-constraints.  In _Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)_ , pages 21–35, Virtual. Association for Machine Translation in the Americas.
  * Oka et al. (2020) Yui Oka, Katsuki Chousa, Katsuhito Sudoh, and Satoshi Nakamura. 2020.  Incorporating noisy length constraints into transformer with length-aware positional encodings.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 3580–3585, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Peter et al. (2018) Jan-Thorsten Peter, Eugen Beck, and Hermann Ney. 2018.  Sisyphus, a workflow manager designed for machine translation and automatic speech recognition.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 84–89, Brussels, Belgium. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting bleu scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191.
  * Saboo and Baumann (2019) Ashutosh Saboo and Timo Baumann. 2019.  Integration of dubbing constraints into machine translation.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)_ , pages 94–101, Florence, Italy. Association for Computational Linguistics.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Takase and Okazaki (2019) Sho Takase and Naoaki Okazaki. 2019.  Positional encoding to control output sequence length.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3999–4004, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wilken and Matusov (2019) Patrick Wilken and Evgeny Matusov. 2019.  Novel applications of factored neural machine translation.  _arXiv preprint arXiv:1910.03912_.
  * Zeyer et al. (2018) Albert Zeyer, Tamer Alkhouli, and Hermann Ney. 2018.  RETURNN as a generic flexible neural toolkit with application to translation and speech recognition.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 128–133, Melbourne, Australia. Association for Computational Linguistics.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.