# Evaluating the Robustness of Discrete Prompts

Yoichi Ishibashi1 Danushka Bollegala2 Katsuhito Sudoh1 Satoshi Nakamura1
1 Nara Institute of Science and Technology 2 University of Liverpool
{ishibashi.yoichi.ir3, sudoh<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Discrete prompts have been used for fine-tuning Pre-trained Language Models
for diverse NLP tasks. In particular, automatic methods that generate discrete
prompts from a small set of training instances have reported superior
performance. However, a closer look at the learnt prompts reveals that they
contain noisy and counter-intuitive lexical constructs that would not be
encountered in manually-written prompts. This raises an important yet
understudied question regarding the _robustness_ of automatically learnt
discrete prompts when used in downstream tasks. To address this question, we
conduct a systematic study of the robustness of discrete prompts by applying
carefully designed perturbations into an application using AutoPrompt and then
measure their performance in two Natural Language Inference (NLI) datasets.
Our experimental results show that although the discrete prompt-based method
remains relatively robust against perturbations to NLI inputs, they are highly
sensitive to other types of perturbations such as shuffling and deletion of
prompt tokens. Moreover, they generalize poorly across different NLI datasets.
We hope our findings will inspire future work on robust discrete prompt
learning.111Our codes and the adversarial NLI dataset are available at
https://github.com/LivNLP/prompt-robustness

##  § 1 Introduction

Pre-trained Language Models (PLMs) have been successfully adapted to a wide
range of Natural Language Processing (NLP) tasks using _prompt-based_ learning
Radford et al. (2018, 2019); Brown et al. (2020); Petroni et al. (2019) such
as sentiment classification Gao et al. (2021), natural language inference
(NLI) Schick and Schütze (2021, 2022), relation extraction Shin et al. (2020),
cross-lingual inference Qi et al. (2022). However, manually writing prompts
that generalize well is very challenging for several reasons such as (a) it
might not always be possible to recruit domain-expert human annotators, (b)
human annotators might not be able to cover all corner cases by writing
prompts, and (c) there can be disagreements between human annotators regarding
the coverage of a particular prompt. To address these challenges, automatic
learning of discrete prompts has been proposed such as AdvTrigger Wallace et
al. (2019), AutoPrompt (AP; Shin et al., 2020), WARP Hambardzumyan et al.
(2021), and RLPrompt Deng et al. (2022).

Relation | Method | Prompt | P@1
---|---|---|---
native-language-of (P103) | Manual | The native language of [X] is [Y] | 74.54
| AP BERT | [X]PA communerug speaks proper [Y] | 84.87
| AP RoBERTa | [X]neau optionally fluent!?ẗraditional [Y] | 81.61
profession-of (P106) | Manual | [X] is a [Y] by profession | 0.73
| AP BERT | [X] supporters studied politicians musician turned [Y] | 15.83
| AP RoBERTa | [X] (), astronomers businessman·former [Y] | 19.24
music-played-by (P136) | Manual | [X] plays [Y] music | 0.7
| AP BERT | [X] freaking genre orchestra fiction acid [Y] | 59.95
| AP RoBERTa | [X] blends postwar hostage drama sax [Y] | 52.97
Table 1: Examples of prompts learnt by AP for the fact retrieval task for BERT
and RoBERTa PLMs and the human-written manual prompts. T-REx relation ids are
shown with brackets for each relation type. Precision@1 (P@1) scores are shown
when each prompt is used in fact retrieval.

Although discrete prompt learning methods have achieved good performance in
numerous downstream tasks by automatically learnt prompts, such automatic
prompts seem to be significantly different from the manually-written ones. For
example, Table 1 shows manually-written and AP-learnt prompts for fact
retrieval Petroni et al. (2019). We see that the AP-learnt prompts for BERT
Devlin et al. (2019) and RoBERTa Liu et al. (2019) outperform the manual
prompts in precision1 (P@1) scores. However, the AP-learnt prompts contain
various counter-intuitive language constructs such as punctuation (e.g. ‘(’,
‘?’, ‘!’, ‘)’), spelling errors (e.g. _commuenrug_) etc., which seem unrelated
to the target relation. Similar cases can be observed for AP-learnt prompts
for other tasks as well (see Appendix in Shin et al. (2020)). It is
unrealistic that a human annotator would be able to write such prompts even if
they were able to see the same training instances as used by automatic
methods.

Considering the fact that discrete prompt learning methods are trained in a
few-shot setting where they use only a small number of training instances, the
seemingly counter-intuitive nature of the discrete prompts learnt by automatic
methods raises concerns about their robustness. For example, _How will the
performance of a target task change if we add small random perturbations to
the prompts learnt by AP?_ and _Whether the prompts learnt by AP generalize to
out-of-domain data?_. To study these issues, in this paper we evaluate the
robustness of discrete prompts learnt by automatic prompt learning methods and
compare that with manually-written prompts and direct fine-tuning of PLMs.

An evaluation of the robustness of discrete prompts is important for two main
reasons. First, given that discrete prompt learning methods are learning those
prompts from a small set of training instances, it is important that they
cover the core patterns that generalize to the target task and not simply
capture some random artefacts in the training samples. Second, unlike
embedding-based continuous prompts Li and Liang (2021); Lester et al. (2021),
discrete prompts Wallace et al. (2019); Shin et al. (2020); Deng et al. (2022)
are represented in natural language and supposed to be interpretable. However,
if a discrete prompt learning method is less robust, a seemingly harmless
perturbation such as removing a punctuation character can significantly alter
the performance of a downstream task.

In contrast to the numerous work that has used prompts for fine-tuning PLMs,
to the best of our knowledge, the robustness of discrete prompts to random or
adversarial perturbations has not been systematically studied. To address this
gap, we use AP as a concrete example of a widely-used method and evaluate its
robustness under different types of carefully designed perturbations. However,
we note that our perturbation techniques are not limited to AP and can be used
for any discrete prompt learning method. We compare the performance of AP-
learnt prompts against fine-tuning using Manually-written Prompts (MP), and
Head-based Fine-Tuning (HFT), where we fine-tune both the classifier head and
the PLM parameters.

From our evaluation, we find several interesting facts about the robustness of
discrete prompts as summarized below.

  * •

Overall, when the number of training instances is increased, MP outperforms
both AP and HFT on CB De Marneffe et al. (2019) and MNLI Williams et al.
(2018), two independent benchmark datasets for NLI (§ 3.1). In particular,
the performance of AP on MNLI is much worse than that on CB. This is in
contrast to the superior performance of AP on SICK-E Marelli et al. (2014),
another NLI dataset, as reported by Shin et al. (2020).

  * •

Moreover, we see a performance drop when we use discrete prompts learnt from
CB for MNLI and vice-versa (§ 3.4). These results indicate that the
performance of discrete prompts learnt by AP is highly dataset-dependent and
such discrete prompts do not generalize well across datasets.

  * •

Compared to MP, AP-learnt discrete prompts turn out to be highly sensitive to
the ordering of prompt tokens (§ 3.2).

  * •

Random deletion of prompt tokens decreases performance in both AP and MP (§
3.3).

  * •

We create an adversarial NLI dataset from randomly-sampled test instances from
MNLI and CB, and manually modify the hypothesis sentences with keeping the
corresponding premise sentences unchanged, such that (a) the target label
would not change, and (b) would reverse an entailment label to a contradiction
(or vice-versa). Both AP and MP remain relatively robust against the
perturbations that do not change the target label, but the performance of MP
drops significantly in the label-changing setting (§ 3.5). This shows that AP
is relatively more robust against adversarial perturbations than MP, which
explains AP’s superior performance in various tasks.

##  § 2 Related Work

#### Prompting Methods:

Prompting or _in-context-learning_ has received wide attention as an efficient
method to extract knowledge from PLMs Brown et al. (2020); Petroni et al.
(2019); Cui et al. (2021). However, to manually write prompts one must possess
task-specific domain knowledge. As an alternative, methods that can
automatically learn prompts from training data have been proposed. Two
distinct types of prompts have been learnt in prior work: discrete prompts
(learns lexical sequences), and continuous prompts (learns embeddings).
Continuous prompts Li and Liang (2021); Lester et al. (2021) are parameter
efficient because they learn generalizable task-specific embeddings, with
performance comparable to PLM fine-tuning. However, continuous prompts cannot
be learnt when a PLM is publicly unavailable and the only access to it is via
an API Brown et al. (2020). Moreover, compared to discrete prompts, continuous
prompts are difficult to interpret. Learning discrete prompts Wallace et al.
(2019); Shin et al. (2020); Deng et al. (2022) does not suffer from these
limitations of continuous prompts and can be used with diverse NLP tasks.
Especially, fine-tuning massive PLMs has become computationally costly, which
has made discrete prompt learning an attractive alternative.

#### Analysis of Prompting Methods:

Prior work has analyzed prompts from various viewpoints. Scao and Rush (2021)
studied the effect of training dataset size on fixed-prompt PLM fine-tuning
and head-based fine-tuning and found that prompting is often worth 100s of
instances on average across classification tasks. Kavumba et al. (2022) showed
that the performance of prompt-based models varies significantly depending on
the surface cues in the sentence. Lu et al. (2022) found that ordering of task
input significantly affects the performance. Utama et al. (2021) focused on
the reliance on lexical overlap in sentence pair classification and showed
that prompt-based models fail to make predictions dependent on the lexical
overlap. To the best of our knowledge, the robustness of discrete prompts
under different types of perturbations has not been studied in prior work,
which is the main focus of this paper.

##  § 3 Experiments

Let us first describe experimental settings common to all experiments.

#### Prompting and Fine-Tuning Methods:

We compared the following methods.

  * •

AutoPrompt (AP; Shin et al., 2020) is a representative method of discrete
prompt learning. The learning strategy is based on fill-in-the-blank task
Devlin et al. (2019). First, a manually created prompt template (e.g., [X]
<MASK> <T> ... <T> [Y]) is given, and a prompt token (called a trigger token)
is learnt by replacing <T>, which is a special token representing a trigger
token. In the search for trigger tokens, the probability of <MASK> is
converted into class probability by using label tokens (e.g., {‘ _nobody_ ’, ‘
_nor_ ’} for contradiction Shin et al. (2020)), and trigger tokens are
searched by gradient-guided search Wallace et al. (2019) to find a candidate
set consisting of trigger tokens from a vocabulary of the language model. As a
template for NLI, we used the one given by Shin et al. (2020), and the prompt
tokens were learnt from the training dataset. In our experiments, we used the
official implementation.222https://github.com/ucinlp/autoprompt

  * •

Manually-written Prompts (MP; Schick and Schütze, 2021) is a method for fine-tuning the entire masked language model with training data using manually-written prompts as the input and predicting the <MASK> tokens for the labels (e.g., ‘ _yes_ ’ for entailment). We used the template {hypothesis}? | <MASK>, {premise} and verbalizer (‘ _yes_ ’ for entailment, ‘ _no_ ’ for contradiction, ‘ _maybe_ ’ for neutral) following prior work Schick and Schütze (2021); Scao and Rush (2021). Schick and Schütze (2021) proposed an ensemble-based method with multiple rounds of fine-tuning using different templates. However, because a single template is used in AP, for a fair comparison in our experiments, we fine-tuned a PLM using one MP template.

  * •

Head-based Fine-Tuning (HFT; Devlin et al., 2019) fine-tunes the PLM with a
classifier head. We report the head-based results trained by Scao and Rush
(2021). They trained HFT with a low learning rate ($10^{-5}$) and always with
a large number of steps (at least 250), following the recommendations in prior
work Mosbach et al. (2021); Zhang et al. (2021). Note that HFT is not a
prompt-based method, so it was excluded from some experiments on the
robustness of discrete prompts.

#### Datasets:

We used NLI as an evaluation task to compare the robustness of discrete
prompting methods. The NLI task has been used in multiple previous studies to
evaluate and/or propose novel prompt learning methods because it is a
fundamental task related to many NLP applications Shin et al. (2020); Scao and
Rush (2021); Webson and Pavlick (2022). It is important to use the same NLI
task and datasets in our experiments to facilitate fair comparisons and reach
reproducible conclusions. We used the two datasets: CommitmentBank (CB; De
Marneffe et al., 2019)333https://super.gluebenchmark.com/tasks (a corpus of
short texts), and Multi-Genre Natural Language Inference Corpus (MNLI;
Williams et al., 2018)444https://cims.nyu.edu/~sbowman/multinli/ (a
crowdsourced collection of sentence pairs for NLI). Each sentence pair is
labelled with _entailment_ , _neutral_ , or _contradiction_.

#### PLM:

In our experiments, we used the same pre-trained language model to evaluate
AP, MP, and HFT equally. Specifically, we used RoBERTa-large (355M parameters)
555https://huggingface.co/roberta-large Liu et al. (2019), which has been used
in much prior work in prompt learning Shin et al. (2020); Scao and Rush
(2021). The PLM was trained on five datasets, including
BookCorpus666https://yknzhu.wixsite.com/mbweb, English
Wikipedia777https://en.wikipedia.org/wiki/English_Wikipedia, CC-
News888https://commoncrawl.org/2016/10/news-dataset-available/,
OpenWebText999https://github.com/jcpeterson/openwebtext, and
Stories101010https://arxiv.org/abs/1806.02847. The texts were tokenised using
a byte-level Byte-Pair Encoding (BPE; Sennrich et al., 2016) vocabulary of
size 50,000.

#### Evaluating the Robustness of Prompts:

We used _rate of degradation_ (RoD) Meyers et al. (2020) to evaluate
robustness, which is defined as the decrease in accuracy of the target task
due to the perturbations added to the prompt. If the RoD of a model is small
after the inclusion of a perturbation, the model is considered to be robust
against that perturbation. Specifically, we first calculate the respective
accuracies $\textrm{acc}_{x}$ and $\textrm{acc}_{x^{\ast}}$ on the same
evaluation set for both prompt $x$ and its perturbated version $x^{\ast}$.
Using the average accuracies $\textrm{avg-acc}_{x}$ and $\textrm{avg-
acc}_{x^{\ast}}$ over $M$ prompts ${x_{1},...,x_{M}}$, we calculate the RoD as
$(\textrm{avg-acc}_{x}-\textrm{avg-acc}_{x^{\ast}})/\textrm{avg-
acc}_{x}=1-\textrm{avg-acc}_{x^{\ast}}/\textrm{avg-acc}_{x}$.

###  § 3.1 Effect of the Training Dataset Size

Before moving on to robustness experiments, we first investigate the number of
training instances on which AP and MP perform best, and used the best-
performing AP and MP to evaluate their robustness in the subsequent
experiments.

#### Experimental Settings:

We gradually increased the size of the training dataset following the
experimental setup of Scao and Rush (2021). Specifically, we experimented with
randomly sampled subsets of the training dataset having varying numbers of
instances in $\\{10,15,20,30,50,70,100,150,200\\}$. Because the performance of
few-shot learning methods often varies due to the high feature variance in the
training data, we randomly sampled four subsets per each dataset size and used
them independently for training the models111111NVIDIA RTX A5000 was mainly
used. (i.e. trigger tokens and label tokens for AP, or fine-tuned language
model for MP and HFT) for each subset and report the average accuracy on the
validation data for the four models ($M=4$). We used the matched (example from
the same source as the training set) validation set for MNLI. For CB, we held
out 50 training instances for development as in Scao and Rush (2021) and
evaluated the original validation set as test data.

We searched for the optimal values for the following hyperparameters: the
number of trigger tokens in {3, 5, 10}, the number of label tokens in {3, 5,
10}, and the number of tokens in a candidate set in {10, 50}. We evaluated the
test accuracy using the hyperparameters that had the highest accuracy on the
validation data for each dataset size. In the training of MP, we used AdamW
optimizer Loshchilov and Hutter (2019) with an initial learning rate of
$10^{-5}$ and a learning step of 1,000 following Mosbach et al. (2021).

Method | #Train | Template | #Prompt tokens | #Label tokens per class | Avg. accuracy
---|---|---|---|---|---
|  |  |  |  | CB | MNLI
AP | 200 |  p <MASK> <T> ... <T> h | 10 | 3 | 68.3 | 37.7
MP | 200 |  h? | <MASK>, p | 3 | 1 | 95.1 | 65.5
HFT | - | <CLS> p <SEP> h | 0 | - | - | -
Table 2:  The average accuracy of the experiment with four training subsets of
200 instances. Red represents the task inputs, h represents the hypothesis, p
represents the premise, blue represents the prompt tokens, and <T> represents
a trigger token. Unreported values were marked with ‘-’.

#### Main Results:

Figure 1 shows the performance121212HFT results were obtained from Scao and
Rush (2021), F1-macro for CB and accuracy for MNLI. against the training
dataset size. We see that in both CB and MNLI MP is always superior to AP. For
example, with a dataset of size 200, AP and MP achieved the best accuracy in
CB, MP’s accuracy was 92.7%, while that of AP was lower at 54.2%.

Figure 1:  Performance of AutoPrompt (AP), Manually-written Prompt (MP), and
Head-based Fine-Tuning (HFT) on the scale of dataset size for CB and MNLI.
Means and their 95% confidence intervals are plotted. The accuracy of HFT for
dataset size for CB was not plotted because the accuracy was not reported.

Our results also suggest that the performance of discrete prompts learnt by AP
is highly dataset dependent. Shin et al. (2020) reported results for AP and
HFT on SICK-E Marelli et al. (2014), which is another NLI dataset. They
concluded that AP was always superior to HFT up to training dataset sizes of
1,000 for the same RoBERTa-large PLM that we use. However, our experiments
show the opposite trend (i.e. HFT is superior to AP). This suggests that even
if AP is superior to HFT on a given dataset, it is not guaranteed to be
superior in a different dataset for the same task. This may be due to the
differences in the domain and annotation guidelines for each dataset. For
example, the accuracy of MNLI was quite low on AP, which contrasts with that
of CB. This result suggests that the discrepancies in domains and annotation
guidelines make it difficult for AP to perform consistently.

#### Best Prompts:

Table 2 shows the average accuracy of models trained on 200 instances that
performed well in both CB and MNLI. Note that there are four training subsets
for each dataset size, resulting in corresponding four trained AP prompts and
four PLMs fine-tuned by MP. 131313We show the four best prompts learnt by AP
in Appendix A. In the robustness evaluations in § 3.2 through § 3.5, we used
these learnt APs and MPs. In this paper, (a) trigger tokens learnt by AP, and
(b) manually-written prompts excluding the task inputs and mask tokens are
collectively referred to as the _prompt tokens_.

###  § 3.2 Token Reordering

As seen from Table 1, compared to MPs where the ordering of tokens in a prompt
is manually determined, discrete prompts learnt by AP appear to have no
obvious ordering among their tokens. To empirically investigate the importance
of the token order in a discrete prompt, we conduct an experiment where we
randomly shuffle the prompt tokens and measure the effect on the downstream
task performance.

#### Experimental Procedure:

Given a discrete prompt, we first randomly reordered its prompt tokens (e.g.
shaded in blue in Table 2). Next, we used the reordered prompt with the PLM to
make entailment predictions for the test instances in the CB and MNLI
datasets. Finally, the entailment prediction accuracy (Acc) obtained with the
reordered prompts was computed. We repeated this evaluation 10 times for each
prompt and report the averaged values and the corresponding RoD values.

#### Main Results:

From Table 3 we see that the accuracy drops for both AP and MP when the prompt
tokens are randomly reordered. In particular, the accuracy of AP drops
significantly compared to that of MP. For example, the accuracy of AP on CB
drops by ca. 14% due to token reordering, while that for MP drops only by ca.
2%. Intuitively, one would expect that changing the order of prompt tokens in
MP would result in a significant drop in accuracy because the meaning of the
prompts would change. However, we see that this is not the case. This result
shows that the discrete prompts learnt by AP strongly rely on the token order.

#### Additional Analysis:

To further analyze the relationship between the level of perturbation
introduced by reordering prompt tokens in AP and its effect on the
performance, we computed the token-level edit distance (Levenshtein distance;
Levenshtein et al., 1966) between each prompt and its token-shuffled version
as shown in Figure 2. For all four AP prompts, we see that the accuracy drops
when the perturbation noise (i.e. measured by edit distance) increases. This
reconfirms the lack of robustness in discrete prompts learnt by AP to the
random shuffling of prompt tokens.

Method | Metrics | CB | MNLI
---|---|---|---
AP | Acc | 54.2 | 34.3
RoD | 0.21 | 0.10
MP | Acc | 92.7 | 59.3
RoD | 0.03 | 0.09
Table 3:  Performance of reordered prompts. Acc denotes accuracy; RoD denotes
the RoD from before the reordering (Table 2). The largest drops in accuracy
are bolded and the smallest drops are underlined for each method and dataset.
AP relies more strongly on word order than MP.

###  § 3.3 Token Deletion

As seen from Table 1, the discrete prompts learnt by AP perform better than
MP. However, it is often difficult to determine the importance of prompt
tokens to the target task due to their lack of interpretability (e.g. prompt
token ‘ _neau_ ’ in Table 1). To understand the significance of individual
prompt tokens to the overall discrete prompt, we conducted an experiment where
we systematically deleted one or more prompt tokens at various positions from
a given discrete prompt and measure the drop (if any) in the performance of
the NLI task.

Figure 2: Edit distance and accuracy of the reordered trigger tokens. We
evaluated them on the validation data of CB. The prompts numbered 0 through 3
each represent the four prompts learnt by AP (shown in Table 9). Note that a
point with an edit distance of zero indicates accuracy with the original
trigger token.

#### Experimental Procedure:

We evaluated two settings of prompt deletion: _single_ and _multiple_ token
deletion. In the single token deletion setting, we deleted one token at
different positions in a given prompt. For AP, we repeated this with each of
the four discrete prompts (shown in Table 2) and report the average accuracy.
In the multiple token deletion setting, we delete $n\in\\{1,3,5,7\\}$ prompt
tokens following three strategies: _Random-deletion_ deletes $n$ prompt tokens
randomly, _Front-deletion_ deletes $n$ consecutive prompt tokens from the
beginning of the prompt, and _Back-deletion_ deletes $n$ tokens counted
backwards from the end of the prompt. In random-deletion, we ran 100 trials
and report the average accuracy. As in the previous experiments, we used four
prompts for AP and report the averaged results.

Task | Method | Metrics | Position of the deleted prompt token | Orig.
---|---|---|---|---
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
CB | AP | Acc | 62.1 | 61.6 | 63.4 | 59.4 | 65.6 | 65.6 | 62.1 | 63.8 | 62.1 | 62.9 | 68.3
RoD | 0.09 | 0.10 | 0.07 | 0.13 | 0.04 | 0.04 | 0.09 | 0.07 | 0.09 | 0.08 | -
MP | Acc | 93.8 | 93.3 | 96.0 | - | - | - | - | - | - | - | 95.1
RoD | 0.01 | 0.02 | -0.01 | - | - | - | - | - | - | - | -
MNLI | AP | Acc | 37.9 | 37.8 | 36.6 | 37.5 | 37.5 | 37.2 | 37.5 | 37.4 | 37.5 | 37.1 | 37.7
RoD | -0.01 | 0.00 | 0.03 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01 | 0.02 | -
MP | Acc | 64.5 | 65.4 | 55.4 | - | - | - | - | - | - | - | 65.5
RoD | 0.02 | 0.00 | 0.15 | - | - | - | - | - | - | - | -
Table 4:  Average accuracy was obtained after deleting a single token at
different positions of a given prompt. The largest drops in accuracy over the
deletion positions are bolded and the smallest drops are underlined for each
method and dataset. Column ‘Orig.’ shows the performance of the original
prompt.

#### Results:

From Table 4 we see that the accuracy of both AP and MP drops even when a
single token is deleted at specific positions. However, the observed trends
differ in CB and MNLI. For example, AP resulted in higher RoD values in CB
compared to MNLI. This shows that the robustness of AP under single token
deletion heavily depends on the dataset. Table 5 shows the results for the
multiple token deletion setting. We see that the performance of both AP and MP
degrades when more tokens are deleted. Interestingly, the accuracy drop in CB
is very small for MP even when all prompt tokens are deleted (i.e., only the
task inputs and <MASK> were used as the input). This suggests that the
performance on CB is less reliant on the prompt tokens in MP.

###  § 3.4 Cross-Dataset Evaluation

Given that discrete prompt learning methods such as AP learn prompts from a
small set of training instances, it is important that the learnt prompts
encode generalizable task-specific features and not random artefacts in the
training sample used. To study the transferability of the learnt discrete
prompts from one dataset to another, we conduct a cross-dataset evaluation as
described next.

Strategy | Method | Metrics | #Deleted Tokens | Orig.
---|---|---|---|---
1 | 3 | 5 | 7
CB
Random | AP | Acc | 56.7 | 56.0 | 55.4 | 54.8 | 68.3
RoD | 0.17 | 0.18 | 0.19 | 0.20 | -
MP | Acc | 93.3 | 94.6 | - | - | 95.1
RoD | 0.02 | 0.01 | - | - | -
Front | AP | Acc | 62.1 | 49.1 | 57.6 | 57.6 | 68.3
RoD | 0.09 | 0.28 | 0.16 | 0.16 | -
MP | Acc | 93.8 | 94.6 | - | - | 95.1
RoD | 0.01 | 0.01 | - | - | -
Back | AP | Acc | 62.9 | 57.6 | 55.8 | 51.3 | 68.3
RoD | 0.08 | 0.16 | 0.18 | 0.25 | -
MP | Acc | 96.0 | 94.6 | - | - | 95.1
RoD | -0.01 | 0.01 | - | - | -
MNLI
Random | AP | Acc | 35.8 | 35.8 | 36.0 | 36.2 | 37.7
RoD | 0.05 | 0.05 | 0.05 | 0.04 | -
MP | Acc | 65.4 | 52.6 | - | - | 65.5
RoD | 0.0 | 0.20 | - | - | -
Front | AP | Acc | 37.9 | 36.5 | 36.2 | 36.0 | 37.7
RoD | -0.01 | 0.03 | 0.04 | 0.05 | -
MP | Acc | 64.5 | 52.6 | - | - | 65.5
RoD | 0.02 | 0.20 | - | - | -
Back | AP | Acc | 37.1 | 36.7 | 35.7 | 36.5 | 37.7
RoD | 0.02 | 0.03 | 0.05 | 0.03 | -
MP | Acc | 55.4 | 52.6 | - | - | 65.5
RoD | 0.15 | 0.20 | - | - | -
Table 5:  Average accuracy was obtained after deleting multiple tokens from a
given prompt. The largest drops in accuracy over the deleted tokens are bolded
and the smallest drops are underlined for each strategy and method.

#### Experimental Procedure:

We used one NLI dataset (e.g. CB) to learn the prompts and then use them to
make entailment predictions in another NLI dataset (e.g. MNLI). We then
measured the drop in accuracy using RoD for this cross-dataset transferability
task with respect to the accuracy of test data from the same dataset.

#### Results:

As seen from Table 6, AP-based prompts do not generalize well across datasets.
For both AP and MP, RoD is larger in the transfer from CB to MNLI than in the
opposite direction. This implies that MNLI is a better dataset for fine-tuning
a PLM for NLI using discrete prompts.

Method | Test Dataset | RoD
---|---|---
CB | MNLI
AP trained on CB | 68.3 | 36.1 | 0.47
AP trained on MNLI | 42.9 | 37.7 | 0.12
MP trained on CB | 95.1 | 43.4 | 0.54
MP trained on MNLI | 43.8 | 65.5 | 0.33
Table 6:  Accuracy and RoD for the cross-dataset evaluation where a method
(AP/MP) is trained on one NLI dataset (CB/MNLI) and the learnt prompts are
used to make entailment predictions in a different NLI dataset.

###  § 3.5 Adversarial Perturbations

Introducing carefully designed adversarial perturbations to the test instances
such as modifications to sentences that might or might not alter the original
target labels have been used as a technique for probing the robustness of
models Goodfellow et al. (2015). Previous studies Samanta and Mehta (2017);
Jin et al. (2020) have shown that pre-trained models can be easily fooled to
make incorrect predictions with seemingly innocuous perturbations to the test
instances. Therefore, we evaluate discrete prompt-based NLI models for their
robustness against adversarially perturbated test instances.

#### Evaluation Dataset:

For this purpose, we asked two annotators to manually edit hypothesis
sentences in NLI test data considering two types of perturbations: (1)
perturbations that do not change reference labels, and (2) perturbations that
change reference labels. An example is shown in Table 7.

For the first type of perturbation, we edited a hypothesis sentence such that
its relationship with the corresponding premise remains unchanged. For the
second type, we edited a hypothesis sentence such that its relationship (e.g.,
from _entailment_ to _contradiction_) will be reversed. The premise and
hypothesis pairs were sampled from CB (validation set) and MNLI (test set).
Because there are ca. 10,000 test instances in MNLI and it is costly to
manually edit sentences, we used 100 randomly-chosen sentence pairs covering
MNLI and CB.

|  Hypothesis |  Label
---|---|---
Original |  The Wither’s only had daughters. |  contradiction
Perturbation |  |
w/o label changes |  The Wither’s did not have sons. |  contradiction
w/ label changes |  The Wither’s had a boy. |  entailment
Table 7:  Examples of our evaluation set consisting of task inputs with
perturbations. The premise sentence is ‘ _The Wither’s eldest boy, one of the
four of the town militia, saluted in the old style with his stick sword._ ’

#### Experimental Procedure:

We computed the RoD of average accuracies obtained with original and
adversarial test instances. Specifically, we used the AP prompts in Table 2
under three settings: (a) original (without perturbations), (b) perturbations
without label changes, and (c) perturbations with label changes. Then, we
calculate RoD from (a) to (b) and (a) to (c) as shown in Table 8.

Perturbation | Method | Metrics | CB | MNLI
---|---|---|---|---
Original | AP | Acc | 54.5 | 40.5
RoD | - | -
MP | Acc | 95.5 | 71.0
RoD | - | -
Perturbation w/o label changes | AP | Acc | 55.5 | 43.2
RoD | -0.02 | -0.07
MP | Acc | 93.0 | 66.7
RoD | 0.03 | 0.06
Perturbation w/ label changes | AP | Acc | 42.3 | 39.4
RoD | 0.22 | 0.03
MP | Acc | 41.8 | 61.2
RoD | 0.56 | 0.14
Table 8:  Accuracy and RoD in prompts for task inputs that include
perturbations. The RoD here is the rate of degradation in the average accuracy
from the original without perturbations to perturbations without label changes
or perturbations with label changes. The largest drops in accuracy are bolded
and the smallest drops are underlined for each perturbation and method.

#### Results:

Overall, we see that the RoD of AP is consistently smaller than that of MP in
both CB and MNLI under both types of perturbations. However, it is also clear
that the accuracy obtained with AP is much smaller than that with MP. For the
perturbations without label changes, both AP and MP show small RoD values,
compared to those with label changes.141414w/o label change modifications
slightly increase the average length of a hypothesis and AP seems to better
exploit this extra information for inference resulting in a slight improvement
in accuracy (negative RoD). This shows that both AP and MP are relatively
robust against modifications to the hypotheses that do not significantly alter
the meaning. However, when stronger perturbations are introduced that would
result in label changes, the accuracy of both AP and MP drops significantly.
151515MP is less robust compared to AP, likely as a result of overfitting to
strongly perturbed training data during fine-tuning the PLM. This is a concern
because it shows that neither AP nor MP is sufficiently robust to correctly
predict the target labels when the hypothesis sentences in test data are
adversarially modified.

##  § 4 Conclusion

We investigated the robustness of discrete prompts under different
perturbations. We found that although discrete prompts remain relatively
robust against token deletion, it is highly sensitive to other types of
perturbations such as token shuffling. For adversarial perturbations to the
input, discrete prompts were robust to weak perturbations without label
changes, but AP was more robust than MP for perturbations with label changes.
Moreover, they generalize poorly across different datasets annotated for NLI.
We hope our analysis will inspire future work to develop methods that learn
both accurate as well as robust discrete prompts.

##  § 5 Limitations

Possible limitations of this work are:

  * •

We chose popular discrete prompt methods of AP and MP and did not investigate
other methods in this work. Our analysis procedure can still be applied to
other discrete prompts such as AvgTrigger Wallace et al. (2019).

  * •

We chose RoBERTa-large following previous studies of HFT Scao and Rush (2021)
and AP Shin et al. (2020) for reproducible and identical comparisons with
them. Other PLMs would lead to different results, but they can also be
investigated in the same way as in this work.

  * •

This work focuses on NLI because it is a fundamental natural language
understanding task and still difficult even with PLMs Brown et al. (2020).
Other complex downstream tasks are worth investigating for a deeper
understanding of prompt-based approaches in future work.

  * •

The results and conclusions are from the English datasets and would differ in
other languages. However, our methodologies do not depend on English and can
be applied to other languages as important future studies.

  * •

Since there was a performance gap between MP/HFT and AP, the accuracies by the
perturbations could be affected. However, this work does not aim to find the
best prompt learning method but to analyze the robustness of discrete prompts
for a deeper understanding of them.

##  § 6 Ethical Considerations

Our adversarial dataset came from existing datasets of CB and MNLI. We
visually checked the instances in the data development and found no instances
with ethical concerns.

One should also be aware of social biases (e.g. gender stereotypes) in PLM.
RoBERTa, the PLM we used in our experiments, is known to have gender biases
Sharma et al. (2021). Since we used it as-is in order to follow the
experimental conditions of previous studies using RoBERTa, our current results
are possibly influenced by such biases. However, the consideration of the
prompt robustness of this work would not pose or magnify such ethical
concerns.

##  § 7 Acknowledgments

This research was supported by the JSPS KAKENHI (Grants-in-Aid for Scientific
Research) JP22H03654.

## References

  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners.
  * Cui et al. (2021) Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. 2021.  Template-based named entity recognition using BART.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 1835–1845, Online. Association for Computational Linguistics.
  * De Marneffe et al. (2019) Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019.  The commitmentbank: Investigating projection in naturally occurring discourse.  In _proceedings of Sinn und Bedeutung_ , volume 23.
  * Deng et al. (2022) Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu. 2022.  Rlprompt: Optimizing discrete text prompts with reinforcement learning.  _CoRR_ , abs/2205.12548.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021.  Making pre-trained language models better few-shot learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3816–3830, Online. Association for Computational Linguistics.
  * Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015.  Explaining and harnessing adversarial examples.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Hambardzumyan et al. (2021) Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021.  WARP: Word-level Adversarial ReProgramming.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4921–4933, Online. Association for Computational Linguistics.
  * Jin et al. (2020) Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020.  Is BERT really robust? A strong baseline for natural language attack on text classification and entailment.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 8018–8025. AAAI Press.
  * Kavumba et al. (2022) Pride Kavumba, Ryo Takahashi, and Yusuke Oda. 2022.  Are prompt-based models clueless? In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2333–2352, Dublin, Ireland. Association for Computational Linguistics.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Levenshtein et al. (1966) Vladimir I Levenshtein et al. 1966.  Binary codes capable of correcting deletions, insertions, and reversals.  In _Soviet physics doklady_ , volume 10, pages 707–710. Soviet Union.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022\.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8086–8098, Dublin, Ireland. Association for Computational Linguistics.
  * Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014.  A SICK cure for the evaluation of compositional distributional semantic models.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26-31, 2014_ , pages 216–223. European Language Resources Association (ELRA).
  * Meyers et al. (2020) Bennet Meyers, Michael Deceglie, Chris Deline, and Dirk Jordan. 2020.  Signal processing on pv time-series data: Robust degradation analysis without physical models.  _IEEE Journal of Photovoltaics_ , 10(2):546–553.
  * Mosbach et al. (2021) Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021.  On the stability of fine-tuning BERT: misconceptions, explanations, and strong baselines.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Qi et al. (2022) Kunxun Qi, Hai Wan, Jianfeng Du, and Haolan Chen. 2022.  Enhancing cross-lingual natural language inference by prompt-learning from cross-lingual templates.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1910–1923, Dublin, Ireland. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffery Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Radford et al. (2018) Alex Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Samanta and Mehta (2017) Suranjana Samanta and Sameep Mehta. 2017.  Towards crafting text adversarial samples.  _CoRR_ , abs/1707.02812.
  * Scao and Rush (2021) Teven Le Scao and Alexander M. Rush. 2021.  How many data points is a prompt worth? In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 2627–2636. Association for Computational Linguistics.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 255–269, Online. Association for Computational Linguistics.
  * Schick and Schütze (2022) Timo Schick and Hinrich Schütze. 2022.  True few-shot learning with Prompts—A real-world perspective.  _Transactions of the Association for Computational Linguistics_ , 10:716–731.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_. The Association for Computer Linguistics.
  * Sharma et al. (2021) Shanya Sharma, Manan Dey, and Koustuv Sinha. 2021.  Evaluating gender bias in natural language inference.  _CoRR_ , abs/2105.05541.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020.  AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4222–4235, Online. Association for Computational Linguistics.
  * Utama et al. (2021) Prasetya Utama, Nafise Sadat Moosavi, Victor Sanh, and Iryna Gurevych. 2021.  Avoiding inference heuristics in few-shot prompt-based finetuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9063–9074, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019.  Universal adversarial triggers for attacking and analyzing NLP.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2153–2162, Hong Kong, China. Association for Computational Linguistics.
  * Webson and Pavlick (2022) Albert Webson and Ellie Pavlick. 2022.  Do prompt-based models really understand the meaning of their prompts? In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ , pages 2300–2344. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Zhang et al. (2021) Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav Artzi. 2021\.  Revisiting few-sample BERT fine-tuning.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.

##  Appendix A Supplementary Materials/Appendix

Table 9 shows the best prompts learnt by AP, which are used in the robustness
evaluations.

Prompt ID |  Prompt learnt by AP | Label tokens | Accuracy
---|---|---|---
0 |  p <MASK> strikers <MASK> <MASK> Ever Want å£« Console Encyclopedia Sie ANC h |  |  entailment: 1927, 1897, 1904
---
contradiction: personally, skeptics, squarely
neutral: æµ, ä¸Ĭ, ä¹
69.64
1 |  p <MASK> diagnoses undert fueling Hist setups prev bound advertisers paper records h |  |  entailment: 1930, 1830, 1890
---
contradiction: contradict, straight, favors
neutral: à¨, annabin, kb
75.00
2 |  p <MASK> maximize useful courts <MASK> malink rooms Scrib home interested Service h |  |  entailment: 4000, 1830, THEN
---
contradiction: yet, preferring, Ps
neutral: ĭ, Username, ãĥ«
57.14
3 |  p <MASK> fever <MASK> <MASK> EL <MASK> <MASK> <MASK> ARE ENE cue h |  |  entailment: 1890, 1886, 1889
---
contradiction: yet, endorsing, contradict
neutral: ctory, boolean, Boolean
71.43
Table 9:  Four prompts learnt by AP in CB. Red represents the task inputs, h
represents the hypothesis, p represents the premise, blue represents the
prompt tokens (trigger tokens). <MASK> tokens in the trigger tokens of some
prompts are those used to initialize trigger tokens.


# DeMPT: Decoding-enhanced Multi-phase Prompt Tuning for Making LLMs Be Better
Context-aware Translators

Xinglin Lyu♣, Junhui Li♣, Yanqing Zhao♠, Min Zhang♠, Daimeng Wei♠,
Shimin Tao♠, Hao Yang♠, Min Zhang♣
♣School of Computer Science and Technology, Soochow University, Suzhou, China
♠Huawei Translation Services Center, Beijing, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>Corresponding author: Junhui Li.

###### Abstract

Generally, the decoder-only large language models (LLMs) are adapted to
context-aware neural machine translation (NMT) in a concatenating way, where
LLMs take the concatenation of the source sentence (i.e., intra-sentence
context) and the inter-sentence context as the input, and then to generate the
target tokens sequentially. This adaptation strategy, i.e., concatenation
mode, considers intra-sentence and inter-sentence contexts with the same
priority, despite an apparent difference between the two kinds of contexts. In
this paper, we propose an alternative adaptation approach, named Decoding-
enhanced Multi-phase Prompt Tuning (DeMPT), to make LLMs discriminately model
and utilize the inter- and intra-sentence context and more effectively adapt
LLMs to context-aware NMT. First, DeMPT divides the context-aware NMT process
into three separate phases. During each phase, different continuous prompts
are introduced to make LLMs discriminately model various information. Second,
DeMPT employs a heuristic way to further discriminately enhance the
utilization of the source-side inter- and intra-sentence information at the
final decoding phase. Experiments show that our approach significantly
outperforms the concatenation method, and further improves the performance of
LLMs in discourse modeling.111As this paper is under review, we will release
our code and datasets later.

DeMPT: Decoding-enhanced Multi-phase Prompt Tuning for Making LLMs Be Better
Context-aware Translators

Xinglin Lyu♣, Junhui Li♣††thanks:  Corresponding author: Junhui Li., Yanqing
Zhao♠, Min Zhang♠, Daimeng Wei♠, Shimin Tao♠, Hao Yang♠, Min Zhang♣ ♣School of
Computer Science and Technology, Soochow University, Suzhou, China ♠Huawei
Translation Services Center, Beijing, China<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Context-aware neural machine translation (NMT) goes beyond sentence-level NMT
by incorporating inter-sentence context at the document level Zhang et al.
(2018); Miculicich et al. (2018); Voita et al. (2018, 2019b, 2019a); Bao et
al. (2021); Sun et al. (2022), aiming to address discourse-related challenges
such as zero pronoun translation Wang et al. (2019), lexical translation
consistency Lyu et al. (2021, 2022), and discourse structure Hu and Wan
(2023). A recent paradigm shift has been witnessed in context-aware NMT with
the emergence of the decoder-only large language models (LLMs) BigScience
(2022); Google (2022); MetaAI (2023b, a); OpenAI (2023). These generative
language models, trained on extensive public data, have gained significant
attention in the natural language processing (NLP) community. In adapting LLMs
to context-aware NMT, a common strategy involves concatenating multiple source
sentences as a prefix and generating translations token-by-token, relying on
the prefix and previously predicted target tokens, as shown in Figure 1 (a).
However, a critical observation of this strategy reveals a potential drawback
– the equal prioritization of the inter- and intra-sentence contexts during
token generation. Importantly, the intra-sentence context inherently contains
richer parallel semantic information with the target sentence and should be
given a higher priority than the inter-sentence context. Consequently, we
propose that separately modeling and utilizing the inter- and intra-sentence
contexts should explicitly inform LLMs of the document-level context and the
current sentence itself, thus being able to prevent the misallocation of
attention weights to source-side tokens Bao et al. (2021); Li et al. (2023).
Inspired by the success of prompt tuning Li and Liang (2021); Liu et al.
(2022); Tan et al. (2022), our alternative approach, named Decoding-Enhanced
Multi-phase Prompt Tuning (DeMPT), aims to enhance LLMs’ adaptability to
context-aware NMT, as shown in Figure 1 (b).

Figure 1: Comparison of different strategies for adapting LLMs to context-
aware NMT. The concatenation strategy (left) treats inter-sentence and intra-
sentence (referred to as the "source sentence" context in the figure) with
equal importance. In contrast, our approach (right) divides context-aware NMT
into three distinct phases, enabling LLMs to selectively model and leverage
both inter- and intra-sentence contexts.

Specifically, we divide the whole procedure of context-aware NMT into three
phases: inter-sentence context encoding, intra-sentence context encoding, and
decoding. Following Li and Liang (2021); Liu et al. (2022), we sequentially
and differentially adapt LLMs for each phase, utilizing phase-specific
trainable prompts. This phased tuning method enables LLMs to independently
capture and model both inter- and intra-sentence contexts, facilitating a
better understanding of their differences. Importantly, our approach only
divides the original input into three parts without significantly increasing
computational load. As a result, there is no substantial decrease in inference
speed compared to the concatenating method, as detailed in Section 4.3.

Furthermore, during the decoding phase, we propose a heuristic method to
emphasize the difference between inter- and intra-sentence contexts, and avoid
long-distance issue when utilizing inter-sentence context. Specifically, at
each decoding step, we use LLMs to predict the next token three times. The
decoding states used for each prediction directly concatenate with the
representations of two contexts in a discriminative manner. Finally, we
combine three probability distributions to search for the next token as the
output from the target vocabulary. This method enables LLMs to learn not only
to properly capture inter-sentence context in addressing discourse-related
issues but also to recognize a difference between inter- and intra-sentence
contexts, allowing for effective utilization of both types of contexts.

In summary, our contributions can be outlined as follows:

  * •

We propose a novel multi-phase prompt tuning approach to divide context-aware
NMT into three phases, making LLMs aware of the distinction between inter- and
intra-sentence contexts.

  * •

We introduce a enhanced decoding method that discriminately utilize both
context types. This allows LLMs not only properly capture inter-sentence
context in addressing discourse-related issues, but also be aware of the
importance of the intra-sentence context.

  * •

We validate our approach using llama-2-7b and bloomz-7b1-mt as foundation
models, demonstrating its effectiveness across five context-aware translation
directions. Extensive analyses further highlight the substantial enhancement
in LLMs’ ability for context-aware NMT.

##  2 Methodology

In this section, we describe our decoding-enhanced multi-phase approach for
adapting LLMs to context-aware NMT in details. Specifically, we break down the
whole procedure of context-aware NMT into three phases (Section 2.1), i.e.,
inter-sentence context encoding, intra-sentence encoding, and decoding.
Additionally, we discriminatively enhance the utilization of inter- and intra-
sentence contexts during the decoding phase (Section 2.2). Finally, we
describe our phase-aware prompts and training objective in Section 2.3 and
Section 2.4, respectively.

For a given document pair $(\mathcal{S},\mathcal{T})$ with $K$ sentences, we
will construct $K$ training instances. Each training instance is denoted as a
tuple $(\mathcal{C},S,T)$. Here $S=x|_{1}^{|S|}$ represents $k$-th current
source sentence with $|S|$ tokens, i.e., intra-sentence context, and
$T=y|_{1}^{|T|}$ is the $k$-th target sentence with $|T|$ tokens.
$\mathcal{C}$ denotes the $z$ previous sentences of $S$, i.e., the inter-
sentence context of $S$. We denote the hidden size of the LLM as $d$, and $L$
as the number of transformer layers within it.

###  2.1 Multi-phase Encoding and Decoding

We implement our approach based on deep prompt tuning Li and Liang (2021); Liu
et al. (2022). Next, we use training instance $(\mathcal{C},S,T)$ as an
example to describe the multi-phase approach. Figure 2 illustrates the
procedure of multi-phase prompt tuning.

#### Inter-sentence Context Encoding Phase.

In the inter-sentence context encoding phase (Phase 1 in Figure 2), we first
concatenate all sentences in $\mathcal{C}$ into a sequence, and then utilize
the LLM to encode $\mathcal{C}$ by incorporating the trainable prompt:

$H_{\mathcal{C}}^{1:L}=\text{LLM}(\mathcal{C},{\bf P}_{\mathcal{C}}),$ (1)

where $H_{\mathcal{C}}^{1:L}\in\mathbb{R}^{L\times|\mathcal{C}|\times d}$ is
the sequence of activations for $\mathcal{C}$, ${\bf
P}_{\mathcal{C}}\in\mathbb{R}^{L\times 2q\times d}$ is the current-phase
trainable prompt, and $q$ is a hyper-parameter for the length of the prompt.
${\bf P}_{\mathcal{C}}$ aims to adapt the LLM for better modeling the inter-
sentence context. Same as basic deep prompting, at the $l$-th transformer
block, we inject corresponding prompt in ${\bf P}_{\mathcal{C}}$ into encoding
procedure of $\mathcal{C}$ as follows:

$H_{\mathcal{C}}^{l}=\text{FFN}\left(\text{Multi-
Attn}\left(\textbf{K}_{\mathcal{C}},\textbf{V}_{\mathcal{C}},\textbf{Q}_{\mathcal{C}}\right)\right),$
(2) $\textbf{Q}_{\mathcal{C}}=H_{\mathcal{C}}^{l-1},$ (3)
$\textbf{K}_{\mathcal{C}}=[{\bf
P}_{\mathcal{C}}[l,:q,:];H_{\mathcal{C}}^{l-1}],$ (4)
$\textbf{V}_{\mathcal{C}}=[{\bf
P}_{\mathcal{C}}[l,q:,:];H_{\mathcal{C}}^{l-1}],$ (5)

where $H_{\mathcal{C}}^{l}\in\mathbb{R}^{|\mathcal{C}|\times d}$ is the output
of the $l$-th transformer block. FFN and Multi-Attn are the feed-forward
network sublayer and multi-head self-attention sublayer, respectively.222For
simplicity, we omit the normalization and residual operations in this paper.
$[\cdot;\cdot]$ and $[\cdot:\cdot]$ are the concatenating and slicing
operations, respectively.

Figure 2: Illustration of pipeline of multi-phase prompt tuning LLM for
context-aware NMT. Red lines illustrate the procedure of enhanced decoding
phase.

#### Intra-sentence Context Encoding Phase.

In the intra-sentence context encoding phase (Phase 2 in Figure 2), the LLM
encodes the intra-sentence context $S$ by conditioning on the past activations
of the inter-sentence context $H_{\mathcal{C}}^{1:L}$ and trainable prompt:

$H_{S}^{1:L}=\text{LLM}(S,H_{\mathcal{C}}^{1:L},{\bf P}_{S}),$ (6)

where $H_{{S}}^{1:L}\in\mathbb{R}^{L\times|S|\times d}$ is the sequence of
activations for $S$, and ${\bf P}_{S}\in\mathbb{R}^{L\times 2q\times d}$
denotes current-phase prompt. Similarly, at the $l$-th transformer block, we
incorporate $H_{{\mathcal{C}}}$ and ${\bf P}_{S}$ into the encoding procedure
of $S$ as follows:

$H_{S}^{l}=\text{FFN}\left(\text{Multi-
Attn}\left(\textbf{K}_{S},\textbf{V}_{S},\textbf{Q}_{S}\right)\right),$ (7)
$\textbf{Q}_{S}=H_{S}^{l-1},$ (8) $\textbf{K}_{S}=[{\bf
P}_{S}[l,:q,:];H_{\mathcal{C}}^{l-1};H_{S}^{l-1}],$ (9) $\textbf{V}_{S}=[{\bf
P}_{S}[l,q:,:];H_{\mathcal{C}}^{l-1};H_{S}^{l-1}],$ (10)

where $H_{S}^{l}$ is output of the $l$-th transformer block, which fuses
$H_{\mathcal{C}}^{l-1}$, the $l-1$ layer output of the inter-sentence context
encoding.

#### Decoding Phase.

In the decoding phase (Phase 3 in Figure 2), given the past activations
$H_{S}$ and trainable prompt, we call the LLM again to generate the hidden
state for predicting the probability of the target sentence:

$H_{T}^{1:L}=\text{LLM}(T,H_{{S}}^{1:L},{\bf P}_{T}),$ (11)

where $H_{T}^{1:L}\in\mathbb{R}^{L\times|T|\times d}$ is the sequence of
activations for $T$, and ${\bf P}_{T}\in\mathbb{R}^{L\times 2q\times d}$ is
current-phase prompt. Similarly, we inject $S$ and ${\bf P}_{T}$ into the
decoding procedure of $T$ as follows:

$H_{T}^{l}=\text{FFN}\left(\text{Multi-
Attn}\left(\textbf{K}_{T},\textbf{V}_{T},\textbf{Q}_{T}\right)\right),$ (12)
$\textbf{Q}_{T}=H_{T}^{l-1},$ (13) $\textbf{K}_{T}=[{\bf
P}_{T}[l,:q,:];H_{S}^{l-1};H_{T}^{l-1}],$ (14) $\textbf{V}_{T}=[{\bf
P}_{T}[l,q:,:];H_{S}^{l-1};H_{T}^{l-1}],$ (15)

where $H_{T}^{l}\in\mathbb{R}^{|T|\times d}$ is the decoding state of the
$l$-th transformer block. Finally, we refer the $t$-th decoding state as
$h_{t}^{L}$ (i.e., $H_{T}^{L}=h_{t}^{L}|_{t=1}^{|T|+1}$) which is used to
predict the next token $y_{t}$:

$p\left(y_{t}|S,\mathcal{C},y_{<t}\right)=\text{Softmax}\left(h_{t}^{L}W\right),$
(16)

where $W\in\mathbb{R}^{d\times|\mathcal{V}|}$ is parameter of LLM-Head layer
and $|\mathcal{V}|$ is the vocabulary size.

###  2.2 Enhanced Decoding Phase

Figure 3: Illustration of the procedure of our proposed decoding-enhanced
approach at the $t$-th decoding step of the decoding phase.

As shown in Figure 2, both the inter-sentence context representation
$H_{\mathcal{C}}^{1:L}$ and the intra-sentence context representation
$H_{S}^{1:L}$ are used as keys and values when generating hidden states of
next phase. Meanwhile, hidden states of decoding phase, i.e.,
$h_{i}^{L}|_{i=1}^{|T|}$ are used to predict next tokens. On the one hand,
while the decoding hidden states incorporate both inter- and intra-sentence
contexts, there is no explicit differentiation between the two when predicting
next tokens. On the other hand, the inter-sentence context representation
$H_{\mathcal{C}}^{1:L}$ and decoding hidden states $H_{T}^{1:L}$ are mediated
by hidden states of phases 2, i.e., $H_{S}^{1:L}$. This may result in a long-
distance issue such that the inter-sentence context are not properly aligned
by target-side tokens.

Therefore, to address above two issues, we propose an enhanced decoding phase
with an aim to more effectively utilize both the inter- and intra-sentence
contexts. Inspired by Kuang et al. (2018), we move both the two types of
inter- and intra-sentence contexts closer to target words to achieve a tight
interaction between them. Specifically, we concatenate the decoding states
with the two types of representations to predict the next target words. As
shown in Figure 3, the enhanced next word prediction $p_{e}$ is a combination
of three distributions with different inputs:

$\begin{split}p_{e}\left(y_{t}|S,\mathcal{C},y_{<t}\right)=&\lambda_{1}\times\hat{p}\left(y_{t}|S,\mathcal{C},y_{<t}\right)\\\
&+\lambda_{2}\times\bar{p}\left(y_{t}|S,\mathcal{C},y_{<t}\right)\\\
&+\left(1-\lambda_{1}-\lambda_{2}\right)\times
p\left(y_{t}|S,\mathcal{C},y_{<t}\right),\end{split}$ (17)

where $\lambda_{1}$ and $\lambda_{2}$ control the contribution of
$\hat{p}\left(y_{t}|\cdot\right)$ and $\bar{p}\left(y_{t}|\cdot\right)$,
respectively, which can be further computed as:

$\hat{p}\left(y_{t}|S,\mathcal{C},y_{<t}\right)=\text{Softmax}\left(\hat{h}_{t}^{L}W\right),$
(18)
$\bar{p}\left(y_{t}|S,\mathcal{C},y_{<t}\right)=\text{Softmax}\left(\bar{h}_{t}^{L}W\right),$
(19)
$\hat{h}_{t}^{L}=\text{FFN}\left([\tilde{H}_{\mathcal{C}}^{L};\tilde{H}_{S}^{L};h_{t}^{L}]\right),$
(20) $\bar{h}_{t}^{L}=\text{FFN}\left([\tilde{H}_{S}^{L};h_{t}^{L}]\right),$
(21)

where $W$ is same as in Eq. 16, $\tilde{H}_{S}^{L}\in\mathbb{R}^{d}$ and
$\tilde{H}_{\mathcal{C}}^{L}\in\mathbb{R}^{d}$ are the averaged $H_{S}^{L}$
and $H_{\mathcal{C}}^{L}$ at token level, respectively.

###  2.3 Phase-aware Prompts

We emphasize the LLM needs to play various roles across three phases, and
maintaining similar prompts across different phases may not be reasonable.
Thus, we empower LLM to distinguish different phases by introducing a type
embedding and a transfer layer333Different from the multi-layer perceptron
(MLPs) used for reparameterization, our transfer layer is shared-parameter for
all prompts. Thus, there are fewer trainable parameters during the training of
our model. We compare the number of trainable parameters among different
tuning methods in Table 3 and analyze the effect of the transfer layer in
Appendix D. for these prompts:

${\bf P}_{r}=\left(\tanh\left({\bf
O}_{r}W_{1}\right)\right)W_{2}+\text{TypeEmb}\left(r\right),$ (22)

where ${\bf O}_{r}\in\mathbb{R}^{L\times 2q\times d}$ is randomly initialized
prompt, $W_{1},W_{2}\in\mathbb{R}^{d\times d}$ are trainable parameters, and
TypeEmb($\cdot$) is type embeddings layer of prompts.
$r\in\\{\mathcal{C},S,T\\}$ represents either phase 1, phase 2, or phase 3.

###  2.4 Training Objective

We employ the cross-entropy loss as the training objective of our model. Given
a training instance $(\mathcal{C},S,T)$, its training loss is defined as:

$\mathcal{L}\left(\mathcal{C},S,T\right)=-\frac{1}{|T|}\sum_{t=1}^{|T|}\text{log}~{}p_{e}\left(y_{t}|S,\mathcal{C},y_{<t}\right).$
(23)

Notably, the parameters in LLM, including $W$ in Eq. 16, 18, 19, are frozen
during training.

##  3 Experimentation

Model | ZH$\rightarrow$EN | FR$\rightarrow$EN | DE$\rightarrow$EN | ES$\rightarrow$EN | RU$\rightarrow$EN
---|---|---|---|---|---
BLEU | COMET | BLEU | COMET | BLEU | COMET | BLEU | COMET | BLEU | COMET
Trans. | 29.86 | 0.8406 | 38.53 | 0.8545 | 41.44 | 0.8682 | 48.74 | 0.8783 | 32.25 | 0.8169
llama-2-7b as foundation model
MT-LoRA | 27.43 | 0.8511 | 38.18 | 0.8647 | 40.96 | 0.8712 | 47.52 | 0.8733 | 33.00 | 0.8311
MT-PT | 31.32 | 0.8565 | 41.92 | 0.8675 | 43.56 | 0.8752 | 51.32 | 0.8819 | 35.46 | 0.8333
CMT-PT | 31.13 | 0.8387 | 42.01 | 0.8699 | 43.11 | 0.8762 | 51.66 | 0.8823 | 35.91 | 0.8396
MPT | *33.21 | 0.8645 | †43.11 | 0.8744 | *43.88 | 0.8824 | †52.01 | 0.8913 | †36.49 | 0.8456
DeMPT | *33.89 | 0.8658 | †43.71 | 0.8816 | *44.69 | 0.8899 | †53.10 | 0.8979 | †36.55 | 0.8438
bloomz-7b1-mt as foundation model
MT-LoRA | 25.79 | 0.8466 | 35.67 | 0.8601 | 35.17 | 0.8522 | 46.32 | 0.8644 | 28.01 | 0.8012
MT-PT | 30.99 | 0.8520 | 40.49 | 0.8661 | 37.76 | 0.8579 | 50.68 | 0.8823 | 30.27 | 0.8106
CMT-PT | 30.82 | 0.8504 | 40.31 | 0.8639 | 38.01 | 0.8601 | 50.26 | 0.8832 | 29.80 | 0.8108
MPT | *31.81 | 0.8601 | *41.11 | 0.8766 | †38.99 | 0.8669 | *51.33 | 0.8910 | *30.99 | 0.8201
DeMPT | *32.46 | 0.8649 | *41.92 | 0.8790 | †40.06 | 0.8703 | *52.25 | 0.8990 | *31.79 | 0.8253

Table 1: Results of different systems on sacreBLEU and COMET metrics.
DeMPT/MPT is our proposed Multi-phase Prompt Tuning approach with/without
Decoding-enhanced strategy (in Sec. 2.2). Scores with bold indicate the best
performance. * or † indicates the gains are statistically significant over MT-
PT or CMT-PT with $p$<0.01 Koehn (2004).

Model | ZH$\rightarrow$ | FR$\rightarrow$ | DE$\rightarrow$ | ES$\rightarrow$ | RU$\rightarrow$
---|---|---|---|---|---
Trans. | 47.63 | 54.41 | 58.29 | 62.52 | 48.79
llama-2-7b as foundation model
MT-LoRA | 44.83 | 54.52 | 57.72 | 62.18 | 49.06
MT-PT | 49.49 | 57.87 | 60.89 | 65.02 | 52.59
CMT-PT | 49.53 | 58.27 | 61.23 | 65.89 | 53.34
MPT | 51.56 | 59.56 | 62.15 | 67.14 | 54.18
DeMPT | 52.68 | 60.33 | 63.11 | 67.95 | 54.34
bloomz-7b1-mt as foundation model
MT-LoRA | 43.23 | 51.82 | 51.12 | 61.77 | 43.29
MT-PT | 49.48 | 56.81 | 55.40 | 64.71 | 46.14
CMT-PT | 49.61 | 57.05 | 55.81 | 65.12 | 46.09
MPT | 50.22 | 57.93 | 56.69 | 66.25 | 47.29
DeMPT | 50.62 | 58.30 | 57.34 | 67.12 | 48.00

Table 2: Results of different systems on BlonDe metric.

We build our approach upon two open-source LLMs, namely,
llama-2-7b444https://huggingface.co/meta-llama/Llama-2-7b-hf and
bloomz-7b1-mt555https://huggingface.co/bigscience/bloomz-7b1-mt. We verify the
effectiveness of our proposed approach on five translation tasks, including
{Chinese (ZH), French (FR), German (DE), Spanish (ES), Russian
(RU)}$\rightarrow$English (EN).

###  3.1 Experimental Settings

#### Datasets and Preprocessing.

The corpus of all translation tasks is extracted from New-Comentary-v18. See
Appendix A for splitting and statistics of the training set, valid set, and
test set. We use the tokenizer of foundation models to process the input data
and no any other preprocessing is performed.

#### Baselines.

We compare our approach against four baselines:

  * •

Transformer (Trans.): It is an encoder-decoder Transformer-base model Vaswani
et al. (2017) that is trained from scratch.

  * •

MT-LoRA: It is a tuned LLM adapted to NMT task via the tuning method of Low-
Rank Adaptation Hu et al. (2022), which makes large-scale pre-training models
adapt to a new task by injecting a trainable rank decomposition matrice into
each layer of the Transformer architecture.

  * •

MT-PT: It is a tuned LLM adapted to NMT task via the deep prompt tuning with
MLPs reparameterization,666We attempt to remove reparameterization but
experience a significant decline in performance. which only tunes continuous
prompts with a frozen language model.

  * •

CMT-PT: Similar to MT-PT, it is also a tuned LLM via the deep prompt tuning
with MLPs reparameterization. Unlike MT-PT, it utilizes inter-sentence context
within the concatenation strategy, as depicted in Figure 1 (a).

Among them, Transformer, MT-LoRA, and MT-PT are context-agnostic systems while
CMT-PT is a context-aware system. For a fair comparison, we ensure that all
context-aware systems, including CMT-PT, MPT, and DeMPT, incorporate identical
inter-sentence context.

#### Model Setting and Training.

For the Transformer model, we implement it upon Fairseq Ott et al. (2019). For
MT-LoRA models, we set the rank of trainable matrices as 16 which performs
best in our preliminary experiment. For all MT-PT models, CMT-PT models, and
our models, we set the prompt length $q$ as 64. For the incorporation of
inter-sentence context in CMT-PT models and our models, we consider a dynamic
$z$, in which the total tokens are no more than 256. In enhanced decoding, we
consider the three next word predictions to be equally important by setting
both $\lambda_{1}$ and $\lambda_{2}$ to 1/3. More details of training are
provided in Appendix B.

#### Evaluation.

We use sacreBLEU (accuracy-related metric)777Signature:
nrefs:1|case:mixed|eff:no|tok:13a|
smooth:exp|version:2.3.1 Post (2018), COMET (semantics-related metric) with
the wmt22-comet-da model888https://github.com/Unbabel/COMET Rei et al. (2020),
and BlonDe (discourse-related metric) Jiang et al. (2022) as the evaluation
metrics.

###  3.2 Experimental Results

The main experimental results are presented in Tables 1 and 2. Additionally, a
comparison of the number of trainable parameters is presented in Table 3
across different tuning methods. From these results we have the following
observations:

  * •

The encoder-decoder Transformer model (Trans.) performs better than the LLMs
with LoRA tuning in most translation directions in BLEU score. For example,
when utilizing llama-2-7b as the foundation model, Transformer surpasses MT-
LoRA an average of 0.75 BLEU score across all translation tasks. However, MT-
LoRA model outperforms Trans. in terms of COMET, suggesting that translations
from LLMs may align more closely with human preferences.

  * •

The MT-PT model presents superior performance compared to the MT-LoRA model in
terms of BLEU, COMET, and BlonDe. Taking bloomz-7b1-mt as the foundation
model, the MT-PT model outperforms the MT-LoRA model by an average of 3.84
BLEU score, 3.51 BlonDe score, and 0.0047 COMET score. Nevertheless, the MT-PT
model sacrifices efficiency for performance, introducing more trainable
parameters (13.87% vs. 0.12%).

  * •

Leveraging the inter-sentence context is helpful in alleviating discourse-
related issues. For example, with bloomz-7b1-mt used as the foundation model,
the CMT-PT model, despite underperforming in BLEU and COMET compared to the
MT-PT model, excels in discourse-related BlonDe scores (averaging 57.66 vs.
57.17).

  * •

Our MPT/DeMPT model outperforms all baselines across all translation tasks.
For example, when using llama-2-7b as the foundation model, our MPT model
achieves an average gain of 1.62/1.45/2.03 in BLEU/COMET/BlonDe score compared
to the CMT-PT model. Furthermore, our decoding-enhance strategy enhances the
capacity of LLMs in context-aware NMT, with DeMPT outperforming MPT in
BLEU/COMET/BlonDe score (averaging 42.39/0.8758/59.68 vs. 41.74/0.8716/58.91).

  * •

The model built upon llama-2-7b as the foundation model outperforms the one
using bloomz-7b1-mt, suggesting that llama-2-7b serves as a more robust
foundation model for translation tasks.

| MT-LoRA | MT-PT/CMT-PT | DeMPT
---|---|---|---
Trainable Para. | 0.12% | 13.87% | 3.11%

Table 3: Proportion of trainable parameters against total parameters for
different tuning methods.

##  4 Discussion

In this section, we use bloomz-7b1-mt as the foundation model to discuss and
analyze our approach. See Appendix C$\sim$E for further discussions.

###  4.1 Effect of Length of Inter-sentence Context

For efficient training, we define the inter-sentence context in Section 2 as
previous sentences with a total tokens not exceeding 256. We are curious about
the potential impact of inter-sentence length on the performance of our
approach. Consequently, we extend the inter-sentence context length from 256
to 1024 and assess the performance of our approach in the ZH$\rightarrow$EN
task.

Figure 4 shows the performance trend of the CMT-PT model and our DeMPT model.
As the length of the inter-sentence context increases, both models exhibit a
slight enhancement in both BLEU and BlonDe scores. Interestingly, our model
with a 256-token inter-sentence context outperforms the CMT-PT model with a
1024-token inter-sentence context in both BLEU and BlonDe scores. This further
suggests the effectiveness of our approach in harnessing the capabilities of
LLMs for context-aware NMT compared to the concatenation strategy.

Figure 4: Performance of CMT-PT and our DeMPT on ZH$\rightarrow$EN test set
when using different inter-sentence context lengths.

###  4.2 Effect of Prompt Length

Figure 5: Performance of MT-PT, CMT-PT, and our DeMPT on ZH$\rightarrow$EN
test set when using different lengths of the trainable prompts.

As our approach is implemented based on deep prompt tuning, next we compare
the impact of the trainable prompt length for MT-PT, CMT-PT, and our DeMPT.

Figure 5 shows the performance curves when increasing the prompt length from
32 to 128. We observe that increased prompt length tends to enhance
performance for both BLEU and BlonDe, yet the gains exhibit diminishing
returns. This finding is consistent with that in Li and Liang (2021); Lester
et al. (2021); Tan et al. (2022). We also observe that DeMPT with a prompt
length of 64 outperforms both MT-PT and CMT-PT with a prompt length of 128 on
both metrics, suggesting the superiority of our approach over the
concatenation strategy in enhancing LLMs’ capacity for context-aware NMT.

###  4.3 Comparison of Inference Speed

Model | Speed | BLEU
---|---|---
MT-PT | 0.75 sec/sent. | 30.99
CMT-PT | 0.77 sec/sent. | 30.82
MPT | 0.78 sec/sent. | 31.81
DeMPT | 0.79 sec/sent. | 32.46
Table 4: Comparison of inference speed on ZH$\rightarrow$EN translation task.
Speed is measured on the test set using 4 GPUs. sec/sent. means seconds spent
for decoding each sentence. Note that the reparameterization is not needed
during inference Li and Liang (2021).

Table 4 compares the inference speed of different models on ZH$\rightarrow$EN
translation task. Our MPT and DeMPT models, dividing the context-aware NMT
process into three separate phases, demonstrates comparable inference speed to
the single-phase MT-PT and CMT-PT models, with only a marginal drop of 0.02
seconds per sentence in decoding. This illustrates the efficiency of our
approach without introducing significant computational overhead.

###  4.4 Performance on Contrastive Test Set

We evaluate the models’ ability to resolve discourse inconsistencies using the
contrastive test set proposed by Voita et al. (2019a), which focuses on four
discourse phenomena such as deixis, lexicon consistency (lex.c), ellipsis
inflection (ell.infl), and verb phrase ellipsis (ell.VP) in
English$\rightarrow$Russian translation. Within the test set, each instance
comprises a positive translation and several negative ones that vary by only
one specific word. The purpose of the contrastive test set is to assess
whether a model is more inclined to generate a correct translation as opposed
to incorrect variations.

Table 5 lists the accuracy of translation prediction on the contrastive test
set for MT-PT, CMT-PT and DeMPT. Compared to the context-agnostic MT-PT model,
both context-aware CMT-PT and DeMPT models show substantial improvements
across the four discourse phenomena. Additionally, DeMPT demonstrates the best
performance, surpassing CMT-PT by an average accuracy margin of 3.8.

Model | deixis | lex.c | ell.infl | ell.VP | Avg.
---|---|---|---|---|---
MT-PT | 50.0 | 45.7 | 53.0 | 28.6 | 44.3
CMT-PT | 80.2 | 46.1 | 74.3 | 75.3 | 68.9
DeMPT | 80.1 | 55.7 | 75.9 | 79.3 | 72.7

Table 5: Accuracy [%] of translation prediction for four discourse phenomena
on the English $\rightarrow$ Russian contrastive test set.

###  4.5 Human Evaluation

We use the Direct Assessment (DA) method Graham et al. (2017) to manually
assess the quality of translations generated by DeMPT and CMT-PT. In this
assessment, human evaluators compare the meaning of the MT output with a
human-produced reference translation, working within the same language.

Specifically, we randomly select 5 documents with a total of 200 groups of
sentences from the ZH$\rightarrow$EN test set. To avoid potential bias in
evaluation, we recruit 6 professional translators and ensure each translation
from DeMPT or CMT-PT is scored twice by two translators. Table 6 shows the DA
scores for CMT-PT and DeMPT. Our DeMPT outperforms CMT-PT by 7.14 DA score,
providing strong evidence for the effectiveness of our approach. Further
details and results regarding the DA can be found in Appendix C.

Model | Score_1 | Score_2 | Average
---|---|---|---
CMT-PT | 79.00 | 80.17 | 79.59
DeMPT | 86.17 (+7.17) | 87.30 (+7.13) | 86.73 (+7.14)

Table 6: Human DA scores for CMT-PT and DeMPT on ZH$\rightarrow$EN translation
task.

##  5 Related Work

#### Large Language Models for Context-aware Machine Translation.

While traditional context-aware neural machine translation (NMT) has seen
considerable progress in recent years Jean et al. (2017); Wang et al. (2017);
Voita et al. (2018); Maruf et al. (2019); Kang et al. (2020); Bao et al.
(2021); Sun et al. (2022); Bao et al. (2023), the effective integration of
large language models (LLMs) to model inter-sentence context and enhance
context-aware translation remains an area of limited exploration. Existing
studies mainly focus on the assessment of LLMs’ ability in discourse modeling.
For example, Wang et al. (2023) approach context-aware NMT as a task involving
long sequence generation, employing a concatenation strategy, and conduct
comprehensive evaluations of LLMs such as ChatGPT and GPT-4. Their focus
includes the impact of context-aware prompts, comparisons with translation
models, and an in-depth analysis of discourse modeling ability. Similarly,
Karpinska and Iyyer (2023) engage professional translators to evaluate LLMs’
capacity in context-aware NMT. In contrast, Wu et al. (2024) compare the
effectiveness of various parameter-efficient fine-tuning methods on
moderately-sized LLMs for context-aware NMT. Besides, Wu and Hu (2023) explore
the prompt engineering with GPT language models specifically for document-
level (context-aware) MT while Li et al. (2024) experiment with combining
sentence-level and document-level translation instructions of varying lengths
to fine-tune LLMs.

#### Prompt Tuning for Large Language Model.

Liu et al. (2021) and Li and Liang (2021) propose to make LLMs adapt to
various tasks by adding trainable prompts (also called continuous prompts) to
the original input sequences. In this paradigm, only the continuous prompts
are updated during training. Liu et al. (2022) further introduce deep prompt
tuning, extending the idea by inserting trainable prompts into all layers of
LLMs, rather than just the embedding layer. While these approaches lay the
groundwork for a general framework, our focus lies in augmenting the
performance of LLMs specifically for inter-sentence context modeling in
context-aware NMT. Notably related, Tan et al. (2022) propose a multi-phase
tuning approach to enhance the sentence-level translation performance of a
multilingual GPT. However, the exploration of effective LLM tuning for
addressing discourse-related challenges in the context-aware NMT domain
remains underdeveloped.

##  6 Conclusion

In this paper, we have examined the hypothesis that it is crucial to
differentially model and leverage inter-sentence context and intra-sentence
context when adapting LLMs to context-aware NMT. This stems from our
observation that intra-sentence context exhibits a stronger correlation with
the target sentence compared to inter-sentence context, owing to its richer
parallel semantic information. To this end, we have proposed a novel decoding-
enhanced multi-phase prompt tuning (DeMPT) approach to make LLMs aware of the
differences between inter- and intra-sentence contexts, and further improve
LLMs’ capacity in discourse modeling. We have evaluated our approach using two
foundation models and present experimental results across five translation
directions. Experimental results and discussions have demonstrated a
significant enhancement in the performance of LLMs in context-aware NMT,
manifesting as improved translation accuracy and a reduction in discourse-
related issues.

## Limitations

Owing to resource limitations, our work is restricted to moderate-scale LLMs,
specifically those with 7 billion parameters, and a confined window size of
inter-sentence context. It is imperative to acknowledge that the results of
our research may differ when employing larger models and an extended window
size for inter-sentence context. We acknowledge these limitations and consider
them as avenues for future exploration.

## References

  * Bao et al. (2023) Guangsheng Bao, Zhiyang Teng, and Yue Zhang. 2023.  Target-side augmentation for document-level machine translation.  In _Proceedings of ACL_ , pages 10725–10742.
  * Bao et al. (2021) Guangsheng Bao, Yue Zhang, Zhiyang Teng, Boxing Chen, and Weihua Luo. 2021.  G-transformer for document-level machine translation.  In _Proceedings of ACL_ , pages 3442–3455.
  * BigScience (2022) BigScience. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _Computing Research Repository_ , arXiv:2211.05100.
  * Google (2022) Google. 2022.  Palm: Scaling language modeling with pathways.  _J. Mach. Learn. Res._ , 24:240:1–240:113.
  * Graham et al. (2017) Yvette Graham, Qingsong Ma, Timothy Baldwin, Qun Liu, Carla Parra, and Carolina Scarton. 2017.  Improving evaluation of document-level machine translation quality estimation.  In _Proceedings of EACL_ , pages 356–361.
  * Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _Proceedings of ICLR_.
  * Hu and Wan (2023) Xinyu Hu and Xiaojun Wan. 2023.  Exploring discourse structure in document-level machine translation.  In _Proceedings of EMNLP_ , pages 13889–13902.
  * Jean et al. (2017) Sebastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017.  Does neural machine translation benefit from larger context?  _Computing Research Repository_ , arXiv:1704.05135.
  * Jiang et al. (2022) Yuchen Eleanor Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Jian Yang, Haoyang Huang, Rico Sennrich, Ryan Cotterell, Mrinmaya Sachan, and Ming Zhou. 2022.  BlonDe: An automatic evaluation metric for document-level machine translation.  In _Proceedings of NAACL_ , pages 1550–1565, Seattle, United States.
  * Kang et al. (2020) Xiaomian Kang, Yang Zhao, Jiajun Zhang, and Chengqing Zong. 2020.  Dynamic context selection for document-level neural machine translation via reinforcement learning.  In _Proceedings of EMNLP_ , pages 2242–2254.
  * Karpinska and Iyyer (2023) Marzena Karpinska and Mohit Iyyer. 2023.  Large language models effectively leverage document-level context for literary translation, but critical errors persist.  In _Proceedings of WMT_ , pages 419–451.
  * Koehn (2004) Philipp Koehn. 2004.  Statistical significance tests for machine translation evaluation.  In _Proceedings of EMNLP_ , pages 388–395.
  * Kuang et al. (2018) Shaohui Kuang, Junhui Li, António Branco, Weihua Luo, and Deyi Xiong. 2018.  Attention focusing for neural machine translation by bridging source and target embeddings.  In _Proceedings of ACL_ , pages 1767–1776.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of EMNLP_ , pages 3045–3059.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of ACL-IJCNLP_ , pages 4582–4597, Online.
  * Li et al. (2023) Yachao Li, Junhui Li, Jing Jiang, Shimin Tao, Hao Yang, and Min Zhang. 2023.  P-Transformer: Towards Better Document-to-Document Neural Machine Translation.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 31:3859–3870.
  * Li et al. (2024) Yachao Li, Junhui Li, Jing Jiang, and Min Zhang. 2024.  Enhancing document-level translation of large language model via translation mixed-instructions.  _Computing Research Repository_ , arXiv:2401.08088.
  * Liu et al. (2022) Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022.  P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks.  In _Proceedings of ACL_ , pages 61–68.
  * Liu et al. (2021) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021.  Gpt understands, too.  _Computing Research Repository_ , arXiv:2103.10385.
  * Lyu et al. (2021) Xinglin Lyu, Junhui Li, Zhengxian Gong, and Min Zhang. 2021.  Encouraging lexical translation consistency for document-level neural machine translation.  In _Proceedings of EMNLP_ , pages 3265–3277.
  * Lyu et al. (2022) Xinglin Lyu, Junhui Li, Shimin Tao, Hao Yang, and Min Zhang. 2022.  Modeling consistency preference via lexical chains for document-level neural machine translation.  In _Proceedings of EMNLP_ , pages 6312–6326.
  * Maruf et al. (2019) Sameen Maruf, André F. T. Martins, and Gholamreza Haffari. 2019.  Selective attention for context-aware neural machine translation.  In _Proceedings of NAACL_ , pages 3092–3102.
  * MetaAI (2023a) MetaAI. 2023a.  Llama 2: Open foundation and fine-tuned chat models.  _Computing Research Repository_ , arXiv:2307.09288.
  * MetaAI (2023b) MetaAI. 2023b.  Llama: Open and efficient foundation language models.  _ArXiv_ , abs/2302.13971.
  * Miculicich et al. (2018) Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018.  Document-level neural machine translation with hierarchical attention networks.  In _Proceedings of EMNLP_ , pages 2947–2954.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _Computing Research Repository_ , arXiv:2303.08774.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of NAACL-HLT: Demonstrations_.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of WMT_ , pages 186–191.
  * Rajbhandari et al. (2020) Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020.  Zero: Memory optimizations toward training trillion parameter models.  In _SC20: Proceedings of High Performance Computing, Networking, Storage and Analysis_ , pages 1–16.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of EMNLP_ , pages 2685–2702.
  * Sun et al. (2022) Zewei Sun, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Shujian Huang, Jiajun Chen, and Lei Li. 2022.  Rethinking document-level neural machine translation.  In _Findings of ACL_ , pages 3537–3548.
  * Tan et al. (2022) Zhixing Tan, Xiangwen Zhang, Shuo Wang, and Yang Liu. 2022.  MSP: Multi-stage prompting for making pre-trained language models better translators.  In _Proceedings of ACL_ , pages 6131–6142.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of NIPS_ , pages 5998–6008.
  * Voita et al. (2019a) Elena Voita, Rico Sennrich, and Ivan Titov. 2019a.  Context-aware monolingual repair for neural machine translation.  In _Proceedings of EMNLP-IJCNLP_ , pages 877–886.
  * Voita et al. (2019b) Elena Voita, Rico Sennrich, and Ivan Titov. 2019b.  When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion.  In _Proceedings of ACL_ , pages 1198–1212.
  * Voita et al. (2018) Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018.  Context-aware neural machine translation learns anaphora resolution.  In _Proceedings of ACL_ , pages 1264–1274.
  * Wang et al. (2023) Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023.  Document-level machine translation with large language models.  In _Proceedings of EMNLP_ , pages 16646–16661.
  * Wang et al. (2019) Longyue Wang, Zhaopeng Tu, Xing Wang, and Shuming Shi. 2019.  One model to learn both: Zero pronoun prediction and translation.  In _Proceedings of EMNLP-IJCNLP_ , pages 921–930.
  * Wang et al. (2017) Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017.  Exploiting cross-sentence context for neural machine translation.  In _Proceedings of EMNLP_ , pages 2826–2831.
  * Wu et al. (2024) Minghao Wu, Thuy-Trang Vu, Lizhen Qu, George Foster, and Gholamreza Haffari. 2024.  Adapting large language models for document-level machine translation.  _Computing Research Repository_ , arXiv:2401.06468.
  * Wu and Hu (2023) Yangjian Wu and Gang Hu. 2023.  Exploring prompt engineering with GPT language models for document-level machine translation: Insights and findings.  In _Proceedings of the Eighth Conference on Machine Translation_ , pages 166–169.
  * Zhang et al. (2018) Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018.  Improving the transformer translation model with document-level context.  In _Proceedings of EMNLP_ , pages 533–542.

##  Appendix A Datasets

#### Statistics and Splitting of Datasets.

We provide the detailed statistic in Table 7. For all translation tasks, we
randomly select 80% document pairs from the corpus as the training set. Both
the test set and validation set include 150 document pairs each, randomly
sampled from the remaining 20% of document pairs in the corpus. Regarding
sentence preprocessing across all datasets, we segment the sentences with the
tokenizer from the respective foundation model. No additional preprocessing
steps are performed. Datasets are downloaded from
https://data.statmt.org/news-commentary/v18.

Dataset | ZH$\rightarrow$EN | FR$\rightarrow$EN | DE$\rightarrow$EN | ES$\rightarrow$EN | RU$\rightarrow$EN
---|---|---|---|---|---
#Doc | #Sent | #Doc | #Sent | #Doc | #Sent | #Doc | #Sent | #Doc | #Sent
Training | 8,622 | 342,495 | 7,915 | 310,489 | 8,417 | 333,201 | 9,677 | 378,281 | 7,255 | 272,100
Validation | 150 | 6,061 | 150 | 5,890 | 150 | 5,866 | 150 | 5,782 | 150 | 5,691
Test | 150 | 5,747 | 150 | 5,795 | 150 | 5,967 | 150 | 5,819 | 150 | 5,619

Table 7: Statistics of training, validation, and test sets for five
translation tasks. #Doc and #Sent denote the numbers of Document and Sentence,
respectively.

Figure 6: Scoring criterion for Direct Assessment. We group the score into
five ranges, i.e., 0-20, 21-40, 41-60, 61-80, 81-100.

##  Appendix B Training Details

For all Transformer NMT models, we use the transformer-base setting as in
Vaswani et al. (2017), where the learning rate is set to 1e-4. The Transformer
NMT models are trained on 4$\times$ NVIDIA V100 32GB GPUs with a batch size of
4096. For the models with prompt tuning in Section 3, including MT-PT, CMT-PT,
and our MPT and DeMPT models, the length of the trainable prompt is set as 64.
During both training and inference, the model generates only the current
target sentence, operating in a many-to-one translation mode. For all fine-
tuning models in this paper, we set the training epoch to 4, and the warm-up
rate to 0.1. We use the log learning rate decay strategy with a maximum
learning rate of 5e-5. We collate a mini-batch by counting the total tokens
inside the batch and set the batch size as 4096. All fine-tuning models are
trained on 4 $\times$ NVIDIA A800 GPUs with Deespeed Zero 2 offload setting
Rajbhandari et al. (2020).999https://github.com/microsoft/DeepSpeed

Figure 7: A case study for the CMT-PT model and our DeMPT model on
ZH$\rightarrow$EN translation task.

##  Appendix C Details of Human Evaluation

#### Criterion and Recruitment.

Given a source sentence, its translation from MT (i.e., CMT-PT and our DeMPT),
and its human-produced reference translation, the evaluators are asked to give
a score ranging from 0 to 100. Figure 6 presents the detailed criterion of
scoring. We recruit evaluators from professional translators with at least
five years of experience in translation.

#### Statistics of Translation Errors.

We manually count the number of bad cases from our DeMPT model. The bad cases
fall into two categories: (1) the DA score is 60 or lower; (2) the DA score is
lower than that of the translation from CMT-PT. The main types of the bad
cases are Mistranslation  (Mis.), Unnoticed Omission (UO), Inappropriate
Expression (IE), and Grammatical Error (GE). We present detailed statistics in
Table 8. The statistics indicate the bad cases mainly come from Mistranslation
and Unnoticed Omission. Meanwhile, our DeMPT model outperforms the CMT-PT
model in 86.5% DA cases.

#### Case Study.

We present a case in Figure 7 to illustrate how our DeMPT model outperforms
the CMT-PT model. In this case, we compare the translations of two consecutive
sentences from our model and the CMT-PT model. First, we notice that the CMT-
PT model translates the source word 美国 in the two sentences into US and
America, respectively. However, our model consistently translates them into
US. Second, our model uses for its part, a phase with more coherent
preference, as the translation of 同时 , instead of At the same time adopted in
the translation from the CMT-PT model. Both of them demonstrate the
superiority of our proposed approach in discourse modeling.

Group | Type of Bad Case
---|---
Mis. | UO | IE | GE | Total (Perc.)
1 | 6 | 3 | 1 | 2 | 12 (6.0%)
2 | 9 | 7 | 6 | 5 | 27 (13.5%)

Table 8: Statistics of bad cases from our DeMPT model on ZH$\rightarrow$EN
translation task. Perc. denotes the percentage of bad cases against the total
of DA cases.

##  Appendix D Effect of Transfer Layer and Type Embedding

As in Eq. 22 within Section 2.3, we introduce two sublayers: a non-linear
transfer sublayer and a type embedding sublayer for the trainable prompt in
each phase. This design enhances the awareness of LLMs regarding the
distinctions in inputs across the three tuning phases, allowing them to adapt
to specific roles at each phase. We investigate the effect of these two
sublayers.

As shown in Table 9, our observations reveal that the transfer sublayer holds
greater importance than the type embedding sublayer. Removing either the non-
linear transfer sublayer (w/o Transfer.) or the type embedding sublayer (w/o
Embed.) results in a performance drop of 0.84/0.0048/0.39 or 0.45/0.0036/0.007
in BLEU/COMET/BlonDe metrics.

Model | BLEU | COMET | BlonDe
---|---|---|---
MT-PT | 30.99 | 0.8520 | 49.48
CMT-PT | 30.82 | 0.8504 | 49.61
DeMPT | 32.46 | 0.8649 | 50.62
w/o Transfer. | 31.62 | 0.8601 | 50.23
w/o Embed. | 32.01 | 0.8613 | 50.55
w/o CTX. | 31.98 | 0.8593 | 49.89

Table 9: Comparison of performances of the DeMPT variants on ZH$\rightarrow$EN
test set. w/o Trans. or w/o Embed. denotes the variant without the non-linear
transfer sublayer or type embedding sublayer in Eq. 22. w/o CTX. means the
inter-sentence context is not available, i.e., context-agnostic DeMPT system.

##  Appendix E Effect of Inter-sentence Context

We implement the context-agnostic (sentence-level) DeMPT system to analyze the
effect of the inter-sentence context. More specifically, we replace the input
of LLMs in the inter-sentence context encoding phase with the intra-sentence
context. In other words, we encode the intra-sentence context twice to keep
the multi-phase tuning strategy in DeMPT while making the inter-sentence
context unavailable.

As shown in the last row of Table 9 (i.e., w/o CTX), we find that the inter-
sentence context is crucial for the alleviation of discourse-related issues.
The BlonDe score drops by 0.73 when the inter-sentence context is unavailable.
Meanwhile, our DeMPT also significantly improves the performance of LLMs in
context-agnostic MT, e.g., + 0.99 BLEU score and + 0.0073 COMET score compared
to the MT-PT model.