# Gender-specific Machine Translation with Large Language Models

Eduardo Sánchez${}^{\text{$\alpha$ $\beta$}}$ Pierre
Andrews${}^{\text{$\alpha$}}$ Pontus Stenetorp${}^{\text{$\beta$}}$
Mikel Artetxe${}^{\text{$\gamma$}}$ Marta R. Costa-jussà${}^{\text{$\alpha$}}$
${}^{\text{$\alpha$}}$Meta AI
${}^{\text{$\beta$}}$UCL Centre for Artificial Intelligence
${}^{\text{$\gamma$}}$Reka AI
{eduardosanchez, mortimer<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Decoder-only Large Language Models (LLMs) have demonstrated potential in
machine translation (MT), albeit with performance slightly lagging behind
traditional encoder-decoder Neural Machine Translation (NMT) systems. However,
LLMs offer a unique advantage: the ability to control the properties of the
output through prompts. In this study, we harness this flexibility to explore
LLaMa’s capability to produce gender-specific translations for languages with
grammatical gender. Our results indicate that LLaMa can generate gender-
specific translations with competitive accuracy and gender bias mitigation
when compared to NLLB, a state-of-the-art multilingual NMT system.
Furthermore, our experiments reveal that LLaMa’s translations are robust,
showing significant performance drops when evaluated against opposite-gender
references in gender-ambiguous datasets but maintaining consistency in less
ambiguous contexts. This research provides insights into the potential and
challenges of using LLMs for gender-specific translations and highlights the
importance of in-context learning to elicit new tasks in LLMs.

##  1 Introduction

Decoder-only Large Language Models (LLMs) have shown Machine Translation (MT)
capabilities inferior to but competitive with encoder-decoder Neural Machine
Translation (NMT) systems Chowdhery et al. (2022); Agrawal et al. (2023);
Zhang et al. (2023); Bawden and Yvon (2023); Zhu et al. (2023); Jiao et al.
(2023); Hendy et al. (2023). However, LLMs have been proven to allow for more
control over the properties of the output Garcia et al. (2023); Moslem et al.
(2023); Pilault et al. (2023). While NMT models are trained to accept a single
sequence and output its translation, LLMs make it possible to condition the
output format with a prompt.

The task of providing two translations for a gender-ambiguous source has been
addressed mainly through post-editing, the most popular solution being Google
Translate’s post-translation gender rewriter Johnson (2020). The proposed
system produces a single sentence that is then switched into the opposite
gender using a second language-specific model. This approach is limited by
having to train language-specific gender-switching models and the breadth of
patterns it can cover.

Given the flexibility of prompting, we explore the capacity of LLMs to produce
gender-specific translations for languages with grammatical gender, while
maintaining a robust translation quality and minimizing gender bias. We use
in-context examples (ICEs) to elicit the task of translating from a single
source to two gender-specific targets (Figure 1).

We evaluate the quality of the gender-specific translations on two aspects:
translation accuracy (measured in BLEU points) and gender bias mitigation
(measured in coreference resolution accuracy).

Figure 1: Example of gender-specific translation template with two ICEs. The
bold text denotes the prompt and the blue text denotes the model’s output.
While the source sentence "I have friends who are orphans." is ambiguous,
i.e., it isn’t possible to infer the gender of the direct object of the
sentence, the two produced sentences have clear grammatical gender markers, so
that they can be translated as "I have (male) friends who are orphans." and "I
have (female) friends who are orphans." respectively.

We show that it is possible to generate gender-specific translations with
accuracy and gender bias mitigation competitive with NLLB, the SOTA massively
multilingual NMT system at the time of writing this paper. LLaMa achieves on
average under 2 BLEU points less than NLLB; when only accounting for languages
explicitly included in LLaMa, this gap lowers to less than 1 BLEU point. We
also demonstrate the robustness of the gender-specific translation method,
showing steep decreases in performance when using the opposite gender as an
evaluation reference in a highly gender-ambiguous dataset
(MultilingualHolisticBias), but exhibiting almost no variance in less gender-
ambiguous datasets (FLoRes).

The structure of the article consists of giving insight into the related work
in the field of MT with LLMs and their gender biases in Section 2, outlining
the proposed methodology in Section 3, listing the employed datasets, models
and evaluation metrics in Section 4, discussing experiments and results in
Section 5 and highlighting the conclusions of our work in Section 6.

##  2 Related Work

#### In-context learning

In-context learning (ICL) is a well-known emergent ability of LLMs (Brown et
al., 2020; Chowdhery et al., 2022), where new tasks are learned using only a
few supervised examples in the form of demonstrations (Dong et al., 2023)
Given a set of $n$ labeled examples
$E=\\{(x_{1},y_{1}),(x_{2},y_{2}),...,(x_{n},y_{n})\\}$ and a prompt head $H$,
a templating function $T(x,y)$ and an incomplete templating function
$\hat{T}(x)$, the output $\hat{y}$ corresponding to the input $\hat{x}$ is
calculated according to the following formula:

$\hat{y}=\operatorname*{arg\,max}_{y^{\prime}_{j}\in Y}P(\hat{y}|X)$

with

$X=H\oplus\mathop{\bigoplus}_{i=1}^{n}T(x_{i},y_{i})\oplus\hat{T}(\hat{x})$

where $Y$ denotes the set of all possible outputs and $\oplus$ denotes the
concatenation operation.

Several authors have investigated how the choice of ICEs impacts the
performance of LLMs. Min et al. (2022) explore the role of ICEs and show that
ground truth demonstrations are not necessary for effective ICL and that other
aspects, such as label space, input text distribution, and sequence format are
instead the key drivers of task performance. Agrawal et al. (2023) analyze the
impact of factors such as the choice and number of examples on the output
translation quality. They show that a single good prompt optimized to maximize
translation quality on the development dataset can elicit learned information
from the pre-trained language model. Tanwar et al. (2023) study ICL in a
cross-lingual setting, proposing a prompt construction strategy aimed at
mitigating the lack of alignment between input and output spaces in
multilingual LLMs. Wei et al. (2023b) examined the reliance of LLMs on the
ICEs as the scale of the models grew. They find that larger models are able to
override semantic priors from their pre-trained dataset more easily than
smaller models, rendering the increasing capacity to reply on ICL an emergent
ability in itself.

#### MT and controlled output with LLMs

A few papers have evaluated the quality of MT using different models and GPT-
based commercial products, such as PALM Chowdhery et al. (2022), XGLM Agrawal
et al. (2023), GLM Zhang et al. (2023), BLOOM Bawden and Yvon (2023), OPT Zhu
et al. (2023) or ChatGPT Jiao et al. (2023); Hendy et al. (2023). They
conclude that the translation quality comes close but remains behind the
performance of NMTs. Using LLMs can, however, allow for more control over the
properties of the output without further finetuning, such as specifying the
language variety and style of the translation Garcia et al. (2023), producing
terminology-constrained translations Moslem et al. (2023) or using an
iterative prompting process to clarify ambiguities in the source sentence
Pilault et al. (2023). Challenges persist in the area of hallucinations Zhang
et al. (2023); Guerreiro et al. (2023) and in performance in low-resource
languages Bawden and Yvon (2023); Zhu et al. (2023).

#### Bias in LLMs

There has been extensive work on bias for Natural Language Processing Blodgett
et al. (2020); Pennington et al. (2014); Bolukbasi et al. (2016); Caliskan et
al. (2017); Costa-jussà (2019). In particular, there has been growing interest
in detecting bias in LLMs. Lin et al. (2021) propose a benchmark to evaluate
truthfulness in LLMs while Gehman et al. (2020) present a dataset to discover
neural degeneration that leads to toxic outputs. Shaikh et al. (2023) expand
the previous work to analyze how chain-of-thought (CoT) prompting Wei et al.
(2023a) increases a model’s likelihood to produce biased outputs previously
mitigated through alignment.

#### Gender Bias in MT

More narrowly, some authors have worked in analyzing and mitigating biases in
Machine Translation. Prates et al. (2018) studied the bias of the commercial
translation system Google Translate and found that it yields male defaults
much more frequently than what would be expected from US demographic data.
Costa-jussà et al. (2022) investigate the role of model architecture in the
level of gender bias, while Měchura (2022) looks at the source sentences and
elaborates a taxonomy of the features that induce gender bias into the
translations. Others have looked more closely at the challenge of gender bias
mitigation. Stafanovičs et al. (2020) assume that it’s not always possible to
infer all the necessary information from the source sentence alone and a
method that uses word-level annotations containing information about the
subject’s gender to decouple the task of performing an unbiased translation
from the task of acquiring gender-specific information. Saunders and Byrne
(2020) treat the mitigation as a domain adaptation problem, using transfer
learning on a small set of trusted, gender-balanced examples to achieve
considerable gains with a fraction of the from-scratch training costs. Fleisig
and Fellbaum (2022) develop a framework to make NMT systems suitable for
gender bias mitigation through adversarial learning, adjusting the training
objective at fine-tuning time. Finally, Wang et al. (2022) focus on existing
biases in person name translation, applying a data augmentation technique
consisting of randomly switching entities, obtaining satisfactory results.

##  3 Proposed Methodology

Given the capacity of LLMs to produce text given a context, we leverage LLMs’
ICL abilities to elicit gender-specific translations via prompting. Our
template consists of the English name of the source language between square
brackets, followed by the source sentence. This is followed by two
translations preceded by the English name of the target language. Each
translation represents, respectively, the masculine and the feminine
translation of the source sentence. The three sentences are joined by a
carriage return character (`’\n’`). An example of a gender-specific prompt can
be found in Figure 1.

We compare the accuracy of the gender-specific translations produced by an LLM
with the translations produced by an NMT model of the MultilingualHolisticBias
dataset, which contains gender-specific translations from English to 25
languages. We verify the robustness of our approach by inverting the gender of
the reference and calculating the decrease in translation accuracy. We also
compare the gender bias of the gender-specific translations of an LLM with
respect to an NMT model by evaluating coreference resolution accuracy on the
WinoMT and BUG datasets. Finally, we generate gender-specific translations
with FLoRes, a general-purpose dataset, and verify the similarity between both
translations in a context where very little gender ambiguity when translating
from English to other languages is expected.

##  4 Experimental framework

###  4.1 Data

#### MultilingualHolisticBias

To extend bias analyses beyond commonly studied languages, Costa-jussà et al.
(2023) provided expert-curated translations of a subset (325 sentences) of
Holistic Bias Smith et al. (2022) into 50 languages. Additionally,
MultilingualHolisticBias presents separate translations for each noun class or
grammatical gender for those languages that make use of them. An example of an
entry of the MultilingualHolisticBias dataset can be found in Table 1. For
this study, we used the subset of languages that make use of grammatical
genders or noun classes, as it allows us to benchmark the models’ output
against gender-specific translations. We have also verified that the selected
25 languages have a very high correlation between grammatical gender and
natural gender, allowing us to establish a relationship between gender bias
and the accuracy of coreference resolution in a model. A complete list of
languages used from the MultilingualHolisticBias dataset can be found in
Appendix A.

| source sentence | masculine | feminine | neutral | generic
---|---|---|---|---|---
est |  What do you think about Hasidic children? |  - |  - |  - |  Mida arvate lastest, kes on hassiidid?
spa |  What do you think about Hasidic children? |  ¿Qué piensas de los niños judíos jasídicos? |  ¿Qué piensas de las niñas judías jasídicas? |  - |  -
kat |  What do you think about Hasidic children? |  - |  - |  ras pikrob sasid bav+svebze? (ras pikrob khasid bavshvebze?) |  -
Table 1: Example of entries in the full MultilingualHolisticBias dataset. From
the 50 languages present in MultilingualHolisticBias, we selected those that
exhibit grammatical gender differences, like Spanish (spa). The sentence in
Georgian (kat) has been transcribed into Latin script by the authors.

#### WinoMT & BUG

Aimed at estimating the gender bias of an MT system, Stanovsky et al. (2019)
concatenate Winogender Rudinger et al. (2018) and WinoBias Zhao et al.
(2018)’s coreference test sets for 8 languages where the grammatical gender
tends to be aligned with the biological gender Craig (1986); Mucchi-Faina
(2005); Corbett (2007), naming the new dataset WinoMT. This benchmark relies
on language-specific automatic measures for alignment and morphological
analysis, therefore it doesn’t need gold translations for evaluation.
Stanovsky et al.’s (2019) dataset was expanded by Levy et al. (2021) to
include 108K diverse real-world English sentences in a new benchmark dubbed
BUG. Given the size of the BUG dataset, we only use the Gold BUG split, made
up of 1700 uniformly sampled sentences across every pattern and domain
described in the dataset. The list of languages into which the WinoMT & BUG
datasets are translated in this work can be found in Appendix A.

#### FLoRes

The Facebook Low Resource (FLoRes) MT benchmark is a massively multilingual
general domain dataset introduced by Guzmán et al. (2019) and Goyal et al.
(2021), and expanded by the NLLB Team et al. (2022). In its last version, it
includes 200 languages. In this work, we use FLoRes’s test set to validate the
generalization of our results beyond highly ambiguous datasets such as
MultilingualHolisticBias and WinoMT + BUG.

#### Usage of datasets

We use MultilingualHolisticBias to generate gender-specific translations in 25
languages with LLaMa and compare them with NLLB outputs. We translate the
WinoMT and BUG datasets to 7 languages and we leverage Stanovsky et al.’s
(2019) reference-less evaluation
method111https://github.com/gabrielStanovsky/mt_gender to measure the gender
bias of the translations both across models and across genders. We translate
both WinoMT’s stereotypical and non-stereotypical datasets (where gender
correspondence is aligned or not with stereotypical and non-stereotypical
gender role assignments, as defined by Zhao et al. (2018), who use statistics
provided by the US Department of
Labor222https://www.bls.gov/cps/cpsaat11.htm). We also use BUG’s gold dataset
to verify overall (stereotypical/non-stereotypical balanced) performance.

Given the structure of the templates of the MultilingualHolisticBias
sentences, it is possible that the model learns to force the grammatical
gender of every word into masculine or feminine, rather than only setting the
gender when there is ambiguity. In a general domain, where ambiguities are not
prevalent, the masculine and the feminine translations should have minimal
variations among each other. We translate FLoRes’s test set to a group of
languages and verify the similarity between masculine and feminine outputs.

###  4.2 Models

We compare the performance of two models with different architectures on the
task of MT: LLaMA Touvron et al. (2023a), a decoder-only model and NLLB NLLB
Team et al. (2022), an encoder-decoder model.

We use the NLLB-200 version with 3 billion parameters. For generation, we
follow the standard setting (beam search with beam size 5, limiting the
translation length to 100 tokens).

For LLaMa we use the 7 billion parameters version. We prompt LLaMa with ICEs
(Figure 1) to elicit the translation task.

###  4.3 Evaluation

To facilitate comparison with previous works using MultilingualHolisticBias
Costa-jussà et al. (2023), we use the sacrebleu implementation of spBLEU Goyal
et al. (2021) to compute the translation quality with ‘add-k = 1’ smoothing.
When using MultilingualHolisticBias, to avoid test set contamination, we
employ a leave-one-out cross-validation (LOOCV) strategy to ensure no example
is used as ICE of itself. This is possible because the test set sentences are
independent of each other. We run evaluations with masculine, feminine, and
both references. The first two evaluations are meant to establish the
preference of the translation toward each gender, while the third evaluation
establishes the correctness of the translation, regardless of the employed
gender.

Together with BLEU, we use Stanovsky et al.’s (2019) reference-less
coreference resolution metric as a way of determining the level of gender bias
in the translations.

##  5 Analysis and Results

###  5.1 Producing gender-specific translations with LLaMa

We investigate the capability of LLaMa to learn to produce gender-specific
translations with ICL. We prompt LLaMa with 5, 16, and 32 ICEs produced with
the masculine and feminine translations from MultilingualHolisticBias. In case
no masculine or feminine translation is available for a given example, we use
the neutral translation. Hereinafter all experiments are performed with this
setting.

For NLLB, we calculate three BLEU scores on the output: one with the masculine
reference, one with the feminine reference and one with both. In the case of
LLaMa, we calculate two BLEU score for each gender-specific output: one with
the corresponding gender’s reference and one with both references, for a total
of four BLEU scores per generation.

After prompting LLaMa with the gender-specific template shown in Figure 1, we
find that LLaMa’s performance surpasses NLLB for all four reported metrics in
7 out of 25 languages, with NLLB surpassing LLaMa in 10 out of 25 languages.
If we exclude the languages for which LLaMa in which LLaMa hasn’t been
explicitly trained, we find that LLaMa surpasses NLLB in 5 out of 14
languages, while NLLB surpasses LLaMa in 4 out of 14 languages. Overall, BLEU
scores for LLaMa are on average less than 1 point lower than with NLLB. We
summarize the results in Table 2.

| NLLB | LLaMa | $\Delta$
---|---|---|---
arb | 31.65 | 10.42 | -21.23
bel | 10.92 | 16.82 | 5.90
bul | 44.22 | 12.54 | -31.68
cat | 37.93 | 56.87 | 18.94
ces | 19.52 | 17.13 | -2.39
dan | 54.32 | 49.69 | -4.63
deu | 23.62 | 41.81 | 18.19
ell | 37.39 | 17.62 | -19.77
fra | 52.59 | 68.36 | 15.77
ita | 31.69 | 35.52 | 3.83
lit | 26.90 | 14.81 | -12.09
lvs | 15.24 | 11.25 | -3.99
mar | 19.76 | 23.42 | 3.66
nld | 24.66 | 24.16 | -0.50
por | 37.30 | 27.76 | -9.54
ron | 22.11 | 20.08 | -2.03
rus | 35.46 | 30.59 | -4.87
slk | 40.89 | 25.09 | -15.80
slv | 33.11 | 3.30 | -29.81
spa | 57.60 | 67.33 | 9.73
swe | 45.94 | 36.20 | -9.74
tam | 14.25 | 10.65 | -3.60
tha | 14.63 | 19.17 | 4.54
ukr | 17.72 | 32.51 | 14.79
urd | 9.92 | 26.43 | 16.51
Average |  |  | -1.96
Average* |  |  | -0.93
Table 2: Average BLEU scores across the four evaluation metrics (masculine
reference, feminine reference, masculine to both references, feminine to both
references) for gender-specific generation in LLaMa-7B compared to NLLB-3B. To
make the results comparable, we have included twice the evaluation with both
references of NLLB in the reported averages. In the case of LLaMa, we selected
the best result out the three sets of experiments with different number of
ICEs (5, 16 and 32). The underlined values denote the best-performing model
per language. The value Average* denotes the average among only the languages
explicitly present in LLaMa.

We see a clear trend (with exceptions) of performance increase as the number
of ICEs grows, which suggests that further gains are still possible. In some
cases, jumping from 16 to 32 ICEs makes LLaMa miss the task specification,
hallucinating a set of incoherent characters or either partially or totally
failing to produce an output. The similarity of MultilingualHolisticBias’s
ICEs could play a big role in the competitiveness of the performance, which
highlights the importance of relevant examples for ICL. The full results can
be found in Appendix B.

Given these positive results, we wanted to estimate the significance of the
gender-specificity when translating. We inverted the references for each
gender of the LLaMa output (evaluating masculine translations with their
corresponding feminine reference and vice versa). We found a steep decrease in
performance with the wrong references (Table 3), which suggests a strong
gender-specificity of the dataset with respect to the BLEU score and validates
the significance of the experiments performed above. Detailed numbers are
available in Appendix B.

| right ref. | wrong ref. | $\Delta$
---|---|---|---
arb | 10.31 | 3.03 | -7.28
bel | 16.72 | 12.46 | -4.26
bul | 12.36 | 6.57 | -5.79
cat | 56.32 | 28.96 | -27.36
ces | 16.98 | 13.73 | -3.25
dan | 49.6 | 35.91 | -13.69
deu | 41.52 | 28.39 | -13.13
ell | 17.42 | 7.52 | -9.9
fra | 68.13 | 55.45 | -12.68
ita | 35.18 | 13.3 | -21.88
lit | 14.54 | 4.78 | -9.76
lvs | 11.14 | 2.13 | -9.01
mar | 23.38 | 4.32 | -19.06
nld | 24.14 | 14.43 | -9.71
por | 27.66 | 19.13 | -8.53
ron | 19.7 | 14.23 | -5.47
rus | 30.43 | 23.7 | -6.73
slk | 24.7 | 10.19 | -14.51
slv | 2.24 | 4.05 | 1.81
spa | 67.18 | 40.69 | -26.49
swe | 35.75 | 22.05 | -13.7
tam | 22.71 | 3.62 | -19.09
tha | 23.81 | 3.72 | -20.09
ukr | 32.19 | 19.84 | -12.35
urd | 26.3 | 20.09 | -6.21
Average |  |  | -11.93
Table 3: Average BLEU score across a number of ICEs and gender of using the
right gender reference for evaluation vs using the wrong reference. The
underlined values denote the best-performing evaluation.

###  5.2 Gender bias evaluation of LLaMa

To assess the gender bias of LLaMa’s gender-specific translations, we
translate some of the test sets of WinoMT and BUG, as described in Section
4.1, with MultilingualHolisticBias as ICEs. These test sets present examples
that require unambiguous coreference resolution or grammatical gender
utilization regardless of stereotypical associations. Stanovsky et al. (2019)
and Levy et al. (2021) found that several (encoder-decoder) NMTs are
significantly prone to translate based on gender stereotypes rather than more
meaningful context. We verify to which degree these errors are reproduced by
LLaMa in gender-specific translations. As a baseline, we translate the
selected WinoMT and BUG test sets with NLLB as well.

When performing the translation of WinoMT’s and BUG’s datasets, we found that
the phenomenon of empty or incomplete (only one of our two genders) outputs
continues to occur. Since a gender bias analysis is not defined over an empty
sentence, we excluded incomplete outputs for our gender bias evaluation.

We report balanced results for the stereotypical dataset in Table 4. We find
that LLaMa tends to perform better on high-resource languages it has likely
been more exposed to, with NLLB surpassing LLaMa 4 out of 7 times and LLaMa
overperforming NLLB 3 out of 7 times. In all cases, except for Russian and
Ukrainian, both systems have a higher-than-random accuracy, with improved
performance on the male side over the female side.

| NLLB | LLaMa (m.) | LLaMa (f.)
---|---|---|---
acc. ($\uparrow$) |  $\Delta_{G}$ ($\downarrow$) | acc. | $\Delta_{G}$ | acc. | $\Delta_{G}$
arb | 80.6 | 7.2 | 55.0 | 22.6 | 55.1 | 22.8
ces | 76.0 | 10.0 | 55.3 | 15.6 | 55.3 | 15.6
deu | 78.0 | 7.1 | 80.9 | 6.9 | 80.9 | 6.9
ita | 58.6 | 20.0 | 63.3 | 19.7 | 63.3 | 19.6
spa | 69.3 | 13.2 | 77.5 | 5.9 | 77.5 | 6.0
rus | 55.3 | 37.9 | 47.5 | 40.2 | 47.0 | 39.6
ukr | 43.4 | 36.5 | 38.5 | 22.9 | 38.5 | 22.8
Table 4: Noun gender prediction accuracy on the subset of WinoMT’s
stereotypical dataset’s fully generated gender-specific translations with
LLaMa, compared to NLLB’s prediction accuracy. $\Delta_{G}$ denotes the
F1-score difference between male and female nouns.

When analyzing the results for the anti-stereotypical dataset (Table 5), the
performance of LLaMa is almost on par with NLLB, albeit almost always
inferior. Overall, the average difference is of little more than 1% of
accuracy between both systems.

| NLLB | LLaMa (m.) | LLaMa (f.)
---|---|---|---
acc. ($\uparrow$) |  $\Delta_{G}$ ($\downarrow$) | acc. | $\Delta_{G}$ | acc. | $\Delta_{G}$
arb | 49.4 | 31.2 | 45.7 | 33.4 | 45.5 | 36.6
ces | 55.0 | 30.3 | 50.3 | 7.7 | 50.2 | 7.6
deu | 59.5 | 16.4 | 52.1 | 24.1 | 52.1 | 24.1
ita | 36.7 | 31.5 | 35.4 | 24.9 | 35.3 | 24.8
spa | 47.1 | 38.8 | 45.6 | 32.7 | 45.6 | 32.7
rus | 32.5 | 4.1 | 29.9 | 0.9 | 29.9 | 0.9
ukr | 31.9 | 36.4 | 45.0 | 8.2 | 45.0 | 8.3
Table 5: Noun gender prediction accuracy on the subset of WinoMT’s anti-
stereotypical dataset’s fully generated gender-specific translations with
LLaMa, compared to NLLB’s prediction accuracy. $\Delta_{G}$ denotes the
F1-score difference between male and female nouns.

However, the performance on BUG’s gold dataset, a balanced dataset constructed
from a more diverse domain, favors NLLB over LLaMa in almost every language,
except for Italian and Spanish, as can be seen on Table 6.

It is interesting to see that, in all three analyzed sets, the difference in
performance between LLaMa’s male and female outputs is almost non-existent,
suggesting a reliance on coreference resolution during the gender-specific
translation task, rather than forcing the gender of the nouns in the
translation. Our results suggest very little variance across languages, gender
outputs, and number of ICEs, further backing the hypothesis that when
prompting LLaMa with the proposed gender-specific template, it does a more
complex analysis than simple gender matching, relying on more complex methods
such as coreference resolution to determine each of the outputs.

| NLLB | LLaMa (m.) | LLaMa (f.)
---|---|---|---
acc. ($\uparrow$) |  $\Delta_{G}$ ($\downarrow$) | acc. | $\Delta_{G}$ | acc. | $\Delta_{G}$
arb | 69.7 | 18.1 | 46.8 | 48.0 | 46.8 | 47.9
ces | 71.9 | 16.9 | 39.2 | 25.2 | 39.2 | 25.2
deu | 79.3 | 24.6 | 78.3 | 32.5 | 78.3 | 32.5
ita | 50.5 | 27.1 | 52.3 | 39.9 | 52.3 | 39.9
spa | 53.5 | 33.7 | 53.3 | 37.4 | 53.3 | 37.4
rus | 47.5 | 20.9 | 40.9 | 40.4 | 40.9 | 40.4
ukr | 45.6 | 24.9 | 43.2 | 11.5 | 43.2 | 11.5
Table 6: Noun gender prediction accuracy on the subset of BUG’s gold dataset’s
fully generated gender-specific translations with LLaMa, compared to NLLB’s
prediction accuracy. $\Delta_{G}$ denotes the F1-score difference between male
and female nouns.

###  5.3 Generalizing to the general domain: generating gender-specific
translations from FLoRes dataset

We already verified in Subsection 5.1 the high significance of gender for the
accuracy of the outputs. We also assess the difference in performance for each
produced gender when there aren’t major gender ambiguities to translate. In
this case, a robust model should not have significant differences between both
genders. We translate FLoRes’s test set into a subset of the languages used in
previous experiments. Given that FLoRes is a general domain dataset,
ambiguities should not be prevalent and both outputs should tend to be the
same. We use MultilingualHolisticBias as ICEs, reproducing the setting in Sub-
section 5.1 and we compare the BLEU scores of both outputs. The list of
languages we translate into for this experiment can be found in Table 10.

| 5 | 16 | 32
---|---|---|---
masc. | fem. | $\Delta$ | masc. | fem. | $\Delta$ | masc. | fem. | $\Delta$
bel | 1.07 | 1.11 | 0.04 | 1.25 | 1.29 | 0.04 | 0.00 | 2.29 | 2.29
cat | 25.97 | 25.02 | 0.95 | 26.07 | 25.27 | 0.80 | 26.62 | 26.23 | 0.39
ces | 16.47 | 16.42 | 0.05 | 15.15 | 14.92 | 0.23 | 15.55 | 15.36 | 0.19
dan | 25.05 | 25.01 | 0.04 | 25.45 | 25.05 | 0.40 | 25.31 | 25.09 | 0.22
deu | 26.33 | 25.54 | 0.21 | 26.47 | 23.74 | 2.73 | 28.14 | 24.12 | 4.02
fra | 37.39 | 37.04 | 0.35 | 38.20 | 37.74 | 0.46 | 37.73 | 37.21 | 0.52
ita | 24.23 | 23.88 | 0.35 | 24.96 | 24.48 | 0.48 | 25.44 | 23.50 | 1.94
lit | 2.17 | 2.02 | 0.15 | 2.03 | 1.99 | 0.04 | 2.23 | 2.18 | 0.05
lvs | 1.60 | 1.57 | 0.03 | 1.80 | 1.69 | 0.11 | 1.73 | 1.64 | 0.09
nld | 23.08 | 22.53 | 0.55 | 23.74 | 22.04 | 1.70 | 23.21 | 18.26 | 4.95
por | 39.28 | 38.40 | 0.88 | 38.88 | 37.89 | 0.99 | 39.11 | 37.87 | 1.24
ron | 24.38 | 24.18 | 0.20 | 25.31 | 25.17 | 0.14 | 25.07 | 24.92 | 0.15
rus | 19.82 | 19.86 | 0.04 | 21.53 | 21.46 | 0.07 | 21.59 | 21.59 | 0.00
spa | 24.45 | 24.05 | 0.40 | 24.57 | 23.78 | 0.79 | 25.03 | 23.77 | 0.74
swe | 30.07 | 29.26 | 0.81 | 30.41 | 29.67 | 0.74 | 30.76 | 30.25 | 0.51
tha | 2.10 | 1.74 | 0.36 | 1.69 | 1.50 | 0.19 | 0.00 | 0.00 | 0.00
ukr | 19.63 | 19.59 | 0.04 | 19.53 | 19.27 | 0.26 | 19.26 | 19.04 | 0.22
avg |  |  | 0.31 |  |  | 0.63 |  |  | 1.02
Table 7: BLEU scores for each output of LLaMa’s gender-specific translation on
FLoRes’s testset. $\Delta$ denotes the difference between male and female
translations. Since FLoRes’s sentences are not expected to contain a high rate
of ambiguity, a correct translation should tend to be identical in both
outputs. Results are given for 5, 16 and 32 in-context examples.

The results show minor differences between both genders, suggesting a robust
gender-specific generation based on coreference resolution rather than on
mechanically switching the grammatical gender of the words of the sentence.
This result is aligned with what experiments in Sub-section 5.2 indicated.

##  6 Conclusions

In this paper, we explored the capabilities and limitations of decoder-only
LLMs in producing gender-specific translations. Our study encompassed a range
of experiments across 25 languages and four datasets using LLaMa-7B, with NLLB
as a baseline.

To gauge LLaMa’s capacity for gender-specific translations, we employed
gender-specific templates in our prompts. We observed that LLaMa outperforms
NLLB 7 out of 25 times vs 10 out of 25 times, with NLLB scoring under 2 BLEU
points on average above LLaMa and under 1 BLEU points when only measuring
languages explicitly included in LLaMa. This suggests that it is possible to
elicit accurate gender-specific translations with carefully constructed prompt
templates and highly relevant ICEs, but more work is required to get LLMs on
par with encoder-decoder models such as NLLB.

To analyze the gender bias of the gender-specific translations produced by
LLaMa, we employed an automatic reference-less metric based on correct
coreference resolution. We show that, while exhibiting slightly higher gender
bias than NLLB translations, LLaMa’s gender-specific translations consistently
ranked above random for coreference resolution, suggesting a competitive
baseline to improve the task of gender-specific MT.

Furthermore, we examined LLaMa’s translation performance on a general-domain
benchmark (FLoRes). We found that LLaMa’s translations tended to converge to a
similar output for both masculine and feminine genders, reaffirming the
model’s robustness to contextual ambiguity.

In conclusion, our study shed light on the capabilities of LLMs to produce
gender-specific translations, and the role of ICEs to elicit the task
successfully. Our results indicate that LLaMa can generate gender-specific
translations with competitive accuracy and reduced gender bias when compared
to NLLB. Our experiments also reveal that LLaMa’s translations are robust,
showing significant performance drops when evaluated against opposite-gender
references in gender-ambiguous datasets but maintaining consistency in less
ambiguous contexts. While the quality of the gender-specific translations is
competitive, they continue to fall slightly behind NMT models in accuracy and
gender bias. These shortcomings should be addressed if the advantages and
flexibility of decoder-only models are to be leveraged as an alternative to
encoder-decoder models for Machine Translation.

## Limitations

Even though we performed a diverse set of experiments, some limitations arise
due to the vastness of the research space we’re dealing with. The study
heavily relies on the effectiveness of prompt engineering, specifically in
providing accurate ICEs. The conclusions drawn are thus constrained by the
quality and relevance of the prompts used. Variations in prompt structure or
content could yield different results. Moreover, the gender bias dataset used
for fine-tuning can inherently introduce biases, impacting the model’s
behavior in gender-related translation scenarios.

The study focuses on a particular model, LLaMa, leaving out an exploration of
alternative architectures that could yield different results. A notable
constraint is the version of the model employed in this study. At the time of
the performance of the experiments, the latest version of LLaMa was LLaMa-1
(simply known as LLaMa). At the time of publishing of this paper, the latest
LLaMa version is LLaMa-2 (Touvron et al., 2023b). The results with LLaMa-2 are
likely to be greater or equal to those with LLaMa-1, opening up the
possibility of more nuanced conclusions about the performance of LLMs with
respect to encoder-decoder NMT systems. Additionally, LLaMa-2 includes an
instruction-finetuned version, which could mitigate some of the issues
encountered (e.g., hallucinations) and could reduce the number of ICEs needed
to elicit the task of gender-specific translation.

The study primarily offers quantitative analysis, often overlooking
qualitative insights that could provide a richer understanding of the
effectiveness and limitations of gender-related translations. Additionally,
the evaluation focuses on short contexts, which might not encapsulate the
complexities of real-world, long-form content. Moreover, the evaluation method
of BLEU can excessively penalize errors in short sentences, leading to some
distortions in the results of the experiments with MultilingualHolisticBias. A
character-level metric like chrF (Popović, 2015) could be used in future works
to address this shortcoming.

In conclusion, while our study contributes valuable insights into the
landscape of gender-related translations and bias mitigation in the context of
machine learning, its limitations underscore the need for continued research.
Adapting to diverse contextual gender nuances, improving evaluation
methodologies, exploring alternative model architectures, addressing biases
comprehensively, and considering wider ethical and societal implications are
all directions that future research could take to build upon the foundation
laid by this study.

## Ethics Statement

The evaluation of translation quality heavily relies on both automatic metrics
and subjective human assessments. While metrics like BLEU offer quantitative
insights, they might not capture the entirety of translation quality nuances.
Human judgments are subjective and can vary based on individual preferences
and biases, potentially affecting the overall evaluation process.

The understanding of nuanced gender contexts is intricate and can be
challenging even for humans. The study tends to approach gender in a binary
manner, which might not account for social perceptions among some of the users
of these languages. This limitation is inherent in the current state of the
field and warrants future investigations into better representation and
handling of gender-related nuances.

Furthermore, although the MultilingualHolisticBias dataset presents an
awareness of other biases like racial bias, it predominantly addresses gender
bias, potentially overlooking a spectrum of biases present in language models.
Moreover, the stereotypical and non-stereotypical datasets were built based on
the US Department of Labor data. Since we work with a variety of world
languages, the proportions stated on these datasets might not reflect the
realities of the users of the wide range of languages employed in this study.

## References

  * Agrawal et al. (2023) Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023.  In-context examples selection for machine translation.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 8857–8873, Toronto, Canada. Association for Computational Linguistics.
  * Bawden and Yvon (2023) Rachel Bawden and François Yvon. 2023.  Investigating the translation performance of a large multilingual language model: the case of bloom.
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020.  Language (technology) is power: A critical survey of “bias” in NLP.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5454–5476, Online. Association for Computational Linguistics.
  * Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai. 2016.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  In _NIPS_.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Caliskan et al. (2017) Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017.  Semantics derived automatically from language corpora contain human-like biases.  _Science_ , 356(6334):183–186.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022\.  Palm: Scaling language modeling with pathways.
  * Corbett (2007) Greville G. Corbett. 2007.  _Gender and noun classes_, 2 edition, volume 3, page 241–279. Cambridge University Press.
  * Costa-jussà (2019) Marta R. Costa-jussà. 2019.  An analysis of gender bias studies in natural language processing.  _Nature Machine Intelligence_ , 1(11):495–496.
  * Costa-jussà et al. (2023) Marta R. Costa-jussà, Pierre Andrews, Eric Smith, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cynthia Gao, Daniel Licht, and Carleigh Wood. 2023.  Multilingual holistic bias: Extending descriptors and patterns to unveil demographic biases in languages at scale.
  * Costa-jussà et al. (2022) Marta R. Costa-jussà, Carlos Escolano, Christine Basta, Javier Ferrando, Roser Batlle, and Ksenia Kharitonova. 2022.  Interpreting gender bias in neural machine translation: Multilingual architecture matters.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(11):11855–11863.
  * Craig (1986) Colette G. Craig, editor. 1986.  _Noun Classes and Categorization_.  John Benjamins Publishing Company.
  * Dong et al. (2023) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023.  A survey on in-context learning.
  * Fleisig and Fellbaum (2022) Eve Fleisig and Christiane Fellbaum. 2022.  Mitigating gender bias in machine translation through adversarial learning.
  * Garcia et al. (2023) Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Fangxiaoyu Feng, Melvin Johnson, and Orhan Firat. 2023.  The unreasonable effectiveness of few-shot learning for machine translation.
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020\.  Realtoxicityprompts: Evaluating neural toxic degeneration in language models.  In _Findings_.
  * Goyal et al. (2021) Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2021.  The flores-101 evaluation benchmark for low-resource and multilingual machine translation.
  * Guerreiro et al. (2023) Nuno M. Guerreiro, Duarte Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023.  Hallucinations in large multilingual translation models.
  * Guzmán et al. (2019) Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019.  Two new evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english.
  * Hendy et al. (2023) Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023.  How good are gpt models at machine translation? a comprehensive evaluation.
  * Jiao et al. (2023) Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023.  Is chatgpt a good translator? yes with gpt-4 as the engine.
  * Johnson (2020) Melvin Johnson. 2020.  A scalable approach to reducing gender bias in google translate.  Accessed: September 5th, 2023.
  * Levy et al. (2021) Shahar Levy, Koren Lazar, and Gabriel Stanovsky. 2021.  Collecting a large-scale gender bias dataset for coreference resolution and machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2470–2480, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lin et al. (2021) Stephanie C. Lin, Jacob Hilton, and Owain Evans. 2021.  Truthfulqa: Measuring how models mimic human falsehoods.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Měchura (2022) Michal Měchura. 2022.  A taxonomy of bias-causing ambiguities in machine translation.  In _Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP)_ , pages 168–173, Seattle, Washington. Association for Computational Linguistics.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Moslem et al. (2023) Yasmin Moslem, Rejwanul Haque, John D. Kelleher, and Andy Way. 2023.  Adaptive machine translation with large language models.
  * Mucchi-Faina (2005) Angelica Mucchi-Faina. 2005.  Visible or influential? language reforms and gender (in)equality.  _Social Science Information_ , 44(1):189–215.
  * NLLB Team et al. (2022) NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022.  No language left behind: Scaling human-centered machine translation.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Pilault et al. (2023) Jonathan Pilault, Xavier Garcia, Arthur Bražinskas, and Orhan Firat. 2023.  Interactive-chain-prompting: Ambiguity resolution for crosslingual conditional generation with interaction.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Prates et al. (2018) Marcelo O. R. Prates, Pedro H. C. Avelar, and Luís C. Lamb. 2018.  Assessing gender bias in machine translation - A case study with google translate.  _CoRR_ , abs/1809.02208.
  * Rudinger et al. (2018) Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018.  Gender bias in coreference resolution.  In _North American Chapter of the Association for Computational Linguistics_.
  * Saunders and Byrne (2020) Danielle Saunders and Bill Byrne. 2020.  Reducing gender bias in neural machine translation as a domain adaptation problem.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7724–7736, Online. Association for Computational Linguistics.
  * Shaikh et al. (2023) Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. 2023\.  On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4454–4470, Toronto, Canada. Association for Computational Linguistics.
  * Smith et al. (2022) Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. 2022.  “I’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9180–9211, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Stafanovičs et al. (2020) Artūrs Stafanovičs, Toms Bergmanis, and Mārcis Pinnis. 2020.  Mitigating gender bias in machine translation with target gender annotations.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 629–638, Online. Association for Computational Linguistics.
  * Stanovsky et al. (2019) Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019.  Evaluating gender bias in machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1679–1684, Florence, Italy. Association for Computational Linguistics.
  * Tanwar et al. (2023) Eshaan Tanwar, Subhabrata Dutta, Manish Borthakur, and Tanmoy Chakraborty. 2023\.  Multilingual LLMs are better cross-lingual in-context learners with alignment.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6292–6307, Toronto, Canada. Association for Computational Linguistics.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a.  Llama: Open and efficient foundation language models.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b.  Llama 2: Open foundation and fine-tuned chat models.
  * Wang et al. (2022) Jun Wang, Benjamin Rubinstein, and Trevor Cohn. 2022.  Measuring and mitigating name biases in neural machine translation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2576–2590, Dublin, Ireland. Association for Computational Linguistics.
  * Wei et al. (2023a) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023a.  Chain-of-thought prompting elicits reasoning in large language models.
  * Wei et al. (2023b) Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. 2023b.  Larger language models do in-context learning differently.
  * Zhang et al. (2023) Biao Zhang, Barry Haddow, and Alexandra Birch. 2023.  Prompting large language model for machine translation: A case study.
  * Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018\.  Gender bias in coreference resolution: Evaluation and debiasing methods.  In _North American Chapter of the Association for Computational Linguistics_.
  * Zhu et al. (2023) Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023.  Multilingual machine translation with large language models: Empirical results and analysis.

##  Appendix A Languages

Language code | Name | Script
---|---|---
arb | Modern Standard Arabic | Arabic
bel | Belarusian | Cyrillic
bul | Bulgarian | Cyrillic
cat | Catalan | Latin
ces | Czech | Latin
dan | Danish | Latin
deu | German | Latin
ell | Greek | Greek
fra | French | Latin
ita | Italian | Latin
lit | Lithuanian | Latin
lvs | Standard Latvian | Latin
mar | Marathi | Devanagari
nld | Dutch | Latin
por | Portuguese | Latin
ron | Romanian | Latin
rus | Russian | Cyrillic
slk | Slovak | Latin
slv | Slovenian | Latin
spa | Spanish | Latin
swe | Swedish | Latin
tam | Tamil | Tamil
tha | Thai | Thai
ukr | Ukrainian | Cyrillic
urd | Urdu | Arabic
Table 8: Complete list of languages analyzed in this work. Language code | Name | Script
---|---|---
arb | Modern Standard Arabic | Arabic
ces | Czech | Latin
deu | German | Latin
ita | Italian | Latin
rus | Russian | Cyrillic
spa | Spanish | Latin
ukr | Ukrainian | Cyrillic
Table 9: List of languages to which WinoMT and BUG datasets are translated. Language code | Name | Script
---|---|---
bel | Belarusian | Cyrillic
cat | Catalan | Latin
ces | Czech | Latin
dan | Danish | Latin
deu | German | Latin
fra | French | Latin
ita | Italian | Latin
lit | Lithuanian | Latin
lvs | Standard Latvian | Latin
nld | Dutch | Latin
por | Portuguese | Latin
ron | Romanian | Latin
rus | Russian | Cyrillic
spa | Spanish | Latin
swe | Swedish | Latin
tha | Thai | Thai
ukr | Ukrainian | Cyrillic
Table 10: Complete list of languages evaluated in FLoRes’s test set.

##  Appendix B Full results

| masc. | fem. | both refs. (m.) | both refs. (f.)
---|---|---|---|---
NLLB | LLaMa | NLLB | LLaMa | NLLB | LLaMa | NLLB | LLaMa
|  | 8.02 |  | 7.72 |  | 8.1 |  | 8.25
arb | 33.95 | 9.91 | 24.07 | 10.7 | 34.29 | 9.96 | 34.29 | 11.12
|  | 0.3 |  | 0.15 |  | 0.3 |  | 0.18
|  | 13.69 |  | 18.51 |  | 14.04 |  | 18.52
bel | 10.72 | 13.32 | 10.47 | 19.74 | 11.24 | 13.55 | 11.24 | 19.82
|  | 1.42 |  | 1.13 |  | 1.43 |  | 1.9
|  | 11.22 |  | 13.5 |  | 10.99 |  | 14.43
bul | 46.6 | 10.82 | 36.1 | 13.31 | 47.14 | 10.68 | 47.14 | 14.13
|  | 9.4 |  | 11.53 |  | 9.33 |  | 12.57
|  | 52.12 |  | 55.56 |  | 51.55 |  | 58.25
cat | 41.96 | 49.85 | 25.49 | 54.13 | 42.14 | 49.08 | 42.14 | 56.91
|  | 53.9 |  | 58.73 |  | 53.46 |  | 61.4
|  | 10.37 |  | 10.26 |  | 10.98 |  | 10.64
ces | 19.19 | 15.32 | 19.21 | 14.47 | 19.83 | 15.8 | 19.83 | 14.77
|  | 17.43 |  | 16.53 |  | 17.82 |  | 16.72
|  | 43.3 |  | 45.33 |  | 43.03 |  | 46.09
dan | 57.16 | 39.22 | 45.61 | 41.03 | 57.25 | 38.95 | 57.25 | 41.75
|  | 48.59 |  | 50.61 |  | 48.29 |  | 51.27
|  | 35.21 |  | 33.56 |  | 35.64 |  | 34.23
deu | 25.15 | 40.63 | 19.04 | 39.88 | 25.14 | 41.2 | 25.14 | 40.54
|  | 42.98 |  | 40.05 |  | 43.33 |  | 40.89
|  | 15.19 |  | 14.75 |  | 15.46 |  | 15.31
ell | 40.08 | 17.86 | 29.31 | 16.98 | 40.09 | 17.9 | 40.09 | 17.72
|  | 0.0 |  | 0.0 |  | 0.0 |  | 0.0
|  | 61.61 |  | 62.31 |  | 61.63 |  | 63.34
fra | 56.06 | 67.85 | 42.46 | 68.4 | 55.91 | 67.75 | 55.91 | 69.42
|  | 66.42 |  | 68.02 |  | 66.31 |  | 68.86
|  | 28.27 |  | 32.82 |  | 27.81 |  | 34.44
ita | 33.33 | 30.56 | 24.24 | 34.97 | 34.6 | 29.98 | 34.6 | 36.88
|  | 32.31 |  | 38.05 |  | 31.83 |  | 39.9
|  | 10.51 |  | 11.05 |  | 10.77 |  | 12.8
lit | 29.98 | 14.41 | 16.37 | 14.66 | 30.63 | 14.71 | 30.63 | 15.47
|  | 13.88 |  | 14.42 |  | 14.19 |  | 14.8
|  | 5.0 |  | 11.71 |  | 5.03 |  | 12.22
lvs | 16.53 | 7.64 | 10.79 | 14.63 | 16.82 | 7.64 | 16.82 | 15.09
|  | 7.12 |  | 14.28 |  | 7.19 |  | 14.56
|  | 16.21 |  | 24.88 |  | 16.23 |  | 25.28
mar | 19.86 | 19.25 | 16.48 | 27.5 | 21.35 | 19.32 | 21.35 | 27.61
|  | 0.0 |  | 0.0 |  | 0.0 |  | 0.0
|  | 19.74 |  | 23.02 |  | 19.25 |  | 23.34
nld | 26.33 | 20.1 | 19.68 | 23.34 | 26.33 | 19.79 | 26.33 | 23.81
|  | 22.19 |  | 26.08 |  | 21.86 |  | 26.49
|  | 23.96 |  | 26.21 |  | 23.85 |  | 27.04
por | 39.52 | 25.64 | 31.78 | 29.53 | 38.96 | 25.47 | 38.96 | 30.47
|  | 25.79 |  | 28.98 |  | 25.5 |  | 29.76
|  | 16.61 |  | 17.35 |  | 17.71 |  | 19.33
ron | 22.26 | 17.07 | 19.37 | 17.92 | 23.4 | 18.05 | 23.4 | 19.06
|  | 18.89 |  | 20.5 |  | 19.67 |  | 21.26
|  | 19.99 |  | 18.93 |  | 19.93 |  | 19.82
rus | 36.3 | 30.8 | 32.15 | 28.44 | 36.69 | 30.9 | 36.69 | 29.14
|  | 31.79 |  | 29.06 |  | 31.72 |  | 29.78
|  | 15.89 |  | 16.35 |  | 16.12 |  | 18.47
slk | 45.09 | 21.05 | 27.52 | 20.18 | 45.48 | 21.31 | 45.48 | 20.39
|  | 24.94 |  | 24.46 |  | 25.16 |  | 25.28
|  | 0.0 |  | 0.47 |  | 0.0 |  | 0.92
slv | 35.39 | 0.0 | 24.35 | 0.36 | 36.36 | 0.0 | 36.36 | 0.79
|  | 0.0 |  | 4.48 |  | 0.0 |  | 8.73
|  | 59.4 |  | 62.08 |  | 59.04 |  | 64.23
spa | 63.21 | 63.95 | 40.19 | 66.34 | 63.51 | 63.67 | 63.51 | 68.06
|  | 66.22 |  | 68.14 |  | 65.59 |  | 69.35
|  | 29.57 |  | 34.72 |  | 29.47 |  | 36.7
swe | 49.33 | 31.52 | 36.81 | 36.6 | 48.81 | 31.42 | 48.81 | 38.74
|  | 33.04 |  | 38.46 |  | 32.96 |  | 40.32
|  | 8.61 |  | 12.05 |  | 8.69 |  | 13.24
tam | 14.25 | 3.81 | 13.99 | 3.37 | 14.37 | 3.93 | 14.37 | 4.35
|  | 0.0 |  | 0.0 |  | 0.0 |  | 0.0
|  | 7.08 |  | 23.64 |  | 7.08 |  | 23.64
tha | 12.83 | 9.42 | 14.82 | 28.92 | 15.43 | 9.42 | 15.43 | 28.92
|  | 0.0 |  | 0.0 |  | 0.0 |  | 0.0
|  | 26.12 |  | 24.66 |  | 26.64 |  | 25.27
ukr | 17.53 | 29.21 | 16.9 | 27.91 | 17.73 | 29.61 | 17.73 | 28.9
|  | 33.33 |  | 31.04 |  | 33.79 |  | 31.88
|  | 18.27 |  | 26.27 |  | 18.37 |  | 26.68
urd | 10.77 | 24.06 | 7.31 | 28.53 | 10.8 | 24.18 | 10.8 | 28.94
|  | 0.0 |  | 0.0 |  | 0.0 |  | 0.0
Table 11: BLEU scores for double (gender-specific) sentence generation in LLaMa-7B compared to NLLB-3B. For each language/gender combination, the results for 5 (top) 16 (middle), and 32 (bottom) in-context examples are shown. | masculine output | feminine output
---|---|---
right ref. | opposite ref. | right ref. | opposite ref.
arb | 8.02 | 1.91 | 7.72 | 2.62
9.91 | 2.67 | 10.7 | 3.39
0.3 | 0.2 | 0.15 | 0.17
bel | 13.69 | 11.85 | 18.51 | 13.07
13.32 | 11.45 | 19.74 | 11.15
1.42 | 0.82 | 1.13 | 1.18
bul | 11.22 | 5.78 | 13.5 | 7.36
10.82 | 5.77 | 13.31 | 7.17
9.4 | 4.89 | 11.53 | 6.55
cat | 52.12 | 26.63 | 55.56 | 30.12
49.85 | 23.9 | 54.13 | 28.59
53.9 | 26.65 | 58.73 | 31.26
ces | 10.37 | 9.78 | 10.26 | 8.31
15.32 | 13.29 | 14.47 | 10.5
17.43 | 15.04 | 16.53 | 12.42
dan | 43.3 | 30.26 | 45.33 | 30.76
39.22 | 26.67 | 41.03 | 27.09
48.59 | 35.55 | 50.61 | 36.27
deu | 35.21 | 23.8 | 33.56 | 21.53
40.63 | 28.77 | 39.88 | 26.24
42.98 | 30.26 | 40.05 | 26.52
ell | 15.19 | 5.38 | 14.75 | 5.79
17.86 | 7.04 | 16.98 | 7.99
0.0 | 0.0 | 0.0 | 0.0
fra | 61.61 | 49.14 | 62.31 | 48.91
67.85 | 54.82 | 68.4 | 56.07
66.42 | 53.34 | 68.02 | 55.33
ita | 28.27 | 11.79 | 32.82 | 13.73
30.56 | 10.8 | 34.97 | 13.16
32.31 | 11.63 | 38.05 | 14.8
lit | 10.51 | 2.91 | 11.05 | 5.27
14.41 | 4.34 | 14.66 | 5.21
13.88 | 4.29 | 14.42 | 4.27
lvs | 5.0 | 0.98 | 11.71 | 2.06
7.64 | 1.43 | 14.63 | 2.38
7.12 | 1.87 | 14.28 | 2.2
mar | 16.21 | 3.37 | 24.88 | 3.58
19.25 | 4.51 | 27.5 | 4.13
0.0 | 0.0 | 0.0 | 0.0
nld | 19.74 | 11.59 | 23.02 | 11.92
20.1 | 12.9 | 23.34 | 13.76
22.19 | 13.86 | 26.08 | 15.0
por | 23.96 | 16.41 | 26.21 | 17.66
25.64 | 17.53 | 29.53 | 20.5
25.79 | 17.75 | 28.98 | 20.27
ron | 16.61 | 13.09 | 17.35 | 13.44
17.07 | 13.17 | 17.8 | 13.53
18.89 | 14.76 | 20.5 | 13.7
rus | 28.34 | 22.91 | 27.49 | 20.58
29.77 | 24.01 | 28.91 | 21.69
31.79 | 25.28 | 29.06 | 22.11
slk | 21.53 | 9.12 | 21.13 | 8.91
23.11 | 9.72 | 22.71 | 9.51
24.94 | 10.64 | 24.46 | 9.74
slv | 0.0 | 0.0 | 3.89 | 7.1
0.0 | 0.0 | 4.48 | 8.1
0.0 | 0.0 | 4.48 | 8.1
spa | 62.02 | 38.41 | 63.13 | 38.91
64.1 | 40.12 | 65.21 | 40.62
66.22 | 40.76 | 68.14 | 39.34
swe | 30.11 | 19.02 | 34.12 | 20.03
31.56 | 20.47 | 35.57 | 21.48
33.04 | 20.8 | 38.46 | 23.3
tam | 14.31 | 2.91 | 24.82 | 3.02
17.45 | 3.56 | 27.96 | 3.67
0.0 | 0.0 | 0.0 | 0.0
tha | 15.41 | 3.01 | 25.92 | 3.12
18.55 | 3.66 | 29.06 | 3.77
0.0 | 0.0 | 0.0 | 0.0
ukr | 29.34 | 18.25 | 28.49 | 16.38
30.77 | 19.68 | 29.91 | 17.81
33.33 | 20.43 | 31.04 | 19.25
urd | 18.27 | 15.21 | 26.27 | 18.8
24.06 | 19.71 | 28.53 | 20.46
0.0 | 0.0 | 0.0 | 0.0
Table 12: Comparison of BLEU score using the right gender reference for
evaluation vs using the wrong reference for both gender-specific outputs of
LLaMa. The underlined values denote the best-performing evaluation.