# Exterior degree of infinite groups

Rashid Rezaei  Department of Mathematics, Malayer University, Malayer, Iran
<EMAIL_ADDRESS>and  Francesco G. Russo  DIEETCAM, Universitá Degli
Studi di Palermo, Viale Delle Scienze, Edificio 8, 90128, Palermo, Italy
<EMAIL_ADDRESS>
###### Abstract.

The exterior degree of a finite group has been introduced in [P. Niroomand and
R. Rezaei, On the exterior degree of finite groups, Comm. Algebra 39 (2011),
335–343] and the present paper is devoted to study the exterior degree of
infinite groups. We find some inequalities of combinatorial nature, which
generalize those of the finite case and allow us to get structural
restrictions for the whole group.

###### Key words and phrases:

Exterior degree, exterior center, centralizers.

###### 2010 Mathematics Subject Classification:

Primary 20J99; Secondary 20E10, 20P05.

##  1\. Introduction

The structure of a finite group may be strongly restricted, once we have
information about invariants which are related to the number of elements
satisfying a given property. For instance, the property of being commutative
has motivated some authors to introduce the so–called commutativity degree of
a finite group $E$, defined as the ratio

$\mathrm{d}(E)=\frac{|\\{(x,y)\in E\times E\ |\
[x,y]=1\\}|}{|E|^{2}}=\frac{1}{|E|^{2}}\underset{x\in
E}{\sum}|C_{E}(x)|=\frac{k(E)}{|E|},$

where $k(E)$ is the number of the $E$–conjugacy classes $[x]_{E}=\\{x^{g}\ |\
g\in E\\}$ that constitute $E$. There is a wide production on $\mathrm{d}(E)$
and its generalizations in the last decades and we recall here [4, 5, 6, 7,
18, 19]. The exterior degree of $E$ is a more recent invariant, studied for
similar purposes by the second author in [13] and with combinatorial
techniques in [9, 14, 16, 17]. The exterior degree of $E$ is defined as

$\mathrm{d}^{\wedge}(E)=\frac{|\\{(x,y)\in E\times E\ |\ x\wedge
y=1_{{}_{E\wedge
E}}\\}|}{|E|^{2}}=\frac{1}{|E|}\sum^{k(E)}_{i=1}\frac{|C^{\wedge}_{E}(x_{i})|}{|C_{E}(x_{i})|},\,\,\,(*)$

where the last equality is proved in [13, Lemma 2.2], and one of the main
results in [13] shows that

$\mathrm{d}^{\wedge}(E)\leq\mathrm{d}(E)$

so that the commutativity degree and the exterior degree are connected.

Since there is literature for the commutativity degree of infinite groups (see
again [5, 6, 7, 18, 19]), it is natural to look for a corresponding treatment
of the exterior degree. To the best of our knowledge, this precise point is
not available in literature and has motivated us to write the present paper.
One of the main difficulties is due to the fact that we need of a meaningful
topological structure over infinite groups, which should agree with the
discrete topology of the finite case. This forced us to look for projective
limits of finite groups, where we have a good theory of measure and may
proceed by analogy with the finite case in a compatible way with respect to
the topology which we will consider.

Section 2 is devoted to justify the use of projective limits of finite groups
in our context and we will introduce some technical notions which will be
helpful in Section 3, where the main results are placed. The terminology and
the notations will follow those of [4, 12, 13, 14, 15, 16, 17, 18, 19].

##  2\. Preliminaries

Eick [3] has recently described the low dimensional homology of an infinite
pro–$p$–group of finite coclass and central exponent $t\geq 1$

$C_{t}=\mathbb{Z}^{d_{t}}_{p}\rtimes C_{p^{t}}=\langle
t_{1},\ldots,t_{d_{t}}\rangle\rtimes\langle g\rangle=\langle
g,t_{1},\ldots,t_{d_{t}}\ |\ g^{p^{t}}=1,$
$g^{-1}t_{1}g=t^{-1}_{d_{t}},g^{-1}t_{i}g=t_{i-1}t^{-e_{i}}_{d_{t}}\ (1<i\leq
d_{t}),[t_{i},t_{j}]=1\ (1\leq j<i\leq d_{t})\rangle,$

where $p$ is a prime, $e_{i}=1$ if $p^{t-1}$ divides $i-1$, $e_{i}=0$ if
$p^{t-1}$ does not divide $i-1$, $\mathbb{Z}^{d_{t}}_{p}$ is the direct
product of $d_{t}=p^{t-1}(p-1)$ copies of the group $\mathbb{Z}_{p}$ of
$p$–adic integers, $C_{p^{t}}$ is a cyclic group of order $p^{t}$ acting
uniserially on $\mathbb{Z}^{d_{t}}_{p}$.

The above presentation of $C_{t}$ can be found in [3, Proof of Theorem 7] (see
also [10, §7]). The relevance of $C_{t}$ is emphasized by [10, Theorem 7.4.12,
Corollary 7.4.13], which characterize an arbitrary infinite pro–$p$–group of
finite coclass to be constructed always as $C_{t}$.

Before to proceed, it is good to recall some basic notions on pro–$p$–groups
from [10]. On a compact (Hausdorff) group $G$ it is possible to introduce the
filter basis

$\mathcal{P}(G)=\\{N=\overline{N}\triangleleft G\ |\ G/N\ \mathrm{is\ a\
finite\ }p-\mathrm{group}\\}$

and $G$ is said to be a pro–$p$–group if

$G=\lim_{N\in\mathcal{P}(G)}G/N,$

that is, if $G$ is a projective limit of finite $p$–groups (see [10,
Definitions 7.1.12, 7.2.1, 7.2.3, 7.2.4]). Here the open subgroups of $G$ are
exactly those closed subgroups of $p$–power index [10, Lemma 7.2.2]. Of
course, the topology of $G$ is the unique topology induced by
$\mathcal{P}(G)$. The notion of coclass of a pro–$p$–group can be found in
[10, Definition 7.4.1] and is originally due to Leedham–Green and Newman: a
finite $p$–group of order $p^{n}$ and nilpotency class $c$ has coclass r if
$n-c=r$; a pro–$p$–group $G$ has coclass $r$ if there exists some $u\geq 2$
such that $G/\gamma_{i}(G)$ has coclass $r$ for all $i\geq u$, where
$\gamma_{i}(G)$ denotes the $i$–th term of the lower central series of $G$.

From [10], the rank of a pro–$p$–group $G$ is defined by

$\mathrm{rk}(G)={\underset{H=\overline{H}\leq G}{\mathrm{sup}}}d(H),$

where $d(H)$ denotes the minimal number of elements which are necessary to
generate topologically $H$. If $G$ is a torsion–free pro–$p$–group, then
$l=\mathrm{rk}(G)=\mathrm{tf}(G)$ is called torsion–free rank. Now we know
from [3, p.148] (see also [10, §9]) that the second homology group
$H_{2}(G,\mathbb{Z}_{p})$ (with coefficients in $\mathbb{Z}_{p}$) of an
infinite pro–$p$–group $G$ of finite coclass is an abelian pro–$p$–group of
the form $H_{2}(G,\mathbb{Z}_{p})=T(G)\times F(G)$, where $T(G)$ is a finite
$p$–group and $F(G)\simeq\mathbb{Z}^{l}_{p}$. As usual in these situations,
$H_{2}(G,\mathbb{Z}_{p})$ is called Schur multiplier of $G$ and the following
bound was proved few years ago.

######  Theorem 2.1 (See [3], Theorem A).

An infinite pro–p–group $G$ of finite coclass and central exponent $t$ has
$\mathrm{tf}(H_{2}(G,\mathbb{Z}_{p}))=\frac{1}{2}d_{t}$ for all $p>2$ and
$t>1$. In particular, this is true for $G=C_{t}$.

The case $p=2$ of Theorem 2.1 needs to be treated separately (see [3, p.148
and §7]). Some of our main results deal with generalizations of Theorem 2.1
and [11, Theorems 3.2, 3.3].

From [11], the complete nonabelian tensor square $G\widehat{\otimes}G$ of a
pro–$p$–group $G$, which generalizes the nonabelian tensor square in [2], is
the group topologically generated by the symbols $x\widehat{\otimes}y$,
subject to the relations
$xy\widehat{\otimes}z=(x^{y}\widehat{\otimes}z^{y})(y\widehat{\otimes}z)$ and
$y\widehat{\otimes}tz=(y\widehat{\otimes}z)(y^{z}\widehat{\otimes}t^{z})$ for
all $x,y,z,t\in G$, where $x^{y}=y^{-1}xy$ and so on for $z^{y},y^{z},t^{z}$.
It is straightforward to check that $G\widehat{\otimes}G$ is a pro–$p$–group.
The subgroup $\widehat{\nabla}(G)=\overline{\langle x\otimes x\ |\ x\in
G\rangle}$ is a closed central subgroup of $G\widehat{\otimes}G$ and the
quotient group

$G\widehat{\otimes}G/\widehat{\nabla}(G)=G\widehat{\wedge}G$

is called complete nonabelian exterior square of the pro–$p$–group $G$. In
analogy with the homological methods which appear in [2, 12], the maps

$\widehat{\kappa}:x\widehat{\otimes}y\in G\widehat{\otimes}G\mapsto[x,y]\in
G^{\prime}\ \mathrm{and}\ \widehat{\kappa^{\prime}}:x\widehat{\wedge}y\in
G\widehat{\wedge}G\mapsto[x,y]\in G^{\prime},$

are epimorphisms of pro–$p$–groups such that
$\ker\widehat{\kappa}\simeq\widehat{\nabla}(G)$ and
$\ker\widehat{\kappa^{\prime}}\simeq H_{2}(G,\mathbb{Z}_{p})$. We note that
$H_{2}(G,\mathbb{Z}_{p})$ is a closed central subgroup of
$G\widehat{\wedge}G$. Also

$\widehat{\varepsilon}:x\widehat{\otimes}y\in G\widehat{\otimes}G\mapsto
x\widehat{\wedge}y\in G\widehat{\wedge}G\ \mathrm{and}\
\widehat{\varphi}:x\widehat{\otimes}x\in\widehat{\nabla}(G)\mapsto
x\widehat{\wedge}x\in H_{2}(G,\mathbb{Z}_{p})$

are epimorphisms of pro–$p$–groups. More details on $\widehat{\kappa}$,
$\widehat{\kappa^{\prime}}$, $\widehat{\varepsilon}$ and $\widehat{\varphi}$
can be found in [11, §2]. In virtue of all we have said, the famous diagram in
[2] becomes the following, whose rows are central extensions of
pro–$p$–groups.

$\begin{CD}1@>{}>{}>\widehat{\nabla}(G)@>{}>{}>G\widehat{\otimes}G@>{\widehat{\kappa}}>{}>G^{\prime}@>{}>{}>1\hskip
42.67912pt\\\
@V{\widehat{\varphi}}V{}V@V{\widehat{\varepsilon}}V{}V\Big{\|}\hskip
42.67912pt(**)\\\
1@>{}>{}>H_{2}(G,\mathbb{Z}_{p})@>{}>{}>G\widehat{\wedge}G@>{\widehat{\kappa^{\prime}}}>{}>G^{\prime}@>{}>{}>1\hskip
42.67912pt\\\ \end{CD}$

We have information on the topology of $G\widehat{\wedge}G$ by the next
result.

######  Theorem 2.2 (See [11], Theorem 2.1).

If $G=\lim_{N\in\mathcal{P}(G)}G/N$ is a pro–$p$–group, then
$G\widehat{\otimes}G=\lim_{(N,M)\in\mathcal{P}(G)\times\mathcal{P}(G)}G/N\otimes
G/M$ is a pro–$p$–group. The same is true for $G\widehat{\wedge}G$.

Roughly speaking, Theorem 2.2 shows a passage under projective limit for the
operator $\widehat{\wedge}$ and ensures a topological structure on
$G\widehat{\wedge}G$. This allows us to generalize the exterior centralizer
$C_{E}^{\wedge}(x)$ of an element $x$ of a finite group $E$ and the exterior
center $Z^{\wedge}(E)$, recently studied in [12, 13, 14, 15, 16, 17]. The
interest for $C_{E}^{\wedge}(x)$ and $Z^{\wedge}(E)$ is due to the fact that
they provide criteria to decide if we have a capable group or not (see [1, 12,
13]), that is, if our group is isomorphic or not to the inner automorphism of
another group.

Some basic properties, studied in [2, 12] for finite groups, may be adapted to
the infinite case in the sense of the next result.

######  Lemma 2.3.

Let $G$ be a pro–p–group and $x,y,z,t\in G$. Then

$(x^{-1}\widehat{\otimes}y)^{x}=(x\widehat{\otimes}y)^{-1}=(x\widehat{\otimes}y^{-1})^{y};\,\,\,\,\,\,\,\,\,\,(z\widehat{\otimes}t)^{xy}\
(g\widehat{\otimes}y)=(g\widehat{\otimes}y)\ (z\widehat{\otimes}t)^{yx};$
$z\widehat{\otimes}(y^{x}y^{-1})=(x\widehat{\otimes}y)^{z}\
(x\widehat{\otimes}y)^{-1};\,\,\,\,\,\,\,\,\,\,(x(x^{-1})^{y})\widehat{\otimes}t=(x\widehat{\otimes}y)\
((x\widehat{\otimes}y)^{-1})^{t};$
$(z\widehat{\otimes}t)^{(x\widehat{\otimes}y)}=(z\widehat{\otimes}t)^{[x,y]};\,\,\,\,\,\,\,\,\,\,[x\widehat{\otimes}y,z\widehat{\otimes}t]=(x(x^{-1})^{y})(t^{z}t^{-1}).$

The same rules are true if we replace $\widehat{\wedge}$ with
$\widehat{\otimes}$.

###### Proof.

It is enough to repeat the computations in [2, Propositions 1,2,3], replacing
the role of $\otimes$ with that of $\widehat{\otimes}$ (resp., $\wedge$ with
$\widehat{\wedge}$). ∎

The complete exterior centralizer of a pro–$p$–group $G$ is the set

$\widehat{C_{G}}(x)=\\{a\in G\ |\
a\widehat{\wedge}x=1_{{}_{G\widehat{\wedge}G}}\\}$

which turns out to be a subgroup of $G$, because if
$a,b\in\widehat{C_{G}}(x)$, then the rules in Lemma 2.3 allow us to conclude
that

$ab^{-1}\widehat{\wedge}x=(b^{-1}\widehat{\wedge}x)^{a}\
(a\widehat{\wedge}x)={((b\widehat{\wedge}x)^{-1})}^{ab^{-1}}\
(a\widehat{\wedge}x)=1_{G\widehat{\wedge}G}$

so $ab^{-1}\in\widehat{C_{G}}(x)$. The complete exterior center of $G$ is the
set

$\widehat{Z}(G)=\\{g\in G\ |\
1_{{}_{G\widehat{\wedge}G}}=g\widehat{\wedge}y\in G\widehat{\wedge}G,\forall
y\in G\\}$

and a similar argument shows that it is a subgroup of $G\widehat{\wedge}G$.
Analogously, $\widehat{C_{G}}(x)$ is a subgroup of $C_{G}(x)$ and
$\widehat{Z}(G)$ is a subgroup of $Z(G)$.

Of course, if $G$ is a finite group, $\widehat{\wedge}$ is exactly the
nonabelian exterior square $\wedge$ in [2] and $\widehat{\otimes}$ is the
nonabelian tensor square in $\otimes$. Then
$\widehat{C_{G}}(x)=C^{\wedge}_{G}(x)$ and $\widehat{Z}(G)=Z^{\wedge}(G)$ so
we have a significant approach in order to generalize most of the results in
[2, 3, 12].

######  Lemma 2.4.

Let $G$ be a pro–$p$–group, $A$ and $B$ two closed subgroups of $G$ such that
$A\cap B=\\{1\\}$, $N$ a closed normal subgroup of $G$ and $x\in G$. Then

  * (i)

$\widehat{C_{G}}(x)$ is a closed normal subgroup of $C_{G}(x)$;

  * (ii)

$\widehat{Z}(G)={\underset{g\in G}{\bigcap}}\widehat{C_{G}}(g)$ is a closed
subgroup of $Z(G)$;

  * (iii)

$1\longrightarrow\widehat{C_{G}}(x)\cap
N\longrightarrow\widehat{C_{G}}(x)\longrightarrow\widehat{C_{G/N}}(xN)$ is a
short exact sequence of pro–$p$–groups;

  * (iv)

$1\longrightarrow\widehat{Z}(G)\cap
N\longrightarrow\widehat{Z}(G)\longrightarrow\widehat{Z}(G/N)$ is a short
exact sequence of pro–$p$–groups;

  * (v)

$\widehat{C_{A\times B}}(ab)=\widehat{C_{A}}(a)\times\widehat{C_{B}}(b)$ for
all $a\in A$ and $b\in B$;

  * (vi)

$\widehat{Z}(A\times B)=\widehat{Z}(A)\times\widehat{Z}(B)$.

###### Proof.

(i). If $g\in\widehat{C_{G}}(x)$ and $y\in\widehat{C_{G}}(x)$, then
$g^{y}\widehat{\wedge}x=(g\widehat{\wedge}x)^{y}=1_{G\widehat{\wedge}G}$ thus
$\widehat{C_{G}}(x)$ is normal in $C_{G}(x)$. The fact that
$\widehat{C_{G}}(x)$ is the stabilizer of a point, under the action of the
operator $\widehat{\wedge}$, ensures that it is closed.

(ii). The equality of $\widehat{Z}(G)$ with the intersection of the complete
exterior centralizers is obvious. It is obvious also that $\widehat{Z}(G)$ is
contained in $Z(G)$. Finally, the intersection of closed is closed, then
$\widehat{Z}(G)$ is closed in $Z(G)$.

(iii). Consider the natural epimorphism of pro–$p$–groups

$\widehat{\pi}:g\wedge h\in G\widehat{\wedge}G\mapsto gN\widehat{\wedge}hN\in
G/N\widehat{\wedge}G/N.$

If $y\in\widehat{C_{G}}(x)$, then
$\widehat{\pi}(y)=yN\in\widehat{C_{G/N}}(xN)$. On the other hand, if
$\widehat{\pi}(y)=1$ for $y\in\widehat{C_{G}}(x)$, then $y\in
N\cap\widehat{C_{G}}(x)$. The result follows.

(iv). It follows from (ii) and (iii).

(v). We may overlap [12, Proof of Proposition 2.6], mutatis mutandis.

(vi). It follows from (ii) and (v). ∎

Now, we may give a structural result which is related to
$H_{2}(G,\mathbb{Z}_{p})$.

######  Proposition 2.5.

Let $G$ be a pro–$p$–group. $C_{G}(x)/\widehat{C_{G}}(x)$ is isomorphic as
pro–$p$–group with a subgroup of $H_{2}(G,\mathbb{Z}_{p})$.

###### Proof.

The map

$\widehat{\psi}:y\in
C_{G}(x)\longmapsto\widehat{\psi}(y)=x\widehat{\wedge}y\in
H_{2}(G,\mathbb{Z}_{p})$

is a homomorphism of pro–$p$–groups. Note that $H_{2}(G,\mathbb{Z}_{p})$ is a
factor group of $\widehat{\nabla}(G)$, which is a closed central subgroup of
$G\widehat{\otimes}G$. Furthemore, the elements of $\widehat{\nabla}(G)$ are
fixed under the action of $G$. This allows us to conclude that the elements of
$H_{2}(G,\mathbb{Z}_{p})$ are fixed by the action of $G$. On the other hand,
it is clear that $\ker\widehat{\psi}=\widehat{C_{G}}(x)$. Then the result
follows. ∎

An interesting consequence is listed below.

######  Corollary 2.6.

If a pro–$p$–group $G$ has an element $x$ such that
$C_{G}(x)\not=\widehat{C_{G}}(x)$, then $H_{2}(G,\mathbb{Z}_{p})$ is
nontrivial.

##  3\. Main theorems

The present section illustrates our main results. Firstly, we show an upper
bound for $H_{2}(G,\mathbb{Z}_{p})$, when $G$ is a pro–$p$–group. This is in
harmony with the results in [3, 8, 11].

######  Theorem 3.1.

Let $G$ be a pro–$p$–group, $\mathrm{rk}(G/\widehat{Z}(G))=n$,
$\mathrm{rk}(H_{2}(G,\mathbb{Z}_{p}))=m$ and
$\mathrm{tf}(H_{2}(G,\mathbb{Z}_{p}))=l$ . Then $H_{2}(G,\mathbb{Z}_{p})$ is
an abelian pro–$p$–group. Furthermore,

  * (i)

if $H_{2}(G,\mathbb{Z}_{p})$ is finite, then $|Z(G)/\widehat{Z}(G)|$ divides
$|H_{2}(G,\mathbb{Z}_{p})|^{n}$;

  * (ii)

if $H_{2}(G,\mathbb{Z}_{p})$ is infinite, then
$\mathrm{rk}(Z(G)/\widehat{Z}(G))\leq m^{n}$. In particular, if
$H_{2}(G,\mathbb{Z}_{p})$ is torsion–free, then
$\mathrm{tf}(Z(G)/\widehat{Z}(G))\leq l^{n}$.

###### Proof.

From $(**)$, $H_{2}(G,\mathbb{Z}_{p})$ is a closed central subgroup of
$G\widehat{\wedge}G$ and so it is a closed abelian subgroup of
$G\widehat{\wedge}G$. From Theorem 2.2, $G\widehat{\wedge}G$ is a
pro–$p$–group and each closed subgroup of a pro–$p$–group is again a
pro–$p$–group [10, Lemma 7.2.9 (i)]. Then $H_{2}(G,\mathbb{Z}_{p})$ is an
abelian pro–$p$–group.

Now we proceed to prove (i) and (ii). Assume that
$G/\widehat{Z}(G)=\overline{\langle\bar{x}_{1},\ldots\bar{x}_{n}\rangle}$ for
some elements
$\bar{x}_{1}=x_{1}\widehat{Z}(G),\ldots,\bar{x}_{n}=x_{n}\widehat{Z}(G)$ of
$G/\widehat{Z}(G)$. Define

$\widehat{\xi}:x\in Z(G)\longmapsto(x\
\widehat{\wedge}x_{1},\ldots,x\widehat{\wedge}x_{n})\in
H_{2}(G,\mathbb{Z}_{p})^{n}.$

$\widehat{\xi}$ is a homomorphism of pro–$p$–groups, because for all $x,y\in
Z(G)$ we have

$xy\widehat{\wedge}x_{i}=(x\widehat{\wedge}x_{i})\,\,(y\widehat{\wedge}x_{i})^{x}=(x\widehat{\wedge}x_{i})\,\,(y\widehat{\wedge}x_{i})$

for every $i\in\\{1,\ldots n\\}$.

We claim that $\ker\widehat{\xi}=\widehat{Z}(G)$. It is easy to check that
$\widehat{Z}(G)\leq\ker\widehat{\xi}$. On the other hand, if
$x\in\ker\widehat{\xi}$, then $x\widehat{\wedge}x_{i}=1_{G\widehat{\wedge}G}$
for every $i\in\\{1,\ldots n\\}$. It is enough to show that
$x\widehat{\wedge}y=1_{G\widehat{\wedge}G}$ for every $y\in G$ in order to
finish our proof. If $y\in G\setminus\widehat{Z}(G)$, then we may always write
$y=x^{\alpha_{1}}_{1}\ldots x^{\alpha_{n}}_{n}$, where
$\alpha_{1},\ldots\alpha_{n}$ are integers. Thus,

$x\widehat{\wedge}y=x\widehat{\wedge}(x^{\alpha_{1}}_{1}\ldots
x^{\alpha_{n}}_{n})=(x\widehat{\wedge}x^{\alpha_{1}}_{1})\ldots(x\widehat{\wedge}x^{\alpha_{n}}_{n})$
$=(x\widehat{\wedge}x_{1})^{\alpha_{1}}\ldots(x\widehat{\wedge}x_{n})^{\alpha_{n}}=1_{G\widehat{\wedge}G}.$

We have then proved that $\widehat{\xi}$ is a monomorphism of pro–$p$–groups.
Then $\widehat{\xi}$ allows us to embed $Z(G)/\widehat{Z}(G)$ in
$H_{2}(G,\mathbb{Z}_{p})^{n}$. In case (i) this means that, if
$H_{2}(G,\mathbb{Z}_{p})$ is finite, then $H_{2}(G,\mathbb{Z}_{p})^{n}$ is
finite and consequently $Z(G)/\widehat{Z}(G)$ is finite, subject to the
corresponding arithmetic condition in (i). In case (ii)
$H_{2}(G,\mathbb{Z}_{p})$ is infinite abelian of finite (topological) rank.
Consequently, each closed subgroup of $H_{2}(G,\mathbb{Z}_{p})$ has the same
property. We conclude that (ii) is true. ∎

######  Remark 3.2.

It is well known that a finite group $E$ is capable if and only if
$Z^{\wedge}(E)=1$. Then the previous result may be useful to find criteria for
the size of $\widehat{Z}(G)$ in a pro–$p$–group $G$. This aspect has been
studied in [1, 12] in the finite case.

In order to generalize to the infinite case the exterior degree, we proceed in
the following way. Let $G$ be a pro–$p$–group. On the pro–$p$–group $G\times
G$, we may consider the crossing pairing

$\widehat{f}:(x,y)\in G\times G\longmapsto x\widehat{\wedge}y\in
G\widehat{\wedge}G,$

described in [11, §2], which is continuous and bilinear for all $x,y\in G$.
This map $\widehat{f}$ appears, for instance, in the universal property of
$G\widehat{\wedge}G$. On the other hand, we know that there exists a unique
normalized Haar measure $\mu$ on a pro–$p$–group $G$ (the reader may find also
a more general situation in [5, 6, 7, 18, 19]) and this is true also for
$G\times G$, when we consider the product measure $\mu\times\mu$. Then

$C=\widehat{f}^{-1}(1_{G\widehat{\wedge}G})=\\{(x,y)\in G\times G\ |\
x\widehat{\wedge}y=1_{G\widehat{\wedge}G}\\}\subseteq G\times G$

is a closed subgroup of $G\times G$ and so it is meaningful to define the
exterior degree of a pro–$p$–group $G$ as

$\widehat{\mathrm{d}}(G)=(\mu\times\mu)(C).$

In case $G$ is finite and $\mu$ is the counting measure on $G$, we find
exactly $(*)$.

The next lemmas show that some methods of [4, 7, 13, 18, 19] can be adapted
here. The idea is in fact to relate $\widehat{\mathrm{d}}(G)$ with
$\mathrm{d}(G)$.

######  Lemma 3.3.

Let $G$ be a pro–$p$–group and $x\in G$. Then

$\widehat{\mathrm{d}}(G)=\int_{G}\mu(\widehat{C_{G}}(x))d\mu(x),$

where

$\mu(\widehat{C_{G}}(x))=\int_{G\times G}\chi_{{}_{C}}(x,y)d\mu(y),$

and $\chi_{{}_{C}}$ denotes the characteristic map of $C$.

###### Proof.

Fubini-Tonelli’s Theorem implies:

$\widehat{\mathrm{d}}(G)=(\mu\times\mu)(C)=\int_{G\times
G}\chi_{{}_{C}}(d\mu\times d\mu)$
$=\int_{G}\left(\int_{G}\chi_{{}_{C}}(x,y)d\mu(x)\right)d\mu(y)=\int_{G}\mu(\widehat{C_{G}}(x))d\mu(x).$

∎

######  Lemma 3.4.

Let $H$ be a closed subgroup of a pro–$p$–group $G$ and $k\geq 1$. Then

$\mu(H)=\left\\{\begin{array}[]{lcl}\frac{1}{p^{k}},&\,if&|G:H|=p^{k}\\\
0,&\,if&|G:H|=\infty.\end{array}\right.$

###### Proof.

It is a well–known fact, which can be found for instance in [7]. ∎

######  Lemma 3.5.

A pro–$p$–group $G$ has $0\leq\widehat{\mathrm{d}}(G)\leq 1$. In particular,

  * (i)

$\widehat{\mathrm{d}}(G)=0$ if and only if $|G:\widehat{C_{G}}(x)|=\infty$ for
all but finitely many $x\in G$;

  * (ii)

$\widehat{\mathrm{d}}(G)=1$ if and only if $\widehat{Z}(G)=G$.

###### Proof.

Since $\mu$ is monotone, positive–defined and normalized,
$0=(\mu\times\mu)(\\{(1,1)\\})\leq\widehat{\mathrm{d}}(G)=(\mu\times\mu)(C)\leq(\mu\times\mu)(G)=1.$
Now (i) follows easily from Lemmas 3.3 and 3.4. (ii) is clear from the
definition of $\widehat{\mathrm{d}}(G)=1$. ∎

A fundamental difference with the finite case is the following.

######  Remark 3.6.

$\mathbb{Z}_{p}$ is an infinite abelian pro–$p$–group topologically generated
by 1 element (but not generated by 1 element in the abstract sense) such that
$\widehat{Z}(\mathbb{Z}_{p})=\mathbb{Z}_{p}$,
$\widehat{\mathrm{d}}(\mathbb{Z}_{p})=1$ and $H_{2}(G,\mathbb{Z}_{p})$ is
trivial.

We may show the main result of this section.

######  Theorem 3.7.

A pro–$p$–group $G$ satisfies the following inequality

$\widehat{\mathrm{d}}(G)\leq\mathrm{d}(G)-\left(\frac{p-1}{p}\right)\left(\mu(Z(G))-\mu(\widehat{Z}(G))\right).$

Furthermore, if $H_{2}(G,\mathbb{Z}_{p})$ is finite, then

$\widehat{\mathrm{d}}(G)\geq\mu(\widehat{Z}(G))+\frac{1}{|H_{2}(G,\mathbb{Z}_{p})|}\left(\mathrm{d}(G)-\mu(\widehat{Z}(G))\right).$

###### Proof.

We begin to prove the upper bound. Let $x\not\in\widehat{Z}(G)$. Using Lemma
3.3,

$\widehat{\mathrm{d}}(G)=\int_{G}\mu(\widehat{C_{G}}(x))d\mu(x)$
$=\mu(\widehat{Z}(G))+\int_{Z(G)-\widehat{Z}(G)}\mu(\widehat{C_{G}}(x))d\mu(x)+\int_{G-Z(G)}\mu(\widehat{C_{G}}(x))d\mu(x)$

but the monotonicity of $\mu$ implies
$\mu(\widehat{C_{G}}(x))\leq\mu(C_{G}(x))$ and Lemma 3.4 implies
$\mu(\widehat{C_{G}}(x))=|G:\widehat{C_{G}}(x)|^{-1}\leq\frac{1}{p}$, thus

$\leq\mu(\widehat{Z}(G))+\frac{1}{p}\left(\mu(Z(G))-\mu(\widehat{Z}(G))\right)+\int_{G-Z(G)}\mu(C_{G}(x))d\mu(x)$
$=\mu(\widehat{Z}(G))+\frac{1}{p}\left(\mu(Z(G))-\mu(\widehat{Z}(G))\right)+\mathrm{d}(G)-\mu(Z(G))$
$=\mathrm{d}(G)-\left(\frac{p-1}{p}\right)\left(\mu(Z(G))-\mu(\widehat{Z}(G))\right).$

For the lower bound, Lemmas 3.3, 3.4 and Proposition 2.5 imply

$\widehat{\mathrm{d}}(G)=\int_{G}\mu(\widehat{C_{G}}(x))d\mu(x)=\mu(\widehat{Z}(G))+\int_{G-\widehat{Z}(G)}\frac{\mu(C_{G}(x))}{|C_{G}(x):\widehat{C_{G}}(x)|}d\mu(x)$
$\geq\mu(\widehat{Z}(G))+\frac{1}{|H_{2}(G,\mathbb{Z}_{p})|}\int_{G-\widehat{Z}(G)}\mu(C_{G}(x))d\mu(x)$
$=\mu(\widehat{Z}(G))+\frac{1}{|H_{2}(G,\mathbb{Z}_{p})|}\left(\int_{G}\mu(C_{G}(x))d\mu(x)-\mu(\widehat{Z}(G))\right)$
$=\mu(\widehat{Z}(G))+\frac{1}{|H_{2}(G,\mathbb{Z}_{p})|}\left(\mathrm{d}(G)-\mu(\widehat{Z}(G))\right).$

∎

Theorem 3.7 allows us to conclude another interesting inequality.

######  Corollary 3.8.

A pro–$p$–group $G$ satisfies always the inequality
$\widehat{\mathrm{d}}(G)\leq\mathrm{d}(G)$. In particular,
$\widehat{\mathrm{d}}(G)=\mathrm{d}(G)$ implies $\widehat{Z}(G)=Z(G)$.

For instance, $\widehat{\mathrm{d}}(G)=\mathrm{d}(G)$, whenever
$H_{2}(G,\mathbb{Z}_{p})$ is trivial. This happens for the infinite
pro–2–group (with $r\geq 1$ arbitrary)

$D=\langle a,t\ |\ a^{2^{r}}=1,a^{-1}ta=t^{-1}\rangle=\mathbb{Z}_{2}\rtimes
C_{2^{r}},$

described also in [3, §1]. Its exterior degree is computed below.

######  Example 3.9.

It is clear that $Z(D)=\widehat{Z}(D)=1$. For $i=0$,
$\mu(\widehat{C_{D}}(t^{i}))=1$; for all $i\not=0$,
$\mu(\widehat{C_{D}}(t^{i}))=1/2^{r}$; for all $i$ and $1\leq j\leq 2^{r}-1$,
$\mu(\widehat{C_{D}}(a^{j}t^{i}))=0$. By Lemma 3.3 we find that

$\widehat{\mathrm{d}}(D)=\mu(\widehat{Z}(D))+\int_{\langle
t\rangle-\widehat{Z}(D)}\mu(\widehat{C_{D}}(x))d\mu(x)+\int_{D-\langle
t\rangle}\mu(\widehat{C_{D}}(x))d\mu(x)$ $=\frac{1}{2^{r}}\mu(\langle
t\rangle-\\{1\\})=\frac{1}{2^{r}}\mu(\langle t\rangle)=\frac{1}{4^{r}}.$

The importance of the condition
$\widehat{\mathrm{d}}(G)=\mathrm{d}(G)=\mathrm{d}^{\wedge}(G)$ is illustrated
in [13] in finite case. The following result is an application of Theorem 3.7
and, at the same time, is a generalization of the corresponding results of
[13] to the infinite case.

######  Corollary 3.10.

Assume that $G$ is a pro–$p$–group.

  * (i)

If $G$ is abelian of $\mathrm{rk}(G)>1$, then
$\widehat{\mathrm{d}}(G)\leq\frac{p^{2}+p-1}{p^{3}}$ and the equality holds if
and only if $G/\widehat{Z}(G)\simeq C_{p}\times C_{p}$.

  * (ii)

If $G$ is nonabelian and $\widehat{Z}(G)\not=Z(G)$, then
$\widehat{\mathrm{d}}(G)\leq\frac{p^{3}+p-1}{p^{4}}.$

###### Proof.

(i). Abelian pro–$p$–groups of $\mathrm{rk}(G)=1$ are procyclic pro–$p$–groups
and they are either isomorphic to $C_{p^{k}}$ (for some $k\geq 1$) or to
$\mathbb{Z}_{p}$. Both of them have $\widehat{Z}(G)=Z(G)=G$ by Lemma 3.5. Let
$\widehat{Z}(G)\not=Z(G)=G$ and $\mathrm{d}(G)=1$. For all
$x\not\in\widehat{Z}(G)$, arguing as in Theorem 3.7, we find that
$|G:\widehat{C_{G}}(x)|\geq p$, $|\widehat{C_{G}}(x):\widehat{Z}(G)|\geq p$
and $\mu(\widehat{Z}(G))\leq 1/p^{2}$. From Theorem 3.7, we have

$\widehat{\mathrm{d}}(G)\leq
1-\left(\frac{p-1}{p}\right)\left(1-\frac{1}{p^{2}}\right)=\frac{p^{2}+p-1}{p^{3}}.$

Now assume $G/\widehat{Z}(G)\cong C_{p}\times C_{p}$. From Lemma 3.3, we get

$\widehat{\mathrm{d}}(G)=\int_{G}\mu(\widehat{C_{G}}(x))d\mu(x)=\mu(\widehat{Z}(G))+\int_{G-\widehat{Z}(G)}\mu(\widehat{C_{G}}(x))d\mu(x)$
$=\frac{1}{p}+\mu(\widehat{Z}(G))\Big{(}1-\frac{1}{p}\Big{)}=\frac{p^{2}+p-1}{p^{3}}.$

Conversely,

$\frac{p^{2}+p-1}{p^{3}}=\widehat{\mathrm{d}}(G)=\int_{G}\mu(\widehat{C_{G}}(x))d\mu(x)\leq\frac{1}{p}+\mu(\widehat{Z}(G))\Big{(}1-\frac{1}{p}\Big{)}$

implies $\mu(\widehat{Z}(G))\geq 1/p^{2}$ and then
$\mu(\widehat{Z}(G))=1/p^{2}$. We conclude necessarily that
$G/\widehat{Z}(G)\cong C_{p}\times C_{p}.$

(ii). Assume that $G$ is nonabelian and $\widehat{Z}(G)\not=Z(G)$. Then we may
argue as (i) above, getting $\mathrm{d}(G)\leq(p^{2}+p-1)/p^{3}$, $\mu(G)\leq
1/p^{2}$ and $\mu(\widehat{Z}(G))\leq 1/p^{3}$. Again an application of
Theorem 3.7 allows us to conclude the proof. ∎

We end with two theorems which describe the exterior degree for quotients and
direct products. The first case is the following.

######  Theorem 3.11.

Let $N$ be a closed normal subgroup of a pro–$p$–group $G$. Then

$\widehat{\mathrm{d}}(G)\leq\widehat{\mathrm{d}}(G/N)$

and the equality holds if $N\leq\widehat{Z}(G)$.

###### Proof.

Assume that $\lambda$, $\mu$ and $\nu$ are corresponding Haar measure of $N$,
$G$ and $G/N$ respectively. The Extended Weil’s Formula (see for instance [19,
Equation $(*)$, p. 126]), implies

$\widehat{\mathrm{d}}(G)=\int_{G}\mu(\widehat{C_{G}}(x))d\mu(x)\leq\int_{G}\mu(\widehat{C_{G}}(x)N)d\mu(x)$
$=\int_{\frac{G}{N}}\int_{N}\mu(\widehat{C_{G}}(xn)N)d\lambda(n)d\nu(xN).$

On the other hand, one can see without difficulties that

$\mu(\widehat{C_{G}}(xn)N)=\nu(\widehat{C_{G}}(xn)N/N)\leq\nu(\widehat{C_{G/N}}(xN))$

therefore,

$\widehat{\mathrm{d}}(G)\leq\int_{\frac{G}{N}}\int_{N}\nu(\widehat{C_{G/N}}(xN))d\lambda(n)d\nu(xN)$
$=\int_{\frac{G}{N}}\nu(\widehat{C_{G/N}}(xN))\left(\int_{N}d\lambda(n)\right)d\nu(xN)=\int_{\frac{G}{N}}\nu(\widehat{C_{G/N}}(xN))d\nu(xN)$
$=\widehat{\mathrm{d}}(G/N).$

Now assume that $N$ is contained in $\widehat{Z}(G)$, then the canonical map
$G\widehat{\wedge}G\rightarrow G/N\widehat{\wedge}G/N$ will be an isomorphism.
Therefore $\widehat{C_{G}}(xn)N/N=\widehat{C_{G/N}}(xN)$ for all $x\in G$ and
all inequalities should be changed into equalities and so
$\widehat{\mathrm{d}}(G)=\widehat{\mathrm{d}}(G/N).$ ∎

The second case describes a theorem of splitting of probabilities.

######  Theorem 3.12.

Let $G$ be a pro–$p$–group and $H$ be a pro–$q$–group for two different primes
$p$ and $q$. Then

$\widehat{\mathrm{d}}(G\times H)=\widehat{\mathrm{d}}(G)\ \
\widehat{\mathrm{d}}(H).$

###### Proof.

Assume that $\mu$ and $\nu$ are the Haar measure on $G$ and $H$ respectively.
Then $\lambda=\mu\times\nu$ will be the Haar measure on $G\times H$.
Forevermore, for all $(x,y)\in G\times H$, we have $\widehat{C_{G\times
H}}((x,y))=\widehat{C_{G}}(x)\times\widehat{C_{H}}(y)$, then

$\widehat{\mathrm{d}}(G\times H)=\int_{G\times H}\lambda(\widehat{C_{G\times
H}}((x,y)))d\lambda((x,y))$
$=\int_{G}\int_{H}\mu(\widehat{C_{G}}(x))\nu(\widehat{C_{H}}(y))d\mu(x)d\nu(y)$
$\int_{G}\mu(\widehat{C_{G}}(x))d\mu(x)\int_{H}\nu(\widehat{C_{H}}(y))d\nu(y)=\widehat{\mathrm{d}}(G)\
\ \widehat{\mathrm{d}}(H).$

∎

## References

  * [1] F.R. Beyl, U. Felgner and P. Schmid, On groups occurring as center factor groups, J. Algebra 61 (1979), 161–177.
  * [2] R. Brown, D.L. Johnson and E.F. Robertson, Some computations of non–abelian tensor products of groups,  J. Algebra 111 (1987), 177–202.
  * [3] B. Eick, Schur multiplicators of infinite pro–$p$–groups with finite coclass, Israel J. Math. 166 (2008), 147–156.
  * [4] A. Erfanian, P. Lescot and R. Rezaei, On the relative commutativity degree of a subgroup of a finite group, Comm. Algebra 35 (2007), 4183–4197.
  * [5] A. Erfanian and F.G. Russo, Probability of mutually commuting n-tuples in some classes of compact groups, Bull. Iranian Math. Soc. 24 (2008), 27–37.
  * [6] A. Erfanian and F.G. Russo, Miscellaneous Topics in Group Theory: Mashhad–Naples, 2005–2008, Aracne, 2009, Rome.
  * [7] A. Erfanian and R. Rezaei, On the commutativity degree of compact groups, Arch. Math. (Basel) 93 (2009), 345–356.
  * [8] A. Erfanian, F.G. Russo and N.H. Sarmin, Some considerations on the nonabelian tensor square of crystallographic groups, Asian–Eur. J. Math. 4 (2011), 271–282.
  * [9] A. Erfanian, P. Niroomand and R. Rezaei, On the multiple exterior degree of finite groups, eprint, Cornell University Library, arXiv:1108.1303, 2011.
  * [10] C.R. Leedham–Green and S. McKay, The Structure of Groups of Prime Power Order, Oxford University Press, Oxford, 2002.
  * [11] P. Moravec, On the Schur multipliers of finite $p$–groups of given coclass,  Israel J. Math. 185 (2011), 189–205.
  * [12] P. Niroomand and F.G. Russo, A note on the exterior centralizer, Arch. Math. (Basel) 93 (2009), 505–512.
  * [13] P. Niroomand and R. Rezaei, On the exterior degree of finite groups, Comm. Algebra 39 (2011), 335–343.
  * [14] P. Niroomand and R. Rezaei, The exterior degree of a pair of finite groups, eprint, Cornell University Library, arxiv: 1101.4312, 2011.
  * [15] P. Niroomand and M. Parvizi , On the structure of groups whose exterior or tensor square is a $p$–group, J. Algebra 352 (2012), 347–353.
  * [16] P. Niroomand, R. Rezaei and F.G. Russo, Commuting powers and exterior degree of finite groups, J. Korean Math. Soc. 49 (2012), 855–865.
  * [17] P. Niroomand, Some results on the exterior degree of extra–special group, Ars Combinatoria (2012), to appear.
  * [18] R. Rezaei and F.G. Russo, Bounds for the relative $n$-th nilpotency degree in compact groups, Asian–Eur. J. Math. 4 (2011), 495–506.
  * [19] R. Rezaei and F.G. Russo, $n$-th relative nilpotency degree and relative $n$-isoclinism classes, Carpathian J. Math. 27 (2011), 123–130.


# On the Language Coverage Bias for Neural Machine Translation

Shuo Wang1 Zhaopeng Tu2 Zhixing Tan1 Shuming Shi2 Maosong Sun1,3 Yang Liu1,3,4
1Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, Tsinghua
University
3Beijing Academy of Artificial Intelligence 4Institute for AIR, Tsinghua
University
<EMAIL_ADDRESS>1{zxtan, sms<EMAIL_ADDRESS>
2Tencent AI Lab
2{zptu<EMAIL_ADDRESS>

###### Abstract

Language coverage bias, which indicates the content-dependent differences
between sentence pairs originating from the source and target languages, is
important for neural machine translation (NMT) because the target-original
training data is not well exploited in current practice. By carefully
designing experiments, we provide comprehensive analyses of the language
coverage bias in the training data, and find that using only the source-
original data achieves comparable performance with using full training data.
Based on these observations, we further propose two simple and effective
approaches to alleviate the language coverage bias problem through explicitly
distinguishing between the source- and target-original training data, which
consistently improve the performance over strong baselines on six WMT20
translation tasks. Complementary to the translationese effect, language
coverage bias provides another explanation for the performance drop caused by
back-translation Marie et al. (2020). We also apply our approach to both back-
and forward-translation and find that mitigating the language coverage bias
can improve the performance of both the two representative data augmentation
methods and their tagged variants Caswell et al. (2019).

##  1 Introduction

(a) English-Original (b) German-Original

Figure 1: Example of language coverage bias illustrated by word clouds that
are plotted at the English side of sentence pairs in the En-De test sets from
WMT10 to WMT18. The test sets consist of English-original and German-original
sentence pairs.

In recent years, there has been a growing interest in investigating the effect
of original languages in parallel data on neural machine translation Barrault
et al. (2020); Edunov et al. (2020); Marie et al. (2020). Several studies have
shown that target-original test examples111Target-original test examples are
sentence pairs that are translated from the target language into the source
language. can lead to distortions in automatic and human evaluations, which
should be omitted from machine translation test sets Barrault et al. (2019);
Zhang and Toral (2019); Graham et al. (2020). Another branch of studies report
that target-original test data leads to discrepant conclusions: back-
translation only benefits the translation of target-original test data while
harms that of source-original test data Edunov et al. (2020); Marie et al.
(2020). They attribute these phenomena to the reason that human-translated
texts (i.e., translationese) exhibit formal and stylistic differences that set
them apart from the texts originally written in that language Baker et al.
(1993); Volansky et al. (2015); Zhang and Toral (2019).

Complementary to the translationese bias, which is content-independent
Volansky et al. (2015), we identify another important problem, namely language
coverage bias, which refers to the content-dependent differences in data
originating from different languages. These differences stem from the
diversity of regions and cultures. While the degree of the translationese bias
varies across different translators Toral (2019), language coverage bias is an
intrinsic bias between the source- and target-original data, which is hardly
affected by the ability of the translator. Figure 1 shows an example, where
the contents in English- and German-original texts differ significantly due to
language coverage bias.

To investigate the effect of language coverage bias in the training data on
NMT models, we propose an automatic method to identify the original language
of each training example, which is generally unknown in practical corpora.
Experimental results on three large-scale translation corpora show that there
exists a significant performance gap between NMT models trained on the source-
and target-original data, which have different vocabulary distributions,
especially for content words. Since the target-original training data performs
poorly in translating content words, using only the source-original data
achieves comparable performance with using full training data. These findings
motivate us to explore other data utilization methods rather than
indiscriminately mixing the source- and target-original training data.

We propose to alleviate the language coverage bias problem by explicitly
distinguishing between the source- and target-original training data.
Specifically, two simple and effective methods are employed: bias-tagging and
fine-tuning. Experimental results show that both approaches consistently
improve the performance on six WMT20 translation tasks. Language coverage bias
also provides another explanation for the failure of back-translation on the
source-original test data, complementary to the translationese effect Marie et
al. (2020). We further validate our approach in the monolingual data
augmentation scenario, where the language coverage bias problem would be more
severe due to the newly introduced monolingual data.

#### Contributions

The main contributions of our work are listed as follows:

  * •

We demonstrate the necessity of studying the language coverage bias for NMT,
and identify that using the target-original data can cause poor translation
adequacy on content words.

  * •

We address the language coverage bias induced by the target-original data by
explicitly distinguishing the original languages, which can significantly
improve the translation performance on six WMT20 translation tasks.

  * •

We show that alleviating the language coverage bias also benefits monolingual
data augmentation, which can improve both back- and forward-translation and
their tagged variants Caswell et al. (2019).

##  2 Experimental Setup

#### Data

We conducted experiments on six WMT20 benchmarks Barrault et al. (2020),
including English$\Leftrightarrow$German (En$\Leftrightarrow$De),
English$\Leftrightarrow$Chinese (En$\Leftrightarrow$Zh), and
English$\Leftrightarrow$Japanese (En$\Leftrightarrow$Ja) news translation
tasks. The preprocessed training corpora contain 41.0M, 21.8M, and 13.0M
sentence pairs for En$\Leftrightarrow$De, En$\Leftrightarrow$Zh, and
En$\Leftrightarrow$Ja, respectively. We used the monolingual data that is
publicly available in WMT20 to train the proposed original language detection
model (Section 3.1) and data augmentation (Section 4.2). The Appendix lists
details about the data preprocessing.

For En$\Leftrightarrow$De and En$\Leftrightarrow$Zh, we used newstest2019 as
the validation sets. For En$\Leftrightarrow$Ja, we split the official
validation set released by WMT20 into two parts by the original language and
only used the corresponding part for each direction. We used newstest2020 as
the test sets for all the six tasks. We reported the Sacre BLEU Post (2018),
as recommended by WMT20.

#### Model

We used the Transformer-Big Vaswani et al. (2017) model, which consists of a
6-layer encoder and a 6-layer decoder, and the hidden size is 1024. Recent
studies showed that training on large batches can further boost model
performance Ott et al. (2018); Wu et al. (2018). Accordingly, we followed
their settings to train models with batches of approximately 460k tokens.
Please refer to the Appendix for more details about model training. We
followed Ng et al. (2019) to use the Transformer-Big decoder as our language
models, which are used to detect the original language and measure translation
fluency. Language models are also trained with large batches Ott et al.
(2018).

##  3 Observing Language Coverage Bias

In this study, we first establish the existence of language coverage bias
(Section 3.2), and show how the bias affects NMT performance (Section 3.3). To
this end, we propose an automatic method to detect the original language of
each training example (Section 3.1), which is often not available in large-
scale parallel corpora Riley et al. (2020).

###  3.1 Detecting Original Languages

#### Detection Method

Intuitively, we use a large-scale monolingual dataset to estimate the
distribution of the contents covered by each language. For each training
example, we compare its similarities to the distributions of source and target
languages, based on which we determine its original language.

Formally, let $\mathcal{D}_{s}$ and $\mathcal{D}_{t}$ denote the source-side
and target-side distributions of the covered contents. Given a training
example $\langle\mathbf{x},\mathbf{y}\rangle$, the probability that it is
covered by one language (represented as $\mathcal{D}_{s}$ and
$\mathcal{D}_{t}$) can be expressed as

$\displaystyle P(\mathcal{D}_{s}|\langle\mathbf{x},\mathbf{y}\rangle)$
$\displaystyle=\frac{P(\mathcal{D}_{s})P(\langle\mathbf{x},\mathbf{y}\rangle|\mathcal{D}_{s})}{P(\langle\mathbf{x},\mathbf{y}\rangle)},$
$\displaystyle P(\mathcal{D}_{t}|\langle\mathbf{x},\mathbf{y}\rangle)$
$\displaystyle=\frac{P(\mathcal{D}_{t})P(\langle\mathbf{x},\mathbf{y}\rangle|\mathcal{D}_{t})}{P(\langle\mathbf{x},\mathbf{y}\rangle)}.$

We use a score function to denote the difference between the two
probabilities:

$\begin{split}\text{\em score}=\log
P(\mathcal{D}_{s}|\langle\mathbf{x},\mathbf{y}\rangle)-\log
P(\mathcal{D}_{t}|\langle\mathbf{x},\mathbf{y}\rangle),\\\ =\log
P(\langle\mathbf{x},\mathbf{y}\rangle|\mathcal{D}_{s})-\log
P(\langle\mathbf{x},\mathbf{y}\rangle|\mathcal{D}_{t})+c,\end{split}$

where $c=\log P(\mathcal{D}_{s})-\log P(\mathcal{D}_{t})$, which has a
constant value when the source and target monolingual datasets are given.
Intuitively, examples with higher score values are more likely to be source-
original while those with lower score values are more likely to be target-
original data. We train language models $\bm{\theta}^{lm}_{s}$ and
$\bm{\theta}^{lm}_{t}$ on the source- and target-language monolingual data to
estimate the conditional probabilities:

$\displaystyle P(\langle\mathbf{x},\mathbf{y}\rangle|\mathcal{D}_{s})$
$\displaystyle=P(\mathbf{x}|\bm{\theta}^{lm}_{s}),$ $\displaystyle
P(\langle\mathbf{x},\mathbf{y}\rangle|\mathcal{D}_{t})$
$\displaystyle=P(\mathbf{y}|\bm{\theta}^{lm}_{t}).$

Accordingly, the score can be rewritten as

$\begin{split}\text{\em score}=\log P(\mathbf{x}|\bm{\theta}^{lm}_{s})-\log
P(\mathbf{y}|\bm{\theta}^{lm}_{t})+c.\end{split}$ (1)

We label examples as source-original if their score values are positive, and
the other examples as target-original. To find a specific constant for each
language pair, we tune the value of $c$ to obtain the best classification
performance on the validation sets, where the original languages are known.

Method | En-Zh | En-Ja | En-De
---|---|---|---
FT | 83.6 | 83.7 | 86.6
Ours | 84.4 | 91.5 | 88.7
Table 1: F1 scores of detecting original languages in the test sets. “FT”
denotes the forward translation classifier proposed by Riley et al. (2020).

#### Detection Accuracy

We evaluated the detection method on the mixture of the test sets of
bidirectional translation tasks in WMT20 for each language pair. For
comparison, we re-implemented the CNN-based forward-translation (FT)
classifier proposed by Riley et al. (2020). The FT classifier and the language
models used in our method were trained on the same monolingual data sets.
Table 1 shows that our method outperforms the FT classifier in all language
pairs. In addition, our model also outperforms the FT approach on detecting
noisy training data, which leads to an improvement in translation performance
(please refer to Table 11 in the Appendix for more results).

###  3.2 Existence of Language Coverage Bias

In this section, we validate the existence of language coverage bias by (1)
comparing the performance of NMT models trained on data with different
original languages, and (2) directly calculating the divergence between the
vocabulary distributions of the source- and target-original data.

(a) En$\Rightarrow$Zh (b) Zh$\Rightarrow$En

Figure 2: Translation performance on the validation sets of the
En$\Leftrightarrow$Zh translation task for different ratios of most source-
and target-original training examples.

#### Translation Performance

Once all the training examples are assigned a score by the detection method
(Eq. (1)), we regard $R\%$ of examples with the highest scores as the most
source-original examples, and $R\%$ of examples with the least scores as the
most target-original examples. We investigate the effect of $R\%$ on
translation performance, as shown in Figure 2. Clearly, using the most source-
original examples significantly outperforms using its target-original
counterparts, demonstrating that the source- and target-original data indeed
differ greatly from each other. To rule out the effect of data scale, we treat
50% of data with the highest scores as source-original data, and the same
amount of data with the least scores as target-original data in the following
experiments by default.

Since some recent works find that BLEU might be affected by the translationese
problem Edunov et al. (2020); Freitag et al. (2020), we have also conducted a
side-by-side human-evaluation on the Zh$\Rightarrow$En development set, where
500 randomly sampled examples were evaluated by six persons (agreement Fleiss
(1971): Fleiss’ Kappa=0.46). 37.0% of outputs using the source-original data
are better than using target-original data, and 21.0% are worse. By manually
checking the outputs, we find using only the target-original data tends to
omit important source contents (e.g., named entities) either by totally
ignoring some contents or by using pronouns instead. The human-evaluation
shows the same trend with the BLEU score presented in Figure 2. Given that
conducting human-evaluation on all the six translation tasks is time-consuming
and labor-intensive, we use automatic measures to further investigate this
problem in Section 3.3.

Data | JS ($\times 10^{-5}$)
---|---
All | Content | Function
Random | 4 | 10 | 0
S vs T | 745 | 1503 | 261
Table 2: JS divergence of the vocabulary distributions between the source- and
target-original data (“S vs T”) on the training set of WMT20
En$\Leftrightarrow$Zh. “All”, “Content”, and “Function” denote all words,
content words, and function words respectively. For reference, we also report
the JS divergence between randomly selected 50% examples and the others
(“Random”).

#### Vocabulary Distributions

Complementary to previous studies that focus on the content-independent
stylistic difference Volansky et al. (2015) between translationese and
original texts Riley et al. (2020); Edunov et al. (2020); Marie et al. (2020),
we investigate the content-dependent language coverage bias between the
source- and target-original data in this experiment. Intuitively, if the
language coverage bias exists, the vocabulary distributions of the source- and
target-original data should differ greatly from each other, since the covered
issues tend to have different frequencies between them D’Alessio and Allen
(2000). We use the Jensen-Shannon (JS) divergence Lin (1991) to measure the
difference between two vocabulary distributions $p$ and $q$:

$\begin{split}\mathrm{JS}\left(p||q\right)=\frac{1}{2}\left(\mathrm{KL}(p||\frac{p+q}{2})+\mathrm{KL}(q||\frac{p+q}{2})\right),\end{split}$

where $\mathrm{KL}(\cdot||\cdot)$ is the KL divergence Kullback and Leibler
(1951) of two distributions.

Table 2 shows the JS divergence of the vocabulary distributions between the
source- and target-original data. We also divide the words into content words
and functions words based on their POS tags, since content words are more
related to the language coverage bias, while the function words are more
related to the stylistic and structural differences between the translationese
and original texts Lembersky et al. (2011); Volansky et al. (2015). The JS
divergence between the source- and target-original data are 186$\times$ larger
than that between randomly split data, which is mainly due to the difference
between content words. Results for different ratios $R\%$ and other language
pairs can be found in Appendix (Tables 12 and 13), where the trend holds in
all cases, supporting our claim of the existence of language coverage bias.

###  3.3 Effect of Language Coverage Bias

In this section, we investigate the effect of language coverage bias on NMT
models.

Data | En-Zh | En-Ja | En-De
---|---|---|---
Origin | $\Rightarrow$ | $\Leftarrow$ | $\Rightarrow$ | $\Leftarrow$ | $\Rightarrow$ | $\Leftarrow$
Target | 33.2 | 20.6 | 30.5 | 15.4 | 39.3 | 37.4
Source | 36.5 | 27.8 | 35.3 | 17.9 | 41.7 | 42.5
Both | 36.6 | 27.5 | 34.9 | 18.5 | 42.3 | 42.2

Table 3: Sacre BLEU of using different sets of training data on validation
sets. We highlight the highest score in bold and the second-highest score with
underlines.

Data | En$\Rightarrow$Zh | En$\Leftarrow$Zh
---|---|---
Origin | noun | verb | adj | noun | verb | adj
Target | 67.6 | 52.0 | 64.3 | 53.8 | 38.0 | 57.0
Source | 69.7 | 54.0 | 66.2 | 61.8 | 44.1 | 63.9
Both | 69.9 | 54.1 | 65.9 | 61.2 | 43.8 | 63.4

Table 4: Translation adequacy of different types of content words measured by
F-measure Neubig et al. (2019). The results are reported on the validation
sets.

#### Using only the source-original data achieves comparable performance with
using full data.

Table 3 lists the translation performances of NMT models trained on only the
source- or target-original data and on both of them. The results show that
using only the source-original data significantly outperforms using the
target-original data in all language pairs, which reconfirm the necessity of
studying the language coverage bias for NMT. It should be emphasized that
using only the source-original data (i.e. 50% of the whole training set)
achieves translation performances on par with using full training data. In the
following experiments, we investigate why using target-original data together
cannot further improve the performance.

#### Using additional target-original data does not consistently improve
translation adequacy.

To rule out the effect of translationese and focus on the content-dependent
difference caused by the language coverage bias, we examine the translation
adequacy of content words in Table 4 222We only list the results on
En$\Leftrightarrow$Zh due to space limit. Please refer to Table 14 in the
Appendix for the translation quality on other language pairs.. We follow
Raunak et al. (2020) to use F-measure Neubig et al. (2019) to quantify the
translation accuracy of specific types of words.

Compared with the source-original data, using only the target-original data
greatly reduces the translation accuracy of content words, which we attribute
to the divergence of the content word distributions between the source- and
target-original data. The results also indicate that indiscriminately using
all the training data can not consistently improve the translation adequacy of
content words over using only source-original data, and in some cases using
all the data is even harmful to the adequacy on content words. Table 5 shows
an example, which suggests that using only the target-original data tends to
omit content words. This problem is potentially caused by that some content
words at the source-side are less or even not visible in the target-original
data, and indiscriminately adding target-original data induces a distribution
shift on the content word distribution.

Input |  大闸蟹是巴城最为知名的形象代言人。
---|---
Refer. | The  hairy crab is the most famous image
spokesperson in  Bacheng.
Target | It is one of the city’s most well-known
Orig. | image spokesmen.
Source |  Hairy crabs are the most well-known
Orig. | image spokesmen of  Bacheng.
Both | It is the best-known icon of  Bacheng.

Table 5: An example of the outputs of NMT models trained on different sets of
data. Using the target-original data tends to omit content words.

#### Using additional target-original data only slightly improves the
structural fluency.

Recently, Edunov et al. (2020) claim that using additional back-translated
data can improve translation fluency. Target-original bilingual data is
similar to back-translated data since both of them are constructed by
translating sentences from the target language into the source language. One
question naturally arises: can target-original bilingual data improve the
fluency of NMT models?

To answer the above question, we measure the fluency of outputs with language
models trained on the monolingual data as described in Section 2. Previous
study finds that different perplexities could be caused by specific contents
rather than structural differences Lembersky et al. (2011). Specifically, some
source-original contents are of low frequency in the target-language
monolingual data (e.g., “Bacheng” in Table 5), thus the language model trained
on the target-language monolingual data tends to assign higher perplexities to
outputs containing more source-original content words. To rule out this
possibility and check whether the outputs are structurally different, we
follow Lembersky et al. (2011) to abstract away from the content-specific
features of the outputs to measure their fluency at the syntactic level. Table
6 shows the results. Although using only the source-original data results in
high perplexities measured by vanilla language models, the perplexities of NMT
models trained on different data are close to each other at the syntactic
level. Using additional target-original data only slightly reduces the
perplexity at the syntactic level over using only the source-original data.

Data | No Abs. | Cont. Abs.
---|---|---
Origin | PPL | Diff. | PPL | Diff.
WMT20 En$\Rightarrow$Zh
Target | 38.4 |  $-$6.3% | 14.0 |  $-$2.8%
Source | 44.0 |  $+$7.3% | 14.6 |  $+$1.4%
Both | 41.0 | - | 14.4 | -
WMT20 En$\Leftarrow$Zh
Target | 25.9 |  $-$5.8% | 13.4 |  $-$3.6%
Source | 31.0 |  $+$12.7% | 14.2 |  $+$2.2%
Both | 27.5 | - | 13.9 | -

Table 6: Translation fluency measured by the perplexities (i.e., PPL) of
language models with different levels of lexical abstraction, “Diff.” means
the relative change with respect to “Both”. “No Abs.” denotes no abstraction
(i.e., vanilla LM), “Cont. Abs.” denotes abstracting all content words with
their corresponding POS tags. The results are reported on the validation sets.

##  4 Addressing Language Coverage Bias

Method | En-Zh | En-Ja | En-De | Average
---|---|---|---|---
$\Rightarrow$ | $\Leftarrow$ | $\Rightarrow$ | $\Leftarrow$ | $\Rightarrow$ | $\Leftarrow$
WMT20 Systems |  |  |  |  |  |  |
Shi et al. (2020) | 38.6 | 28.8 |  - |  - |  - |  - | -
Zhang et al. (2020) | 40.8 |  - | 34.8 | 20.4 |  - |  - | -
Molchanov (2020) |  - |  - |  - |  - | 31.9 | 39.6 | -
Our Implemented Systems |  |  |  |  |  |  |
Baseline | 42.3 | 28.4 | 35.8 | 20.9 | 32.3 | 41.4 | 33.5
Tag | 43.4⇑ |  29.2↑ | 36.3 | 21.9⇑ | 32.7 | 42.5⇑ | 34.3
Tune |  43.3⇑ | 29.7⇑ | 36.6↑ |  21.8↑ | 32.9↑ |  42.2↑ | 34.4
Table 7: Sacre BLEU reported on the WMT20 test sets. “Tag” and “Tune” denote
the bias-tagging and fine-tuning, respectively. We highlight the highest score
in bold and the second-highest score with underlines. “$\uparrow$/$\Uparrow$”
denotes significantly better than the baseline with $p<0.05$ and $p<0.01$,
respectively. For comparison, we list three systems that use vanilla
Transformer models trained on the bilingual data in the WMT20 competition.

In Section 3 we show that the target-original data performs poorly in
translating content words due to the language coverage bias. Accordingly,
simply using the full training data without distinguishing the original
languages is sub-optimal for model training. Based on these findings, we
propose to address the language coverage bias by explicitly distinguishing
between the source- and the target-original data (Section 4.1). We then
investigate whether the performance improvement still holds in the monolingual
data augmentation scenario (Section 4.2), where the language coverage bias
problem is more severe due to the newly introduced dataset in source or target
language.

###  4.1 Bilingual Data Utilization

In this section, we aim to improve bilingual data utilization through
explicitly distinguishing between the source- and target-original training
data.

#### Methodology

We distinguish original languages with two simple and effective methods:

  * •

Bias-Tagging: Tagging is a commonly-used approach to distinguishing between
different types of examples, such as different languages Aharoni et al.
(2019); Riley et al. (2020) and synthetic vs authentic examples Caswell et al.
(2019). In this work, we attach a special tag to the source side of each
target-original example, which enables NMT models to distinguish it from the
source-original ones in training.

  * •

Fine-Tuning: Fine-tuning Luong and Manning (2015) is a useful method to help
knowledge transmit among data from different distributions. We pre-train NMT
models on the full training data that consists of both the source- and target-
original data, and then fine-tune them on only the source-original data. For
fair comparison, the total training steps of the pre-training and fine-tuning
stages are the same as the baseline.

#### Translation Performance

Table 7 depicts the results on the benchmarking datasets. For comparison, we
also list the results of several baselines using the vanilla Transformer
architecture trained on the constrained bilingual data in the WMT20
competition Barrault et al. (2020). Clearly, both the bias tagging and fine-
tuning approaches consistently improve translation performance on all
benchmarks, which confirms our claim of the necessity of explicitly
distinguishing target-original examples in model training.

Method | En$\Rightarrow$Zh | En$\Leftarrow$Zh
---|---|---
noun | verb | adj | noun | verb | adj
Baseline | 70.7 | 61.0 | 67.9 | 60.2 | 43.6 | 61.4
Tag | 72.3 | 62.3 | 67.9 | 60.7 | 43.9 | 62.2
Tune | 71.8 | 61.9 | 68.4 | 61.1 | 44.1 | 62.2

Table 8: F-measure of different types of content words on the WMT20
En$\Leftrightarrow$Zh test sets. Results on other languages can be found in
Appendix (Table 15).

Data | En-Zh | En-Ja | En-De
---|---|---|---
Origin | $\Rightarrow$ | $\Leftarrow$ | $\Rightarrow$ | $\Leftarrow$ | $\Rightarrow$ | $\Leftarrow$
Baseline | 13.7 | 13.4 | 17.4 | 15.4 | 16.3 | 17.8
Tag | 13.7 | 13.4 | 17.5 | 15.4 | 16.4 | 17.8
Tune | 13.8 | 13.4 | 17.4 | 15.4 | 16.3 | 17.9

Table 9: PPL at the syntactic level on the test sets. We abstract the content words to rule out the language coverage bias when measuring fluency. # | Monolingual | Bilingual | X$\Rightarrow$En | En$\Rightarrow$X
---|---|---|---|---
Data | Tagging | Fine-Tune | Zh | Ja | De | Ave. | Zh | Ja | De | Ave.
1 | × | n/a | × | 28.4 | 20.9 | 41.4 | 30.2 | 42.3 | 35.8 | 32.3 | 36.8
2 | ✓ | 29.7 | 21.8 | 42.2 | 31.2 | 43.3 | 36.6 | 32.9 | 37.6
3 | ✓ | × | × | 28.9 | 21.2 | 38.1 | 29.4 | 44.2 | 36.8 | 32.8 | 37.9
4 | ✓ | 29.4 | 21.2 | 41.5 | 30.7 | 43.1 | 36.4 | 32.3 | 37.3
5 | × | ✓ | 30.4 | 22.1 | 42.3 | 31.6 | 45.1 | 37.6 | 33.4 | 38.7
6 | ✓ | 30.6 | 22.2 | 42.7 | 31.8 | 44.7 | 36.9 | 33.3 | 38.3
Table 10: Translation performance of augmenting English monolingual data with
different strategies: back-translation for X$\Rightarrow$En tasks ( blue
cells), and forward-translation for En$\Rightarrow$X tasks ( red cells).
“Tagging” denotes adding a special tag to each synthetic sentence pair Caswell
et al. (2019). “Fine-Tune” denotes fine-tuning the pre-trained NMT models on
the source-original bilingual data, as described in Section 4.1.

#### Analysis

Recent studies have shown that generating human-translation like texts as
opposed to original texts can improve the BLEU score Riley et al. (2020). To
validate that the improvement is partially from alleviating the content-
dependent language coverage bias, we examine the translation adequacy of
content words on the test sets, as listed in Table 8. The results indicate
that explicitly distinguishing between the source- and target-original data
improves the translation of content words (e.g., nouns), which is closely
related to the language coverage bias problem. Table 9 lists the translation
fluency at the syntactic level, where the proposed approaches maintain the
syntactic fluency.

###  4.2 Monolingual Data Augmentation

In this section, we aim to provide some insights where monolingual data
augmentation improves translation performance, and investigate whether our
approach can further improve model performance in this scenario that
potentially suffers more from the language coverage bias problem.

For fair comparison across language pairs, we augment NMT models with the same
English monolingual corpus as described in Section 2. We down-sample the
large-scale monolingual corpus to the same amount as that of the bilingual
corpus in each language pair, in order to rule out the effect of the scale of
synthetic data Edunov et al. (2018); Fadaee and Monz (2018). We use back-
translation Sennrich et al. (2016a) to augment the English monolingual data
for the task of translating from another language to English
(“X$\Rightarrow$En”), and use forward-translation for the task in the opposite
translation direction (“En$\Rightarrow$X”). Table 10 lists the results, where
several observations can be made.

#### Explaining Data Augmentation with Language Coverage Bias

Concerning the monolingual data augmentation methods (Rows 3-4), the vanilla
back-translation (Row 3) harms the translation performance on average, while
the vanilla forward-translation improves the performance, which is consistent
with the findings in previous studies Edunov et al. (2020); Marie et al.
(2020). Caswell et al. (2019) have shown that the tagging strategy works for
back-translation while fails for forward-translation, and our results confirm
these findings. Both phenomena can be attributed in part to the language
coverage bias problem. Back-translated data originates from the target
language, and thus suffers more from the language coverage bias problem.
Accordingly, directly using the back-translated data is sub-optimal, while
tagged back-translation recovers translation performance by distinguishing
training examples with different origins, which is consistent with our results
in Table 7. In contrast, the language coverage bias problem does not exist for
source-side monolingual data (i.e. the same original language). Therefore, the
vanilla forward-translation can improve translation performance, while tagged
forward-translation performs worse.

#### Improving Data Augmentation

Our approach (Row 2) achieves comparable improvements of translation
performance with the monolingual data augmentation approaches (e.g. averaged
BLEU: 31.2 vs. 30.7, and 37.6 vs. 37.9), while we do not use additional
monolingual data to train the models.333The monolingual data is only used to
detect the original languages of training data and is invisible in model
training. Combining them can further improve performance (Rows 5-6),
indicating that the two types of approaches are complementary to each other.
This is straightforward, since our approach better exploits the bilingual
data, while data augmentation introduces new knowledge from additional
monolingual data. In addition, our approach consistently improves performance
over both vanilla and tagged augmentation approaches, making it more robust in
practical application across datasets.

##  5 Related Work

Our work is inspired by three lines of research in the NMT community.

###  5.1 Translationese

Recently, the effect of translationese in NMT evaluation has attracted
increasing attention Zhang and Toral (2019); Bogoychev and Sennrich (2019);
Edunov et al. (2020); Graham et al. (2020). Graham et al. (2020) show that the
source-side translationese texts can potentially lead to distortions in
automatic and human evaluations. Accordingly, the WMT competition starts to
use only source-original test sets for most translation directions since 2019.
Our study reconfirms the necessity of distinguishing the source- and target-
original examples and takes one step further to distinguish examples in
training data. Complementary to previous works, we investigate the effect of
language coverage bias on machine translation, which is related to the content
bias rather than the language style difference. Shen et al. (2021) also reveal
the context mismatch between texts from different original languages. To
alleviate this problem, they proposed to combine back- and forward-translation
by introducing additional monolingual data, while we focus on better
exploiting bilingual data by distinguishing the original languages, which is
also helpful for back- and forward-translation.

Lembersky et al. (2011, 2012) propose to adapt machine translation systems to
generate texts that are more similar to human-translations, while Riley et al.
(2020) propose to model human-translated texts and original texts as separate
languages in a multilingual model and perform zero-shot translation between
original texts. Riley et al. (2020) and our work both aim to better utilize
the bilingual training data. They aim to guide NMT models to produce original
text, while we focus on improving translation adequacy by alleviating the
language coverage bias problem.

###  5.2 Data Augmentation

Concerning model training, recent works find that back-translation can harm
the translation of source-original test set, and attribute the quality drop to
the stylistic and content-independent differences between translationese and
original texts Edunov et al. (2020); Marie et al. (2020). In this work, we
empirically show that language coverage bias is another reason for the
performance drop of back-translation, as well as the different performances
between tagged forward-translation and tagged back-translation Caswell et al.
(2019). In addition, we show that our approach is also beneficial for data
augmentation approaches, which can further improve the translation performance
over both back-translation and forward-translation.

###  5.3 Domain Adaptation

Since high-quality and domain-specific parallel data is usually scarce or even
unavailable, domain adaptation approaches are generally employed for
translation in low-resource domains by leveraging out-of-domain data Chu and
Wang (2018). Languages can be also regarded as different domains, since
articles in different languages cover different topics Bogoychev and Sennrich
(2019). Starting from this intuition, we distinguish examples with different
original languages with tagging Aharoni et al. (2019) and fine-tuning Luong
and Manning (2015), which are commonly-used in domain adaptation and multi-
lingual translation tasks.

Our work also benefits domain adaptation: distinguishing original languages in
general domain data consistently improves translation performance of NMT
models in several specific domains (Table 16 in Appendix), making these models
better start points for further domain adaptation.

##  6 Conclusion and Future Work

In this work, we first systematically examine why the language coverage bias
problem is important for NMT models. We conducted extensive experiments on six
WMT20 translation benchmarks. Empirically, we find that source-original data
and target-original data differ significantly at the text content, and using
target-original data together without discrimination is sub-optimal. Based on
these observations, we propose two simple and effective approaches to
distinguish the source- and target-original training data, which obtain
consistent improvements in all benchmarks.

Furthermore, we link language coverage bias to two well-known problems in
monolingual data augmentation, namely the performance drop of back-
translation, and the different behaviors between tagged back-translation and
tagged forward-translation. We show that language coverage bias can be
considered as another reason for these problems, and fine-tuning on the
source-original bilingual training data can further improve performance over
both back- and forward-translation.

Future directions include exploring advanced methods to better alleviate the
language coverage bias problem, as well as validating on other language pairs.
It is also interesting to investigate the language coverage bias problem in
multilingual translation, where we can better understand this problem by
considering language family.

## Acknowledgments

We thank all anonymous reviewers for their insightful comments and suggestions
for this work. We also sincerely thank Cunxiao Du and Wenxiang Jiao for their
valuable help and advice. This work was supported by the National Key R&D
Program of China (No. 2017YFB0202204), National Natural Science Foundation of
China (No. 61925601, No. 61772302), and the Tencent AI Lab Rhino-Bird Focused
Research Program.

## References

  * Aharoni et al. (2019) Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019.  Massively multilingual neural machine translation.  In _NAACL_.
  * Baker et al. (1993) Mona Baker, Gill Francis, and Elena Tognini-Bonelli. 1993.  Corpus linguistics and translation studies: Implications and applications.  _Text and technology: In honour of John Sinclair_ , 233:250.
  * Barrault et al. (2020) Loïc Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, et al. 2020.  Findings of the 2020 conference on machine translation (wmt20).  In _WMT_.
  * Barrault et al. (2019) Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019.  Findings of the 2019 conference on machine translation (WMT19).  In _WMT_.
  * Bogoychev and Sennrich (2019) Nikolay Bogoychev and Rico Sennrich. 2019.  Domain, translationese and noise in synthetic data for neural machine translation.  _arXiv preprint arXiv:1911.03362_.
  * Caswell et al. (2019) Isaac Caswell, Ciprian Chelba, and David Grangier. 2019.  Tagged back-translation.  In _WMT_.
  * Chu and Wang (2018) Chenhui Chu and Rui Wang. 2018.  A survey of domain adaptation for neural machine translation.  In _COLING_.
  * D’Alessio and Allen (2000) Dave D’Alessio and Mike Allen. 2000.  Media bias in presidential elections: A meta-analysis.  _Journal of communication_ , 50(4):133–156.
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding back-translation at scale.  In _EMNLP_.
  * Edunov et al. (2020) Sergey Edunov, Myle Ott, Marc’Aurelio Ranzato, and Michael Auli. 2020.  On the evaluation of machine translation systems trained with back-translation.  In _ACL_.
  * Fadaee and Monz (2018) Marzieh Fadaee and Christof Monz. 2018.  Back-translation sampling by targeting difficult words in neural machine translation.  In _EMNLP_.
  * Fleiss (1971) Joseph L Fleiss. 1971.  Measuring nominal scale agreement among many raters.  _Psychological bulletin_ , 76(5):378.
  * Freitag et al. (2020) Markus Freitag, David Grangier, and Isaac Caswell. 2020.  BLEU might be guilty but references are not innocent.  In _EMNLP_.
  * Graham et al. (2020) Yvette Graham, Barry Haddow, and Philipp Koehn. 2020.  Statistical power and translationese in machine translation evaluation.  In _EMNLP_.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _ICLR_.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007.  Moses: Open source toolkit for statistical machine translation.  In _ACL_.
  * Kullback and Leibler (1951) Solomon Kullback and Richard A Leibler. 1951.  On information and sufficiency.  _The annals of mathematical statistics_ , 22(1):79–86.
  * Lembersky et al. (2011) Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2011.  Language models for machine translation: Original vs. translated texts.  In _EMNLP_.
  * Lembersky et al. (2012) Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2012.  Adapting translation models to translationese improves smt.  In _EACL_.
  * Lin (1991) Jianhua Lin. 1991.  Divergence measures based on the shannon entropy.  _IEEE Transactions on Information theory_ , 37(1):145–151.
  * Luong and Manning (2015) Minh-Thang Luong and Christopher D Manning. 2015.  Stanford neural machine translation systems for spoken language domains.  In _IWSLT_.
  * Marie et al. (2020) Benjamin Marie, Raphael Rubino, and Atsushi Fujita. 2020.  Tagged back-translation revisited: Why does it really work?  In _ACL_.
  * Molchanov (2020) Alexander Molchanov. 2020.  Promt systems for wmt 2020 shared news translation task.  In _WMT_.
  * Neubig et al. (2019) Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, and Xinyi Wang. 2019.  compare-mt: A tool for holistic comparison of language generation systems.  In _NAACL_.
  * Ng et al. (2019) Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019\.  Facebook fair’s wmt19 news translation task submission.  In _WMT_.
  * Ott et al. (2018) Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018.  Scaling neural machine translation.  In _WMT_.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting bleu scores.  In _WMT_.
  * Raunak et al. (2020) Vikas Raunak, Siddharth Dalmia, Vivek Gupta, and Florian Metze. 2020.  On long-tailed phenomena in neural machine translation.  In _Findings of EMNLP_.
  * Riley et al. (2020) Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. 2020.  Translationese as a language in “multilingual” nmt.  In _ACL_.
  * Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a.  Improving neural machine translation models with monolingual data.  In _ACL_.
  * Sennrich et al. (2016b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b.  Neural machine translation of rare words with subword units.  In _ACL_.
  * Shen et al. (2021) Jiajun Shen, Peng-Jen Chen, Matthew Le, Junxian He, Jiatao Gu, Myle Ott, Michael Auli, and Marc’Aurelio Ranzato. 2021.  The source-target domain mismatch problem in machine translation.  In _EACL_.
  * Shi et al. (2020) Tingxun Shi, Shiyu Zhao, Xiaopu Li, Xiaoxue Wang, Qian Zhang, Di Ai, Dawei Dang, Zhengshan Xue, and Jie Hao. 2020.  Oppo’s machine translation systems for wmt20.  In _WMT_.
  * Toral (2019) Antonio Toral. 2019.  Post-editese: an exacerbated translationese.  In _MT Summit_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NeurIPS_.
  * Volansky et al. (2015) Vered Volansky, Noam Ordan, and Shuly Wintner. 2015.  On the features of translationese.  _Digital Scholarship in the Humanities_ , 30(1):98–118.
  * Wu et al. (2018) Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. 2018.  Pay less attention with lightweight and dynamic convolutions.  In _ICLR_.
  * Zhang and Toral (2019) Mike Zhang and Antonio Toral. 2019.  The effect of translationese in machine translation test sets.  In _WMT_.
  * Zhang et al. (2020) Yuhao Zhang, Ziyang Wang, Runzhe Cao, Binghao Wei, Weiqiao Shan, Shuhan Zhou, Abudurexiti Reheman, Tao Zhou, Xin Zeng, Laohu Wang, Xiaoqian Liu, Xunjuan Zhou, Yongyu Mu, Jingnan Zhang, Yinqiao Li, Bei Li, Tong Xiao, and Jingbo Zhu. 2020.  The niutrans machine translation systems for wmt20.  In _WMT_.

##  Appendix A Appendices

###  A.1 Data Preprocessing

We used all the parallel corpora provided by WMT20 and filtered sentences that
are longer than 250 words. We tokenized English and German sentences with
Moses Koehn et al. (2007), and segmented Chinese and Japanese sentences with
Jieba444https://github.com/fxsjy/jieba and
Mecab555https://taku910.github.io/mecab respectively. We employed Byte pair
encoding (BPE) Sennrich et al. (2016b) with 32K merge operations for all
language pairs. Specifically, we jointly trained the BPE code on both sides in
En$\Leftrightarrow$De and independently learned the BPE code on each side in
En$\Leftrightarrow$Zh and En$\Leftrightarrow$Ja.

As for the monolingual data, we combined the newscrawl data from 2017 to 2019
for English and German. Since the newscrawl corpora for Chinese and Japanese
are significantly smaller, we augmented these two languages with the
commoncrawl corpus. We preprocessed the monolingual data with the same rules
as parallel data. Finally, we randomly selected 41.0M sentences for each
language (i.e., En, De, Zh, Ja), which were used to train the language
detection models. For data augmentation, to rule out the effect of the ratio
between synthetic and authentic data, we down-sampled the monolingual data to
the same amount as the bilingual data for each language pair.

We used spaCy666https://spacy.io to perform the Part-Of-Speech (POS) tagging
for each language. Nouns, verbs, and adjectives belong to content words and
the others belong to function words.

###  A.2 More Details of Model Training

In this work, we generally followed the default hyper-parameters used in
Vaswani et al. (2017) except the batch size. Recent studies showed that
training on large batches can further boost model performance Ott et al.
(2018); Wu et al. (2018). Accordingly, we followed them to train models with
batches of approximately 460k tokens, using Adam (Kingma and Ba, 2015) with
$\beta_{1}=0.9$, $\beta_{2}=0.98$ and $\epsilon=10^{-8}$. We used the same
cosine learning rate schedule as Wu et al. (2018), where the learning rate was
warmed up linearly in the first 10K steps, and then decayed following a cosine
rate within a single cycle. By default, NMT models and language models were
both trained for 30k steps with the aforementioned batch size. Each model was
trained using 8 NVIDIA V100 GPUs for about 20 hours.

###  A.3 Effect of Detection Methods on Translation Performance

To further compare our proposed original language detection method and the FT
classifier Riley et al. (2020), we fine-tune the NMT model pre-trained on the
whole training set using the source-original data detected by the two methods.
Note that the two detection methods are developed using the same monolingual
data sets. For fair comparison, the fine-tuning sets are of the same amount
(50% of the whole training set) between the two methods in this experiment.
Table 11 lists the results, indicating that our method performs better in
detecting original languages in large-scale parallel data.

Fine-Tune Data | BLEU
---|---
× | 27.5
FT | 27.8
Ours | 28.4
Table 11: Effect of original language detection methods. The results are reported on the validation set of the Zh$\Rightarrow$En translation task. Data | 10% | 30% | 50%
---|---|---|---
All | Content | Function | All | Content | Function | All | Content | Function
Random | 20 | 51 | 0 | 7 | 18 | 0 | 4 | 10 | 0
S vs T | 2660 | 5096 | 961 | 1371 | 2731 | 486 | 745 | 1503 | 261
Table 12: JS divergence ($\times 10^{-5}$) of the vocabulary distributions between the source- and target-original training data (“S vs T”) for different labeled ratios on En$\Leftrightarrow$Zh. For reference, we also report the JS divergence between two sets of randomly selected examples (“Random”, non-overlap). Data | En-Zh | En-Ja | En-De
---|---|---|---
All | Content | Function | All | Content | Function | All | Content | Function
Random | 4 | 10 | 0 | 8 | 17 | 0 | 2 | 4 | 0
S vs T | 745 | 1503 | 261 | 1687 | 2910 | 666 | 870 | 1622 | 250
Table 13: JS divergence ($\times 10^{-5}$) of the vocabulary distributions between the source- and target-original training data for different language pairs. 50% examples are treated as source-original and the others are treated as target-original. For reference, we also report the JS divergence between randomly selected 50% examples and the others (“Random”, non-overlap). Data | En$\Rightarrow$Ja | En$\Leftarrow$Ja | En$\Rightarrow$De | En$\Leftarrow$De
---|---|---|---|---
Origin | noun | verb | adj | noun | verb | adj | noun | verb | adj | noun | verb | adj
Target | 60.9 | 47.7 | 62.1 | 44.5 | 29.8 | 46.2 | 70.5 | 54.3 | 58.4 | 70.8 | 53.6 | 67.0
Source | 61.4 | 51.8 | 63.5 | 49.5 | 31.8 | 50.1 | 72.1 | 55.0 | 60.3 | 75.3 | 55.3 | 70.2
Both | 61.3 | 51.7 | 63.2 | 50.7 | 32.1 | 50.4 | 72.7 | 56.6 | 60.4 | 74.9 | 55.7 | 71.0
Table 14: Translation adequacy of different types of content words measured by F-measure Neubig et al. (2019). The results are reported on the validation sets. Method | En$\Rightarrow$Ja | En$\Leftarrow$Ja | En$\Rightarrow$De | En$\Leftarrow$De
---|---|---|---|---
noun | verb | adj | noun | verb | adj | noun | verb | adj | noun | verb | adj
Baseline | 62.0 | 53.0 | 59.1 | 54.0 | 35.5 | 50.4 | 66.7 | 48.1 | 53.6 | 75.0 | 54.0 | 70.2
Tag | 62.5 | 53.3 | 61.1 | 55.7 | 36.5 | 52.4 | 67.1 | 48.6 | 54.0 | 76.1 | 54.4 | 70.7
Tune | 62.8 | 53.7 | 61.7 | 55.3 | 36.9 | 51.8 | 67.1 | 48.7 | 54.0 | 75.7 | 54.8 | 70.8
Table 15: Translation adequacy of different types of content words measured by F-measure Neubig et al. (2019). The results are reported on the test sets. Domain | Baseline | Ours
---|---|---
Business | 40.4 | 40.8
Crime | 34.8 | 35.5
Entertainment | 28.8 | 30.0
Politics | 39.5 | 40.3
Sci-Tech | 38.2 | 39.9
Sport | 31.5 | 31.5
World | 38.8 | 38.9
Overall | 36.6 | 37.2
Table 16: Transformer performance on the validation set of the
En$\Rightarrow$Zh task. We split the whole validation set into several parts
by the domain tag. “Ours” denotes the “Bias-Tagging” approach as described in
Section 4.1. The results indicate that distinguishing data with different
original languages in the general domain training data can improve the
performance of NMT models in many specific domains, making the models better
start points for further domain adaptation.

###  A.4 Divergence of Vocabulary Distributions

In this section, we report the JS divergence of the vocabulary distributions
in more cases. Table 12 lists the results for different ratios $R\%$ on
En$\Leftrightarrow$Zh, and Table 13 shows the results on all language pairs.
The results show that the divergence of vocabulary distributions between the
source- and target-original data is substantially larger than that between
randomly split data, which reconfirms the existence of language coverage bias.

###  A.5 Effect of Language Coverage Bias for Other Language Pairs

Table 14 lists the translation adequacy of NMT models trained on only the
source- or target-original data and on both of them. The results are reported
on En$\Leftrightarrow$De and En$\Leftrightarrow$Ja, which exhibit the same
trend as that on En$\Leftrightarrow$Zh (Table 4 in the main paper), indicating
that the target-original data performs poorly in translating content words.

###  A.6 Translation Adequacy on Test Sets for Other Language Pairs

We report the translation adequacy on test sets for En$\Leftrightarrow$De and
En$\Leftrightarrow$Ja in Table 15, corresponding to Table 8 in the main paper.
The results show that explicitly distinguishing the source- and target-
original training data can consistently improve the translation adequacy for
content words on all the six translation tasks.

###  A.7 Translation Performance in Specific Domains

We evaluate NMT models trained with and without explicit distinguishing
between the source- and target-original data in several specific domains. The
results are shown in Table 16, suggesting that our method can improve the
translation performance of NMT models in several specific domains, which can
be combined with further domain adaptation approaches.