$z\in\mathbb{C}$ and $(m,n)\in A(z)$:

  1. (A1)

$|\omega_{m,n}|>5R$,
${|\omega_{m,n}|/|\lambda_{m,n}|}\in\left({5/6,5/4}\right)$,

  2. (A2)

$|z/\omega_{m,n}|<\frac{4}{5}\mathds{1}_{\\{|z|\leqslant
4R\\}}+\frac{4}{7}\mathds{1}_{\\{|z|>4R\\}}$, and

  3. (A3)

$|z/\lambda_{m,n}|<1/2$.

The inequality in (A1) follows from
$|\omega_{m,n}|\geqslant|\lambda_{m,n}|-R>6R-R=5R$, and the upper and lower
bounds on $|\omega_{m,n}|/|\lambda_{m,n}|$ are due to
$|\omega_{m,n}|\geqslant|\lambda_{m,n}|-R>\frac{5}{6}|\lambda_{m,n}|$ and
$|\lambda_{m,n}|\geqslant|\omega_{m,n}|-R>\frac{4}{5}|\omega_{m,n}|$. To show
(A2), consider the cases $|z|\leqslant 4R$ and $|z|>4R$ separately. If
$|z|\leqslant 4R$, then $|z/\omega_{m,n}|<4R/5R=4/5$, whereas if $|z|>4R$,
then $|\omega_{m,n}|\geqslant|\lambda_{m,n}|-R>2|z|-R>\frac{7}{4}|z|$ and so
$|z/\omega_{m,n}|<4/7$. Finally, (A3) follows directly by the definition of
$A(z)$.

Now, using (A2), we have

$\displaystyle|h_{\mathrm{aux}}(z)|$ $\displaystyle\leqslant\prod_{(m,n)\in
A(z)}\frac{|1-z/\lambda_{m,n}||\exp(z/\lambda_{m,n})|}{|1-z/\omega_{m,n}||\exp(z/\omega_{m,n})|}$
$\displaystyle\leqslant\Bigg{(}\mathds{1}_{\\{|z|>4R\\}}\;+\mathds{1}_{\\{|z|\leqslant
4R\\}}\prod_{\begin{subarray}{c}(m,n)\in A(z)\\\
|\omega_{m,n}|\leqslant\frac{7}{4}|z|\leqslant
7R\end{subarray}}\frac{\left(1+\frac{1}{2}\right)e^{\frac{1}{2}}}{\left(1-\frac{4}{5}\right)e^{-\frac{4}{5}}}\Bigg{)}\prod_{\begin{subarray}{c}(m,n)\in
A(z)\\\
|\omega_{m,n}|>\frac{7}{4}|z|\end{subarray}}\frac{|1-z/\lambda_{m,n}||\exp(z/\lambda_{m,n})|}{|1-z/\omega_{m,n}||\exp(z/\omega_{m,n})|}$
$\displaystyle\lesssim_{\,\gamma,R}\Bigg{|}\prod_{(m,n)\in\widetilde{A}(z)}\frac{(1-z/\lambda_{m,n})\exp(z/\lambda_{m,n})}{(1-z/\omega_{m,n})\exp(z/\omega_{m,n})}\Bigg{|},\quad\forall
z\in\mathbb{C},$ (45)

where we set

$\widetilde{A}(z)=\\{(m,n)\in
A(z):|\omega_{m,n}|>{\textstyle\frac{7}{4}}|z|\\}.$

Next, for $z\in\mathbb{C}$ and $(m,n)\in A(z)$, it follows by (A2) that
$1-{z/\omega_{m,n}}$ and $1-{z/z_{m,n}}$ lie in the domain
$\mathbb{C}\setminus\mathbb{R}_{\leqslant 0}$ of the complex logarithm, which
we denote by $\mathrm{Log}$. We can thus write

$\displaystyle\prod_{(m,n)\in\widetilde{A}(z)}\frac{(1-z/\lambda_{m,n})\exp(z/\lambda_{m,n})}{(1-z/\omega_{m,n})\exp(z/\omega_{m,n})}$
$\displaystyle=\,$
$\displaystyle\prod_{(m,n)\in\widetilde{A}(z)}\exp{\left[\mathrm{Log}\left(1-\frac{z}{\lambda_{m,n}}\right)+\frac{z}{\lambda_{m,n}}-\mathrm{Log}\left(1-\frac{z}{\omega_{m,n}}\right)-\frac{z}{\omega_{m,n}}\right]}$
$\displaystyle=\,$
$\displaystyle\prod_{(m,n)\in\widetilde{A}(z)}\exp{\left[-\sum_{k=2}^{\infty}\frac{1}{k}\left(\frac{z}{\lambda_{m,n}}\right)^{k}+\sum_{k=2}^{\infty}\frac{1}{k}\left(\frac{z}{\omega_{m,n}}\right)^{k}\right]}$
$\displaystyle=\,$
$\displaystyle\exp{\Bigg{[}\sum_{(m,n)\in\widetilde{A}(z)}\sum_{k=2}^{\infty}\frac{z^{k}}{k}\left(\frac{1}{\omega_{m,n}^{k}}-\frac{1}{\lambda_{m,n}^{k}}\right)\Bigg{]}},$

which together with (45) yields

$|h_{\mathrm{aux}}(z)|\lesssim_{\,\gamma,R}\exp\Bigg{[}\sum_{(m,n)\in\widetilde{A}(z)}\sum_{k=2}^{\infty}\frac{|z|^{k}}{k}\left|\frac{1}{\omega_{m,n}^{k}}-\frac{1}{\lambda_{m,n}^{k}}\right|\Bigg{]},\quad\forall
z\in\mathbb{C}.$ (46)

Using (A1), we further have

$\displaystyle\left|\frac{1}{\omega_{m,n}^{k}}-\frac{1}{\lambda_{m,n}^{k}}\right|$
$\displaystyle=\frac{\left|\lambda_{m,n}-\omega_{m,n}\right|\sum_{\ell=0}^{k-1}|\lambda_{m,n}|^{k-1-\ell}|\omega_{m,n}|^{\ell}}{\left|\omega_{m,n}\right|^{k}\left|\lambda_{m,n}\right|^{k}}$
$\displaystyle\leqslant\frac{R\cdot|\omega_{m,n}|^{k-1}\sum_{\ell=0}^{k-1}\left(\frac{6}{5}\right)^{\ell}}{\left|\omega_{m,n}\right|^{2k}\left(\frac{4}{5}\right)^{k}}$
$\displaystyle\leqslant 5R\cdot{1.5}^{k}\left|\omega_{m,n}\right|^{-(k+1)},$
(47)

for all $z\in\mathbb{C}$, $(m,n)\in A(z)$, and all $k\in\mathbb{N}$. Next,
defining $c_{\mathrm{aux}}(z,R)\vcentcolon=\frac{7|z|}{4}\vee 5R$, (A1) and
(A2) imply that
$\widetilde{A}(z)\subset\\{(m,n)\in\mathbb{Z}^{2}:|\omega_{m,n}|>c_{\mathrm{aux}}(z,R)\\}$,
for all $z\in\mathbb{C}$, and so (46) and (47) together yield

$\displaystyle|h_{\mathrm{aux}}(z)|$
$\displaystyle\leqslant\exp\Bigg{[}\sum_{\begin{subarray}{c}(m,n)\in\widetilde{A}(z)\end{subarray}}\sum_{k=2}^{\infty}{\frac{5R|z|^{k}}{k}}\cdot
1.5^{k}|\omega_{m,n}|^{-(k+1)}\Bigg{]}$
$\displaystyle\leqslant\exp\Bigg{[}\sum_{\begin{subarray}{c}(m,n)\in\mathbb{Z}^{2}\\\
|\omega_{m,n}|>c_{\mathrm{aux}}(z,R)\end{subarray}}\quad\sum_{k=2}^{\infty}{\frac{5R|z|^{k}}{k}}\cdot
1.5^{k}|\omega_{m,n}|^{-(k+1)}\Bigg{]}$
$\displaystyle=\exp\Bigg{[}\sum_{k=2}^{\infty}{\frac{5R|z|^{k}}{k}}\cdot
1.5^{k}\cdot\sum_{\begin{subarray}{c}(m,n)\in\mathbb{Z}^{2}\\\
|\omega_{m,n}|>c_{\mathrm{aux}}(z,R)\end{subarray}}|\omega_{m,n}|^{-(k+1)}\Bigg{]}.$
(48)

Recall that $R\geqslant 3\gamma$, and so
$c_{\mathrm{aux}}(z,R)=\frac{7|z|}{4}\vee
5R\geqslant\frac{7|z|}{4}\vee\frac{20\gamma}{\sqrt{2}}$ implies

$c_{\mathrm{aux}}(z,R)-\gamma/\sqrt{2}\geqslant
0.95\,c_{\mathrm{aux}}(z,R)\geqslant 1.6\,|z|.$ (49)

Now, write $K_{\gamma}$ for the square of side length $\gamma$ centered at
$(0,0)$ and use $R\geqslant 3\gamma$ and (49) to obtain the following bound

$\displaystyle\sum_{|\omega_{m,n}|>c_{\mathrm{aux}}(z,R)}|\omega_{m,n}|^{-(k+1)}$
$\displaystyle\leqslant\sum_{|\omega_{m,n}|>c_{\mathrm{aux}}(z,R)}\left(\frac{|\omega_{m,n}|+\frac{\gamma}{\sqrt{2}}}{|\omega_{m,n}|}\right)^{k+1}\int_{(m\gamma,n\gamma)+K_{\gamma}}|w|^{-(k+1)}\left|\mathrm{d}w\right|$
(50)
$\displaystyle\leqslant\left(1+\frac{\gamma}{5R\sqrt{2}}\right)^{k+1}\int_{|w|\geqslant
c_{\mathrm{aux}}(z,R)-\frac{\gamma}{\sqrt{2}}}|w|^{-(k+1)}\left|\mathrm{d}w\right|$
$\displaystyle\leqslant\left(1+\frac{1}{15\sqrt{2}}\right)^{k+1}\frac{2\pi}{k-1}\left(c_{\mathrm{aux}}(z,R)-\gamma/\sqrt{2}\right)^{1-k}$
$\displaystyle\leqslant 1.04^{k+1}\cdot 2\pi\left(1.6|z|\right)^{1-k}$
$\displaystyle<11\cdot 0.65^{k}|z|^{1-k},$ (51)

for all $z\in\mathbb{C}\setminus\\{0\\}$ and $k\geqslant 2$. We now use (51)
in (48) to obtain

$\displaystyle|h_{\mathrm{aux}}(z)|$
$\displaystyle\lesssim_{\,\gamma,R}\exp\Bigg{[}\sum_{k=2}^{\infty}\frac{5R|z|^{k}}{k}\cdot
1.5^{k}\cdot 11\cdot 0.65^{k}|z|^{1-k}\Bigg{]}$
$\displaystyle\leqslant{\Bigg{[}R\,|z|\cdot\frac{55}{2}\sum_{k=2}^{\infty}0.975^{k}\Bigg{]}}$
$\displaystyle=e^{1100R\,|z|},\quad\forall z\in\mathbb{C}.$ (52)

We are now ready to bound $h$, and we do so by treating the cases $|z|>3R$ and
$|z|\leqslant 3R$ separately.

Case $|z|>3R\,$: We analyze $h(z)$ as a product
$h(z)=\prod_{k=1}^{5}h_{k}(z)$, where

$\displaystyle h_{1}(z)$
$\displaystyle=\exp\Bigg{[}z\sum_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\setminus\mathcal{I}_{s}\\\
|\lambda_{m,n}|\leqslant
2|z|\end{subarray}}\left(\frac{1}{\lambda_{m,n}}-\frac{1}{\omega_{m,n}}\right)\Bigg{]},$
$\displaystyle h_{2}(z)$
$\displaystyle=\frac{d(z,\Omega_{\gamma})}{d(z,\Lambda)}\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}-z|\leqslant
2R\end{subarray}}\frac{1-z/\lambda_{m,n}}{1-z/\omega_{m,n}},$ $\displaystyle
h_{3}(z)$
$\displaystyle=\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}-z|>2R\\\ |\omega_{m,n}|\leqslant
2R\end{subarray}}\frac{1-z/\lambda_{m,n}}{1-z/\omega_{m,n}},$ $\displaystyle
h_{4}(z)$
$\displaystyle=\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}-z|>2R\\\ |\omega_{m,n}|>2R\\\ |\lambda_{m,n}|\leqslant
2|z|\end{subarray}}\frac{1-z/\lambda_{m,n}}{1-z/\omega_{m,n}},\quad\text{and}$
$\displaystyle h_{5}(z)$
$\displaystyle=\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}|>2|z|\end{subarray}}\frac{(1-z/\lambda_{m,n})\exp(z/\lambda_{m,n})}{(1-z/\omega_{m,n})\exp(z/\omega_{m,n})},$

and bound the functions $h_{j}$ in order.

Bounding $h_{1}$: Note that $|\lambda_{m,n}|>3R$ implies $|\omega_{m,n}|>2R$,
and hence
$|\lambda_{m,n}|\geqslant|\omega_{m,n}|-R>\frac{1}{2}|\omega_{m,n}|$. We thus
have the following bound for
$(m,n)\in\mathcal{I}^{\prime}\setminus\mathcal{I}_{s}$:

$\displaystyle\left|\frac{1}{\lambda_{m,n}}-\frac{1}{\omega_{m,n}}\right|$
$\displaystyle\leqslant\mathds{1}_{\\{|\lambda_{m,n}|\leqslant
3R\\}}(2s^{-1}+\gamma^{-1})+\mathds{1}_{\\{|\lambda_{m,n}|>3R\\}}\frac{|\lambda_{m,n}-\omega_{m,n}|}{|\omega_{m,n}||\lambda_{m,n}|}$
$\displaystyle\leqslant\mathds{1}_{\\{|\lambda_{m,n}|\leqslant
3R\\}}(2s^{-1}+\gamma^{-1})+2R|\omega_{m,n}|^{-2},$

and therefore, as $\\#\\{(m,n)\in\mathcal{I}^{\prime}:|\lambda_{m,n}|\leqslant
3R\\}\leqslant 9\theta R^{2}$,

$\displaystyle|h_{1}(z)|$
$\displaystyle\leqslant\exp\Bigg{[}|z|\sum_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|z_{m,n}|\leqslant
2|z|\end{subarray}}\left|\frac{1}{z_{m,n}}-\frac{1}{\omega_{m,n}}\right|\Bigg{]},$
$\displaystyle\leqslant\exp\Bigg{[}9\theta
R^{2}(2s^{-1}+\gamma^{-1})|z|+2R|z|\cdot\sum_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\omega_{m,n}|\leqslant
2|z|+R\end{subarray}}\left|\omega_{m,n}\right|^{-2}\Bigg{]}.$ (53)

Recalling that $\log|z|>\log(3R)>\log(3)>0$, we can bound in a manner similar
to (51) to obtain

$\displaystyle\sum_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\omega_{m,n}|\leqslant 2|z|+R\end{subarray}}\left|\omega_{m,n}\right|^{-2}$
$\displaystyle\leqslant\left(\frac{\gamma+\frac{\gamma}{\sqrt{2}}}{\gamma}\right)^{2}\int_{\frac{\gamma}{\sqrt{2}}\leqslant|w|\leqslant
2|z|+R+\frac{\gamma}{\sqrt{2}}}|w|^{-2}\left|\mathrm{d}w\right|$
$\displaystyle\lesssim_{\,\gamma,R}\log|z|.$ (54)

Using this in (53) thus yields

$|h_{1}(z)|\leqslant e^{c_{1}|z|\log{|z|}},$ (55)

for some $c_{1}=c_{1}(s,\theta,\gamma,R)>0$ and all $z\in\mathbb{C}$ such that
$|z|>3R$.

Bounding $h_{2}$: We write
$|h_{2}|=h_{2}^{\mathrm{num}}/h_{2}^{\mathrm{den}}$, where

$h_{2}^{\mathrm{num}}(z)=\frac{1}{d(z,\Lambda)}\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}-z|\leqslant
2R\end{subarray}}\frac{|\lambda_{m,n}-z|}{|\lambda_{m,n}|},\quad\text{and}\quad
h_{2}^{\mathrm{den}}(z)=\frac{1}{d(z,\Omega_{\gamma})}\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}-z|\leqslant
2R\end{subarray}}\frac{|\omega_{m,n}-z|}{|\omega_{m,n}|}.$

In order to bound $h_{2}^{\mathrm{num}}$, we observe that one of the following
two circumstances occurs:

  * –

The distance from $z$ to $\Lambda$ is minimized at $\lambda_{0,0}$, and so
$d(z,\Lambda)=|z|>3R>1$.

  * –

The distance from $z$ to $\Lambda$ is minimized at a point $\lambda_{m,n}$,
where $(m,n)\in\mathcal{I}^{\prime}$, and so the term $d(z,\Lambda)$ cancels
with one of the factors $|\lambda_{m,n}-z|$ in the product over
$\\{(m,n)\in\mathcal{I}^{\prime}:|\lambda_{m,n}-z|\leqslant 2R\\}$.

These facts lead to the following bound

$h_{2}^{\mathrm{num}}(z)\leqslant\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}-z|\leqslant 2R\end{subarray}}\frac{|\lambda_{m,n}-z|\vee
1}{|z|-|\lambda_{m,n}-z|}\leqslant 2^{16\theta R^{2}}.$ (56)

For $h_{2}^{\mathrm{den}}$, we similarly observe that either
$(\mathrm{Re}\,z,\mathrm{Im}\,z)\in[-\frac{\gamma}{2},\frac{\gamma}{2}]^{2}$,
or $d(z,\Omega_{\gamma})$ cancels with a factor $|\omega_{m,n}-z|$ in the
product over $\\{(m,n)\in\mathcal{I}^{\prime}:|\lambda_{m,n}-z|\leqslant
2R\\}$. In either case the numerators of the terms remaining in the product
satisfy $|\omega_{m,n}-z|\geqslant\frac{\gamma}{2}$, and we thus have

$h_{2}^{\mathrm{den}}(z)\geqslant\left(\\!\frac{\gamma}{\sqrt{2}}\wedge{1}\\!\right)\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}-z|\leqslant 2R\end{subarray}}\frac{(\gamma/2)\wedge
1}{|\omega_{m,n}-\lambda_{m,n}|+|\lambda_{m,n}-z|+|z|}\geqslant\left(\\!\frac{\gamma}{\sqrt{2}}\wedge
1\\!\right)\left(\frac{(\gamma/2)\wedge 1}{3R+|z|}\right)^{16\theta R^{2}}.$
(57)

The inequalities (56) and (57) together yield

$|h_{2}(z)|\lesssim_{\,\theta,\gamma,R}|z|^{16\theta
R^{2}}\lesssim_{\,\theta,R}e^{|z|}.$ (58)

Bounding $h_{3}$: Recall that $|\lambda_{m,n}|>\frac{s}{2}$ for all
$(m,n)\in\mathcal{I}^{\prime}\setminus\mathcal{I}_{s}$, and there is at most
one $(m^{\prime},n^{\prime})\in\mathcal{I}_{s}\setminus\\{(0,0)\\}$, and for
this $(m^{\prime},n^{\prime})$ we have
$|\lambda_{m^{\prime},n^{\prime}}|\geqslant\rho$, by assumption (i). We thus
get

$\displaystyle|h_{3}(z)|$
$\displaystyle=\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}-z|>2R\\\ |\omega_{m,n}|\leqslant
2R\end{subarray}}\frac{|1-z/\lambda_{m,n}|}{|1-z/\omega_{m,n}|}\leqslant\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}-z|>2R\\\ |\omega_{m,n}|\leqslant
2R\end{subarray}}\frac{|\omega_{m,n}|}{|\lambda_{m,n}|}\frac{|\lambda_{m,n}-z|}{|\lambda_{m,n}-z|-R}$
$\displaystyle\leqslant\rho^{-1}\prod_{|\omega_{m,n}|\leqslant
2R}\frac{2R}{(s/2)\wedge
1}\,\frac{2R}{2R-R}\leqslant\rho^{-1}\left(\frac{4R}{(s/2)\wedge
1}\right)^{16\gamma^{-2}R^{2}}$
$\displaystyle\lesssim_{\,s,\gamma,R}\rho^{-1}.$ (59)

Bounding $h_{4}$: We write
$|h_{4}|=h_{4}^{\mathrm{num}}/h_{4}^{\mathrm{den}}$, where

$h_{4}^{\mathrm{num}}(z)=\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}-z|>2R\\\ |\omega_{m,n}|>2R\\\ |\lambda_{m,n}|\leqslant
2|z|\end{subarray}}\left|1-\frac{\omega_{m,n}-\lambda_{m,n}}{\omega_{m,n}-z}\right|,\quad\text{and}\quad
h_{4}^{\mathrm{den}}(z)=\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}-z|>2R\\\ |\omega_{m,n}|>2R\\\ |\lambda_{m,n}|\leqslant
2|z|\end{subarray}}\left|1-\frac{\omega_{m,n}-\lambda_{m,n}}{\omega_{m,n}}\right|.$

Now, fix a $z\in\mathbb{C}$ with $|z|>3R$ and write
$z=z^{\prime}+\omega_{k,\ell}$, where $(k,\ell)\in\mathbb{Z}^{2}$ and
$(\mathrm{Re}\,z^{\prime},\mathrm{Im}\,z^{\prime})\in[-\frac{\gamma}{2},\frac{\gamma}{2}]^{2}$.
Then, as

$\\{(m,n)\in\mathcal{I}^{\prime}:|\lambda_{m,n}-z|\\!>\\!2R,|\lambda_{m,n}|\leqslant
2|z|\\}\subset\\{(m,n)\in\mathbb{Z}^{2}:|\omega_{m,n}-z|\\!>\\!R,|\omega_{m,n}-z|\leqslant
3|z|+R\\},$

and $\Omega_{\gamma}-\omega_{k,\ell}=\Omega_{\gamma}$, we have the following
bound

$\displaystyle|h_{4}^{\mathrm{num}}(z)|$
$\displaystyle\leqslant\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}-z|>2R\\\ |\omega_{m,n}|>2R\\\ |\lambda_{m,n}|\leqslant
2|z|\end{subarray}}\left(1+\frac{|\omega_{m,n}-\lambda_{m,n}|}{|\omega_{m,n}-z|}\right)\leqslant\prod_{\begin{subarray}{c}(m,n)\in\mathbb{Z}^{2}\\\
R<|\omega_{m,n}-z|\leqslant
3|z|+R\end{subarray}}\left(1+\frac{R}{|\omega_{m,n}-z|}\right)$
$\displaystyle=\prod_{\begin{subarray}{c}(m,n)\in\mathbb{Z}^{2}\\\
R<|\omega_{m,n}-\omega_{k,\ell}-z^{\prime}|\leqslant
3|z|+R\end{subarray}}\left(1+\frac{R}{|\omega_{m,n}-\omega_{k,\ell}-z^{\prime}|}\right)$
$\displaystyle\leqslant\exp\Bigg{[}\sum_{\begin{subarray}{c}(m,n)\in\mathbb{Z}^{2}\\\
R<|\omega_{m,n}-z^{\prime}|\leqslant
3|z|+R\end{subarray}}\frac{R}{|\omega_{m,n}-z^{\prime}|}\Bigg{]},$ (60)

where in the last inequality we used $\log(1+x)\leqslant x$ for $x\geqslant
0$. Now, $|\omega_{m,n}-z^{\prime}|>R$ and $R\geqslant 3\gamma$ imply
$|\omega_{m,n}|>R-\frac{\gamma}{\sqrt{2}}>\sqrt{2}\gamma$ and so
$|\omega_{m,n}-z^{\prime}|\geqslant|\omega_{m,n}|-\frac{\gamma}{\sqrt{2}}>\frac{1}{2}|\omega_{m,n}|$.
We therefore have

$\displaystyle\sum_{R<|\omega_{m,n}-z^{\prime}|\leqslant
3|z|+R}\frac{1}{|\omega_{m,n}-z^{\prime}|}$
$\displaystyle\leqslant\sum_{R-\frac{\gamma}{\sqrt{2}}<|\omega_{m,n}|\leqslant
3|z|+R+\frac{\gamma}{\sqrt{2}}}\frac{2}{|\omega_{m,n}|}$
$\displaystyle\leqslant\sum_{R-\frac{\gamma}{\sqrt{2}}<|\omega_{m,n}|\leqslant
3|z|+R+\frac{\gamma}{\sqrt{2}}}2\frac{|\omega_{m,n}|+\frac{\gamma}{\sqrt{2}}}{|\omega_{m,n}|}\int_{(m\gamma,n\gamma)+K_{\gamma}}|w|^{-1}|\mathrm{d}w|$
$\displaystyle\lesssim_{\,\gamma,R}\frac{R}{R-\frac{\sqrt{2}}{2}}\int_{R-\sqrt{2}\gamma<|\omega|\leqslant
3|z|+R+\sqrt{2}\gamma}|w|^{-1}|\mathrm{d}w|$
$\displaystyle\lesssim_{\,\gamma,R}|z|.$ (61)

As $z$ was arbitrary, (61) holds for all $z\in\mathbb{C}$ with $|z|>3R$. Using
(61) in (60) thus yields

$|h_{4}^{\mathrm{num}}(z)|\leqslant e^{c_{4}^{\mathrm{num}}|z|}$ (62)

for some $c_{4}^{\mathrm{num}}=c_{4}^{\mathrm{num}}(\gamma,R)>0$ and all
$z\in\mathbb{C}$ with $|z|>3R$.

The quantity $h_{4}^{\mathrm{den}}$ is bounded from below in a similar
fashion:

$\displaystyle|h_{4}^{\mathrm{den}}(z)|$
$\displaystyle=\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}-z|>2R\\\ |\omega_{m,n}|>2R\\\ |\lambda_{m,n}|\leqslant
2|z|\end{subarray}}\left|1-\frac{\omega_{m,n}-\lambda_{m,n}}{\omega_{m,n}}\right|\geqslant\prod_{\begin{subarray}{c}(m,n)\in\mathbb{Z}^{2}\\\
2R<|\omega_{m,n}|\leqslant
2|z|+R\end{subarray}}\left(1-\frac{R}{|\omega_{m,n}|}\right),$
$\displaystyle\geqslant\exp\Bigg{[}\sum_{2R<|\omega_{m,n}|\leqslant
2|z|+R}-\frac{2R}{|\omega_{m,n}|}\Bigg{]},$ (63)

where in the last inequality we used $\log(1+x)\geqslant 2x$ for
$x\in\left[-\frac{1}{2},0\right]$. Another integral bound yields

$\sum_{2R<|\omega_{m,n}|\leqslant
2|z|+R}\frac{1}{|\omega_{m,n}|}\lesssim_{\,\gamma,R}|z|,$

which together with (63) gives

$|h_{4}^{\mathrm{den}}(z)|\geqslant e^{-c_{4}^{\mathrm{den}}|z|},$ (64)

for some $c_{4}^{\mathrm{den}}=c_{4}^{\mathrm{den}}(\gamma,R)>0$ and all
$z\in\mathbb{C}$ with $|z|>3R$. Combining (62) and (64) thus yields

$|h_{4}(z)|\leqslant e^{(c_{4}^{\mathrm{num}}+c_{4}^{\mathrm{den}})|z|},$ (65)

for all $z\in\mathbb{C}$ with $|z|>3R$.

Bounding $h_{5}$: Note that $|\lambda_{m,n}|>2|z|$ implies
$|\lambda_{m,n}|>2|z|>6R$, and so
$\\{(m,n)\in\mathcal{I}^{\prime}:|\lambda_{m,n}|>2|z|\\}=A(z)$. We thus have
$h_{5}=h_{\mathrm{aux}}$, which satisfies (52).

Bounding $h$: We combine (55), (58), (59), (65), and (52) to obtain

$|h(z)|\lesssim_{\,s,\theta,\gamma,R}\rho^{-1}e^{(1+c_{4}^{\mathrm{num}}+c_{4}^{\mathrm{den}}+1100R)|z|+c_{1}|z|\log|z|}\leqslant\rho^{-1}e^{d|z|\log|z|},$
(66)

for some $d=d(s,\theta,\gamma,R)>0$ and all $z\in\mathbb{C}$ with $|z|>3R$.
This completes the derivation of the desired upper bound on $h$ in the case
$|z|>3R$.

Case $|z|\leqslant 3R$: We write $h(z)=h_{6}(z)h_{7}(z)$, where

$h_{6}(z)=\frac{d(z,\Omega_{\gamma})}{d(z,\Lambda)}\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}|\leqslant
6R\end{subarray}}\frac{1-z/\lambda_{m,n}}{1-z/\omega_{m,n}},\quad\text{and}\quad
h_{7}(z)=\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}|>6R\end{subarray}}\frac{(1-z/\lambda_{m,n})\exp(z/\lambda_{m,n})}{(1-z/\omega_{m,n})\exp(z/\omega_{m,n})}.$

Note that $|\lambda_{m,n}|>6R$ and $|z|\leqslant 3R$ together imply
$|\lambda_{m,n}|>2|z|$, and so $h_{7}=h_{\mathrm{aux}}$. Hence it only remains
to bound $h_{6}$. To this end, write
$|h_{6}|=h_{6}^{\mathrm{num}}/h_{6}^{\mathrm{den}}$, where

$h_{6}^{\mathrm{num}}(z)=\frac{z}{d(z,\Lambda)}\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}|\leqslant
6R\end{subarray}}\frac{\lambda_{m,n}-z}{\lambda_{m,n}}\quad\text{and}\quad
h_{6}^{\mathrm{den}}(z)=\frac{z}{d(z,\Omega_{\gamma})}\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}|\leqslant
6R\end{subarray}}\frac{\omega_{m,n}-z}{\omega_{m,n}}.$

Now, the term $d(z,\Lambda)$ cancels with either $z$ or one of the factors
$\lambda_{m,n}-z$, and similarly, $d(z,\Omega_{\gamma})$ cancels with either
$z$ or one of the factors $\omega_{m,n}-z$. In either case the numerators of
the terms remaining in the product satisfy
$|\omega_{m,n}-z|\geqslant\frac{\gamma}{2}$. We again recall that
$|\lambda_{m,n}|>\frac{s}{2}$ for all
$(m,n)\in\mathcal{I}^{\prime}\setminus\mathcal{I}_{s}$, and there is at most
one $(m^{\prime},n^{\prime})\in\mathcal{I}_{s}\setminus\\{(0,0)\\}$, and for
this $(m^{\prime},n^{\prime})$ we have
$|\lambda_{m^{\prime},n^{\prime}}|\geqslant\rho$. These observations together
yield the following bounds:

$\displaystyle|h_{6}^{\mathrm{num}}(z)|$ $\displaystyle\leqslant
3R\rho^{-1}\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}|\leqslant 6R\end{subarray}}\frac{(|\lambda_{m,n}|+|z|)\wedge
1}{(s/2)\vee 1}\leqslant 3R\rho^{-1}\left(\frac{(9R)\wedge 1}{(s/2)\vee
1}\right)^{144\theta R^{2}}$ $\displaystyle|h_{6}^{\mathrm{den}}(z)|$
$\displaystyle\geqslant\prod_{\begin{subarray}{c}(m,n)\in\mathcal{I}^{\prime}\\\
|\lambda_{m,n}|\leqslant 6R\end{subarray}}\frac{(\gamma/2)\wedge
1}{|\omega_{m,n}-\lambda_{m,n}|+|\lambda_{m,n}|}\geqslant\left(\frac{(\gamma/2)\wedge
1}{7R}\right)^{144\theta R^{2}},\quad\text{for }z\in\mathbb{C}\text{ s.t.
}|z|\leqslant 3R.$

Therefore $|h_{6}(z)|\lesssim_{\,s,\theta,\gamma,R}\rho^{-1}$, for $z$ with
$|z|\leqslant 3R$, which together with (52) gives

$|h(z)|\lesssim_{\,s,\theta,\gamma,R}\rho^{-1}e^{1100R}|z|,\quad\text{for
}z\in\mathbb{C}\text{ s.t. }|z|\leqslant 3R.$ (67)

The inequalities (66) and (67) can now be combined to yield the bound (44),
concluding the proof.

∎

###### Proof of Lemma 13.

Consider first the case when $y\in\mathcal{S}(\mathbb{R})$ is a Schwartz
function. We then have $y\in L^{2}(\mathbb{R})$, and thus

$\displaystyle\left\langle
y,\pi(\lambda)\varphi\right\rangle_{M^{q}(\mathbb{R})\times
M^{p}(\mathbb{R})}$
$\displaystyle=\left\langle\mathcal{V}_{\varphi}y,\mathcal{V}_{\varphi}\pi(\lambda)\varphi\right\rangle_{L^{q}(\mathbb{R}^{2})\times
L^{p}(\mathbb{R}^{2})}$
$\displaystyle=\iint_{\mathbb{R}^{2}}(\mathcal{V}_{\varphi}y)(s,\xi)\;\overline{(\mathcal{V}_{\varphi}\pi(\lambda)\varphi)(s,\xi)}\;\mathrm{d}s\mathrm{d}\xi$
$\displaystyle=\int_{\mathbb{R}}y(t)\overline{\left(\pi(\lambda)\varphi\right)(t)}\;\mathrm{d}t$
(68) $\displaystyle=(\mathcal{V}_{\varphi}y)(\lambda)$ $\displaystyle=e^{-\pi
i\tau\nu}e^{-\pi|\lambda|^{2}/2}(\textfrak{B}\hskip 1.0pty)(\bar{\lambda}),$
(69)

where (68) follows from [7, Thm. 3.2.1], and (69) is [7, Prop. 3.4.1].

Now take an arbitrary $y\in M^{q}(\mathbb{R})$. As $\mathcal{S}(\mathbb{R})$
is dense in $M^{q}(\mathbb{R})$ for $q\in[1,\infty)$ (see [7, Prop. 11.3.4]),
we can take a sequence
$\\{y_{n}\\}_{n=1}^{\infty}\subset\mathcal{S}(\mathbb{R})$ such that $y_{n}\to
y$ in $M^{q}(\mathbb{R})$. The calculation above thus shows that

$\langle y_{n},\pi(\lambda)\varphi\rangle_{M^{q}(\mathbb{R})\times
M^{p}(\mathbb{R})}=e^{-\pi i\tau\nu}e^{-\pi|\lambda|^{2}/2}(\textfrak{B}\hskip
1.0pty_{n})(\bar{\lambda}),\quad\forall n\in\mathbb{N}.$ (70)

Furthermore, as the dual pairing is continuous, we have

$\langle y_{n},\pi(\lambda)\varphi\rangle_{M^{q}(\mathbb{R})\times
M^{p}(\mathbb{R})}\to\langle
y,\pi(\lambda)\varphi\rangle_{M^{q}(\mathbb{R})\times
M^{p}(\mathbb{R})}\quad\text{as }n\to\infty.$

On the other hand, by the isometry property (18) we also have
$\|\textfrak{B}\hskip 1.0pty_{n}-\textfrak{B}\hskip
1.0pty\|_{\mathcal{F}^{q}(\mathbb{C})}=\|y_{n}-y\|_{M^{q}(\mathbb{R})}\to 0$
as $n\to\infty$. Thus, as the evaluation functional $F\mapsto
F(\overline{\lambda})$ is continuous on $\mathcal{F}^{q}(\mathbb{C})$ (see
[28, Lem. 2.32]), we obtain $(\textfrak{B}\hskip
1.0ptf_{n})(\overline{\lambda})\to(\textfrak{B}\hskip
1.0ptf)(\overline{\lambda})$, which together with (70) and (69) establishes
the claim of the proposition. ∎

###### Proof of Lemma 14.

(i) Suppose that $\mathcal{A}$ is bounded below. Then the operator
$\tilde{\mathcal{A}}:X\to\mathrm{Im}(\mathcal{A})$ given by
$\tilde{\mathcal{A}}(x)=\mathcal{A}(x)$, for $x\in X$, is a continuous map
between Banach spaces, and has a continuous inverse. In other words,
$\tilde{\mathcal{A}}$ is an isomorphism between Banach spaces. Thus
$\tilde{\mathcal{A}}^{*}:(\mathrm{Im}(\mathcal{A}))^{*}\to X^{*}$ is also an
isomorphism between Banach spaces, and so, by the inverse mapping theorem [31,
Cor. 2.12], so is
$(\tilde{\mathcal{A}}^{*})^{-1}:X^{*}\to(\mathrm{Im}(\mathcal{A}))^{*}$.
Consider now an arbitrary $f\in X^{*}$, and set
$h=(\tilde{\mathcal{A}}^{*})^{-1}f$. As $h$ is a continuous linear functional
on $\mathrm{Im}(\mathcal{A})\subset Y$, it follows by the Hahn-Banach theorem
[31, Thm. 3.6] that $h$ can be extended to a continuous linear functional
$h_{Y}$ defined on $Y$. Now, since
$h_{Y}\\!\mid_{\mathrm{Im}(\mathcal{A})}=h$, we have

$\displaystyle\langle\mathcal{A}^{*}h_{Y},x\rangle$ $\displaystyle=\langle
h_{Y},\mathop{\mathchoice{\vtop{\halign{#\cr$\hfil\displaystyle\mathcal{A}x\hfil$\crcr\kern
2.0pt\nointerlineskip\cr\@mathmeasure\displaystyle{\mathcal{A}x}\@mathmeasure\displaystyle{\upbrace}\@mathmeasure\displaystyle{\upbraceg}\@mathmeasure\displaystyle{\upbracegg}\@mathmeasure\displaystyle{\upbraceggg}\@mathmeasure\displaystyle{\upbracegggg}$\displaystyle\bracelu\leaders\hbox{$\bracemid$}\hfill\bracemu\leaders\hbox{$\bracemid$}\hfill\braceru$\crcr}}}{\vtop{\halign{#\cr$\hfil\textstyle\mathcal{A}x\hfil$\crcr\kern
2.0pt\nointerlineskip\cr\@mathmeasure\textstyle{\mathcal{A}x}\@mathmeasure\textstyle{\upbrace}\@mathmeasure\textstyle{\upbraceg}\@mathmeasure\textstyle{\upbracegg}\@mathmeasure\textstyle{\upbraceggg}\@mathmeasure\textstyle{\upbracegggg}$\textstyle\bracelu\leaders\hbox{$\bracemid$}\hfill\bracemu\leaders\hbox{$\bracemid$}\hfill\braceru$\crcr}}}{\vtop{\halign{#\cr$\hfil\scriptstyle\mathcal{A}x\hfil$\crcr\kern
2.0pt\nointerlineskip\cr\@mathmeasure\scriptstyle{\mathcal{A}x}\@mathmeasure\scriptstyle{\upbrace}\@mathmeasure\scriptstyle{\upbraceg}\@mathmeasure\scriptstyle{\upbracegg}\@mathmeasure\scriptstyle{\upbraceggg}\@mathmeasure\scriptstyle{\upbracegggg}$\scriptstyle\bracelu\leaders\hbox{$\bracemid$}\hfill\bracemu\leaders\hbox{$\bracemid$}\hfill\braceru$\crcr}}}{\vtop{\halign{#\cr$\hfil\scriptscriptstyle\mathcal{A}x\hfil$\crcr\kern
2.0pt\nointerlineskip\cr\@mathmeasure\scriptscriptstyle{\mathcal{A}x}\@mathmeasure\scriptscriptstyle{\upbrace}\@mathmeasure\scriptscriptstyle{\upbraceg}\@mathmeasure\scriptscriptstyle{\upbracegg}\@mathmeasure\scriptscriptstyle{\upbraceggg}\@mathmeasure\scriptscriptstyle{\upbracegggg}$\scriptscriptstyle\bracelu\leaders\hbox{$\bracemid$}\hfill\bracemu\leaders\hbox{$\bracemid$}\hfill\braceru$\crcr}}}}\limits_{\mathclap{\in\mathrm{Im}(\mathcal{A})}}\rangle=\langle
h,\mathcal{A}x\rangle=\langle h,\tilde{\mathcal{A}}x\rangle=$
$\displaystyle=\langle\tilde{\mathcal{A}}^{*}h,x\rangle=\langle
f,x\rangle\quad\text{for }x\in X,$

and thus, as $x\in X$ was arbitrary, we deduce that $\mathcal{A}^{*}h_{Y}=f$.
Finally, since $f$ was arbitrary, we have that $\mathcal{A}^{*}$ is
surjective.

(ii) Let $f$ be an arbitrary element of $X^{*}$ with $\|f\|=1$, and let $g\in
Y^{*}$ be such that $\mathcal{A}^{*}g=f$ and $a\|g\|\leqslant 1$. Note that
then $g\neq 0$, and so $g/\|g\|$ is a well-defined element of $Y^{*}$ of unit
norm. Therefore,

$\|\mathcal{A}x\|\geqslant\left|\left<\mathcal{A}x,\frac{g}{\|g\|}\right>\right|=\|g\|^{-1}\left|\langle
x,\mathcal{A}^{*}g\rangle\right|\geqslant a\left|\langle x,f\rangle\right|,$

for all $x\in X$. Taking the supremum of the right-hand side over $f\in X^{*}$
and using the fact that $\sup_{f\in X^{*},\|f\|=1}|\langle x,f\rangle|=\|x\|$
yields $\|\mathcal{A}x\|\geqslant a\|x\|$, as desired. ∎

###### Proof of Lemma 15.

For $m,n\in\mathbb{Z}$ write

$\textstyle
K_{m,n}=\left[\frac{s}{\sqrt{2}}\left(m_{\lambda}-\frac{1}{2}\right),\frac{s}{\sqrt{2}}\left(m_{\lambda}+\frac{1}{2}\right)\right)\times\left[\frac{s}{\sqrt{2}}\left(n_{\lambda}-\frac{1}{2}\right),\frac{s}{\sqrt{2}}\left(n_{\lambda}+\frac{1}{2}\right)\right),$

and, for $\lambda\in\Lambda$, let $(m_{\lambda},n_{\lambda})$ be the (unique)
element of $\mathbb{Z}^{2}$ such that $\lambda\in K_{m,n}$. Note that, as
$\mathrm{sep}(\Lambda)=s$, every $K_{m,n}$ contains at most one element of
$\Lambda$. Next, define the functions

$A(z)=\frac{2}{s}\sum_{\lambda\in\Lambda}a_{\lambda}\mathds{1}_{K_{m_{\lambda},n_{\lambda}}}(z)\quad\text{and}\quad\tilde{f}(z)=\max_{|w|\leqslant
s}|f(z+w)|.$

We then have

$\sum_{\lambda\in\Lambda}|\theta_{\lambda}||f(z-\lambda)|\leqslant\sum_{\lambda\in\Lambda}|\theta_{\lambda}|\;\tilde{f}\left(z-{s(m_{\lambda}+in_{\lambda})}/{\sqrt{2}}\right)\leqslant\int_{\mathbb{C}}A(w)\tilde{f}(z-w)|\mathrm{d}w|=(A\ast\tilde{f})(z),$

for all $z\in\mathbb{C}$. Therefore, using [7, Prop. 11.1.3], we obtain

$\|A\ast\tilde{f}\|_{L^{q}(\mathbb{C})}\leqslant\|\tilde{f}\|_{L^{1}(\mathbb{C})}\|A\|_{L^{q}(\mathbb{C})}\lesssim
s^{-2/p}\|f\|_{W(L^{\infty},L^{1})}\|\theta\|_{\ell^{p}(\Lambda)},$

where the last equality follows by computing the norm of $A$ explicitly. ∎

###### Proof of Lemma 19.

Note that it suffices to prove the claim for $j=n-1$, as the general statement
then follows by induction. To this end, divide $K_{n}$ into four disjoint
squares of side length $\sqrt{2}\left(2^{n-1}+\frac{1}{2}\right)$. By the
pigeonhole principle, one of these squares must contain at least
$2^{2(n-1)}+1$ points of $K_{n}\cap Y$. Denote this square by $K^{\prime}$,
and Let $K_{n-1}$ be the square which contains $K^{\prime}$ and satisfies
property (i) in the statement of the Lemma. Then $\\#(K_{n-1}\cap
Y)\geqslant\\#(K^{\prime}\cap Y)\geqslant 2^{2(n-1)}+1$, and so $K_{n-1}$
satisfies (ii), as desired. ∎


# A Study of Continual Learning Under Language Shift

Evangelia Gogoulou
RISE, KTH
<EMAIL_ADDRESS>
&Timothée Lesort
Université de Montréal, MILA-Quebec AI Institute
<EMAIL_ADDRESS>
Magnus Boman
KTH, KI, MedTechLabs
<EMAIL_ADDRESS>
&Joakim Nivre
RISE, Uppsala University
<EMAIL_ADDRESS>

###### Abstract

The recent increase in data and model scale for language model pre-training
has led to huge training costs. In scenarios where new data become available
over time, updating a model instead of fully retraining it would therefore
provide significant gains. In this paper, we study the benefits and downsides
of updating a language model when new data comes from new languages – the case
of continual learning under language shift. Starting from a monolingual
English language model, we incrementally add data from Norwegian and Icelandic
to investigate how forward and backward transfer effects depend on the pre-
training order and characteristics of languages, for different model sizes and
learning rate schedulers. Our results show that, while forward transfer is
largely positive and independent of language order, backward transfer can be
either positive or negative depending on the order and characteristics of new
languages. To explain these patterns we explore several language similarity
metrics and find that syntactic similarity appears to have the best
correlation with our results.

##  1 Introduction

Large-scale pre-training on large amounts of text data is the current paradigm
for training language models. Even though these models have been shown to have
good generalization capabilities for various downstream tasks, they will
sooner or later become obsolete given the current rate of producing new data
sources that differ from the data used to train the model Lazaridou et al.
(2021). Continual learning methods aim to update a model with new data while
retaining the previous model knowledge in different scenarios of data
distribution shift Parisi et al. (2019); Hadsell et al. (2020); Lesort et al.
(2021); Wang et al. (2023).

In this paper, we focus on the specific case of language shift, where each new
data source is in a different language, and study how the model learns and
forgets when sequentially trained on multiple languages. More precisely, we
study the training of generative language models with standard next-word
prediction objectives. We evaluate their capacity to predict the next token in
an increasing number of languages. From a continual learning perspective, this
multilingual scenario is particularly interesting since the distribution shift
happening between linguistically dissimilar languages is expected to have a
big impact on the model knowledge. In addition, it allows us to study the
cross-lingual transfer effects as a monolingual model becomes progressively
multilingual.

The main question we investigate in this paper is: How do the characteristics
of languages and their order in the task sequence influence model performance?
We focus on how the model performs on new languages – forward transfer – and
on previous languages – backward transfer. We do this by training generative
language models on data from English, Icelandic and Norwegian, in different
sequential orders. Furthermore, we investigate whether increasing the model
capacity alleviates catastrophic forgetting of previous languages or changes
the transfer effects in other respects. Before we study the transfer effects
in depth, we investigate what learning rate policy is best suited for the
language shift scenario. Finally, in an attempt to explain the transfer effect
patterns, we explore several language similarity metrics, including a novel
metric based on token distribution similarity.

Our contributions are the following:

  * •

We propose a multilingual scenario as a challenging corner case for the study
of continual pre-training of language models.

  * •

We show that, in this scenario, despite the distribution shifts, there is in
general positive forward transfer from earlier to later languages, improving
over monolingual baselines.

  * •

We also show that, by contrast, the existence of positive or negative backward
transfer is dependent on the specific languages and the order in which they
are added.

  * •

We find that, in our setup, the optimal learning rate policy for continual
learning coincides with the optimal policy for transfer.

  * •

We find that increasing model size improves the final performance on all
languages, but does not significantly impact backward transfer.

  * •

We study the relation between transfer effects and language similarity and
find that syntactic similarity is best aligned with our results.

##  2 Related Work

Our work investigates continual pre-training of multilingual models. In this
section, we first review related work on continual learning, move on to
continual pre-training of language models, and finally discuss related work on
multilingual models with a focus on continual multilingual approaches.

##### Continual Learning

Continual learning in the deep neural network literature generally addresses
the problem of learning without forgetting on sequences of datasets French
(1999); Lange et al. (2019); Lesort et al. (2020); Hadsell et al. (2020). The
most classical set-up for studying this problem is image classification
Rebuffi et al. (2017); Aljundi et al. (2019); Douillard et al. (2021); Hayes
et al. (2020). However, recent progress in natural language processing has led
to an increased interest in continual learning also in this area Caccia et al.
(2021); Ke and Liu (2022); Scialom et al. (2022); Winata et al. (2023); de
Masson d'Autume et al. (2019). Notably, the increased availability of large
pre-trained models, such as BLOOM (Scao et al., 2022), Gopher (Rae et al.,
2021), Pythia Biderman et al. (2023)) and LLaMA (Touvron et al., 2023),
provides good foundations for continual learning where just slight adaptations
are needed for downstream tasks. Nevertheless, those large pre-trained models
still have shortcomings when sequences of tasks are too different from the
pre-training data or learning objectives Ostapenko et al. (2021); Abnar et al.
(2021). Consequently, in this paper, we are interested in the continual pre-
training of language models where the whole model is continuously trained. We
also study the influence of model size on continual performance as in several
previous studies Ramasesh et al. (2022); Ostapenko et al. (2021); Mirzadeh et
al. (2021); Lesort et al. (2022).

##### Continual Language Pre-Training

Continual pre-training of language models has been applied for example to data
that evolves through time Han et al. (2021); Jin et al. (2022); Jang et al.
(2021, 2022); Loureiro et al. (2022). In this setup, one of the challenges is
to disentangle knowledge that is outdated and can be updated from the
knowledge that is time-invariant. In our setup, we assume all the factual
knowledge to be invariant and are interested in incorporating more and more
knowledge while minimizing forgetting as in domain adaptation pre-training Ke
et al. (2023); Scialom et al. (2022); Gururangan et al. (2021); Qin et al.
(2022). One of the advantages of continual pre-training is that the learning
objective is self-supervised, therefore it could lead to a scalable approach
that could automatically accumulate knowledge through long sequences of tasks
Lesort et al. (2022); Scialom et al. (2022). As the capacity of algorithms to
continuously learn highly depends on the data distribution shift between
training stages Lesort et al. (2021), we study the sequential multilingual
setup, as a corner case where the input distribution significantly changes
between training stages. Indeed, the change of language will deeply change the
token distribution. We argue that this type of scenario can be used as a
benchmark to evaluate continual pre-training algorithms. The harder
distributional shift between languages is expected to make pre-training more
challenging than with monolingual data.

##### Multilingual Language Models

Multilingual Transformer-based language models are typically pre-trained
jointly on massive amounts of data in multiple languages. Standard examples of
multilingual Transformer encoder models are mBERT Devlin et al. (2019) and
XLM-R Conneau et al. (2020), while mT5 Xue et al. (2021) is an encoder-decoder
architecture for multiple languages. Following the recent development of
Transformer decoder architectures, several multilingual GPT-style Radford et
al. (2018) models have been proposed, such as BLOOM Scao et al. (2022), a
$176B$ parameter Transformer decoder trained on $59$ different languages and
GPT-SW3 Ekgren et al. (2023), a suite of generative language models, ranging
from $126M$ to $40B$ parameters, which are trained on $320B$ tokens from a mix
of Nordic languages and English. Joint pre-training is used as a baseline for
measuring the transfer effects between languages in our setup, while our work
focuses on multilingual models where each new language is added to the model
one by one, starting from scratch. This setup is motivated by the huge
potential cost benefit of updating a large-scale language model with neither
forgetting nor re-training. Our model architecture, data, and training process
closely follow GPT-SW3 Ekgren et al. (2023).

Related to continual learning applied to pre-trained multilingual models,
Badola et al. (2022) propose a method for continual fine-tuning of mBERT on
new multilingual training data from the same downstream task but with an
unbalanced distribution between languages. Garcia et al. (2021) study the
scenario of continual machine translation for multiple languages, while Coria
et al. (2022) investigate catastrophic forgetting in the setup of continual
fine-tuning mBERT on two sequence labelling tasks, one language at a time.
Related to that, Liu et al. (2021) propose a method based on gradient episodic
memory that retains the language modeling performance of mBERT, while fine-
tuning the model for sentence retrieval. Mao et al. (2022) propose a novel
continual multilingual fine-tuning method, applied to XLM-R, which opts for
both minimum forgetting in the source languages and reduction of the fine-
tuning loss. Concurrent with our work, Winata et al. (2023) studies continual
pre-training of language models. However, they mostly aim at improving
algorithmic solutions while we investigate the transfer effects between
languages with simple continual training techniques.

##  3 Method

We start with the learning scenario and general setup and then move on to
technical details concerning model architecture, training and evaluation
procedures, datasets, and learning rate scheduling. We conclude by introducing
the language similarity metrics used in our analysis of the experimental
results. More details about hyperparameter settings and datasets can be found
in Appendix A–B, while information about computational infrastructure and
resource consumption is given in Appendix D.

###  3.1 Learning Scenario and Setup

We consider a scenario where a generative language model is trained in
successive stages given a data input stream $D={D_{1},D_{2},\ldots D_{n}}$. In
this scenario, each $D_{i}$ contains data from a different language. After
each stage, we evaluate the model performance on each of the languages seen
during training by computing the cross-entropy loss on the corresponding hold-
out test sets. In order to study how forward and backward transfer effects
depend on the order and characteristics of the languages involved, we run
multiple experiments where we systematically vary the order of languages.

In this study, we experiment with data from three languages, namely English
(en), Icelandic (is), and Norwegian (no). We consistently take the initial
data $D_{1}$ from English, and then add Icelandic and/or Norwegian in all
possible orders, which gives us four different training sequences: en_is,
en_is_no, en_no, en_no_is. All models are randomly initialized before being
trained on the first language of the sequence. For each successive continual
learning stage $i$ ($i>1$), all model parameters are initialized from the last
checkpoint obtained after stage $i-1$ and then updated using the standard
next-step prediction language modeling objective with data batch $D_{i}$ as
input.

The size of the training sets is constant across languages, which allows us to
study the effect of language shift independently from data set size. Regarding
tokenization, we make the assumption that a multilingual tokenizer, covering
all the languages studied, is available to us before training. This is often
the case in practice as well, given the abundance of openly available
tokenizers with large language coverage Conneau et al. (2020); Scao et al.
(2022), although it is an unrealistic assumption for many low-resource
languages.

For comparison, we also train two types of baseline models: one monolingual
model for each language (en, is, no), trained on the corresponding training
batch $D_{i}$, and one multilingual model (en+is+no) trained on the
concatenation of the three training sets $D_{1}+D_{2}+D_{3}$.

###  3.2 Model Architecture and Training

We use the standard GPT architecture Radford et al. (2018), as implemented in
the Nvidia NeMo library.111https://github.com/NVIDIA/NeMo To be able to study
the role of model capacity, we run experiments with two model sizes, namely
$126$M and $356$M parameters. The exact model hyperparameter values are
presented in Table 5 in Appendix A. For tokenization, all models use the GPT-
SW3 tokenizer Ekgren et al. (2023); Stollenwerk (2023), which is a BPE
tokenizer with $64$K tokens, trained on the Nordic Pile Öhman et al. (2023), a
multilingual corpus including English, Norwegian and Icelandic, Swedish and
Danish. For each language pre-training stage during continual learning, the
model is trained for $35000$ steps (approximately $36$M sequences of $2048$
tokens each). The monolingual baseline models are also trained for $35000$
steps, while the multilingual baseline model is trained for $105000$ steps,
thereby covering the training sets for all three languages.

###  3.3 Datasets

Models are trained on a subset of the Nordic Pile corpus Öhman et al. (2023).
(More information about the total corpus and its data weighting scheme can be
found in Appendix B.) The list of datasets used per language, together with
the number of training sequences per dataset is presented in Table 1. The
validation and test set per language each consists of $51K$ samples and has
the same proportions of datasets as the training set.

Lang | Datasets | N
---|---|---
en | The Pile Gao et al. (2020) | $32$
Wiki en | $4$
is | Icelandic Gigaword Barkarson et al. (2022) | $24$
Wiki is | $0.23$
mc4 (is) Xue et al. (2021) | $6$
OSCAR (is) Ortiz Suárez et al. (2020) | $6$
no | NCC Kummervold et al. (2022) | $18$
Wiki no | $0.37$
mc4 (no) Xue et al. (2021) | $11$
OSCAR (no) Ortiz Suárez et al. (2020) | $7$

Table 1: Data resources per language (Lang) and number of training sequences
in millions per dataset (N).

###  3.4 Learning Rate

As part of our experiments, we investigate which learning rate policy works
best for our continual learning setup. We are interested in both the policy
that leads to the lowest overall loss (on all languages), known as the
continual performance, and the loss on the latest language learned, the local
performance or transfer.

An important question is how to resolve the different practices adopted in
continual learning and NLP. In the continual learning tradition, the learning
rate schedule is normally either kept constant Ke et al. (2023); Lesort et al.
(2022); Scialom et al. (2022) or progressively decreased Winata et al. (2023);
Mirzadeh et al. (2020). By contrast, in the NLP setup, the learning rate is
often first warmed up and then decreased Ruder (2016).

In our experiments, we are not interested in a consistent decrease in learning
rate, since it could not be scaled to a high number of tasks without
vanishing. Therefore, we experiment with constant learning rate and cosine
annealing for the second and third training stages of the $126$M models (see
Figure 1). The learning rate used in the first training stage always follows
the cosine annealing policy. Following Ekgren et al. (2023) for the cosine
annealing scheduler, we start by warming up the learning rate for $250$ steps,
increasing its value to the maximum value, $6\mathrm{e}{-4}$ and then
decreasing it according to the cosine decay function until reaching its
minimum value, $6\mathrm{e}{-5}$, where it will stay constant for the last
$4900$ steps. The second learning rate policy tested is keeping it constant at
$6\mathrm{e}{-5}$, which is the last learning rate value of the cosine
annealing scheduler.

Our hypothesis here is that the constant learning rate will keep the model
parameters closer to the model initialization at the beginning of the training
stage, leading to a smaller catastrophic forgetting effect in the previous
languages.

Figure 1: Evolution of learning rates when using cosine annealing and constant
rate scheduling.

###  3.5 Language Similarity Metrics

One of our hypotheses is that the degree of shift happening when moving from
one language to another will have a significant effect on the model knowledge
in the different languages. In order to investigate this hypothesis, we
explore two types of metrics for estimating the similarity between languages
and their distributions.

First, the linguistic similarity between two languages is used, as estimated
by the precomputed
distance222http://www.cs.cmu.edu/~dmortens/projects/7_project/ of their
corresponding lang2vec language vectors which are part of the URIEL database
Littell et al. (2017). Different types of features are considered, as follows:
_syntax_ (SYN) vectors that encode syntactic features, _inventory_ (INV)
vectors that include phonetic features, _phonology_ (PHON) vectors that
represent phonological properties of the language.333While phonetic and
phonological features are not directly relevant for written texts, they can be
seen as proxies for orthographical features.

Second, we introduce a data-driven metric of similarity between two language
corpora, which we call _token distribution similarity_ (TDS). The idea is that
similarity should take into account the difference in token distributions
between two datasets. Hence, we create one vector per dataset, counting the
number of occurrences for each token, and then compute the cosine similarity
between vectors of different datasets. Note that we use this metric in a
multilingual setup, but it could also be used to compare datasets in the same
language. This is similar to the metric of Garcia et al. (2021), which
estimates the similarity between two vocabularies by the percentage of token
overlap between them. The advantage of our method is that we do not take into
account only the list of common tokens, but also the frequency of their
occurrence.

Given a sample of $50K$ sequences for each language, we first tokenize it
using the model tokenizer and then create a vector where position $i$
corresponds to the frequency of vocabulary token $i$ in the sample. The next
step is to compute the cosine similarity of the two token frequency vectors,
which is used as a proxy for language similarity.

The similarity values between pairs of languages en, no, and is are shown in
Table 2. Interestingly, the degree of similarity of each pair of languages
varies with the metric. Norwegian and Icelandic have the highest similarity
according to the TDS metric, but their similarity value according to the URIEL
features is mixed: their syntactic similarity is the lowest of all three
language pairs, but they have highly similar phonological and phonetic
features. Our hypothesis is that positive transfer will be observed for
languages with the highest TDS value, as this metric derives from the overlap
between the data inputs, but we also expect syntactic similarity to be
relevant for next-token prediction.

Language pair | TDS | SYN | INV | PHON
---|---|---|---|---
en, no | $0.59$ | $0.43$ | $0.40$ | $0.43$
en, is | $0.46$ | $0.21$ | $0.40$ | $0.43$
no, is | $0.67$ | $0.13$ | $1.00$ | $0.99$

Table 2: Language similarity values for each language pair, as estimated by
our token distribution similarity (TDS) metric and by URIEL feature vectors
for syntactic (SYN), phonetic (INV) and phonological (PHON) features Littell
et al. (2017).

##  4 Results

In this section, we first present the results of our experiments concerning
learning rate scheduling. We then move on to our main research question about
transfer effects and conclude with the comparison of different model sizes.

###  4.1 Learning Rate Scheduling

The results of our experimentation with constant learning rate and cosine
annealing are presented in Table 3. It can be observed that the learning rate
policy that is optimal for continual learning (s.s.f., meaning stages “seen so
far”) coincides with the optimal learning rate for the last pre-training
stage. The table shows that the learning rate policy that consistently leads
to the best performance is cosine annealing, which contradicts our initial
hypothesis that a constant learning rate would reduce forgetting. Admittedly,
the variations in the results between the different learning rate policies are
small, which can be due to the restricted hyperparameter search in our setup.
Nevertheless, we decided to use the cosine annealing learning rate scheduler
for all our later experiments.444The minimum and maximum learning rate values
for the $356$M model can be found in Table 6 in Section A.

Model: GPT 126M | Stage 2 | Stage 3
---|---|---
Order: en $\rightarrow$ is $\rightarrow$ no | s.s.f. | current | s.s.f. | current
cos. anneal. $\rightarrow$ cos. anneal. | 7.01 | 2.58 | 11.38 | 3.00
cos. anneal. $\rightarrow$ constant | 7.01 | 2.58 | $11.45$ | $3.14$
constant $\rightarrow$ constant | $7.03$ | $2.64$ | $11.54$ | $3.15$
constant $\rightarrow$ cos. anneal. | $7.03$ | $2.64$ | $11.55$ | $3.09$
Order: en $\rightarrow$ no $\rightarrow$ is | s.s.f. | current | s.s.f. | current
cos. anneal. $\rightarrow$ cos. anneal. | 6.44 | 3.03 | $12.16$ | $5.30$
cos. anneal. $\rightarrow$ constant | 6.44 | 3.03 | 12.02 | 5.21
constant $\rightarrow$ constant | $6.57$ | $3.18$ | $12.18$ | $5.31$
constant $\rightarrow$ cos. anneal. | $6.57$ | $3.18$ | $12.37$ | $5.46$

Table 3: Cumulative test loss in languages seen so far (s.s.f) and performance
on the current stage (current) at the end of second and third pre-training
stages. The lowest loss on the seen so far stages is marked in bold, and the
lowest loss on current stage data is underlined. _The best learning rate
policy for the current and past stages is the same._

###  4.2 Transfer Results

Table 4 shows the model test loss on the three languages, measured after each
training stage, for all our experiments. Starting from the baselines, the
monolingual models $126$M and $356$M in most cases reach a lower test loss
than the corresponding jointly pre-trained model. The only exception is
English, where joint pre-training leads to an even lower test loss than the
English monolingual model of the same size. This could be a result of cross-
lingual transfer from Norwegian during pre-training, given the high token
overlap and syntactic similarity, discussed in Section 3.5.

In order to check for potential contamination between the monolingual corpora,
we also computed the loss of each monolingual baseline on the test set of the
other two languages. As seen in Table 4, the English and Icelandic monolingual
models have a considerably higher loss on the respective unseen languages,
which means that the risk of language contamination is low. This is not the
case for the Norwegian model though, which scores a test loss on English that
is at the same level as the test loss on Norwegian. We discuss the potential
effect of data contamination in our setup in Section 5.

Overall, there is a positive forward transfer effect to both Icelandic and
Norwegian, while the degree of catastrophic forgetting (negative backward
transfer) varies with the language and its position in the sequence. The
observed effects in terms of forward transfer, backward transfer, and their
relation to model size are analyzed in the sections below.

Model: GPT 126M
---
Training Stage(s) | en | no | is | TOT
en | $3.45$ | 5.44 | 5.44 | $15.28$
is | 4.60 | 5.79 | $2.55$ | $12.94$
no | 3.50 | $3.11$ | 5.35 | $11.96$
en+is+no (joint) | $2.97$ | $3.37$ | $2.72$ | $9.06$
(en $\rightarrow$) is | $4.43$ | - | $2.58$ |
(en $\rightarrow$ is $\rightarrow$) no | $3.42$ | $3.07$ | $4.96$ | $11.38$
(en $\rightarrow$) no | $3.41$ | $3.03$ | - |
(en $\rightarrow$ no $\rightarrow$) is | $4.32$ | $5.30$ | $2.54$ | $12.16$
Model: GPT 356M
en | $3.08$ | 5.04 | 5.99 | $14.11$
is | 4.34 | 5.50 | $2.33$ | $12.17$
no | 3.20 | $2.82$ | 4.97 | $10.99$
en+is+no (joint) | $2.64$ | $2.99$ | $2.36$ | $7.99$
(en $\rightarrow$) is | $4.01$ | - | $2.23$ |
(en $\rightarrow$ is $\rightarrow$) no | $3.03$ | $2.71$ | $4.48$ | $10.22$
(en $\rightarrow$) no | $3.02$ | $2.74$ | - |
(en $\rightarrow$ no $\rightarrow$) is | $3.91$ | $4.92$ | $2.19$ | $11.02$

Table 4: Test loss in English (en), Norwegian (no), and Icelandic (is)
measured after each language training stage, for two model sizes and four
different language pre-training sequences per size. Past stages are put in
parentheses. Results in small italics show test loss on languages never seen
by the model.

####  4.2.1 Forward Transfer

One part of our main research question is whether pre-training on the first
two languages helps the third one, independently of order. Figure 2
illustrates the progression of model test loss on Icelandic and Norwegian when
learned in the second or third training stage, in comparison to the
monolingual baseline (first stage). This shows evidence of forward transfer
from past languages to new languages as performance improves when the language
is learned late in the sequence. The only case of negative transfer is when
Icelandic is the second language as input to the $126$M model. We hypothesize
that this happens because of the low similarity with the first language
(English), at least the syntactic one as discussed in Section 3.5, in
combination with the limited model capacity.

Figure 2: Test loss on Icelandic and Norwegian when learned at different
stages of the sequence. The test loss at the first stage corresponds to the
performance of the monolingual baseline. _Clear improvement in the loss is
observed when the language is learned later in the sequence, with the
exception of Icelandic for the $126M$ model._

####  4.2.2 Backward Transfer

The second part of our main research question is what happens to previously
seen languages when a new language is learned. Figure 3 visualizes the
evolution of model test loss in English for the models trained on sequences
en_is_no and en_no_is. The first observation we can make is that the loss
follows a pattern that does not relate to the position of the language in the
sequence, but to the choice of language. More specifically, training the model
on Icelandic leads to an increase in the English test loss, which is a sign of
catastrophic forgetting. On the other hand, we observe that training on
Norwegian transfers positively to English, leading to a reduction in English
test loss. In the interesting case of en_is_no, the model is able to recover
its language modeling performance in English at the third training stage, even
after the performance degradation caused by training on Icelandic. As seen in
Figure 4, Norwegian and Icelandic transfer negatively to one another
independently of the order they are positioned in the second and third places
in the sequence.

Figure 3: Model test loss on English for models with varying size, trained on
the sequences en_no_is and en_is_no. _Training on Norwegian leads to positive
backward transfer on English while Icelandic causes forgetting on English,
whenever those languages are learned._ Figure 4:  Second stage: test loss of
the en_no and en_is $126$M models on Norwegian and Icelandic respectively.
Third stage: model test loss on each of Norwegian and Icelandic while pre-
training on the other language. _Learning Norwegian after Icelandic leads to
significant forgetting and vice versa._

###  4.3 Model Size

Our final research question concerns the effect of model size on forward and
backward transfer. As illustrated in Figure 5, the $356$M models reach a lower
final test loss than the $126$M models, which is expected given the increased
capacity of the former. However, contrary to our expectations, the forgetting
pattern does not change when increasing the model size. The trade-off between
test loss in the current language and the cumulative increase of the loss in
the previous languages is visualized in Figure 6. It is observed that
increasing the model size improves the current loss but does not impact
significantly forgetting, independently of the choice of language and stage.

Figure 5:  Cumulative loss over English, Icelandic and Norwegian test sets for
different model sizes. Results are averaged over the two sequence orders
en_no_is and en_is_no. _It is observed that growing the model from $126$M
parameters to $356$M reduces loss by 10% on average._ Figure 6: Test loss of
the stage at hand (current loss) vs forgetting, (sum of loss increase on the
past test set(s) since the previous stage) for en_no_is and en_is_no. Color
corresponds to the current language, marker shape corresponds to the current
stage, and marker size to model size. _For a given current language and stage,
increasing the model size leads to a lower current loss but it does not
significantly impact forgetting._

##  5 Discussion

One of our main results is that the specific ordering of languages in the
sequence matters, especially for backward transfer. Icelandic stands out as
the language with the strongest forgetting effect on both other languages.
Interestingly, Icelandic and Norwegian have the highest overlap in their token
distributions, as measured by the novel TDS metric introduced in Section 3.5.
This contradicts our initial hypothesis that the forgetting effects will be
weaker for pairs of languages with high TDS similarity, and we have to
conclude that the TDS metric is not predictive of the backward transfer
observed in our continual learning setup. Instead, we find that syntactic
similarity (SYN), as reflected in the metric derived from URIEL Littell et al.
(2017), is a more relevant factor since it indicates that English and
Norwegian are more similar to each other than to Icelandic. By contrast,
phonetic (INV) and phonological (PHON) similarity, which is high between
Icelandic and Norwegian for historical reasons, is clearly not relevant.

One potential risk factor that affects the interpretation of our experimental
results, and especially the positive backward transfer from Norwegian to
English, is language contamination. Blevins and Zettlemoyer (2022) find that
standard English corpora contain a significant amount of non-English text,
which is enough to enable the cross-lingual transfer of English models to
other languages. The surprisingly low test loss of the Norwegian monolingual
model on English, as seen in Table 4, could indicate the presence of English
text in the Norwegian corpus, despite the basic language identification
applied to the corpus used Öhman et al. (2023). At present, we cannot say
whether this is the case and, if so, whether its effect is stronger than that
of syntactic similarity and other linguistically motivated factors. Finding
ways to disentangle these factors will be important going forward.

In our experimental setup, we opted for consistently using the existing
multilingual GPT-SW3 tokenizer Ekgren et al. (2023) because it is openly
available and our corpus is included in the data used for training the
tokenizer. However, a multilingual tokenizer typically comes with the drawback
of some languages, most often the low-resource ones, being underrepresented in
the vocabulary. Stollenwerk (2023) evaluate the quality of tokenization of the
GPT-SW3 tokenizer measured as the average number of tokens that a word is
decomposed to (fertility score) and report that the tokenization quality of
Icelandic text is clearly lower than for English and Norwegian texts. This
might well be a third important factor when trying to explain the divergent
transfer effects associated with Icelandic in relation to the other languages.

##  6 Conclusion

In this paper, we study forward and backward transfer in a language modeling
task – standardly used for pre-training of large language models – in a novel
multilingual continual learning scenario. Our experimental results with two
model sizes ($126$M and $356$M) and three languages (English, Norwegian,
Icelandic) indicate that there is a consistent positive forward transfer that
facilitates learning new languages through time. However, depending on the
choice of languages, and the order in which they are learned, new languages
may result in either positive or negative backward transfer. In an attempt to
further analyze the transfer patterns observed, we look into the similarity
between the language distributions from both a data-driven and linguistic
perspective and find that syntactic similarity correlates best with the
backward transfer effects. When increasing the model size, we find that, while
model performance improves for all languages, forgetting is not reduced.

We hope that our findings will help in the design of large-scale continual
learning for language models either in multilingual or monolingual setups. The
next step would be to explore longer task sequences with larger models to
assess how our findings hold at different scales. In addition, we need to
explore a larger and more diverse set of languages to assess the generality of
the transfer patterns. As a more long-term goal, this could pave the way for
realizing the potential to improve resource efficiency while reducing negative
transfer in the computationally extensive setup of continual language model
pre-training.

##  7 Acknowledgements

A significant part of the computations was enabled by the Berzelius resource
provided by the Knut and Alice Wallenberg Foundation at the National
Supercomputer Centre at Linköping University, Sweden, within the project
Continual Learning Methods for Large-Scale Multi-Lingual Language Models
(Berzelius-2023-178). In addition, the authors gratefully acknowledge the HPC
RIVR consortium (www.hpc-rivr.si) and EuroHPC JU (eurohpc-ju.europa.eu) for
funding this research by providing computing resources of the HPC system Vega
at the Institute of Information Science (www.izum.si).

## Limitations

The research reported in this paper comes with several limitations that affect
the generalizability of our conclusions. The first one is that our setup is
restricted both in terms of the number of languages and the range of model
sizes considered. A larger and more diverse set of languages will be required
to put any claims about the relation between linguistic similarity and
transfer effects on a surer footing. With respect to model size scaling,
Ramasesh et al. (2022) report that Transformer architectures trained on image
classification become more resistant to forgetting as the model size
increases, and we may need to explore a wider range of model sizes to
establish whether the same holds for continual learning under language shift.

An additional limitation comes from our limited understanding of the datasets.
The corpus used consists of $350$GB text in different languages and
categories, with a big part coming from Common Crawl, known for its variable
quality Luccioni and Viviano (2021); Kreutzer et al. (2022). Even though the
data has passed through a multi-step preprocessing pipeline Öhman et al.
(2023), no further manual data auditing has been done. Therefore, we cannot
exclude the possibility of hidden spurious correlations or language
contamination affecting our experimental results.

Lastly, the only metric used to study transfer effects in our setup is the
language modeling loss on hold-out data. We expect that measuring zero-shot
performance on specific tasks instead would lead to similar transfer patterns,
but further investigation is needed in order to give a definite answer.

## Ethical Considerations

There are two ethical aspects of this work that are worth commenting upon.
First, the datasets used to train and evaluate language models come from a
variety of sources, as specified in Appendix B and further described in Öhman
et al. (2023), and their sheer size makes it impossible to guarantee that they
do not contain inappropriate material. Second, although the models under
investigation are relatively small by current standards, the experiments have
consumed substantial computational resources (see Appendix D). However, it is
also worth mentioning that an important motivation for the research on
continual learning is to reduce resource consumption in the long run.

## References

  * Abnar et al. (2021) Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. 2021.  Exploring the limits of large scale pre-training.  _arXiv preprint arXiv:2110.02095_.
  * Aljundi et al. (2019) Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. 2019.  Online continual learning with maximal interfered retrieval.  In _Advances in Neural Information Processing Systems_ , volume 32.
  * Badola et al. (2022) Kartikeya Badola, Shachi Dave, and Partha Talukdar. 2022.  Parameter-efficient finetuning for robust continual multilingual learning.  _arXiv preprint arXiv:2209.06767_.
  * Barkarson et al. (2022) Starkaður Barkarson, Steinþór Steingrímsson, and Hildur Hafsteinsdóttir. 2022.  Evolving large text corpora: Four versions of the Icelandic Gigaword Corpus.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 2371–2381.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023.  Pythia: A suite for analyzing large language models across training and scaling.  _arXiv preprint arXiv:2304.01373_.
  * Blevins and Zettlemoyer (2022) Terra Blevins and Luke Zettlemoyer. 2022.  Language contamination helps explains the cross-lingual capabilities of English pretrained models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 3563–3574.
  * Caccia et al. (2021) Lucas Caccia, Jing Xu, Myle Ott, Marc’Aurelio Ranzato, and Ludovic Denoyer. 2021.  On anytime learning at macroscale.  _arXiv preprint arXiv:2106.09563_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451.
  * Coria et al. (2022) Juan Manuel Coria, Mathilde Veron, Sahar Ghannay, Guillaume Bernard, Hervé Bredin, Olivier Galibert, and Sophie Rosset. 2022.  Analyzing BERT cross-lingual transfer capabilities in continual sequence labeling.  In _Proceedings of the First Workshop on Performance and Interpretability Evaluations of Multimodal, Multipurpose, Massive-Scale Models_ , pages 15–25, Virtual. International Conference on Computational Linguistics.
  * de Masson d'Autume et al. (2019) Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019.  Episodic memory in lifelong language learning.  In _Advances in Neural Information Processing Systems_ , volume 32.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Douillard et al. (2021) Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. 2021.  Dytox: Transformers for continual learning with dynamic token expansion.  _arXiv preprint arXiv:2111.11326_.
  * Ekgren et al. (2023) Ariel Ekgren, Amaru Cuba Gyllensten, Felix Stollenwerk, Joey Öhman, Tim Isbister, Evangelia Gogoulou, Fredrik Carlsson, Alice Heiman, Judit Casademont, and Magnus Sahlgren. 2023.  GPT-SW3: An autoregressive language model for the Nordic languages.  _arXiv preprint arXiv:2305.12987_.
  * French (1999) Robert M. French. 1999.  Catastrophic forgetting in connectionist networks.  _Trends in Cognitive Sciences_ , 3(4):128–135.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020.  The Pile: An 800GB dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Garcia et al. (2021) Xavier Garcia, Noah Constant, Ankur P. Parikh, and Orhan Firat. 2021.  Towards continual learning for multilingual machine translation via vocabulary substitution.  _arXiv preprint arXiv:2103.06799_.
  * Gururangan et al. (2021) Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A Smith, and Luke Zettlemoyer. 2021.  Demix layers: Disentangling domains for modular language modeling.  _arXiv preprint arXiv:2108.05036_.
  * Hadsell et al. (2020) Raia Hadsell, Dushyant Rao, Andrei Rusu, and Razvan Pascanu. 2020.  Embracing change: Continual learning in deep neural networks.  _Trends in Cognitive Sciences_ , 24:1028–1040.
  * Han et al. (2021) Rujun Han, Xiang Ren, and Nanyun Peng. 2021.  ECONET: Effective continual pretraining of language models for event temporal reasoning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5367–5380.
  * Hayes et al. (2020) Tyler L Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. 2020.  Remind your neural network to prevent catastrophic forgetting.  In _European Conference on Computer Vision_ , pages 466–483. Springer.
  * Jang et al. (2022) Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. 2022.  TemporalWiki: A lifelong benchmark for training and evaluating ever-evolving language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6237–6250.
  * Jang et al. (2021) Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2021.  Towards continual knowledge learning of language models.  _arXiv preprint arXiv:2110.03215_.
  * Jin et al. (2022) Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. 2022.  Lifelong pretraining: Continually adapting language models to emerging corpora.  In _Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models_, pages 1–16.
  * Ke and Liu (2022) Zixuan Ke and Bing Liu. 2022.  Continual learning of natural language processing tasks: A survey.  _arXiv preprint arXiv:2211.12701_.
  * Ke et al. (2023) Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023.  Continual pre-training of language models.  In _The Eleventh International Conference on Learning Representations_.
  * Kreutzer et al. (2022) Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. 2022.  Quality at a glance: An audit of web-crawled multilingual datasets.  _Transactions of the Association for Computational Linguistics_ , 10:50–72.
  * Kummervold et al. (2022) Per Kummervold, Freddy Wetjen, and Javier de la Rosa. 2022.  The Norwegian colossal corpus: A text corpus for training large Norwegian language models.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 3852–3860.
  * Lange et al. (2019) Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2019.  Continual learning: A comparative study on how to defy forgetting in classification tasks.  _arXiv preprint arXiv:1909.08383_.
  * Lazaridou et al. (2021) Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021.  Mind the gap: Assessing temporal generalization in neural language models.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 29348–29363.
  * Lesort et al. (2022) Timothée Lesort, Oleksiy Ostapenko, Diganta Misra, Md Rifat Arefin, Pau Rodríguez, Laurent Charlin, and Irina Rish. 2022.  Challenging Common Assumptions about Catastrophic Forgetting.  _arXiv preprint arXiv:2207.04543_.
  * Lesort et al. (2021) Timothée Lesort, Massimo Caccia, and Irina Rish. 2021.  Understanding continual learning settings with data distribution drift analysis.  _arXiv preprint arXiv:2104.01678_.
  * Lesort et al. (2020) Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Díaz-Rodríguez. 2020.  Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges.  _Information Fusion_ , 58:52 – 68.
  * Littell et al. (2017) Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017.  URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 8–14.
  * Liu et al. (2021) Zihan Liu, Genta Indra Winata, Andrea Madotto, and Pascale Fung. 2021.  Preserving cross-linguality of pre-trained models via continual learning.  In _Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)_ , pages 64–71.
  * Loureiro et al. (2022) Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-collados. 2022.  TimeLMs: Diachronic language models from Twitter.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 251–260.
  * Luccioni and Viviano (2021) Alexandra Luccioni and Joseph Viviano. 2021.  What’s in the box? an analysis of undesirable content in the Common Crawl corpus.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 182–189.
  * Mao et al. (2022) Yuren Mao, Yaobo Liang, Nan Duan, Haobo Wang, Kai Wang, Lu Chen, and Yunjun Gao. 2022.  Less-forgetting multi-lingual fine-tuning.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 14917–14928.
  * Mirzadeh et al. (2021) Seyed Iman Mirzadeh, Arslan Chaudhry, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. 2021.  Wide neural networks forget less catastrophically.  _arXiv preprint arXiv:2110.11526_.
  * Mirzadeh et al. (2020) Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. 2020.  Understanding the role of training regimes in continual learning.  _Advances in Neural Information Processing Systems_ , 33:7308–7320.
  * Öhman et al. (2023) Joey Öhman, Severine Verlinden, Ariel Ekgren, Amaru Cuba Gyllensten, Tim Isbister, Evangelia Gogoulou, Fredrik Carlsson, and Magnus Sahlgren. 2023.  The Nordic Pile: A 1.2 TB Nordic dataset for language modeling.  _arXiv preprint arXiv:2303.17183_.
  * Ortiz Suárez et al. (2020) Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2020.  A monolingual approach to contextualized word embeddings for mid-resource languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1703–1714.
  * Ostapenko et al. (2021) Oleksiy Ostapenko, Pau Rodriguez, Massimo Caccia, and Laurent Charlin. 2021.  Continual learning via local module composition.  In _Thirty-Fifth Conference on Neural Information Processing Systems_.
  * Parisi et al. (2019) German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. 2019.  Continual lifelong learning with neural networks: A review.  _Neural Networks_ , 113:54 – 71.
  * Qin et al. (2022) Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022.  Elle: Efficient lifelong pre-training for emerging data.  _arXiv preprint arXiv:2203.06311_.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018.  Improving language understanding by generative pre-training.  OpenAI.
  * Rae et al. (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021.  Scaling language models: Methods, analysis & insights from training gopher.  _arXiv preprint arXiv:2112.11446_.
  * Ramasesh et al. (2022) Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. 2022.  Effect of scale on catastrophic forgetting in neural networks.  In _International Conference on Learning Representations_.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017.  icarl: Incremental classifier and representation learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2001–2010.
  * Ruder (2016) Sebastian Ruder. 2016.  An overview of gradient descent optimization algorithms.  _arXiv preprint arXiv:1609.04747_.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  BLOOM: A 176B-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Scialom et al. (2022) Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022.  Fine-tuned language models are continual learners.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6107–6122.
  * Stollenwerk (2023) Felix Stollenwerk. 2023.  Training and evaluation of a multilingual tokenizer for GPT-SW3.  _arXiv preprint arXiv:2304.14780_.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wang et al. (2023) Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. 2023.  A comprehensive survey of continual learning: Theory, method and application.  _arXiv preprint arXiv:2302.00487_.
  * Winata et al. (2023) Genta Indra Winata, Lingjue Xie, Karthik Radhakrishnan, Shijie Wu, Xisen Jin, Pengxiang Cheng, Mayank Kulkarni, and Daniel Preotiuc-Pietro. 2023.  Overcoming catastrophic forgetting in massively multilingual continual learning.  _arXiv preprint arXiv:2305.16252_.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498.

##  Appendix A Model Details

Table 5 shows the model parameters selected for each model size and Table 6
shows the training hyperparameters used.

Parameter Name | 126M Model | 356M Model
---|---|---
Sequence length | $2048$ | $2048$
Number of layers | $12$ | $24$
Hidden size dimension | $768$ | $1024$
Number of attention heads | $12$ | $16$
Table 5: Model parameters for the two model sizes. Parameter Name | 126M Model | 356M Model
---|---|---
Tensor parallelism | $1$ | $2$
Micro batch size | $4$ | $4$
Global batch size | $1024$ | $1024$
Min learning rate | $6e-5$ | $3e-5$
Max learning rate | $6e-4$ | $3e-4$
Table 6: Hyperparameters used for training the models.

##  Appendix B Dataset Details

Each dataset has a weighted contribution to the total number of training
sequences in that language, with the weight being the dataset category,
normalized by the total size of the language corpus. Table 7 shows a list of
the different datasets, as well as the corresponding category and normalized
weight assigned to each dataset.

Dataset | Category | Category weight | Dataset size (GB) | Normalised weight
---|---|---|---|---
books3 (The Pile) | Books | $1$ | $89$ | $0.43$
PubMed, arXiv (The Pile) | Articles | $0.9$ | $33$ | $0.15$
Stackexchange (The Pile) | Miscellaneous | $1$ | $35$ | $0.17$
NCC | Miscellaneous | $1$ | $39$ | $0.49$
Icelandic Gigaword | Miscellaneous | $1$ | $9.6$ | $0.69$
Pile Openwebtext | Web CC | $0.5$ | $58$ | $0.14$
mc4, oscar (no) | Web CC | $0.5$ | $78$ | $0.49$
mc4, oscar (is) | Web CC | $0.5$ | $8.4$ | $0.34$
Wiki en | Wiki | $1.5$ | $15$ | $0.1$
Wiki no | Wiki | $1.5$ | $0.5$ | $0.01$
Wiki is | Wiki | $1,5$ | $0.05$ | $0.006$
Table 7: List of all datasets together with their corresponding original
dataset size and assigned weighted contribution to the language corpus.

##  Appendix C Detailed Learning Rate Results

Table 8 shows the full list of test loss values when training the $126$M
models on different language orders using combinations of the constant and
cosine annealing learning rate policy.

Model | en | is | no
---|---|---|---
gpt_126m_en_is
cosine annealing | $4.43$ | 2.58 | -
constant | 4.39 | $2.64$ | -
gpt_126m_en_is_no
cosine annealing | $3.42$ | $4.96$ | 3.00
constant | $3.44$ | $4.95$ | $3.15$
constant - cosine annealing | 3.40 | $5.06$ | $3.09$
cosine annealing - constant | $3.45$ | 4.86 | $3.14$
gpt_126m_en_no
cosine annealing | $3.41$ | - | 3.03
constant lr 6e-5 | 3.39 | - | $3.18$
gpt_126m_en_no_is
cosine annealing | $4.32$ | 2.54 | $5.30$
constant lr 6e-5 | $4.28$ | $2.59$ | $5.31$
constant - cosine annealing | $4.37$ | 2.54 | $5.46$
cosine annealing - constant | 4.25 | $2.56$ | 5.21
Table 8: Test loss in English (en), Icelandic (is) and Norwegian (no) for
models trained with different learning rate schedulers on the second and third
training stage. The learning rate scheduler in the first training stage is
always cosine annealing. The lowest value per column and model is indicated in
bold.

##  Appendix D Training Environment and Efficiency

The experiments were performed on two clusters equipped with different numbers
of NVIDIA A100 GPUs: Vega 555https://doc.vega.izum.si/ a GPU cluster part of
the EuroHPC network that consists of 60 GPU nodes of 4 NVidia A100 each and
Berzelius 666https://www.nsc.liu.se/systems/berzelius/ which is located in
Linköping, Sweden and consists of 94 GPU nodes of 8 NVidia A100 each. A single
monolingual training phase of the $126$M model required $352$ GPU hours on
Vega, while the same run for the $356$M model took $352$ GPU hours on
Berzelius.