# Evaluating the Resilience of Variational Quantum Algorithms to Leakage Noise

Chen Ding  Henan Key Laboratory of Quantum Information and Cryptography,
Zhengzhou, Henan 450000, China  Xiao-Yue Xu  Shuo Zhang  Henan Key Laboratory
of Quantum Information and Cryptography, Zhengzhou, Henan 450000, China  He-
Liang Huang<EMAIL_ADDRESS>Henan Key Laboratory of Quantum Information
and Cryptography, Zhengzhou, Henan 450000, China  Hefei National Research
Center for Physical Sciences at the Microscale and School of Physical
Sciences, University of Science and Technology of China, Hefei 230026, China
Shanghai Research Center for Quantum Science and CAS Center for Excellence in
Quantum Information and Quantum Physics, University of Science and Technology
of China, Shanghai 201315, China  Hefei National Laboratory, University of
Science and Technology of China, Hefei 230088, China  Wan-Su Bao
<EMAIL_ADDRESS>Henan Key Laboratory of Quantum Information and Cryptography,
Zhengzhou, Henan 450000, China

###### Abstract

As we are entering the era of constructing practical quantum computers,
suppressing the inevitable noise to accomplish reliable computational tasks
will be the primary goal. Leakage noise, as the amplitude population leaking
outside the qubit subspace, is a particularly damaging source of error that
error correction approaches cannot handle. However, the impact of this noise
on the performance of variational quantum algorithms (VQAs), a type of near-
term quantum algorithms that is naturally resistant to a variety of noises, is
yet unknown. Here, we consider a typical scenario with the widely used
hardware-efficient ansatz and the emergence of leakage in two-qubit gates,
observing that leakage noise generally reduces the expressive power of VQAs.
Furthermore, we benchmark the influence of leakage noise on VQAs in real-world
learning tasks. Results show that, both for data fitting and data
classification, leakage noise generally has a negative impact on the training
process and final outcomes. Our findings give strong evidence that VQAs are
vulnerable to leakage noise in most cases, implying that leakage noise must be
effectively suppressed in order to achieve practical quantum computing
applications, whether for near-term quantum algorithms and long-term error-
correcting quantum computing.

##  I Introduction

While quantum computing is one of the most promising new trends in information
processing, the noisy qubits in the Noisy Intermediate-Scale Quantum (NISQ)
era [1, 2, 3, 4, 5, 6] limit its immediate applicability. Thus, correcting,
mitigating, or avoiding the quantum errors to achieve valid computation
results is a major topic in current era [7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21]. Variational quantum algorithms (VQAs) [22, 23, 24, 25,
26, 27, 28, 29, 30] have emerged as the one of leading candidates to achieve
application-oriented quantum computational advantage on NISQ devices, owing to
their hybrid quantum-classical approach which has potential noise resilience
[31, 32, 33]. To date, VQAs have already been demonstrated successfully for a
variety of machine learning tasks, such as classification [34, 27, 24],
generative modeling [35, 23], optimization [36], and quantum chemistry
problems [37, 38, 39, 40, 41, 42].

Despite the continuing improvement of such experimental demonstrations, a
critical question remains as to how noise affects the performance of VQAs.
This issue has been explored extensively in recent years, with many valuable
findings. Generally, the resillience of VQAs exists for a wide class of noise,
as analyzed in Refs. [31, 32, 33]. Yet such noise resillience is highly
limited as the noise level rising. As suggested in [43, 44, 45, 46, 47], large
noise may leads to problems such as performance degradation or barren
plateaus. Among all the works mentioned above, leakage error is not included
in their error models, which is a notable omission given the severity of
leakage error. Leakage error refers to the accidental qubit activations
outside of the computational space in two-level quantum computers. Usual
quantum error correction protocols cannot correct (and may even exacerbate)
leakage error [48, 49]. Even if the probability of these leakage errors is
modest, they will accumulate and eventually corrupt the error correction code.
Thus, analyzing the impact of leakage error on VQAs is crucial for the near-
term application of quantum computing.

In this paper, we investigate performance of VQAs with the occurrence of
leakage error. We first analyze the effect model of leakage error in the
execution of quantum circuits, and propose an expressibility measure that
predicts the performance of variational ansatz with noiseless environment and
leakage error. Next, based on the assumption of the leakage occurrence during
two-qubit gates, the most typical leakage noise in superconducting quantum
computing, we conduct numerical experiments to benchmark the impact of leakage
error on applications of data fitting and data classification when using the
widely used hardware-efficient ansatz, with varying the system size, circuit
depth, and leakage probability. The simulation results of expressibility, data
fitting and data classification all suggest VQAs are vulnerable to leakage
error. Moreover, our results on chain, ladder and lattice processor also show
that better connectivity architectures also do not contribute to leakage error
immunity. This work facilitates a deeper understanding of how noise impacts
the performance of VQAs, and provides new insight on the key issues of NISQ
techniques.

##  II Theoretical Analysis

In this section, we will theoretically analyze the impact of leakage error on
VQAs, including the measurement results and the expressibility.

###  II.1 Leakage Error in Quantum Circuits

As stated above, the phenomenon of leakage refers to the state activation out
of the usual two-level computation space $\ket{0}$ and $\ket{1}$. Let’s first
take a look at how leakage error affects the output of quantum computing. Take
one-qubit for instance, the leakage noise introduces activated states as

$\ket{\psi}=\alpha_{0}\ket{0}+\alpha_{1}\ket{1}+\alpha_{2}\ket{2}+\alpha_{3}\ket{3}+...$

If high energy levels of the qubits, $\ket{2}$, $\ket{3}$, etc, can be readout
properly, one can directly utilize the post-selection strategy to rule out
events with leakage errors. Thus, here we assume that states leaking to higher
energy levels cannot be readout correctly, where their probabilities of being
misread as $\ket{0}$ and $\ket{1}$ are $\beta$ and $1-\beta$, respectively. In
this case, the read populations of $\ket{0}$ and $\ket{1}$ are

$\displaystyle p_{0}$
$\displaystyle=|\alpha_{0}|^{2}+\beta(1-|\alpha_{0}|^{2}-|\alpha_{1}|^{2}),$
$\displaystyle p_{1}$
$\displaystyle=|\alpha_{1}|^{2}+(1-\beta)(1-|\alpha_{0}|^{2}-|\alpha_{1}|^{2}).$

As a consequence, the expectation of any observable in $\ket{\psi}$ is
deviated from its original value. For instance, the expectation of
$O=\text{Diag}(\lambda_{0},\lambda_{1})$ will be

$\displaystyle\braket{\psi}{O}{\psi}$ $\displaystyle=$
$\displaystyle\lambda_{0}(|\alpha_{0}|^{2}+\beta(1-|\alpha_{0}|^{2}-|\alpha_{1}|^{2}))$
$\displaystyle+$
$\displaystyle\lambda_{1}(|\alpha_{1}|^{2}+(1-\beta)(1-|\alpha_{0}|^{2}-|\alpha_{1}|^{2}))$
$\displaystyle=$
$\displaystyle(|\alpha_{0}|^{2}+|\alpha_{1}|^{2})\braket{\phi}{O}{\phi}$
$\displaystyle+$
$\displaystyle(1-|\alpha_{0}|^{2}-|\alpha_{1}|^{2})(\lambda_{0}\beta+\lambda_{1}(1-\beta)),$

in which
$\ket{\phi}=\frac{1}{\sqrt{|\alpha_{0}|^{2}+|\alpha_{1}|^{2}}}(\alpha_{0}\ket{0}+\alpha_{1}\ket{1})$.

For non-diagonal observable $H$, there are two ways to perform the
measurement—diagonalization and Pauli decomposition. For the diagonalization
method, we implement the measurement by rotating the measurement basis to
diagonalize $H$:

$VHV^{\dagger}=\begin{pmatrix}\lambda_{0}&0\\\ 0&\lambda_{1}\end{pmatrix}$

the expectation will be

$\displaystyle\braket{\psi}{H}{\psi}$ $\displaystyle=$
$\displaystyle\lambda_{0}(|V_{00}\alpha_{0}+V_{01}\alpha_{1}|^{2}+\beta(1-|\alpha_{0}|^{2}-|\alpha_{1}|^{2}))$
$\displaystyle+$
$\displaystyle\lambda_{1}(|V_{10}\alpha_{0}+V_{11}\alpha_{1}|^{2}+(1-\beta)(1-|\alpha_{0}|^{2}-|\alpha_{1}|^{2}))$
$\displaystyle=$
$\displaystyle(|\alpha_{0}|^{2}+|\alpha_{1}|^{2})\braket{\phi}{H}{\phi}$
$\displaystyle+$
$\displaystyle(1-|\alpha_{0}|^{2}-|\alpha_{1}|^{2})(\lambda_{0}\beta+\lambda_{1}(1-\beta)).$

For the Pauli decomposition method, we implement the measurement by sum of
Pauli matrices $H=\sum h_{i}\sigma_{i}$, the expectation will be

$\displaystyle\braket{\psi}{H}{\psi}$ $\displaystyle=$ $\displaystyle
h_{0}+h_{1}\braket{\psi}{\sigma_{x}}{\psi}+h_{2}\braket{\psi}{\sigma_{y}}{\psi}+h_{3}\braket{\psi}{\sigma_{z}}{\psi}$
$\displaystyle=$
$\displaystyle(|\alpha_{0}|^{2}+|\alpha_{1}|^{2})\braket{\phi}{H}{\phi}$
$\displaystyle+$
$\displaystyle(1-|\alpha_{0}|^{2}-|\alpha_{1}|^{2})(h_{0}+(h_{1}+h_{2}+h_{3})(2\beta-1)).$

We observe the two implementation methods will generally yield different
expectation value for the same non-diagonal observable, which is due to the
misreading mechanism stated above, and the limitation of rotation $V$ in the
2-level computational space. Such results suggest the leakage state after a
misreading channel cannot simply be viewed as pure or mixed 2-level states.

The direct effect of leakage is the deviation of the measurement results,
determined by the population of higher-energy states, the misreading
probability $\beta$ and $1-\beta$, the observable $H$, the measurement
implementation. This gives us an intuitive impression on the impact of leakage
error on quantum computing. As more qubits and gate operations are introduced,
the analysis will become complicated, especially for VQAs, which would adjust
the parameters of quantum operations during the training procedure. We thus
propose a more powerful tool to evaluate the performance of VQAs with leakage
error, by estimating the expressibility.

###  II.2 The Expressibility Measure for VQAs with Leakage error

Figure 1: The 3-dimensional simplex $\Delta^{4}$ and uniform distribution
samples in it (corresponding to system size $n=2$). The black dots are the
samples $\mathbf{F}_{\text{uniform}}$, which distributes evenly inside the
simplex. A well-performing ansatz should output similar samples. Our
expressibility measure evaluate minus MMD distance between
$\mathbf{F}_{\text{uniform}}$ and the ansatz’s output samples. The dashed
lines inside the simplex are helpers to view the shape in depth.

VQAs utilize parameterized quantum circuit (PQC) models that can be trained
for a variety of machine learning tasks. Given a fixed circuit structure, the
operation of the PQC is parameterized by the variables $\mathbf{\theta}$, as
illustrated in Fig. 2. We therefore denote the map from the parameters
$\mathbf{\theta}$ to the quantum circuit as $U(\cdot)$. To predict the
performance of the corresponding VQAs before applying the PQC to a certain
application, we need to evaluate the expressibility (or expressivity) of the
map $U(\cdot)$ [50, 43, 51, 52, 53], which indicates its ability to generate
eligible outputs for a general purpose. For instance, Sukin Sim $et~{}al$.
quantify the expressibility by estimating the statistical properties of a
PQC’s outputs, given uniformly distributed parameters $\theta$ and a fixed
initial state [53]. Specifically, they calculate the Kullback-Leibler (KL)
divergence [54] of two distribution functions

$\text{Expr}_{1}(U)=D_{\text{KL}}\left(f_{F_{U}}||f_{F_{\text{Haar}}}\right).$
(1)

In Equation (1), the PQC is assigned with randomly generated parameters
sampled from uniform distribution in $[0,2\pi]$. $f_{F_{U}}$ is the
distribution function of the overlap $F_{U}=|\braket{\psi_{1}}{\psi_{2}}|^{2}$
of the PQC’s arbitrary two outcomes given random parameters $\theta$, and
$f_{F_{\text{Haar}}}$ is the distribution function of the overlap
$F_{\text{Haar}}=|\braket{\phi_{1}}{\phi_{2}}|^{2}$ of arbitrary two states in
the uniform distribution (the Haar distribution) of the Hilbert space. The
initial state of the ansatz is fixed as $\ket{\mathbf{0}}$. By comparing the
fidelity distribution $F_{U}$ and $F_{\text{Haar}}$, Expr1 implies the
ansatz’s capability of approximately generating outputs in Haar distribution,
which meet the demand of applications like finding the ground state for an
arbitrary Hamiltionian, or preparation of an arbitrary pure state.

However, Expr1 assumes the ansatz’s outputs are pure states, which is no more
suitable for expressibility evaluation in the leakage case. As shown in
Subsection II.1, the expectation value of misread leakage state varies with
the choice of measurement basis. To remove the Hilbert output space assumption
of Expr1, we consider the outputs as real-valued vectors—the population of
basis states. By computational basis measurement, the higher-energy states are
read in a fixed way, which can then be viewed as the natural output of the
ansatz. Any state with leakage solely corresponds to one real vector in the
restricted hyperplane in $\mathbb{R}^{2^{n}}$, which is also a standard
simplex in $\mathbb{R}^{2^{n}-1}$:

$\displaystyle\Delta^{2^{n}}:=$
$\displaystyle\\{(p_{0},p_{1},...,p_{2^{n}-2})\in\mathbb{R}^{2^{n}-1}|\sum^{2^{n}-2}_{i=0}p_{i}<1,$
$\displaystyle p_{i}\geq 0\quad\forall i=0,...,2^{n}-2\\}.$ (2)

We thus propose a real-valued output version method for expressibility
estimation, as the uniformity of the ansatz outputs in the simplex
$\Delta^{2^{n}}$. Moreover, to cover higher-order statistics of the ansatz’s
output distribution $\mathbf{F}_{U}$, we directly calculate the distance
between $\mathbf{F}_{U}$ and the uniform distribution
$\mathbf{F}_{\text{uniform}}$ in the simplex. For achieving efficient and
robust high-dimensional distance evaluation, we choose the Maximum Mean
Discrepancy (MMD) [55, 56]

$\text{MMD}(P,Q):=\|\mathbb{E}_{X\sim
P}[\varphi(X)]-\mathbb{E}_{{\color[rgb]{0,0,0}Y\sim
Q}}[\varphi(Y)]\|_{\mathcal{H}},$ (3)

as the distance measure, in which $\varphi(\cdot)$ is the function that maps
the samples $X,Y$ to the reproducing kernel Hilbert space. The expressibility
of the ansatz $U(\mathbf{\theta})$ is defined as the minus MMD distance, as

$\text{Expr}_{2}(U)=-\text{MMD}(\mathbf{F}_{U},\mathbf{F}_{\text{uniform}}).$
(4)

By definition, it is easy to find $\text{Expr}_{2}=0$ if and only if
$\mathbf{F}_{U}=\mathbf{F}_{\text{uniform}}$, while a similar proposition is
yet to be proven for $\text{Expr}_{1}$ [53]. Replacing $\varphi(\cdot)$ by a
kernel
$k(\mathbf{x},\mathbf{y})=\braket{\varphi(x),\varphi(y)}_{\mathcal{H}}$, it is
then efficient to estimate the MMD with the samples of $\mathbf{F}_{U}$ and
$\mathbf{F}_{\text{uniform}}$ (noted as $X_{i},Y_{i},i=1,...,N$,
respectively), as

$\displaystyle\text{MMD}(P,Q)$ $\displaystyle\approx$
$\displaystyle\frac{1}{N^{2}}\left|\sum_{i=1,j=1}^{N}k(X_{i},X_{j})+k(Y_{i},Y_{j})-2k(X_{i},Y_{j})\right|,$
(5)

in which we use the kernel
$k(\mathbf{x},\mathbf{y})=e^{-\frac{\|\mathbf{x}-\mathbf{y}\|^{2}}{4\sigma}}$.
The parameter $\sigma$ is taken as $0.01$ in our calculation to optimize the
prediction performance. The initial state of the ansatz is taken as
$\ket{\mathbf{+}}$. Our proposed method enables us to comprehensively evaluate
whether the output of the variational quantum circuit with different circuit
parameters can be well spread into the entire output space (the simplex), that
is, the high expressibility. Since we assume the outputs are real vectors,
Expr2 is more suitable for predicting ansatz’s performance in machine learning
applications like real data fitting or classification.

Figure 2: A multi-layer ansatz for VQA, in which we only consider leakage
error for the two-qubit gates in the entangling layer (shown in the light
yellow area). In the simulation, we assume the CNOT gates are implemented by a
CZ gate and two Hadamard gates, as $\text{CNOT}=(I\otimes H)\text{CZ}(I\otimes
H)$.

##  III Numerical Experiments

Figure 3: The expressibility difference between ideal ansatz and ansatz with
leakage error, varying with system size $n$, circuit depth $d$, and leakage
probability $L$. In (a), we show a overall 3D heatmap of the expressibility
difference. In (b-f), the five subfigures are the slices taken from (a),
showing the case of different system sizes. In these figures, each lattice
shows the average difference over 20 independent experiments. The average
standard deviation of the means is about $5\cdot 10^{-4}$.

The performance of VQAs heavily depends on the expressibility of the employed
ansatz. However, different machine learning tasks and training methods may
have different requirements for the output quantum states. We thus conduct
numerical experiments to benchmark the actual impact of leakage error in
various machine learning tasks, and validate the capability of prediction by
our expressibility measure. For the sake of typicality, we consider the most
commonly used hardware-efficient PQC ansatz [57, 58, 59, 60, 61], shown in
Fig. 2, which consists of alternating trainable single-qubit layers and fixed
entangling layers. For superconducting transmons, two-qubit gates constitute
the dominant source of leakage, while single-qubit gates and measurement
operations have negligible leakage probabilities which we ignore in the
simulation. We model the leakage in CZ gates as an exchange between $\ket{11}$
and $\ket{02}$ during the CZ gate, i.e.,
$|11\rangle\mapsto\sqrt{1-4L}|11\rangle+{e^{i\phi}}\sqrt{4L}|02\rangle$ and
$|02\rangle\mapsto\sqrt{1-4L}|02\rangle-{e^{-i\phi}}\sqrt{4L}|11\rangle$, in
which leakage probability $L$ is typically $1.25\cdot 10^{-3}$ [62, 63].
Besides, we also assume the transitions phase $\phi=0$ for simplicity. The
misread coefficient $\beta$ is experimentally tunable and may be set as some
specific values to minimize the effects of leakage in a certain computing
task. We focus on the simplest case where higher-energy states are misread as
$\ket{1}$ ($\beta=0$) in the experiments, since it is generally more probable
that higher-level state being misread as $\ket{1}$ other than $\ket{0}$ in
most superconducting quantum systems. Eventually, there are only three
hyperparameters in our experiments: the system size $n$, the depth of the
circuit $d$, and the leakage probability $L$. In all the experiments below, we
take uniformly random initial parameters at the start of training.

###  III.1 Expressibility Estimation

We first estimate the expressibility of the employed ansatz with noiseless
environment and leakage error, respectively, by using the method proposed in
Sec. II.2 In our simulation, the hyperparameters are chosen as: system size
$n=2,...,6$, circuit depth $d=1,...,6$ (We count the depth as the number of
layers, which consist of trainable single-qubit layer and a fixed entangling
layer, in the circuit, as shown in Fig. 2.), leakage probability $L$
proportionally ranges from $1.25\cdot 10^{-4}$ to $1.25\cdot 10^{-2}$. Under
each combination of hyperparameters, we take 10000 samples from the circuit’s
output distribution and the uniform distribution to estimate the
expressibility. Figure 3 shows the expressibility difference between noiseless
ansatz and ansatz with leakage error,
$\text{Expr}_{2}^{\text{noiseless}}-\text{Expr}_{2}^{\text{leakage}}$. It can
be observed that the noiseless ansatz always has a higher expressibility.
Moreover, the expressibility difference rises as the leakage probability $L$
and circuit depth $d$ increase.

###  III.2 Data Fitting

Figure 4: The logarithm of accuracy ratio of ideal ansatz and ansatz with
leakage error, varying with system size $n$, circuit depth $d$, and leakage
probability $L$. In (a), we show a overall 3D heatmap of the ratio logarithm.
In (b-f), the five subfigures are the slices taken from (a), showing the case
of different system sizes. In these figures, each square shows the average
result over 100 independent experiments. The average standard deviation of the
means is about $0.05$.

Next we compare the performance of VQAs with noiseless environment and leakage
error on the application of data fitting—the most fundamental functionality of
machine learning models. We set initial states as $\ket{+}^{\otimes n}$, and
randomly sample vectors from the uniform distribution in $\Delta^{2^{n}-1}$ as
the targets. The loss function is the fidelity between the target distribution
$\bm{q}$ and the output $\bm{p}(\bm{\theta})$ of VQA, as

$f(\bm{\theta})=1-\left(\sum\sqrt{p_{i}(\bm{\theta})q_{i}}\right)^{2}.$

where $\bm{\theta}$ is the trainable parameters in the PQC.

Figure 4 shows the logarithm (in base 10) of the ratio of the loss values of
VQAs with noiseless environment and leakage error at epoch 200. The ratio
rises as the leakage probability $L$ and circuit depth $d$ increase,
suggesting that more leakage occurs with increasing depth and leakage
probability. The highest ratio $10^{6.57}$ is achieved in the case of $n=d=6$,
$L=0.0125$, while the lowest ratio $10^{-0.28}$ is at in the case of $n=4$,
$d=2$, $L=0.000789$. In all the cases of these hyperparameters, the training
of VQAs in the both noiseless and noisy environment converges. Figure 5 shows
the loss curve during the training in the case that highest ratio appears,
which demonstrate the convergence. These results suggest that leakage error
can degrade the performance of data fitting, this overall phenomenon is
consistent with the prediction by expressibility.

Figure 5: The training curve in the case of $n=d=6$, $L=0.0125$. Each point
(from Epoch 1 to Epoch 200) on the curve is an average result over 100
independent experiments. The standard deviations of the means for these
independent experiment results are no more than $0.003$.

###  III.3 Data Classification on the Iris Dataset

Furthermore, we investigate the effect of leakage error on a real-world
application of VQA—the classification of Iris [64, 65], which consists of 150
4-element vectors of three classes. We choose the virginica and versicolor
classes for binary classification. The data vectors are encoded into
amplitudes of 2-qubit quantum states as the circuit input. And we use the mean
square error (MSE) cost function

$C(\bm{\theta})=\sum_{i=1}^{M}(\braket{\psi_{i}}{U(\bm{\theta})^{\dagger}Z^{\otimes
2}U(\bm{\theta})}{\psi_{i}}-y_{i})^{2},$

for training, in which $\bm{\theta}$ are the trainable parameters, $M=100$ is
the number of samples, $\ket{\psi_{i}}$ are the encoded input quantum states
and $y_{i}$ are the corresponding labels.

We use 4-fold cross-validations for evaluating the performance of VQAs with
noiseless environment and leakage error. Figure 6 shows the cross validation
score difference between noiseless VQAs and VQAs with leakage error, at
different circuit depth $d$ and leakage probability $L$. The difference rises
as the leakage probability $L$ and circuit depth $d$ increase, also indicating
that leakage error will degrade the performance of the classification, in
consistency with the results in data fitting experiments.

Figure 6: The cross validation score difference between ideal ansatz and
ansatz with leakage error, with varying circuit depth $d$ and leakage
probability $L$. The system size $n$ is fixed as 2. Each lattice an average
result over 100 independent experiments. The average standard deviation of the
means is about $6\cdot 10^{-4}$.

###  III.4 The Impact of Leakage Error on VQAs on the Quantum Processors with
Different Architectures

Usually, the entangling layers would be slightly changed according to the
topological architecture of the quantum processor, such as chain, ladder, and
lattice (shown in Fig. 7), for the purpose of improving the efficiency of the
quantum hardware. Here, we assume that the an entangling layer is realized by
implementing CZ gates for all nearest-neighbor coupled qubit pairs, and then
compare the expressibility of the ansatz implemented on the quantum processor
with various architectures. Figure 7 shows the expressibility estimation
results for the ansatz with noiseless environment and leakage error, where the
ansatz consists of 9 qubits and 5 depths, and the entangling layers are
assumed to be implemented on the chain, ladder, and lattice processor
architectures. We find that in the noiseless case, the expressibility of these
ansatz rises with increasing processor connectivity, suggesting the advantage
of better connectivity. However, when leakage error occurs, such advantage
vanishes, as the expressibilities of the three ansatz drop to the same level.

Figure 7: The expressibility of ansatz with chain, ladder and lattice
entangling layers. Each bar is an average over 20 independent experiments. The
nearly invisible error bars on them suggest the numerical stability of the
calculated results. The sketches above the bars show the detailed entangling
layer structures, where we use CZ gates instead of CNOT.

##  IV Conclusion

To evaluate the resillience of VQA to leakage noise, we quantify the ansatz
expressibility and perform comparative tests in different types of machine
learning tasks. The results clearly show the detrimental effect of leakage
error on VQA, as it decreases the expressibility and lowers the training
accuracy. In the numerical experiments, the model we choose is typical in
terms of the hardware-efficient ansatz and the leakage occurrence in
superconducting transmons [62, 63, 66, 67]. Furthermore, we note that the
calculation of measurement outcomes in Sec. II.1 is independent of specific
leakage occurrence models, and the expressibility evaluation in III.1 is
independent of the choice of cost functions and training process of VQA.
Thereby, our results are not only typical, but also has a certain generality.
The rising of performance degradation with the increasing of circuit depth and
leakage probability suggests the prominence of developing the leakage-
reduction scheme for qubits and quantum operations, both for near- and long-
term practical quantum computing.

Besides, the generalized expressibility measure we proposed is designed from
the perspective of quantum machine learning with real-valued outputs, and can
in principle be adapted to analyze a wide range of various VQA with different
architectures.

The case considered in our work is the VQAs based a two-level quantum system
in $\ket{0}$ and $\ket{1}$ space. The role of leakage error in high-
dimensional VQAs, which employ high-dimensional quantum manipulation and
detection techniques, may be substantially different, and it may even
constitute a physical activation function. Thus, the impact of leakage in
different scenarios still needs to be re-characterized.

###### Acknowledgements.

H.-L. H. acknowledges support from the Youth Talent Lifting Project (Grant No.
2020-JCJQ-QT-030), National Natural Science Foundation of China (Grants No.
11905294, 12274464), China Postdoctoral Science Foundation, and the Open
Research Fund from State Key Laboratory of High Performance Computing of China
(Grant No. 201901-01).

## References

  * Huang  _et al._ [2020] H.-L. Huang, D. Wu, D. Fan, and X. Zhu, Sci. China Inf. Sci. 63, 180501 (2020).
  * Arute  _et al._ [2019] F. Arute, K. Arya, R. Babbush, D. Bacon, J. C. Bardin, R. Barends, R. Biswas, S. Boixo, F. G. S. L. Brandao, D. A. Buell, _et al._ , Nature 574, 505 (2019).
  * Zhong  _et al._ [2020] H.-S. Zhong, H. Wang, Y.-H. Deng, M.-C. Chen, L.-C. Peng, Y.-H. Luo, J. Qin, D. Wu, X. Ding, Y. Hu, _et al._ , Science 370, 1460 (2020).
  * Wu  _et al._ [2021] Y. Wu, W.-S. Bao, S. Cao, F. Chen, M.-C. Chen, X. Chen, T.-H. Chung, H. Deng, Y. Du, D. Fan, _et al._ , Phys. Rev. Lett. 127, 180501 (2021).
  * Zhu  _et al._ [2022] Q. Zhu, S. Cao, F. Chen, M.-C. Chen, X. Chen, T.-H. Chung, H. Deng, Y. Du, D. Fan, M. Gong, _et al._ , Sci. Bull. 67, 240 (2022).
  * Guo  _et al._ [2021] C. Guo, Y. Zhao, and H.-L. Huang, Phys. Rev. Lett. 126, 070502 (2021).
  * Fowler  _et al._ [2012] A. G. Fowler, M. Mariantoni, J. M. Martinis, and A. N. Cleland, Phys. Rev. A 86, 032324 (2012).
  * AI [2021] G. Q. AI, Nature 595, 383 (2021).
  * Zhao  _et al._ [2021] Y. Zhao, Y. Ye, H.-L. Huang, Y. Zhang, D. Wu, H. Guan, Q. Zhu, Z. Wei, T. He, S. Cao, _et al._ , arXiv:2112.13505 (2021).
  * Erhard  _et al._ [2021] A. Erhard, H. P. Nautrup, M. Meth, L. Postler, R. Stricker, M. Stadler, V. Negnevitsky, M. Ringbauer, P. Schindler, H. J. Briegel, _et al._ , Nature 589, 220 (2021).
  * Andersen  _et al._ [2020] C. K. Andersen, A. Remm, S. Lazar, S. Krinner, N. Lacroix, G. J. Norris, M. Gabureac, C. Eichler, and A. Wallraff, Nat. Phys. 16, 875 (2020).
  * Marques  _et al._ [2022] J. Marques, B. Varbanov, M. Moreira, H. Ali, N. Muthusubramanian, C. Zachariadis, F. Battistel, M. Beekman, N. Haider, W. Vlothuizen, _et al._ , Nat. Phys. 18, 80 (2022).
  * Krinner  _et al._ [2021] S. Krinner, N. Lacroix, A. Remm, A. Di Paolo, E. Genois, C. Leroux, C. Hellings, S. Lazar, F. Swiadek, J. Herrmann, _et al._ , arXiv:2112.03708 (2021).
  * Geller [2021] M. R. Geller, Phys. Rev. Lett. 127, 090502 (2021).
  * Liu  _et al._ [2019] C. Liu, H.-L. Huang, C. Chen, B.-Y. Wang, X.-L. Wang, T. Yang, L. Li, N.-L. Liu, J. P. Dowling, T. Byrnes, _et al._ , Optica 6, 264 (2019).
  * Huang  _et al._ [2021a] H.-L. Huang, M. Narożniak, F. Liang, Y. Zhao, A. D. Castellano, M. Gong, Y. Wu, S. Wang, J. Lin, Y. Xu, _et al._ , Phys. Rev. Lett. 126, 090502 (2021a).
  * Nachman  _et al._ [2020] B. Nachman, M. Urbanek, W. A. de Jong, and C. W. Bauer, Npj Quantum Inf. 6, 84 (2020).
  * Bravyi  _et al._ [2021] S. Bravyi, S. Sheldon, A. Kandala, D. C. Mckay, and J. M. Gambetta, Phys. Rev. A 103, 042605 (2021).
  * Nation  _et al._ [2021] P. Nation, H. Kang, N. Sundaresan, and J. Gambetta, PRX Quantum 2 (2021).
  * Smith  _et al._ [2021] A. Smith, K. Khosla, C. Self, and M. Kim, Sci. Adv. 7 (2021).
  * Ding  _et al._ [2021] C. Ding, Y.-F. Niu, S. Zhang, W.-S. Bao, and H.-L. Huang, arXiv:2109.06805 (2021).
  * Peruzzo  _et al._ [2014] A. Peruzzo, J. McClean, P. Shadbolt, M.-H. Yung, X.-Q. Zhou, P. J. Love, A. Aspuru-Guzik, and J. L. O’Brien, Nat. Commun. 5, 4213 (2014).
  * Huang  _et al._ [2021b] H.-L. Huang, Y. Du, M. Gong, Y. Zhao, Y. Wu, C. Wang, S. Li, F. Liang, J. Lin, Y. Xu, _et al._ , Phys. Rev. Applied 16, 024051 (2021b).
  * Liu  _et al._ [2021] J. Liu, K. H. Lim, K. L. Wood, W. Huang, C. Guo, and H.-L. Huang, Sci. China Phys. Mech. Astron. 64, 290311 (2021).
  * Schuld and Killoran [2019] M. Schuld and N. Killoran, Phys. Rev. Lett. 122, 040504 (2019).
  * Biamonte  _et al._ [2017] J. Biamonte, P. Wittek, N. Pancotti, P. Rebentrost, N. Wiebe, and S. Lloyd, Nature 549, 195 (2017).
  * Havlíček  _et al._ [2019] V. Havlíček, A. D. Córcoles, K. Temme, A. W. Harrow, A. Kandala, J. M. Chow, and J. M. Gambetta, Nature 567, 209 (2019).
  * Saggio  _et al._ [2021] V. Saggio, B. E. Asenbeck, A. Hamann, T. Strömberg, P. Schiansky, V. Dunjko, N. Friis, N. C. Harris, M. Hochberg, D. Englund, _et al._ , Nature 591, 229 (2021).
  * Cerezo  _et al._ [2021] M. Cerezo, A. Arrasmith, R. Babbush, S. C. Benjamin, S. Endo, K. Fujii, J. R. McClean, K. Mitarai, X. Yuan, L. Cincio, _et al._ , Nat. Rev. Phys. 3, 625 (2021).
  * Zhou  _et al._ [2020] L. Zhou, S.-T. Wang, S. Choi, H. Pichler, and M. D. Lukin, Phys. Rev. X 10, 021067 (2020).
  * McClean  _et al._ [2016] J. R. McClean, J. Romero, R. Babbush, and A. Aspuru-Guzik, New J. Phys. 18, 023023 (2016).
  * Gentini  _et al._ [2020] L. Gentini, A. Cuccoli, S. Pirandola, P. Verrucchi, and L. Banchi, Phys. Rev. A 102, 052414 (2020).
  * Sharma  _et al._ [2020] K. Sharma, S. Khatri, M. Cerezo, and P. J. Coles, New J. Phys. 22, 043006 (2020).
  * Cong  _et al._ [2019] I. Cong, S. Choi, and M. D. Lukin, Nat. Phys. 15, 1273 (2019).
  * Lloyd and Weedbrook [2018] S. Lloyd and C. Weedbrook, Phys. Rev. Lett. 121, 040502 (2018).
  * Harrigan  _et al._ [2021] M. P. Harrigan, K. J. Sung, M. Neeley, K. J. Satzinger, F. Arute, K. Arya, J. Atalaya, J. C. Bardin, R. Barends, S. Boixo, _et al._ , Nat. Phys. 17, 332 (2021).
  * Kandala  _et al._ [2017] A. Kandala, A. Mezzacapo, K. Temme, M. Takita, M. Brink, J. M. Chow, and J. M. Gambetta, Nature 549, 242 (2017).
  * Quantum  _et al._ [2020] G. A. Quantum, Collaborators, F. Arute, K. Arya, R. Babbush, D. Bacon, J. C. Bardin, R. Barends, S. Boixo, M. Broughton, B. B. Buckley, _et al._ , Science 369, 1084 (2020).
  * Romero  _et al._ [2018] J. Romero, R. Babbush, J. R. McClean, C. Hempel, P. J. Love, and A. Aspuru-Guzik, Quantum Sci. Technol. 4, 014008 (2018).
  * Cade  _et al._ [2020] C. Cade, L. Mineh, A. Montanaro, and S. Stanisic, Phys. Rev. B 102, 235122 (2020).
  * Hempel  _et al._ [2018] C. Hempel, C. Maier, J. Romero, J. McClean, T. Monz, H. Shen, P. Jurcevic, B. P. Lanyon, P. Love, R. Babbush, _et al._ , Phys. Rev. X 8, 031022 (2018).
  * Nam  _et al._ [2020] Y. Nam, J.-S. Chen, N. C. Pisenti, K. Wright, C. Delaney, D. Maslov, K. R. Brown, S. Allen, J. M. Amini, J. Apisdorf, _et al._ , npj Quantum Inf. 6, 33 (2020).
  * Saib  _et al._ [2021] W. Saib, P. Wallden, and I. Akhalwaya, arXiv:2108.12388 (2021).
  * Wright  _et al._ [2021] J. Wright, M. Gowrishankar, D. Claudino, P. C. Lotshaw, T. Nguyen, A. J. McCaskey, and T. S. Humble, arXiv:2112.15540 (2021).
  * Gowrishankar  _et al._ [2021] M. Gowrishankar, J. Wright, D. Claudino, T. Nguyen, A. McCaskey, and T. S. Humble, in  _IEEE Int. Conf. Quantum Comput. Engr. (QCE)_ (2021) pp. 155–159.
  * Wang  _et al._ [2021] S. Wang, E. Fontana, M. Cerezo, K. Sharma, A. Sone, L. Cincio, and P. Coles, Nat. Commun. 12 (2021).
  * Ito  _et al._ [2021] K. Ito, W. Mizukami, and K. Fujii, arXiv:2106.03390 (2021).
  * Brown  _et al._ [2020] N. C. Brown, A. Cross, and K. R. Brown, in  _IEEE Int. Conf. Quantum Comput. Engr. (QCE)_ (2020) pp. 286–294.
  * Ghosh  _et al._ [2013] J. Ghosh, A. G. Fowler, J. M. Martinis, and M. R. Geller, Phys. Rev. A 88, 062329 (2013).
  * Holmes  _et al._ [2022] Z. Holmes, K. Sharma, M. Cerezo, and P. J. Coles, PRX Quantum 3, 010313 (2022).
  * Tangpanitanon  _et al._ [2020] J. Tangpanitanon, S. Thanasilp, N. Dangniam, M.-A. Lemonde, and D. G. Angelakis, Phys. Rev. Research 2, 043364 (2020).
  * Nakaji and Yamamoto [2021] K. Nakaji and N. Yamamoto, Quantum 5, 434 (2021), arXiv: 2005.12537.
  * Sim  _et al._ [2019] S. Sim, P. D. Johnson, and A. Aspuru-Guzik, Adv. Quantum Technol. 2, 1900070 (2019).
  * Kullback and Leibler [1951] S. Kullback and R. A. Leibler, Ann. Math. Stat. 22, 79 (1951).
  * Gretton  _et al._ [2012] A. Gretton, K. Borgwardt, M. Rasch, B. Scholkopf, and A. Smola, J. Mach. Learn. Res. 13, 723 (2012).
  * Takhanov [2021] R. Takhanov, arXiv:2106.14277 (2021).
  * Zoufal  _et al._ [2019] C. Zoufal, A. Lucchi, and S. Woerner, Npj Quantum Inf. 5, 103 (2019).
  * Campos  _et al._ [2021] E. Campos, A. Nasrallah, and J. Biamonte, Phys. Rev. A 103, 032607 (2021).
  * Schuld  _et al._ [2020] M. Schuld, A. Bocharov, K. M. Svore, and N. Wiebe, Phys. Rev. A 101, 032308 (2020).
  * Schuld  _et al._ [2021] M. Schuld, R. Sweke, and J. J. Meyer, Phys. Rev. A 103, 032430 (2021).
  * Bharti  _et al._ [2022] K. Bharti, A. Cervera-Lierta, T. H. Kyaw, T. Haug, S. Alperin-Lea, A. Anand, M. Degroote, H. Heimonen, J. S. Kottmann, T. Menke, W.-K. Mok, S. Sim, L.-C. Kwek, and A. Aspuru-Guzik, Rev. Mod. Phys. 94, 015004 (2022).
  * Varbanov  _et al._ [2020] B. M. Varbanov, F. Battistel, B. M. Tarasinski, V. P. Ostroukh, T. E. O’Brien, L. DiCarlo, and B. M. Terhal, Npj Quantum Inf. 6, 102 (2020).
  * Wood and Gambetta [2018] C. J. Wood and J. M. Gambetta, Phys. Rev. A 97, 032306 (2018).
  * Anderson [1936] E. Anderson, Ann. Missouri Bot. 23, 457 (1936).
  * FISHER [1936] R. A. FISHER, Ann. Eugen. 7, 179 (1936).
  * Motzoi  _et al._ [2009] F. Motzoi, J. M. Gambetta, P. Rebentrost, and F. K. Wilhelm, Phys. Rev. Lett. 103, 110501 (2009).
  * Chen  _et al._ [2016] Z. Chen, J. Kelly, C. Quintana, R. Barends, B. Campbell, Y. Chen, B. Chiaro, A. Dunsworth, A. G. Fowler, E. Lucero, E. Jeffrey, A. Megrant, J. Mutus, M. Neeley, C. Neill, P. J. J. O’Malley, P. Roushan, D. Sank, A. Vainsencher, J. Wenner, T. C. White, A. N. Korotkov, and J. M. Martinis, Phys. Rev. Lett. 116, 020501 (2016).


# RUBERT: A Bilingual Roman Urdu BERT Using Cross Lingual Transfer Learning

Usama Khalid
AIM Lab, NUCES (FAST)
Islamabad, Pakistan
<EMAIL_ADDRESS>
Mirza Omer Beg
AIM Lab, NUCES (FAST)
Islamabad, Pakistan
<EMAIL_ADDRESS>
Muhammad Umair Arshad
AIM Lab, NUCES (FAST)
Islamabad, Pakistan
<EMAIL_ADDRESS>

###### Abstract

In recent studies it has been shown that Multilingual language models under
perform their monolingual counterparts (Conneau et al., 2020). It is also a
well known fact that training and maintaining monolingual models for each
language is a costly and time consuming process. Roman Urdu is a resource
starved language used popularly on social media platforms and chat apps. In
this research we propose a novel dataset of scraped tweets containing 54M
tokens and 3M sentences. Additionally we also propose RUBERT a bilingual Roman
Urdu model created by additional pretraining of English BERT (Devlin et al.,
2019). We compare its performance with a monolingual Roman Urdu BERT trained
from scratch and a multilingual Roman Urdu BERT created by additional
pretraining of Multilingual BERT (mBERT (Devlin et al., 2019)). We show
through our experiments that additional pretraining of the English BERT
produces the most notable performance improvement.

Figure 1: An abstract representation of a cross lingual transfer of a Roman
Urdu sentence into the existing space of English learned representations. Note
how the existing words are clustered as Nouns(red), Verb(blue) and
objects(green) and the words in the Roman Urdu sentence get mapped to those
existing spaces.

##  1 Introduction

Roman Urdu is a widely popular language used on many social media platforms
and chat apps in South Asian countries like Pakistan and India. However it is
a resource starved language, this means that there is not a single corpus,
tools or techniques to create Large Pretrained Language models and enable out-
of-the-box NLP tasks on Roman Urdu.

Corpus | Reference | Sentence Count | Word Count
---|---|---|---
Urdu NER | (Khana et al., 2016) | 1,738 | 49,021
COUNTER | (Sharjeel et al., 2017) | 3,587 | 105,124
Urdu Fake News | (Amjad et al., 2020) | 5,722 | 312,427
Urdu IMDB Reviews | (Azam et al., 2020) | 608,693 | 14,474,912
Roman Urdu sentences | (Sharf and Rahman, 2018) | 20,040 | 267,779
Roman Urdu Twitter | Proposed | 3,040,153 | 54,622,490
Table 1:  Statistics of the collected Urdu and Roman Urdu corpora. The Urdu
corpora have all been cleaned and transliterated to Roman Urdu. In addition to
this a novel corpus for Roman Urdu has also been proposed.

Pretraining a monolingual Language model from scratch requires thousands of
GBs of data and tens or hundreds of GPU or TPU hours (Yang et al., 2019; Liu
et al., 2019; Farooq et al., 2019; Zafar et al., 2019a, 2018; Thaver and Beg,
2016). While multilingual pretraining can generally improve the monolingual
performance of a model, it is shown that as the number of languages learnt by
a multilingual model increases, the capacity available for each language
decreases (Conneau and Lample, 2019). This phenomenon is also termed as the
curse of multilinguality. It states that initially while increasing the number
of languages contained in a multilingual model leads to better cross-lingual
performance, however there comes a stage when overall performance starts to
degrade for both monolingual and cross-lingual tasks. Recently a lot of work
has been done on multilingual language modeling and it has been shown that
monolingual models tend to outperform their multilingual counterparts in
similar settings (Martin et al., 2020; Virtanen et al., 2019; Pyysalo et al.,
2020; Beg, 2007; Koleilat et al., 2006; Beg, 2006; Baig et al., ). Therefore
in this research we take a bilingual language modeling approach for the
resource starved language Roman Urdu.

Neural Language modeling was first proposed by (Bengio et al., 2003; Collobert
and Weston, 2008; Khawaja et al., 2018; Beg, 2008; Zafar et al., 2019b) who
showed that training neural networks for predicting the next word given the
current, implicitly learnt useful representations, the technique now popularly
known as word embeddings. These embeddings were a leap forward in the field of
NLP by a considerable margin notably after the introduction of techniques like
word2vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014), fastText
(Joulin et al., 2017).

These early techniques (Khawaja et al., 2018) were mostly context-free and a
major shortcoming was that they couldn’t handle Polysemy (one word many
meanings), like the word bank in river bank and bank account. Thus started the
search for contextual embeddings. ELMo (Peters et al., 2018) and ULMFiT
(Howard and Ruder, 2018) were the first to achieve substantial improvements
using LSTM based language models.

Following the line of work of contextual models (Beg, 2009), a different
approach, GPT (Radford et al., 2018) was proposed to improve the performance
on tasks in the GLUE (Wang et al., 2018) benchmark. The new approach to
contextual language modelling was to replace LSTMs entirely with then recently
released Transformers (Vaswani et al., 2017). The hugely popular BERT (Devlin
et al., 2019) was based on a similar strategy as GPT containing 12 levels but
only used the encoder part of the Transformer and looked at sentences
bidirectionally.

Figure 2: The Transformer based BERT base architecture with twelve encoder
blocks.

Although large pre-trained Language models (Javed et al., 2019; Beg and
Dahlin, ) were a huge success, state-of-the-art models were only available for
English. A lot of work is being done to reproduce this for other languages but
this task is many a times difficult if not impossible for Low resource (Seth
and Beg, 2006) and resource starved languages like Roman Urdu (Zafar et al.,
2020). Multilingual Language modeling aims to solve this by pretraining on a
large number of languages (Naeem et al., 2020) to achieve good
generalizability (Rani et al., 2015) so NLP tasks learned for one language
could seamlessly be transferred to multiple languages and also enable zero-
shot cross-lingual transfer for new languages. The authors of the original
BERT contributed a multilingual version mBERT trained on the Wikipedia of 104
languages (Beg et al., 2019). This line of work was continued by XLM (Conneau
and Lample, 2019) which achieved state-of-the-art results in cross-lingual
classification. This was followed by XLM-R (Conneau et al., 2020) who trained
on 2.5TB of text data and outperformed on various multilingual benchmarks
(Javed et al., 2020b).

##  2 Dataset

This section explains the data gathering (Sahar et al., 2019) process and the
preprocessing steps applied to transform the data in a suitable form to enable
the pretraining cycles of BERT and multlingual BERT (Qamar et al., ).

###  2.1 Collection

Little or no work has been done for Roman Urdu and almost no large publicly
available dataset exists for Roman Urdu. Most of our datasets (Beg and Beek,
2013) are transliterated from Urdu (Nacem et al., 2020) using ijunoon’s Urdu
to Roman Urdu transliteration api
111https://www.ijunoon.com/transliteration/urdu-to-roman/. There were also
datasets taken from research works on Roman Urdu (Sharf and Rahman, 2018;
Arshad et al., 2019; Majeed et al., 2020; Zahid et al., 2020). In addition to
the transliterated datasets we propose a novel Roman Urdu dataset consisting
of 3 million Roman Urdu sentences and 54 million tokens. The dataset has been
scraped from twitter where tweets are partially or wholly written in Roman
Urdu. The statistics of the Twitter scraped dataset and all other datsets
(Uzair et al., 2019) is shown in Table 1. All the above mentioned datasets
have also been made publicly available to enable future research for Roman
Urdu 222Datasets link.

###  2.2 Preprocessing

The datasets from various sources have to be standardized and cleaned to
enable them to be passed to pretraining data creation pipeline. The
preprocessing pipeline consists of two steps. In the first step data is
segmented into sentences using end-of-line character. In the second step all
characters except alphabets and numbers are removed, for data transliterated
from Urdu this step also includes characters of Urdu script.

##  3 Methodology

This section explains the steps to transform data for pretraining and defines
the architecture and pretraining methods used to enable cross-lingual transfer
from English to Roman Urdu and create a Bilingual model from a Monolingual
English model.

###  3.1 Architecture

The BERT base architecture consists of 12 layers, 768 hidden nodes, 12
attention heads and 110M parameters (Alvi et al., 2017). An overview of this
architecture is given in 2. For all our pretraining tasks we use the same BERT
base architecture in conjunction with the newly released uncased BERT models
(Bangash et al., 2017). These Uncased models (Karsten et al., 2007) typically
have better performance, but cased versions are useful when case information
is useful in a task such as Part-Of-Speech tagging (Beg et al., 2006) or
Named-Entity-Recognition.

###  3.2 Data generation

The process of data generation (Dilawar et al., 2018) for all pretraining
cycles (Awan and Beg, 2021) is the same. The data generation process involves
two stages. The first stage involves generating a fixed size vocabulary file
(Tariq et al., 2019), we used the same size file as in English BERT model
which is of 30,522 tokens. This was done to enable additional pretraining. In
essence the size of the vocabulary of the BERT pretrained model was not
changed but the words themselves were. Vocabulary generation or tokenization
is performed using the BertWordPieceTokenizer as specified in the official
repository 333https://github.com/google-research/bert. The tokenizer itself is
provided by the HuggingFace team 444https://github.com/huggingface/tokenizers.

The second stage in data generation (Asad et al., 2020) involves creating the
actual data that will be used in the pretraining task. This requires the
vocabulary file as all the tokens from the corpus will be removed except the
ones contained in the vocabulary file. For this step we use the
create_pretraining_data.py script provided by github BERT repository 3. For
larger datasets the pretraining data can be generated in multiple pieces by
dividing the original dataset into multiple parts and creating pretraining
data for each part.

###  3.3 Pretraining

The Multilingual BERT (Javed et al., 2020a) has currently been trained on 104
Languages (Beg and Van Beek, 2010) of which Roman Urdu is not a part of. Thus
for the training phase we take a new approach called additional pretraining
where we run some steps of the pretraining phase for the new language by
replacing the old vocabulary entirely by the vocabulary of the new language.
Roman Urdu uses the same Latin script as English and many other Languages in
addition to this Roman Urdu also consists of many English words such that
these are often used interchangeably in a code switched mode.

This opens up exciting opportunities when building a Roman Urdu Language model
from monolingual English BERT. An abstract representation of this idea is also
shown in Figure 1. Consider the sentence Abbas school main parhata hai (Abbas
teaches in a school). In this sentence the English BERT knows the context of
school thus it is able to produce better word representations for surrounding
words as compared to the case where not a single word’s context is known.

This notion of cross-lingual transfer can also be applied to Mutilingual
models however the learning space is shared with many other languages and the
curse of multilinguality comes into play. These hypothesis are confirmed by
the three types of pretraining Experiments we perform. The first type involves
training from scratch. The second type involves additional pretraining of a
Monolingual English model and the third type involves additional pretraining
of a multilingual model. The results of these experiments are discussed in
detail in Section 4.

##  4 Experiments

In this section we describe the methods used for evaluation and discuss the
results obtained for the three different types of pretraining.

Model | MLM | NSP
---|---|---
RUBERTMonolingual | 0.02 | 0.53
RUBERTMultilingual | 0.11 | 0.91
RUBERTBilingual | 0.23 | 0.95
BERTEnglish | 0.98 | 1.0
Table 2:  A Comparison between the MLM and NSP accuracies of the three types
Roman Urdu model trained and the English BERT model. The Bilingual version of
RUBERT achieves the best performance but it is still a long way off from the
original English BERT.

###  4.1 Evaluation

The original BERT model was evaluated mainly on four popular tasks which are
GLUE (Wang et al., 2019), SQuAD v1.1 (Rajpurkar et al., 2016), SQuAD v2.0
(Rajpurkar et al., 2018) and the SWAG (Zellers et al., 2018) task. However no
well know multilingual evaluation task provides support for Roman Urdu. Thus
for measuring the performance of our pretrained models we use the evaluation
metrics produced in the pretraining phase of each model. The evaluation
metrics are based on data that is put aside during creation of the pretraining
data.

###  4.2 Results

The evaluation of all pretraining cycles is performed for two tasks Masked
Language Modeling (MLM) and Next Sentence Prediction (NSP) which are the same
tasks BERT is trained for. Table 2 shows the accuracies of MLM and NSP tasks
for the models we trained and also for the Pretrained English BERT model for
comparison. The BERT models we have trained using Roman Urdu data are termed
as RUBERT. All three pretraining cycles are carried out only using the
gathered data mentioned in Section 2.1.

The monolingual version of RUBERT is pretrained from scratch while the
Multilingual RUBERT is trained by performing additional pretraining steps on a
pretrained Multilingual BERT. The Bilingual BERT which achieves the best
performance is trained by performing the same additional pretraining steps as
before but on a monolingual English BERT Base model. Each of the pretraining
and additional pretraining cycles are performed for 10K steps with a learning
rate of $2e-5$ except for Monolingual RUBERT which was trained with a learning
rate of $1e-4$, the higher learning rate was necessary when training from
scratch.

##  5 Conclusion

In this research we propose a novel corpus and some transliterated corpora for
Roman Urdu a resource starved language. The novel corpus contains sentences
extracted from scraped tweets. In addition to this we introduce additional
pretraining a new method for cross-lingual transfer of learned representations
to a new low resource language. In this research we have shown that Bilingual
language modeling is able to outperform multilingual modeling. There are some
conditions however to enable cross-lingual transfer which involves the target
language to share vocabulary and linguistic features like typography with the
source language. We hope that the methods and corpora proposed in this paper
will enable further research for resource starved Languages like Roman Urdu.

## References

  * Alvi et al. (2017) Hamza M Alvi, Hareem Sahar, Abdul A Bangash, and Mirza O Beg. 2017.  Ensights: A tool for energy aware software development.  In _2017 13th International Conference on Emerging Technologies (ICET)_ , pages 1–6. IEEE.
  * Amjad et al. (2020) Maaz Amjad, Grigori Sidorov, Alisa Zhila, Helena Gómez-Adorno, Ilia Voronkov, and Alexander Gelbukh. 2020.  “bend the truth”: Benchmark dataset for fake news detection in urdu language and its evaluation.  _Journal of Intelligent & Fuzzy Systems_, 51(Preprint):1–13.
  * Arshad et al. (2019) Muhammad Umair Arshad, Muhammad Farrukh Bashir, Adil Majeed, Waseem Shahzad, and Mirza Omer Beg. 2019.  Corpus for emotion detection on roman urdu.  In _2019 22nd International Multitopic Conference (INMIC)_ , pages 1–6. IEEE.
  * Asad et al. (2020) Muhammad Asad, Muhammad Asim, Talha Javed, Mirza O Beg, Hasan Mujtaba, and Sohail Abbas. 2020.  Deepdetect: detection of distributed denial of service attacks using deep learning.  _The Computer Journal_ , 63(7):983–994.
  * Awan and Beg (2021) Mubashar Nazar Awan and Mirza Omer Beg. 2021.  Top-rank: a topicalpostionrank for extraction and classification of keyphrases in text.  _Computer Speech & Language_, 65:101116.
  * Azam et al. (2020) Nazish Azam, Bilal Tahir, and Muhammad Amir Mehmood. 2020.  Sentiment and emotion analysis of text: A survey on approaches and resources.  _LANGUAGE & TECHNOLOGY_, page 87.
  * (7) Zubair Baig, Mirza Omer Beg, Baber Majid Bhatti, Farzana Ahamed Bhuiyan, Tegawendé F Bissyandé, Shizhan Chen, Mohan Baruwal Chhetri, Marco Couto, João de Macedo, Randy de Vries, et al.  Ahmed, sanam 124 aleti, aldeida 105 aloísio, joão 151 arachchilage, nalin asanka gamagedara 7.
  * Bangash et al. (2017) Abdul Ali Bangash, Hareem Sahar, and Mirza Omer Beg. 2017.  A methodology for relating software structure with energy consumption.  In _2017 IEEE 17th International Working Conference on Source Code Analysis and Manipulation (SCAM)_ , pages 111–120. IEEE.
  * Beg (2008) M Beg. 2008.  Critical path heuristic for automatic parallelization.
  * Beg (2009) Mirza Beg. 2009.  Flecs: A framework for rapidly implementing forwarding protocols.  In _International Conference on Complex Sciences_ , pages 1761–1773. Springer.
  * Beg and Beek (2013) Mirza Beg and Peter van Beek. 2013.  A constraint programming approach for integrated spatial and temporal scheduling for clustered architectures.  _ACM Transactions on Embedded Computing Systems (TECS)_ , 13(1):1–23.
  * Beg et al. (2006) Mirza Beg, Laurent Charlin, and Joel So. 2006.  Maxsm: A multi-heuristic approach to xml schema matching.
  * (13) Mirza Beg and Mike Dahlin.  A memory accounting interface for the java programming language.
  * Beg and Van Beek (2010) Mirza Beg and Peter Van Beek. 2010.  A graph theoretic approach to cache-conscious placement of data for direct mapped caches.  In _Proceedings of the 2010 international symposium on Memory management_ , pages 113–120.
  * Beg et al. (2019) Mirza O Beg, Mubashar Nazar Awan, and Syed Shahzaib Ali. 2019.  Algorithmic machine learning for prediction of stock prices.  In _FinTech as a Disruptive Technology for Financial Institutions_ , pages 142–169. IGI Global.
  * Beg (2006) Mirza Omer Beg. 2006.  Performance analysis of packet forwarding on ixp2400 network processor.
  * Beg (2007) Mirza Omer Beg. 2007.  Flecs: A data-driven framework for rapid protocol prototyping.  Master’s thesis, University of Waterloo.
  * Bengio et al. (2003) Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003.  A neural probabilistic language model.  _Journal of machine learning research_ , 3(Feb):1137–1155.
  * Collobert and Weston (2008) Ronan Collobert and Jason Weston. 2008.  A unified architecture for natural language processing: Deep neural networks with multitask learning.  In _Proceedings of the 25th international conference on Machine learning_ , pages 160–167.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 8440–8451. Association for Computational Linguistics.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  In _Advances in Neural Information Processing Systems_ , pages 7059–7069.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Dilawar et al. (2018) Noman Dilawar, Hammad Majeed, Mirza Omer Beg, Naveed Ejaz, Khan Muhammad, Irfan Mehmood, and Yunyoung Nam. 2018.  Understanding citizen issues through reviews: A step towards data informed planning in smart cities.  _Applied Sciences_ , 8(9):1589.
  * Farooq et al. (2019) Muhammad Umer Farooq, Mirza Omer Beg, et al. 2019.  Bigdata analysis of stack overflow for energy consumption of android framework.  In _2019 International Conference on Innovative Computing (ICIC)_ , pages 1–9. IEEE.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 328–339.
  * Javed et al. (2020a) Abdul Rehman Javed, Mirza Omer Beg, Muhammad Asim, Thar Baker, and Ali Hilal Al-Bayatti. 2020a.  Alphalogger: Detecting motion-based side-channel attack using smartphone keystrokes.  _Journal of Ambient Intelligence and Humanized Computing_ , pages 1–14.
  * Javed et al. (2020b) Abdul Rehman Javed, Muhammad Usman Sarwar, Mirza Omer Beg, Muhammad Asim, Thar Baker, and Hissam Tawfik. 2020b.  A collaborative healthcare framework for shared healthcare plan with ambient intelligence.  _Human-centric Computing and Information Sciences_ , 10(1):1–21.
  * Javed et al. (2019) Hafiz Tayyeb Javed, Mirza Omer Beg, Hasan Mujtaba, Hammad Majeed, and Muhammad Asim. 2019.  Fairness in real-time energy pricing for smart grid using unsupervised learning.  _The Computer Journal_ , 62(3):414–429.
  * Joulin et al. (2017) Armand Joulin, Édouard Grave, Piotr Bojanowski, and Tomáš Mikolov. 2017.  Bag of tricks for efficient text classification.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 427–431.
  * Karsten et al. (2007) Martin Karsten, Srinivasan Keshav, Sanjiva Prasad, and Mirza Beg. 2007.  An axiomatic basis for communication.  _ACM SIGCOMM Computer Communication Review_ , 37(4):217–228.
  * Khana et al. (2016) Wahab Khana, Ali Daudb, Jamal A Nasira, and Tehmina Amjada. 2016.  Named entity dataset for urdu named entity recognition task.  _LANGUAGE & TECHNOLOGY_, page 51.
  * Khawaja et al. (2018) Hussain S Khawaja, Mirza O Beg, and Saira Qamar. 2018.  Domain specific emotion lexicon expansion.  In _2018 14th International Conference on Emerging Technologies (ICET)_ , pages 1–5. IEEE.
  * Koleilat et al. (2006) Walid Koleilat, Joel So, and Mirza Beg. 2006.  Watagent: A fresh look at tac-scm agent design.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Majeed et al. (2020) Adil Majeed, Hasan Mujtaba, and Mirza Omer Beg. 2020.  Emotion detection in roman urdu text using machine learning.  In _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering Workshops_ , pages 125–130.
  * Martin et al. (2020) Louis Martin, Benjamin Müller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020.  Camembert: a tasty french language model.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 7203–7219. Association for Computational Linguistics.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  In _Advances in neural information processing systems (NIPS)_ , pages 3111–3119.
  * Nacem et al. (2020) Saad Nacem, Majid Iqbal, Muhammad Saqib, Muhammad Saad, Muhammad Soban Raza, Zaid Ali, Naveed Akhtar, Mirza Omer Beg, Waseem Shahzad, and Muhhamad Umair Arshad. 2020.  Subspace gaussian mixture model for continuous urdu speech recognition using kaldi.  In _2020 14th International Conference on Open Source Systems and Technologies (ICOSST)_ , pages 1–7. IEEE.
  * Naeem et al. (2020) Bilal Naeem, Aymen Khan, Mirza Omer Beg, and Hasan Mujtaba. 2020.  A deep learning framework for clickbait detection on social area network using natural language cues.  _Journal of Computational Social Science_ , pages 1–13.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of NAACL-HLT_ , pages 2227–2237.
  * Pyysalo et al. (2020) Sampo Pyysalo, Jenna Kanerva, Antti Virtanen, and Filip Ginter. 2020.  Wikibert models: deep transfer learning for many languages.  _CoRR_ , abs/2006.01538.
  * (43) Saira Qamar, Hasan Mujtaba, Hammad Majeed, and Mirza Omer Beg.  Relationship identification between conversational agents using emotion analysis.  _Cognitive Computation_ , pages 1–15.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for squad.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392.
  * Rani et al. (2015) Uzma Rani, Aamer Imdad, and Mirza Beg. 2015.  Case 2: Recurrent anemia in a 10-year-old girl.  _Pediatrics in review_ , 36(12):548–550.
  * Sahar et al. (2019) Hareem Sahar, Abdul A Bangash, and Mirza O Beg. 2019.  Towards energy aware object-oriented development of android applications.  _Sustainable Computing: Informatics and Systems_ , 21:28–46.
  * Seth and Beg (2006) Aaditeshwar Seth and Mirza Beg. 2006.  Achieving privacy and security in radio frequency identification.  In _Proceedings of the 2006 International Conference on Privacy, Security and Trust: Bridge the Gap Between PST Technologies and Business Services_ , pages 1–1.
  * Sharf and Rahman (2018) Zareen Sharf and Saif Ur Rahman. 2018.  Performing natural language processing on roman urdu datasets.  _International Journal Of Computer Science And Network Security_ , 18(1):141–148.
  * Sharjeel et al. (2017) Muhammad Sharjeel, Rao Muhammad Adeel Nawab, and Paul Rayson. 2017.  Counter: corpus of urdu news text reuse.  _Language resources and evaluation_ , 51(3):777–803.
  * Tariq et al. (2019) Muhammad Tariq, Hammad Majeed, Mirza Omer Beg, Farrukh Aslam Khan, and Abdelouahid Derhab. 2019.  Accurate detection of sitting posture activities in a secure iot based assisted living environment.  _Future Generation Computer Systems_ , 92:745–757.
  * Thaver and Beg (2016) Danyal Thaver and Mirza Beg. 2016.  Pulmonary crohn’s disease in down syndrome: A link or linkage problem.  _Case reports in gastroenterology_ , 10(2):206–211.
  * Uzair et al. (2019) Ahmed Uzair, Mirza O Beg, Hasan Mujtaba, and Hammad Majeed. 2019.  Weec: Web energy efficient computing: A machine learning approach.  _Sustainable Computing: Informatics and Systems_ , 22:230–243.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Virtanen et al. (2019) Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019.  Multilingual is not enough: Bert for finnish.  _arXiv_ , pages arXiv–1912.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355.
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In _7th International Conference on Learning Representations, ICLR 2019_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in neural information processing systems_ , pages 5753–5763.
  * Zafar et al. (2019a) Adeel Zafar, Hasan Mujtaba, Sohrab Ashiq, and Mirza Omer Beg. 2019a.  A constructive approach for general video game level generation.  In _2019 11th Computer Science and Electronic Engineering (CEEC)_ , pages 102–107. IEEE.
  * Zafar et al. (2019b) Adeel Zafar, Hasan Mujtaba, Mirza Tauseef Baig, and Mirza Omer Beg. 2019b.  Using patterns as objectives for general video game level generation.  _ICGA Journal_ , 41(2):66–77.
  * Zafar et al. (2020) Adeel Zafar, Hasan Mujtaba, and Mirza Omer Beg. 2020.  Search-based procedural content generation for gvg-lg.  _Applied Soft Computing_ , 86:105909.
  * Zafar et al. (2018) Adeel Zafar, Hasan Mujtaba, Mirza Omer Beg, and Sajid Ali. 2018.  Deceptive level generator.
  * Zahid et al. (2020) Rabail Zahid, Muhammad Owais Idrees, Hasan Mujtaba, and Mirza Omer Beg. 2020.  Roman urdu reviews dataset for aspect based opinion mining.  In _2020 35th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW)_ , pages 138–143. IEEE.
  * Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018.  Swag: A large-scale adversarial dataset for grounded commonsense inference.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 93–104.