# An Artificial Neural Network-Based Model Predictive Control for Three-phase
Flying Capacitor Multi-Level Inverter

Parisa Boodaghi Malidarreh1, Abualkasim Bakeer235, Ihab S. Mohamed45, and
Lantao Liu4

5 Abualkasim Bakeer and Ihab S. Mohamed have contributed equally to this work
as a co-second author. 1Dep. of Power Electronics and Elect. Machines, Iran
University of Science and Technology (IUST), Iran  2Dep. of Elect.
Engineering, Faculty of Engineering, Aswan University, Aswan 81542, Egypt
3Dep. of Elect. Power Eng. and Mechatronics, Tallinn University of Technology,
19086 Tallinn, Estonia  4Luddy School of Informatics, Computing, and
Engineering, Indiana University, Bloomington, IN 47408 USA

###### Abstract

Model predictive control (MPC) has been used widely in power electronics due
to its simple concept, fast dynamic response, and good reference tracking.
However, it suffers from parametric uncertainties, since it directly relies on
the mathematical model of the system to predict the optimal switching states
to be used at the next sampling time. As a result, uncertain parameters lead
to an ill-designed MPC. Thus, this paper offers a model-free control strategy
on the basis of artificial neural networks (ANNs), for mitigating the effects
of parameter mismatching while having a little negative impact on the
inverter’s performance. This method includes two related stages. First, MPC is
used as an expert to control the studied converter in order to provide the
training data; while, in the second stage, the obtained dataset is utilized to
train the proposed ANN which will be used directly to control the inverter
without the requirement for the mathematical model of the system. The case
study herein is based on a four-level three-cell flying capacitor inverter. In
this study, MATLAB/Simulink is used to simulate the performance of the
proposed control strategy, taking into account various operating conditions.
Afterward, the simulation results are reported in comparison with the
conventional MPC scheme, demonstrating the superior performance of the
proposed control strategy in terms of getting low total harmonic distortion
(THD) and the robustness against parameters mismatch, especially when changes
occur in the system parameters.

###### Index Terms:

Model predictive control, Artificial neural network, Multilevel inverter,
Total harmonics distortion.

##  I Introduction

Power converters have been used in a wide range of applications in recent
decades due to their improved performance and efficiency [1]. In this context,
substantial attention has been paid to multilevel inverters (MLIs) that
introduced with certain advantages, such as: (i) reducing the total harmonic
distortion, (ii) reducing common-mode voltage, and (iii) reducing the $dv/dt$
(the rate of voltage change over time in switching instant) stress that leads
to a reduction in electromagnetic emissions. There are a number of well-
established and conventional MLI topologies that have been implemented and
refined over time, such as: flying capacitor (FC), cascaded H-Bridge (CHB),
neutral point clamped (NPC), as well as innovative topologies that combine
different topologies to improve MLIs performance [2, 3]. As a result,
different control methods have been investigated including various linear
methods such as proportional-integral (PI) with pulse width modulation (PWM)
or space vector modulation (SVM) [4, 5], non-linear methods with Hysteresis
[6], and other non-classical (i.e., modern) control methods such as different
techniques of predictive control [7, 6]. However, controlling the MLIs with
the high number of switches brings about dire problems in controlling
algorithms and makes them more complex. Moreover, voltage imbalance among the
MLI capacitors becomes an important concern in designing efficient control
algorithms [8, 9, 10, 11].

Model predictive control, among several control methods described, has
appealing aspects that demonstrate its proper functioning. It provides a fast
transient response, straightforward implementation, and non-linear constraints
consideration [12, 13, 14, 15]. Furthermore, the development of efficient
microprocessors accounts for applying the MPC schemes widely in machine drive,
grid-connected converters, and power supplies [16, 17, 18, 19]. By explicitly
utilizing the model of the system to be controlled, MPC selects, at each time
step, the optimal switching state that minimizes a pre-defined cost function.
However, it has a high online computational burden especially when multiple
time-horizons are considered, as: (i) the cost function should be calculated
for different possible switching states; (ii) the optimization yields the
optimal control sequence (i.e., optimal switching states); (iii) the first
optimal switching state is finally applied to the system in the next sampling
time. Sequentially, it can be quite difficult to employ the MPC to control
MLIs with numerous levels [20, 21, 22, 23, 24, 25]. In addition, as previously
mentioned, MPC directly uses the mathematical model of the system, and thus
the prediction accuracy is sensitive to the change in system parameters. As a
result, it is susceptible to perform incorrectly as a result of parameter
variations that may occur in the real system due to degradation and
temperature effects [26].

To be more specific, some non-modeled variables, such as temperature and
magnetic saturation, have impacts on the real system’s parameters that are so-
called modeling errors or errors in measurements; these effects can
deteriorate the control performance. Therefore, various control techniques
have been proposed for tackling and analyzing the effects of parametric
uncertainties [27, 28]. On the other hand, since MPC possesses a non-linear
structure, these studies were carried out empirically by studying the behavior
of MPC under different uncertainty conditions [28], the obtained results led
to adjust the model parameters such as inductance to mitigate the effects of
the parameter mismatch. Moreover, the negative effects of mismatches in load
resistance were ignored [26, 29]. For instance, authors in [29] examined the
impact of electrical parameters’ variation in six-phase induction motor drive.
It has been shown that the parameter of inductance can affect the performance
of MPC, especially the stator and rotor leakage. While, in [26], the
predictive control-based direct power control with an adaptive parameter
identification technique is proposed to cope with the parameter uncertainty
problem. In this method, the input resistance and inductance of the converter
are estimated using the input current and voltage at each sampling time,
without having extra sensors. Thus, the converter parameters are updated
online to curb the effects of parameter uncertainty.

Furthermore, the use of data-based control approaches, such as ANN-based
control methods, has increased in recent research, particularly in power
electronics and motor drives [30]. Just to name a few, in [31], an ANN-based
MPC control strategy was introduced to improve the performance of a three-
phase inverter with an output LC filter, with comparison to the conventional
MPC scheme. While, in [25], a general ANN-MPC scheme is presented to tackle
the computational challenge in adopting MPC in highly complex power
converters. In [32], the authors proposed a data-based technique for
calculating the voltages of capacitors in order to eliminate the need for
system sensors. While, an ANN-based method is used in [33] to regulate the
weighting factors in the cost function of MPC, which represents one of the
research challenges in Model Predictive Control. Moreover, data-based control
schemes are used in motor drives for estimating rotor speed, rotor flux, and
torque in induction motors [34, 35, 36].

This paper proposes an artificial neural network-based (ANN-based) control
strategy for a three-phase four-level flying capacitor inverter. MPC has been
used as a baseline control scheme for (i) collecting the data required for
training process, considering different operating conditions, and (ii)
assessing our proposed control strategy. The main contributions of the
proposed ANN-based control strategy can be summarized as follows:

  1. 1.

It is an End-to-End (E2E) control scheme that generates directly the optimal
switching states of the inverter, without the need for either the mathematical
model of the inverter and a pre-defined cost function to be minimized at each
time step.

  2. 2.

It reduces the effects of the parameter mismatch problems as it offers a
model-free control strategy based on ANNs.

  3. 3.

The proposed technique exhibits low THD in the output current compared to the
conventional MPC.

  4. 4.

Moreover, the impact of the input features selection on the performance of the
proposed control strategy is studied and evaluated.

  5. 5.

Finally, we provide an open repository of the dataset and simulation files to
the community for further research activities.

This paper is organized as follows. In Section II, the flying capacitor
multilevel inverter (FCMLI) and its mathematical model are briefly described,
whereas the model predictive control strategy is explained in Section III. The
ANN-based MPC control strategy proposed in this work for the FCMLI is
presented in Section IV. In Section V, simulation results for both control
strategies are discussed. Finally, conclusions are given in section VI.

##  II Flying Capacitor Multilevel Inverter

Flying capacitor multilevel inverter (FCMLI) comprises a series of connected
cells which have a flying capacitor and two switches with antiparallel diodes.
These switches are working in a complementary manner to avoid the short-
circuit between the flying capacitor terminals. An $N$-cell FCMLI has $2N$
switches in each leg, and it produces $(N+1)$ voltage levels from
$\frac{-V_{dc}}{2}$ to $\frac{V_{dc}}{2}$, where $V_{dc}$ denotes the input
voltage source. The average voltage of each capacitor should be kept fixed at
multiples of $\frac{V_{dc}}{N}$. The nearest capacitor to the load has
$\frac{V_{dc}}{N}$, whilst the capacitor that is connected to the voltage
source has $V_{dc}$; the capacitor voltage ratio should be set as
$V_{1x}:V_{2x}:V_{3x}=1:2:3$ with $x\in\\{a,b,c\\}$ for generating the four-
level voltage waveform. Fig. 1 shows a three-phase four-level flying capacitor
(FC) inverter with three cells. Each capacitor’s voltage can be estimated
using the inverter current $i_{x}$ and switching states in the corresponding
leg, as given in (1) and (2). In addition, for a resistive-inductive load
(i.e., $RL$-load), the three-phase output voltage can be written as in (3),
where $R$ and $L$ are the load resistance and inductance, respectively. Table
I displays the eight possible switching states of a single-phase FCMLI and the
corresponding output voltage.

Fig. 1: Topology of a three-phase four-level flying capacitor inverter that is
directly connected to an $RL$-load.

$V_{1x}=V_{1x}(0)+\frac{1}{C_{1x}}\int_{0}^{t}i_{x}(S_{2x}-S_{1x})dt$ (1)

$V_{2x}=V_{2x}(0)+\frac{1}{C_{2x}}\int_{0}^{t}i_{x}(S_{3x}-S_{2x})dt$ (2)

$V_{xN}=Ri_{x}+L\frac{di_{x}}{d_{t}}+\frac{1}{3}(V_{aN}+V_{bN}+V_{cN})$ (3)

Table I: Switching states for single-phase FC converter $V_{i}$ | $S_{1x}$ | $S_{2x}$ | $S_{3x}$ | Output voltage $V_{xN}$ | Equation
---|---|---|---|---|---
$V_{0}$ | 0 | 0 | 0 | $-V_{dc}/2$ | $-V_{dc}/2$
$V_{1}$ | 1 | 0 | 0 | $-V_{dc}/6$ | $V_{1x}-V_{dc}/2$
$V_{2}$ | 0 | 1 | 0 | $-V_{dc}/6$ | $V_{2x}-V_{1x}-V_{dc}/2$
$V_{3}$ | 1 | 1 | 0 | $V_{dc}/6$ | $V_{2x}-V_{dc}/2$
$V_{4}$ | 0 | 0 | 1 | $-V_{dc}/6$ | $V_{dc}/2-V_{2x}$
$V_{5}$ | 1 | 0 | 1 | $V_{dc}/6$ | $V_{dc}/2-V_{2x}+V_{1x}$
$V_{6}$ | 0 | 1 | 1 | $V_{dc}/6$ | $V_{dc}/2-V_{1x}$
$V_{7}$ | 1 | 1 | 1 | $V_{dc}/2$ | $V_{dc}/2$

##  III Model Predictive Control for FCMLI

Model predictive control (MPC) is one of the well-established and promising
model-based control methods that is successfully used in power electronics.
The key idea behind (MPC) is the use of the system model for predicting the
future behavior of the variables to be controlled, considering a certain time
horizon. To do so, it needs, in our case, the discrete model of the converter.
The optimal control signal, i.e., switching states as described in Table I,
that minimizes the cost function is then determined. The entire MPC procedure
for FCMLI is shown in Fig. 2, which can be described step-by-step as follows:

  1. 1.

At sampling instant $k$, the controlled variables (namely,
$V^{k}_{1x},V^{k}_{2x},i^{k}_{x}$) should be measured.

  2. 2.

Those controlled variables are then predicted at instant $k+1$ based on the
discrete model of the converter given in (4) and (5), where
$M_{1}=\frac{T_{s}}{L}$, and $M_{2}=1-\frac{RT_{s}}{L}$; while
$V^{k+1}_{1x},V^{k+1}_{2x},i^{k+1}_{x}$ are the flying capacitor voltages and
output current at instant $k+1$.

$\displaystyle
V^{k+1}_{1x}=V^{k}_{1x}+\frac{T_{s}}{C_{1x}}i^{k}_{x}(S^{i}_{2x}-S^{i}_{1x})$
(4) $\displaystyle
V^{k+1}_{2x}=V^{k}_{2x}+\frac{T_{s}}{C_{2x}}i^{k}_{x}(S^{i}_{3x}-S^{i}_{2x})$

$i^{k+1}_{x}=(V_{xN}-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{ON}})M_{1}+i^{k}_{x}M_{2}$
(5)

where $V_{ON}$ is the common mode voltage.

  3. 3.

After defining a proper cost function $J_{i}$, as in (6), it should be
calculated for the current switching states $S_{i}$ based on the desired value
of each controlled variable, namely, $V^{*}_{1x},V^{*}_{2x},i^{*}_{x}$.

$J_{i}=\lambda_{1}(i^{*}_{x}-i^{k+1}_{x})^{2}+\lambda_{2}(V^{*}_{1x}-V^{k+1}_{1x})^{2}+\lambda_{2}(V^{*}_{2x}-V^{k+1}_{2x})^{2}$
(6)

  4. 4.

As the main objective of the optimization problem is to find the optimum
switching state ($S_{opt}$) that minimizes the cost function, the cost
function of the current switching state $J_{i}$ is compared with the smallest
previous value $e$.

  5. 5.

Steps 2) to 4) are repeated for all possible switching states given in Table
I.

  6. 6.

Finally, the optimum switching state ($S_{opt}$) is applied at the next
sampling instant.

In Fig. 2, $N$ refers to the total number of the possible switching states,
which is equal to 8 in this study.

Start$i=1$ Measured variables at instant
$k$$V^{k}_{1x},V^{k}_{2x},i^{k}_{x}$$i\leq N$Predict variables at instant
$k+1$$V^{k+1}_{1x}=V^{k}_{1x}+\frac{T_{s}}{C_{1x}}i^{k}_{x}(S^{i}_{2x}-S^{i}_{1x})$
$V^{k+1}_{2x}=V^{k}_{2x}+\frac{T_{s}}{C_{x}}i^{k}_{x}(S^{i}_{3x}-S^{i}_{2x})$$i^{k+1}_{x}=(V_{xN}-V_{ON})M_{1}+i^{k}_{x}M_{2}$Calculate
cost function at $k+1$
$J_{i}=\lambda_{1}(i^{*}_{x}-i^{k+1}_{x})^{2}+$$\lambda_{2}(V^{*}_{1x}-V^{k+1}_{1x})^{2}+\lambda_{2}(V^{*}_{2x}-V^{k+1}_{2x})^{2}$
$J_{i}\leq e$$e=J_{i},S_{opt}=S_{i}$$i^{++}$Return $S_{opt}$TRUEFALSETRUEFALSE
Fig. 2: Flowchart diagram of MPC scheme for FCMLI.

One of the most critical stages of MPC, as previously stated, is determining
the switching state ($S_{opt}$) that minimizes a given cost function.
Different forms of cost functions have been established, in this regard, which
are used according to the requirements from the MPC design. In this work, the
cost function is expressed as (6), where $i^{*}_{x}$ and $i^{k+1}_{x}$ are the
reference and estimated values of the output current at $k+1$, respectively.
Similarly, $V^{k+1}_{1x},V^{k+1}_{2x}$ and $V^{*}_{1x},V^{*}_{2x}$ refer to
the first and second capacitor voltages and their desired values at $k+1$,
respectively. $\lambda_{1}$ and $\lambda_{2}$ are weighting factors of the
cost function that determines the importance of the related term in the cost
function. For instance, a higher weighting factor indicates the importance of
the variable in determining the optimal switching state, thus it shows the
priority of that variable in determining the optimal switching state. As
previously described, the cost function must be evaluated for all switching
states at each sampling time, which can be difficult in systems with many
possible switching states such as multilevel inverters; thereby, it needs a
strong microprocessor to deal with this high computational burden. In fact,
the major drawback of MPC is its optimization procedure that should be solved
online, which imposes a large amount of real-time calculation. This is an
important motivation for proposing an End-to-End (E2E) control strategy, such
as ANN-Based control strategy, that generates directly the optimal switching
states of the inverter, without the need for either: (i) the mathematical
model of the inverter and (ii) a pre-defined cost function to be minimized
[31].

##  IV Proposed ANN-Based Control Strategy

###  IV-A Overview of ANN

Artificial neural network (ANN) is a subset of machine learning that can be
directly defined as a mathematical relationship between input features and
targets; in other words, it is a mathematical model that uses learning
algorithms to make intelligent decisions based on historical data. It consists
of a set of nodes, so-called neurons, that form layers that are connected and
executed in parallel.

$\vdots$$\vdots$$\vdots$$x_{1}$$x_{2}$$x_{3}$$x_{M}$$H_{1}$$b_{11}$$H_{J}$$b_{1J}$$y_{1}$$b_{21}$$y_{N}$$b_{2N}$$w_{11}$$w_{1J}$$w_{M1}$$w_{MJ}$$w_{11}$$w_{1N}$$w_{J1}$$w_{JN}$Input
LayerHidden LayerOutput Layer

Figure 3: A typical architecture of a two-layer ANN with $M$ inputs, one
hidden layer containing $J$ neurons, and $N$ outputs.

Figure 3 shows the typical architecture of a multi-input multi-output ANN that
has $M$ input features (i.e., $x_{m}\;\forall m=\\{1,\dots,M\\}$) and yields
$N$ output targets (namely, $y_{n}\;\forall n=\\{1,\dots,N\\}$), where the
number of neurons in both input and output layers should be equivalent to the
number of data to be processed and the number of targets, respectively. Each
input feature $x_{m}$ is multiplied by weighting factors $w_{mj}\;\forall
j=\\{1,\dots,J\\}$ and applied to the hidden layer. In the hidden layer, the
bias $b_{1j}$ is added to the sum of the weighted values and an activation
function is applied to it in each neuron; then, the results are multiplied by
another weighting factors $w_{jm}$ before being passed to the output layer.
Those correction factors (i.e., biases) allow us to shift the activation
functions which improves the model’s fit to the given dataset. The
mathematical formulation of the $n^{\text{th}}$ output can be described as:

$\displaystyle y_{n}$
$\displaystyle=h\left(\sum_{j=1}^{J}w_{jn}H_{j}+b_{2n}\right),\quad\forall
n=\\{1,\dots,N\\},$ (7) $\displaystyle H_{j}$
$\displaystyle=h\left(\sum_{m=1}^{M}w_{mj}x_{m}+b_{1j}\right),\quad\forall
j=\\{1,\dots,J\\},$

where $h$ is the activation function, $H_{j}$ are the outputs of the hidden
layer, and $b_{2n}$ refer to the biases of the output layer. Activation
functions $h$ (such as sigmoid, hyperbolic tangent, softmax, and linear) are
the key part of the neural network design that are used to learn complicated
patterns and assess how well the network model learns the training dataset
[37]. Moreover, there exist different learning methods; in this study, the
feed-forward method is used. Unlike recurrent neural networks, connections
between different nodes in various layers do not produce a cycle in the feed-
forward learning approach.

Table II: THD of the output current $i_{x}$ for different sets of input features, where the lowest THD is highlighted in blue. Sample No. | $S_{1}$ | $S_{2}$ | $S_{3}$ | $S_{4}$ | $S_{5}$ | $S_{6}$ | $S_{7}$ | $S_{8}$ | $S_{9}$ | $S_{10}$ | $S_{11}$ | $S_{12}$ | $S_{13}$ | $S_{14}$ | $S_{15}$ | $S_{16}$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$V_{dc}$\text{\,}\;\mathrm{[}\mathrm{V}\mathrm{]}$$ | 360 | 360 | 360 | 342 | 378 | 360 | 360 | 350 | 340 | 360 | 360 | 340 | 350 | 360 | 350 | 350
$T_{s}$\text{\,}\;\mathrm{[}\mathrm{\SIUnitSymbolMicro}\mathrm{s}\mathrm{]}$$ | 30 | 30 | 30 | 20 | 20 | 45 | 45 | 50 | 50 | 50 | 20 | 25 | 20 | 20 | 25 | 40
$R$\text{\,}\;\mathrm{[}\mathrm{\SIUnitSymbolOhm}\mathrm{]}$$ | 10 | 15 | 25 | 7.5 | 15 | 15 | 8 | 9 | 11 | 10 | 10 | 10 | 12 | 7 | 9 | 9
$L$\text{\,}\;\mathrm{[}\mathrm{m}\mathrm{H}\mathrm{]}$$ | 5 | 10 | 12 | 8 | 4.5 | 9 | 10 | 9.5 | 5 | 5.5 | 5 | 5 | 7 | 5 | 5 | 5
$i_{ref}$\text{\,}\;\mathrm{[}\mathrm{A}\mathrm{]}$$ | 17 | 12 | 5 | 12 | 10 | 4 | 5 | 6 | 6 | 4.35 | 12 | 10 | 8 | 15 | 12 | 12
$\mathbf{X_{1}}=\\{V_{c1},V_{c2},i_{ref},i_{x},\Delta V_{c1},\Delta
V_{c2},\Delta i_{x},S_{opt},V_{ph}\\}$
THD [$\%$] | 1.89 | 1.29 | 3.09 | 1.30 | 2.02 | 4.54 | 3.51 | 3.22 | 6.48 | 8.88 | 1.44 | 2.01 | 1.85 | 1.40 | 1.73 | 2.68
$\mathbf{X_{2}}=\\{V_{c1},V_{c2},i_{ref},i_{x},\Delta V_{c1},\Delta
V_{c2},2\Delta i_{x},S_{opt}\\}$
THD [$\%$] | 1.65 | 1.06 | 2.36 | 0.93 | 2.07 | 4.13 | 3.73 | 3.17 | 6.22 | 8.01 | 1.41 | 2.21 | 1.25 | 1.20 | 1.69 | 3.06
$\mathbf{X_{3}}=\\{V_{c1},V_{c2},i_{ref},i_{x},\Delta V_{c1},\Delta
V_{c2},\Delta i_{x},S_{opt}\\}$
THD [$\%$] | 1.90 | 1.32 | 3.28 | 1.36 | 1.94 | 4.38 | 3.78 | 3.37 | 6.29 | 8.11 | 1.47 | 1.98 | 1.43 | 1.22 | 1.74 | 2.69
$\mathbf{X_{4}}=\\{V_{c2},i_{ref},i_{x},\Delta V_{c1},\Delta V_{c2},\Delta
i_{x},S_{opt}\\}$
THD [$\%$] | 1.59 | 1.31 | 3.64 | 1.08 | 2.08 | 4.39 | 3.53 | 3.22 | 6.57 | 7.98 | 1.43 | 2.03 | 1.71 | 1.74 | 1.69 | 2.67
$\mathbf{X_{5}}=\\{i_{ref},i_{x},\Delta V_{c1},\Delta V_{c2},\Delta
i_{x},S_{opt}\\}$
THD [$\%$] | 2.69 | 1.45 | 3.53 | 1.32 | 2.04 | 4.55 | 3.71 | 3.50 | 6.62 | 8.11 | 1.81 | 2.11 | 1.65 | 1.76 | 1.99 | 2.94

###  IV-B ANN-Based Control Strategy

The ANN-based control scheme proposed, in this work, undergoes two main
phases, namely, training and testing phases. As mentioned above, the ANN
should be first trained offline. To do so, MPC has been used as a baseline
control scheme for assessing our proposed control strategy and, in the
meanwhile, for collecting data required for training, validation, and testing
the proposed neural network. The acquired MPC dataset is used to train the ANN
to anticipate the optimal switching state $S_{opt}$ at instant $k+1$.
Afterwards, once the proposed neural network is fine-tuned, it can be
successfully used online for controlling the FCMLI, instead of relying on the
predictive control strategies.

$\vdots$$\vdots$$\vdots$$V_{1x}(k)$$V_{2x}(k)$Apply
$S_{opt}(k+1)$$\text{V}_{0}$$\text{V}_{7}$ANN-Based Control
Strategy$\text{V}_{1}$$i_{ref}(k)$$i_{x}(k)$$\Delta V_{1x}(k)$$\Delta
V_{2x}(k)$$2\Delta i_{x}(k)$$S_{opt}(k)$FCMLI Fig. 4: Block diagram of the
ANN-based control strategy, considering
$\mathbf{X}_{2}=\\{V_{1x},V_{2x},i_{ref},i_{x},\Delta V_{1x},\Delta
V_{2x},2\Delta i_{x},S_{opt}\\}$ as input features.

In this study, the input features of our proposed ANN-based control scheme are
a set of measured variables of the inverter, while the output includes the
optimal switching states $S_{opt}$ that should be applied to the inverter at
next sampling instant $k+1$, as illustrated in Fig. 4. More precisely, the
output includes a vector of eight elements that identifies the eight voltage
vectors ($V_{i}\;\forall i=\\{0,\dots,7\\}$) (see Table I). Each sampling
instant, the index of the optimum voltage vector will be only active with a
value of one, whereas others will be equal to zero.

###  IV-C Input Features Selection and Training Process

Basically, selecting the best set of features is a significant part of
designing the ANN-based control strategy. Based on the type of inverters and
the main purpose of the study, different features can become efficient in both
the training phase and testing phase that assess the capability of the
learning the mathematical model of the system to be controlled and its
dynamics. We have observed that in order to seek the best mapping from the raw
input data to the desired outputs, different combinations of input features
should be first considered and then assessed. To this end, different operating
conditions (i.e., $S_{1},S_{2},\dots,S_{16}$) with different system parameters
(such as, $V_{dc},T_{s},R,L,i_{ref}$) are considered (see Table II) so that
assess five different sets of input features, named:
$\mathbf{X}_{1},\mathbf{X}_{2},\mathbf{X}_{3},\mathbf{X}_{4},\text{
and}\;\mathbf{X}_{5}$. As can be seen Table II, for those input features, the
THD of the output current $i_{x}$ is computed, based on primary samples
acquired by MPC, and compared so that the best input features can be chosen.
We can observe from the preliminary results that the performance of the
proposed control strategy on the basis of $\mathbf{X}_{5}$, in which the
lowest set of features is used, is the worst as the THD is quite higher.
Furthermore, it can be observed that the four-level output voltage $V_{ph}$
that has been added to $\mathbf{X}_{1}$ does not have much improvement. For
that reason, $\mathbf{X}_{2}$ is selected as input features for the proposed
control scheme that given in Fig. 4, due to its significant improvement
compared to other considered sets. $\mathbf{X}_{2}$ is composed of 8
measurable variables, namely: $\\{V_{1x},V_{2x},i_{ref},i_{x},\Delta
V_{1x},\Delta V_{2x},2\Delta i_{x},S_{opt}\\}$, where $\Delta
V_{1x}=(\frac{V_{dc}}{3}-V_{1x}),\Delta
V_{2x}=(\frac{2V_{dc}}{3}-V_{2x}),\Delta i_{x}=(i_{ref}-i_{x})$, and $S_{opt}$
is the optimal switching state at time instant $k$.

Once the effective set of features is selected, MPC is utilized for providing
the required dataset. The dataset is collected taken into account different
uncertainties in the converter parameters such as input voltage, cell
capacitance, $RL$ impedance, reference load current, and sampling period, as
listed in Table III. Only eleven different conditions are applied to the FC
inverter, where the nominal values of the converter parameters are as follows:
$V_{dc}=$360\text{\,}\;\mathrm{V}$$, $C$ = 680 $\mu$F,
$L=$10\text{\,}\;\mathrm{m}\mathrm{H}$$,
$R=$15\text{\,}\;\mathrm{\SIUnitSymbolOhm}$$,
$i_{ref}=$15\text{\,}\;\mathrm{A}$$, The MPC’s sampling time is indicated in
the last column. In this work, the total dataset is randomly divided into
three sets: training set, validation set, and testing set with $70\%$, $15\%$,
and $15\%$ of the total dataset, respectively. The optimum number of hidden
layers is determined by varying the number of hidden layers and observe the
testing accuracy. In addition, the $transcg$ function is used as a training
function. After the training process, the trained ANN can be used to control
the four-level three-phase flying capacitor inverter. The training confusion
matrix that assesses the performance of the training is shown in Fig. 5, where
the lowest validation error is $0.1016$ taken from epoch $701$ and the overall
accuracy is $89.80\%$ with the total instances numbers of $509,788$. The
training results are summarized in Table IV, demonstrating that the training
dataset based on $\mathbf{X}_{2}$ is properly selected, resulting in an
efficient control system.

Table III: Training conditions of ANN-based control strategy. Condition No. |  $V_{dc}=$360\text{\,}\;\mathrm{[}\mathrm{V}\mathrm{]}$$ |  $C=$680\text{\,}\;\mathrm{[}\mathrm{\SIUnitSymbolMicro}\mathrm{F}\mathrm{]}$$ |  $L=$10\text{\,}\;\mathrm{[}\mathrm{m}\mathrm{H}\mathrm{]}$$ |  $R=$15\text{\,}\;\mathrm{[}\mathrm{\SIUnitSymbolOhm}\mathrm{]}$$ |  $i_{ref}=$15\text{\,}\;\mathrm{[}\mathrm{A}\mathrm{]}$$ |  $T_{s}$\text{\,}\;\mathrm{[}\mathrm{\SIUnitSymbolMicro}\mathrm{s}\mathrm{]}$$
---|---|---|---|---|---|---
$C_{1}$ | 0.95 | 0.95 | 1.00 | 0.80 | 0.95 | $30$
$C_{2}$ | 0.90 | 0.85 | 0.95 | 0.70 | 0.90 | $10$
$S_{3}$ | 1.25 | 0.90 | 1.20 | 1.10 | 1.15 | $50$
$C_{4}$ | 1.10 | 1.05 | 1.50 | 1.23 | 1.05 | $60$
$C_{5}$ | 1.00 | 1.10 | 1.05 | 1.40 | 0.75 | $15$
$C_{6}$ | 1.00 | 0.98 | 0.75 | 1.30 | 0.65 | $18$
$C_{7}$ | 1.15 | 1.20 | 0.80 | 1.17 | 0.55 | $50$
$C_{8}$ | 1.00 | 1.07 | 0.88 | 0.77 | 0.85 | $30$
$C_{9}$ | 1.00 | 1.00 | 0.98 | 0.87 | 0.50 | $40$
$C_{10}$ | 1.00 | 1.00 | 0.90 | 0.10 | 2.00 | $5$
$C_{11}$ | 1.00 | 1.00 | 0.90 | 0.10 | 2.00 | $15$
Fig. 5: Training confusion matrix for $70\%$ training set with a misclassification error of $10.2\%$, where the number of correctly and incorrectly classified observations are shown in green and red, respectively. Table IV: Training results. No. instances | Accuracy | Error (Epoch)
---|---|---
$509,788$ | $89.80\%$ | $0.1016(701)$

##  V Simulation Results and Discussion

In this section, the ANN-based control strategy is simulated using
MATLAB/Simulink software and its performance is compared to that of the
conventional MPC. The parameters of the FCMLI given in Fig. 1 are listed in
Table V.

Table V: Parameters of the FCMLI. Parameter | Value
---|---
$R$ | $15\text{\,}\;\mathrm{[}\mathrm{\SIUnitSymbolOhm}\mathrm{]}$
$L$ | $10\text{\,}\;\mathrm{[}\mathrm{m}\mathrm{H}\mathrm{]}$
$C$ | $680\text{\,}\;\mathrm{[}\mathrm{\SIUnitSymbolMicro}\mathrm{F}\mathrm{]}$
$V_{dc}$ | $360\text{\,}\;\mathrm{[}\mathrm{V}\mathrm{]}$
$T_{s}$ | $30\text{\,}\;\mathrm{[}\mathrm{\SIUnitSymbolMicro}\mathrm{s}\mathrm{]}$
$f$ | $50\text{\,}\;\mathrm{[}\mathrm{Hz}\mathrm{]}$
Fig. 6: Dynamic response of ANN-based control strategy: Output current and
voltage when the reference current $i_{ref}$ changes from
$i_{ref}=$10\text{\,}\mathrm{A}$$ to $i_{ref}=$5\text{\,}\mathrm{A}$$ at
$t=$0.05\text{\,}\mathrm{s}$$. Fig. 7: Dynamic response of MPC: output current
and voltage when the reference current $i_{ref}$ changes from
$i_{ref}=$10\text{\,}\mathrm{A}$$ to $i_{ref}=$5\text{\,}\mathrm{A}$$ at
$t=$0.05\text{\,}\mathrm{s}$$.

The dynamic behavior of the ANN-based and MPC control schemes are illustrated
in Figs. 6 and 7, respectively. Both figures show the load output current
$i_{x}$ and four-level output voltage $V_{ph}$ under a sudden change of
reference current $i_{ref}$ from $10\text{\,}\mathrm{A}$ to
$5\text{\,}\mathrm{A}$ at $t=$0.05\text{\,}\mathrm{s}$$. It is observed that
both methods have similar and good performance under dynamic conditions, since
both control strategies track the reference current properly.

Fig. 8: Simulation results of ANN-based control strategy: Output current and
four-level voltage. Fig. 9: Simulation results of MPC: Output current and
four-level voltage

The behavior of our proposed ANN-based control strategy under steady-state
condition is depicted in Fig. 8, while the behavior of the classical MPC is
shown in Fig. 9. It can be seen in the figures that the two control strategies
generate a high-quality sinusoidal output current $i_{x}$ with low distortion.
However, the THD of the output current in the case of an ANN-based control
scheme is slightly lower than that in the case of MPC, namely, 1.34% compared
to 1.28% in case of ANN-based method. For more information, the frequency
spectrum of the two control methods are shown in Figs. 10 and 11. In addition,
the voltages of the capacitors (i.e., $V_{1x},V_{2x}$) obtained by the two
proposed control strategies are depicted in Figs. 12 and 13, respectively. As
can be observed, our proposed control scheme has a proper performance to keep
the voltage at its desired value as well as the MPC scheme.

Fig. 10: Frequency spectrum of output current for the ANN-based control
strategy.

.

Fig. 11: Frequency spectrum of output current for the MPC.

.

Fig. 12: Simulation results of ANN-based control strategy: Voltages of the
capacitors. Fig. 13: Simulation results of MPC: Voltages of the capacitors.
Fig. 14: Performance of our proposed ANN-based control strategy under load
inductance variation from $L=$10\text{\,}\mathrm{mH}$$ to
$L=$5\text{\,}\mathrm{mH}$$ at $t=$0.1\text{\,}\mathrm{s}$$. Fig. 15:
Performance of the MPC scheme under load inductance variation from
$L=$10\text{\,}\mathrm{mH}$$ to $L=$5\text{\,}\mathrm{mH}$$ at
$t=$0.1\text{\,}\mathrm{s}$$.
$S_{1}$$S_{2}$$S_{4}$$S_{5}$$S_{6}$$S_{7}$$S_{9}$$S_{11}$$S_{13}$$S_{14}$$S_{15}$$S_{16}$$0.00$$2.00$$4.00$$6.00$$8.00$$1.65$$1.06$$0.93$$2.07$$4.13$$3.73$$6.22$$1.41$$1.25$$1.2$$1.69$$3.06$$1.55$$1.22$$0.86$$1.96$$4.39$$3.55$$6.36$$1.39$$1.44$$1.1$$1.67$$2.62$Different
Operating ConditionsTHD [%]ANN-BasedMPC Fig. 16: THD comparison of output
current between MPC and ANN-based control strategy.

As previously described, the ANN-based control strategy can be used to deal
with the parameter uncertainty issues that arise in real systems. Thus, to
verify the efficiency of the ANN-based MPC technique in this scope, another
simulation study is carried out, where both control strategies are utilized to
compare their performance under this condition. More precisely, the load
inductance $L$ is changed from $10\text{\,}\mathrm{mH}$ to
$5\text{\,}\mathrm{mH}$ at $t=$0.1\text{\,}\mathrm{s}$$; the obtained results
are depicted in Figs. 14 and 15. By comparison between these two figures, it
can be seen that the proposed ANN-based control scheme has better performance
in such a situation. This can be clearly seen in the THD of the output current
for the ANN-based control scheme that has a 2.78% compared to 3.05% in the
case of MPC after changing the inductance value, which demonstrates that the
proposed control scheme is more robust to the parameter uncertainty than the
classical MPC.

To better assess and show the superiority of the proposed control strategy in
learning the mathematical model of the inverter and its dynamics, a comparison
of the THD of the output current obtained by the two control strategies is
represented in Fig. 16, considering the different operating conditions listed
in Table II. The obtained results demonstrate that the proposed ANN-based
control scheme on the basis of the selected input feature (namely, $X_{2}$)
has good performance compared to the conventional MPC, without requiring a
large amount of training data.

##  VI Conclusions and Future Work

In this paper, we have proposed an ANN-based control strategy for controlling
a three-phase four-level flying capacitor inverter with the aim of generating
a high-quality sinusoidal output current and improving the robustness against
parametric mismatches, especially when compared to conventional MPC scheme.
The proposed control scheme employs the classical MPC to generate a reliable
dataset of inverter variables to be used as input features by the proposed a
feed-forward ANN. Afterwards, once the off-line training is performed based on
the obtained data from MPC, it can be used to directly control the inverter
without the requirement for a mathematical model of the system. Simulation
results reveal that ANN-based control strategy performs better with respect to
the THD under most conditions. Additionally, an experiment is carried out for
assessing its robustness against parameter uncertainty. According to the
results, the proposed control scheme showed better performance than the
conventional MPC under parameter mismatch conditions, owing to the fact that,
unlike MPC, the proposed technique does not dependent on the system model. One
of the possible directions for future work would be the implementation of the
ANN-based control strategy in practical applications.

## References

  * [1] P. Cortés, G. Ortiz, J. I. Yuz, J. Rodríguez, S. Vazquez, and L. G. Franquelo, “Model predictive control of an inverter with output $LC$ filter for UPS applications,” _IEEE Transactions on industrial electronics_ , vol. 56, no. 6, pp. 1875–1883, 2009.
  * [2] P. B. Malidarreh, D. A. Khaburi, and J. Rodriguez, “Capacitor voltage imbalance reduction in flying capacitor modular multilevel converters by using model predictive control,” in _2020 11th Power Electronics, Drive Systems, and Technologies Conference (PEDSTC)_. IEEE, 2020, pp. 1–4.
  * [3] T. G. Habetler, R. Naik, and T. A. Nondahl, “Design and implementation of an inverter output LC filter used for $dv/dt$ reduction,” _IEEE Transactions on Power Electronics_ , vol. 17, no. 3, pp. 327–331, 2002.
  * [4] J. Jung, M. Dai, and A. Keyhani, “Optimal control of three-phase pwm inverter for ups systems,” in _2004 IEEE 35th Annual Power Electronics Specialists Conference (IEEE Cat. No. 04CH37551)_ , vol. 3. IEEE, 2004, pp. 2054–2059.
  * [5] I. S. Mohamed, S. A. Zaid, M. Abu-Elyazeed, and H. M. Elsayed, “Model predictive control-a simple and powerful method to control UPS inverter applications with output $LC$ filter,” in _2013 Saudi International Electronics, Communications and Photonics Conference_. IEEE, 2013, pp. 1–6.
  * [6] ——, “Classical methods and model predictive control of three-phase inverter with output $LC$ filter for UPS applications,” in _2013 International Conference on Control, Decision and Information Technologies (CoDIT)_. IEEE, 2013, pp. 483–488.
  * [7] J. Rodriguez and P. Cortes, _Predictive control of power converters and electrical drives_ , 1st ed. John Wiley & Sons, 2012.
  * [8] S. Kouro, M. Malinowski, K. Gopakumar, J. Pou, L. G. Franquelo, B. Wu, J. Rodriguez, M. A. Pérez, and J. I. Leon, “Recent advances and industrial applications of multilevel converters,” _IEEE Transactions on industrial electronics_ , vol. 57, no. 8, pp. 2553–2580, 2010.
  * [9] S. Kouro, J. Rodriguez, B. Wu, S. Bernet, and M. Perez, “Power the future of industry,” _IEEE Trans. Ind. Electron._ , pp. 26–38, 2012.
  * [10] J. Rodríguez, S. Bernet, B. Wu, J. O. Pontt, and S. Kouro, “Multilevel voltage-source-converter topologies for industrial medium-voltage drives,” _IEEE Transactions on industrial electronics_ , vol. 54, no. 6, pp. 2930–2945, 2007.
  * [11] L. G. Franquelo, J. Rodriguez, J. I. Leon, S. Kouro, R. Portillo, and M. A. Prats, “The age of multilevel converters arrives,” _IEEE industrial electronics magazine_ , vol. 2, no. 2, pp. 28–39, 2008.
  * [12] J. Rodriguez, M. P. Kazmierkowski, J. R. Espinoza, P. Zanchetta, H. Abu-Rub, H. A. Young, and C. A. Rojas, “State of the art of finite control set model predictive control in power electronics,” _IEEE Transactions on Industrial Informatics_ , vol. 9, no. 2, pp. 1003–1016, 2012.
  * [13] P. Cortés and J. Rodríguez, “Three-phase inverter with output $LC$ filter using predictive control for UPS applications,” in _2007 European Conference on Power Electronics and Applications_. IEEE, 2007, pp. 1–7.
  * [14] S. Kwak and J.-C. Park, “Switching strategy based on model predictive control of VSI to obtain high efficiency and balanced loss distribution,” _IEEE Transactions on Power Electronics_ , vol. 29, no. 9, pp. 4551–4567, 2013.
  * [15] I. S. Mohamed, S. A. Zaid, M. Abu-Elyazeed, and H. M. Elsayed, “Improved model predictive control for three-phase inverter with output $LC$ filter,” _International Journal of Modelling, Identification and Control_ , vol. 23, no. 4, pp. 371–379, 2015.
  * [16] F. Defaÿ, A.-M. Llor, and M. Fadel, “Direct control strategy for a four-level three-phase flying-capacitor inverter,” _IEEE Transactions on Industrial Electronics_ , vol. 57, no. 7, pp. 2240–2248, 2010.
  * [17] G. Papafotiou, J. Kley, K. G. Papadopoulos, P. Bohren, and M. Morari, “Model predictive direct torque control—part II: Implementation and experimental evaluation,” _IEEE Transactions on industrial electronics_ , vol. 56, no. 6, pp. 1906–1915, 2008.
  * [18] Z. Song, W. Chen, and C. Xia, “Predictive direct power control for three-phase grid-connected converters without sector information and voltage vector selection,” _IEEE Transactions on Power Electronics_ , vol. 29, no. 10, pp. 5518–5531, 2013.
  * [19] I. S. Mohamed, S. A. Zaid, M. Abu-Elyazeed, and H. M. Elsayed, “Implementation of model predictive control for three-phase inverter with output $LC$ filter on eZdsp F28335 Kit using HIL simulation,” _International Journal of Modelling, Identification and Control_ , vol. 25, no. 4, pp. 301–312, 2016.
  * [20] R. Findeisen, F. Allgöwer, and L. T. Biegler, “Assessment and future directions of nonlinear model predictive control,” in _Lecture Notes in Control and Information Sciences_ , M. Thomaa and M. Morari, Eds. Springer, 2007, vol. 358, no. 7.
  * [21] M. Nauman and A. Hasan, “Efficient implicit model-predictive control of a three-phase inverter with an output $LC$ filter,” _IEEE Transactions on Power Electronics_ , vol. 31, no. 9, pp. 6075–6078, 2016.
  * [22] C. Garcia, M. Mohammadinodoushan, V. Yaramasu, M. Norambuena, A. Davari, Z. Zhang, D. Khaburi, and J. Rodriguez, “FCS-MPC based pre-filtering stage for computational efficiency in a flying capacitor converter,” _IEEE Access_ , 2021.
  * [23] M. Norambuena, C. Garcia, J. Rodriguez, and P. Lezana, “Finite control set model predictive control reduced computational cost applied to a flying capacitor converter,” in _IECON 2017-43rd Annual Conference of the IEEE Industrial Electronics Society_. IEEE, 2017, pp. 4903–4907.
  * [24] C. Garcia, M. Norambuena, J. Rodriguez, M. Khosravi, and D. A. Khaburi, “Finite set model predictive control of a flying capacitor converter with a geometric computational optimization,” in _2017 IEEE Southern Power Electronics Conference (SPEC)_. IEEE, 2017, pp. 1–5.
  * [25] D. Wang, Z. J. Shen, X. Yin, S. Tang, X. Liu, C. Zhang, J. Wang, J. Rodriguez, and M. Norambuena, “Model predictive control using artificial neural network for power converters,” _IEEE Transactions on Industrial Electronics_ , 2021\.
  * [26] S. Kwak, U.-C. Moon, and J.-C. Park, “Predictive-control-based direct power control with an adaptive parameter identification technique for improved AFE performance,” _IEEE Transactions on Power Electronics_ , vol. 29, no. 11, pp. 6178–6187, 2014.
  * [27] C. Martin, M. Bermúdez, F. Barrero, M. R. Arahal, X. Kestelyn, and M. J. Durán, “Sensitivity of predictive controllers to parameter variation in five-phase induction motor drives,” _Control Engineering Practice_ , vol. 68, pp. 23–31, 2017.
  * [28] H. A. Young, M. A. Perez, and J. Rodriguez, “Analysis of finite-control-set model predictive current control with model parameter mismatch in a three-phase inverter,” _IEEE Transactions on Industrial Electronics_ , vol. 63, no. 5, pp. 3100–3107, 2016.
  * [29] B. Bogado, F. Barrero, M. R. Arahal, S. Toral, and E. Levi, “Sensitivity to electrical parameter variations of predictive current control in multiphase drives,” in _IECON 2013 39th Annual Conference of the IEEE Industrial Electronics Society_. IEEE, 2013, pp. 5215–5220.
  * [30] B. Karanayil and M. F. Rahman, “Artificial neural network applications in power electronics and electrical drives,” in _Power Electronics Handbook_. Elsevier, 2011, pp. 1139–1154.
  * [31] I. S. Mohamed, S. Rovetta, T. D. Do, T. Dragicević, and A. A. Z. Diab, “A neural-network-based model predictive control of three-phase inverter with an output $LC$ filter,” _IEEE Access_ , vol. 7, pp. 124 737–124 749, 2019\.
  * [32] X. Liu, L. Qiu, Y. Fang, Z. Peng, and D. Wang, “Finite-level-state model predictive control for sensorless three-phase four-arm modular multilevel converter,” _IEEE Transactions on Power Electronics_ , vol. 35, no. 5, pp. 4462–4466, 2019.
  * [33] T. Dragičević and M. Novak, “Weighting factor design in model predictive control of power electronic converters: An artificial neural network approach,” _IEEE Transactions on Industrial Electronics_ , vol. 66, no. 11, pp. 8870–8880, 2018.
  * [34] M. T. Wishart and R. G. Harley, “Identification and control of induction machines using artificial neural networks,” _IEEE Transactions on Industry Applications_ , vol. 31, no. 3, pp. 612–619, 1995.
  * [35] H.-Y. Lee, J.-l. Lee, S.-O. Kwon, and S.-W. Lee, “Performance estimation of induction motor using artificial neural network,” in _2018 25th International Conference on Systems, Signals and Image Processing (IWSSIP)_. IEEE, 2018, pp. 1–3.
  * [36] X. Sun, L. Chen, Z. Yang, and H. Zhu, “Speed-sensorless vector control of a bearingless induction motor with artificial neural network inverse speed observer,” _IEEE/ASME Transactions on mechatronics_ , vol. 18, no. 4, pp. 1357–1366, 2012.
  * [37] K. Hornik, “Approximation capabilities of multilayer feedforward networks,” _Neural networks_ , vol. 4, no. 2, pp. 251–257, 1991.


# Leveraging Synthetic Audio Data for
End-to-End Low-Resource Speech Translation

Yasmin Moslem
Bering Lab

###### Abstract

This paper describes our system submission to the International Conference on
Spoken Language Translation (IWSLT 2024) for Irish-to-English speech
translation. We built end-to-end systems based on Whisper, and employed a
number of data augmentation techniques, such as speech back-translation and
noise augmentation. We investigate the effect of using synthetic audio data
and discuss several methods for enriching signal diversity.

Leveraging Synthetic Audio Data for
End-to-End Low-Resource Speech Translation

Yasmin Moslem Bering Lab

##  1 Introduction

Resource scarcity and the scattered nature of the data are crucial challenges
for low-resource languages (Lankford et al., 2021; Haddow et al., 2022;
Lovenia et al., 2024). In this sense, Irish is considered a low-resource
language and significantly lacking in speech and language tools and resources
(Barry et al., 2022; Lynn, 2022). Researchers have been employing various data
augmentation techniques to improve the quality of low-resource textual machine
translation (MT) systems. Among these techniques is using synthetic data
generated by back-translation (Sennrich et al., 2016; Edunov et al., 2018;
Dowling et al., 2019; Poncelas et al., 2019; Haque et al., 2020), or large
language models (Moslem et al., 2022). Similarly, in the area of speech, Lee
et al. (2023) showed that models trained solely on synthetic audio datasets
can generalize their performance to human voice data. Nevertheless, Guo et al.
(2023) revealed a consistent decrease in the diversity of the outputs of
language models trained on synthetic textual data. We observe that leveraging
synthetic audio data generated by text-to-speech (TTS) models can be
beneficial for training speech translation models, especially for low-resource
languages. However, it can lack the diversity found in authentic audio signals
in terms of pitch, speed, and background noise.

Speech translation systems can be cascaded systems or end-to-end systems
(Agarwal et al., 2023). Cascaded systems use two models, one for automatic
speech recognition (ASR) and one for textual machine translation (MT). End-to-
end speech translation systems use one model for the whole process; hence, it
is more challenging. In this work, we present end-to-end speech translation
models.

In addition to describing our system submitted to IWSLT 2024, this work
presents the following contributions:

  * •

Showcasing “speech back-translation” as an effective data augmentation
technique for speech translation. In other words, just as back-translation can
improve the output quality of text-to-text MT, generating source-side
synthetic audio data can considerably enhance the performance of speech
translation systems, especially for low-resource languages.

  * •

Introducing a collection of datasets for Irish-to-English speech translation,
three of which comprise 196 hours of synthetic audio.

  * •

Exploring diverse training settings and data processing techniques such as
noise augmentation and voice audio detection (VAD).

  * •

Releasing versions of Whisper models, specifically fine-tuned for Irish-to-
English speech translation.

Dataset | Audio | Translation | Train Hours (H:M) | Train Segments | Test Segments
---|---|---|---|---|---
IWSLT-2023 | Authentic | Authentic | 8:25 | 8,598 | 347
FLEURS | Authentic | Authentic | 16:45 | 3,991 | 0
Bitesize | Authentic | Authentic | 5:15 | 6,149 | 0
SpokenWords | Authentic | MTed | 3:02 | 10,925 | 0
EUbookshop | Synthetic | Authentic | 159:45 | 67,268 | 0
Tatoeba | Synthetic | Authentic | 2:39 | 3,966 | 0
Wikimedia | Synthetic | Authentic | 34:23 | 15,090 | 0
Authentic () | 33:27 | 29,663 | 347
Synthetic ( ) | 196:47 | 86,324 | 0
Authentic () + Synthetic ( ) | 70:29 | 48,719 | 347
Authentic () + Synthetic ( ) | 229:14 | 115,987 | 347
Table 1: Data Statistics: “Audio” and “Translation” columns refer to whether
the data is human-generated or machine-generated. “Train Hours” and “Train
Segments” refer to the size of the training data in terms of duration and
number of utterances, respectively. Finally, “Test Segments” refer to the
number of utterances in the test dataset.

##  2 Authentic Data

The organizers of the IWSLT shared task, provided the IWSLT-2023 dataset,
which consists of training, dev, and test portions. We used both the training
and dev portions for training, and the test portion for evaluation. We also
used the Irish portion of the FLEURS datasets. Moreover, we employed the
bilingual audio-text data available at the Bitesize website for teaching
Irish.111https://huggingface.co/datasets/ymoslem/BitesizeIrish-GA-EN

##  3 Synthetic Data

This section explains diverse approaches for creating synthetic data for
speech translation. We describe each approach, as well as its advantages and
disadvantages.

###  3.1 Machine Translation

When both audio and transcription are available, but there is no translation,
forward MT can be useful as a data augmentation technique. However, there is
the risk of feeding incorrect target translations into the training process.
Forward MT is more sensitive to the quality of the system used to produce the
synthetic data. Compared to back-translation, biases and errors in synthetic
data are intuitively more problematic in forward-translation, since they
directly affect the gold labels (Bogoychev and Sennrich, 2019). Hence, the
used MT system must be of high quality.

We automatically translated the Irish portion of the Spoken Words dataset into
English using the Google Translation API. For quality considerations, we
decided to use this dataset for training only, but not for evaluation. The
dataset consists of 10,925 utterances. Some words are spoken by multiple
narrators.222https://huggingface.co/datasets/ymoslem/SpokenWords-GA-EN-MTed

Figure 1: Comparing authentic (top) and synthetic (bottom) audio signals

###  3.2 Synthetic Audio Data

OPUS (Tiedemann, 2012) hosts several bilingual textual datasets. We extracted
portions of the Tatoeba, Wikimedia, and EUbookshop datasets, comprising 1,983,
7,545 and 33,634 segments, respectively. We extensively filter the datasets
based on the following criteria: removing duplicates, removing segments longer
than 30 words,333https://github.com/ymoslem/MT-Preparation language detection
with fastText (Joulin et al., 2017) (both sides), and Seamless toxicity
filtering (Barrault et al., 2023). Finally, we used Azure Speech service to
generate two sets of audio data, one with a female voice (OrlaNeural) and the
other with a male voice (ColmNeural). As an outcome of this process, we
introduce three new datasets, Tatoeba-Speech-
Irish,444https://huggingface.co/datasets/ymoslem/Tatoeba-Speech-Irish
Wikimedia-Speech-Irish,555https://huggingface.co/datasets/ymoslem/Wikimedia-
Speech-Irish and EUbookshop-Speech-
Irish,666https://huggingface.co/datasets/ymoslem/EUbookshop-Speech-Irish which
together comprise 196 hours of synthetic audio. Table 1 illustrates the
statistics of our datasets.

###  3.3 Audio Signal Processing Augmentation

Synthetic audio data generated by TTS models can have different
characteristics than authentic audio. In addition to quality considerations,
we observe that among the features that distinguish data generated by TTS
systems from authentic data are: 1) lack of noise, and 2) silence differences.

##### Lack of noise:

TTS systems try to mimic studio settings, and produce very clean audio
signals. However, authentic audio signals can include all sorts of
environmental noise, ranging from white noise to background voices of people
and cars. Even in studio settings, some breath signals can occur unless the
audio is extensively edited.

##### Silence variances:

All the synthetic audio signals we generated start at a similar point, with
almost no silence at the beginning of the audio (probably to facilitate mixing
tracks). However, authentic audio signals can start at any point depending on
the recording and processing settings, or whether a signal is truncated from a
longer one.

Figure 1 illustrates an example sentence from the Common Voice dataset uttered
by a human female (non-studio settings) and its synthetic equivalent generated
by Azure TTS system.777Voice name: “Microsoft Server Speech Text to Speech
Voice (ga-IE, OrlaNeural)” The Irish sentence represented here is “Go raibh
maith agaibh as ucht na fíorchaoin fáilte a d’fhear sibh romham.” It can be
translated into English as “Thank you all for that very generous welcome.” The
authentic signal has more noise (both white background noise and sounds of
starting/stopping the recording software), while the synthetic signal does not
show any noise occurrence. Moreover, unlike the authentic signal, the
synthetic data starts almost immediately. Another observation is that this
specific authentic signal has a lower volume than synthetic signals.

####  3.3.1 Voice Activity Detection

One of the most common audio preprocessing techniques is Voice Audio Detection
(VAD). The main idea of VAD is to remove low-amplitude samples from an audio
signal. Low-amplitude samples might represent science or noise samples of
audio signals, which usually occur at the beginning and end of an audio
signal, but can also happen in the midst of longer audio signals. In its basic
form, this can be achieved by removing any sample below an absolute value of a
threshold (e.g. ± 0.001). However, advanced models like Silero
VAD888https://github.com/snakers4/silero-vad can be used as part of the
torchaudio framework, and include more sophisticated options (e.g. minimum
silence duration) to avoid removing important low-amplitude samples like
breath and natural silent durations.

During training, data processed with VAD can either substitute the original
data or augment it, i.e. both processed and unprocessed data can be used
during training. In one of our experiments (cf. Section 4), we used basic VAD
with a threshold of ± 0.001 as a data augmentation technique. When basic VAD
is used (i.e. without taking a minimum silence duration into account), this
can also speed up the audio signal; in other words, the utterance is spoken
faster. At inference time of all the models, we used Silero VAD within Faster-
Whisper based on CTranslate2 (Klein et al., 2020).

####  3.3.2 Noise Augmentation

Mimicking the effect of white noise can take diverse forms, ranging from using
real noise to generating random arrays. To simulate light white noise, we
generated a random array with a distribution scale 0.002 and added it to all
the audio signals in the dataset.

Whisper | Model | Datasets | Data Size | BLEU ↑ | chrF++ ↑ | WER ↓ | Semantic 1 ↑ | Semantic 2 ↑
---|---|---|---|---|---|---|---|---
Medium | A | authentic | 29,663 | 32.38 | 48.95 | 58.85 | 62.09 | 63.28
B | A + synthetic (2d) | 48,719 |  36.34 |  54.08 |  53.35 |  68.31 |  69.93
B++ | A + synthetic (3d) | 115,987 | 38.41 | 57.18 | 51.10 | 69.72 | 71.13
C | B + augmented | 146,157 | 34.09 | 51.40 | 55.83 | 64.26 | 65.56
Table 2: Evaluation Results: Model B++ that uses both authentic data and 3
synthetic audio datasets achieved the best results across all the systems. The
results show that augmenting the training data with synthetic audio (i.e.
Model B and Model B++) outperforms using authentic data only (Model A), while
further signal processing augmentation with white noise and VAD (Model C) did
not help. Moreover, increasing the amount of high-quality synthetic audio data
in Model B++ resulted in better quality than Model B that uses a less amount
of synthetic data.

##  4 Experiments

Our experiments fine-tune Whisper (Radford et al., 2022) for the task of
Irish-to-English speech translation. We experiment with a number of data
augmentation techniques, such as speech back-translation (source-side
synthetic audio data generation), and audio data augmentation with noise and
VAD.

###  4.1 Speech Back-Translation

By the term “speech back-translation”, we refer to generating source-side
synthetic audio for data augmentation of speech translation systems, in the
same manner that back-translation is employed in text-to-text MT systems.
Section 3.2 explains how we created these synthetic audio datasets. In this
set of experiments, we built 3 systems by fine-tuning Whisper Medium. We use
different types of datasets as outlined by Table 1.

  * •

Model A: It uses the authentic data only, namely IWSLT-2023 dataset, FLEURS,
Bitesize, and SpokenWords.

  * •

Model B: It uses the same authentic data used in Model A as well as two
synthetic audio datasets, namely Tatoeba-Speech-Irish, and Wikimedia-Speech-
Irish.

  * •

Model B++: In addition to the authentic and two synthetic datasets used in the
aforementioned models, Model B++ uses a third synthetic dataset, namely
EUbookshop-Speech-Irish.

###  4.2 Noise and VAD Augmentation

  * •

Model C: It uses the same data as Model B, as well as two versions of the data
augmented with basic VAD, and white noise. In other words, we fine-tuned
Whisper-Medium on all the authentic data and two synthetic data as well as two
augmented datasets, one with low-amplitude sample removal, and one with noise
augmentation, as described in Section 3.

###  4.3 Training Arguments

We tried different learning rates and warm-up values. Specifically, we
experimented with warm-up ratios 0%, 1%, and 3% out of 3000 steps, which
corresponds to 0, 30, 90 warm-up steps, respectively. As Table 5 and Table 4
demonstrate, when fine-tuning Whisper Small, changing the warm-up ratio does
not seem to lead to a consistent improvement for the first two sizes of data
used in Model A and Model B. However, increasing the warm-up ratio to 3% when
the size of data is larger as in Model C, seems to slightly improve the
performance. For the learning rate, we used 1e-4 across all the experiments
for the sake of consistency. The batch size was decided based on the compute
capacity of one A100-SXM4-80GB GPU. Hence, we used a batch size of 64 examples
when fine-tuning Whisper Small and a batch size of 16 examples when fine-
tuning Whisper Medium. The max length of generation was set to 225. As this is
an Irish-to-English translation task, both the tokenizer language and model
generation language were set to English. We train the main models with Whisper
Medium for at least two epochs, and save the best performing checkpoint based
on the chrF++ score on the validation dataset. Section 5 elaborates on the
results of these experiments.

###  4.4 Training Epochs

As we reported in the previous section, we used 3000 steps for all the
experiments with Whisper Small, as further training did not seem to improve
the output quality when more than one epoch of data is already reached.
However, Whisper Medium was trained with a smaller batch size due to computing
constrains. We wanted to see the effect of training for at least two epochs.
Hence, we report different step milestones in Table 6. In deep learning
training in general, it is a common practice to use early stopping. However,
for low-resource languages, a smaller value for early stopping can result in
the model not seeing the whole data, which can affect the robustness of the
model. This is especially true if we are not sure if the validation dataset is
well-representative of the task that the model will be actually required to
tackle in the real world. While there is no one rule that applies to all
cases, we recommend taking this point into consideration when training generic
models for low-resource languages.

##  5 Evaluation and Results

To evaluate our systems, we calculated BLEU (Papineni et al., 2002), chrF++
(Popović, 2017), and TER (Snover et al., 2006), as implemented in the
sacreBLEU library999https://github.com/mjpost/sacrebleu (Post, 2018). For
semantic evaluation, we used an embedding-based approach, calculating and
comparing cosine similarity between the vector embeddings of each reference
and the equivalent translation generated by the model. We report the average
of semantic similarity across utterances. We used two models with Sentence-
Transformers (Reimers and Gurevych, 2019), “all-mpnet-base-v2” (Semantic 1)
and “all-MiniLM-L12-v2” (Wang et al., 2020) (Semantic 2). As we fine-tuned all
the models for approximately two epochs, we report the evaluation of the best
performing checkpoint.

For inference, we used Faster-Whisper 101010https://github.com/SYSTRAN/faster-
whisper with the default VAD arguments. We also compared the results without
VAD, and found that applying VAD at inference time is better for all the
models (cf. Appendix A). We used 5 for “beam size” and 2 for “no repeat ngram
size”.

As Table 2 shows, after fine-tuning Whisper Medium on both the authentic and
synthetic audio data (Model B), there are consistent improvements across all
metrics compared to when we fine-tuned it on the authentic audio data only
(Model A). Moreover, Model B++ that uses three synthetic datasets outperforms
Model B that uses only two synthetic datasets. This demonstrates that
augmented authentic audio data with high-quality synthetic audio data can
enhance end-to-end speech translation systems, especially for low-resource
languages like Irish.

Model C uses the same training data as Model B as well as two augmented
versions, one version that applies basic VAD, removing low-amplitude samples
(cf. Section 3.3.1) and another version that injects white background noise
into the data (cf. Section 3.3.2). Although Model C that uses noise and VAD
augmented data still outperforms Model A that uses authentic training data
only, both Model B and B++ that combines authentic data with synthetic data
outperform Model C.

While the choice of augmentation techniques were based on manual observation
of the characteristics of the authentic data and the synthetic data, the
achieved improvements encourage further investigation. In the future, we would
like to conduct more experiments that employ other data augmentation
techniques. Moreover, we would like to measure the effect of adding synthetic
audio data compared to augmenting the authentic data only. Finally, as the
main purpose of this research is to understand the best practices of using
synthetic audio data (i.e. data generated by TTS models) to improve speech
translation quality, we will conduct further study on mimicking authentic data
characteristics to enhance the effect of data augmentation with synthetic
audio data.

## References

  * Agarwal et al. (2023) Milind Agarwal, Sweta Agrawal, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Claudia Borg, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, Mingda Chen, William Chen, Khalid Choukri, Alexandra Chronopoulou, Anna Currey, Thierry Declerck, Qianqian Dong, Kevin Duh, Yannick Estève, Marcello Federico, Souhir Gahbiche, Barry Haddow, Benjamin Hsu, Phu Mon Htut, Hirofumi Inaguma, Dávid Javorský, John Judge, Yasumasa Kano, Tom Ko, Rishu Kumar, Pengwei Li, Xutai Ma, Prashant Mathur, Evgeny Matusov, Paul McNamee, John P. McCrae, Kenton Murray, Maria Nadejde, Satoshi Nakamura, Matteo Negri, Ha Nguyen, Jan Niehues, Xing Niu, Atul Kr. Ojha, John E. Ortega, Proyag Pal, Juan Pino, Lonneke van der Plas, Peter Polák, Elijah Rippeth, Elizabeth Salesky, Jiatong Shi, Matthias Sperber, Sebastian Stüker, Katsuhito Sudoh, Yun Tang, Brian Thompson, Kevin Tran, Marco Turchi, Alex Waibel, Mingxuan Wang, Shinji Watanabe, and Rodolfo Zevallos. 2023.  FINDINGS OF THE IWSLT 2023 EVALUATION CAMPAIGN.  In _Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023)_ , pages 1–61, Toronto, Canada (in-person and online). Association for Computational Linguistics.
  * Barrault et al. (2023) Loic Barrault, Andy Chung, David Dale, Ning Dong (ai), Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Peng-Jen Chen, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Abinesh Ramakrishnan, Alexandre Mourachko, Amanda Kallet, Ann Lee, Anna Sun, Bapi Akula, Benjamin Peloquin, Bernie Huang, Bokai Yu, Brian Ellis, Can Balioglu, Carleigh Wood, Changhan Wang, Christophe Ropers, Cynthia Gao, Daniel Li (fair), Elahe Kalbassi, Ethan Ye, Gabriel Mejia Gonzalez, Hirofumi Inaguma, Holger Schwenk, Igor Tufanov, Ilia Kulikov, Janice Lam, Jeff Wang (pm Ai), Juan Pino, Justin Haaheim, Justine Kao, Prangthip Hasanti, Kevin Tran, Maha Elbayad, Marta R Costa-jussa, Mohamed Ramadan, Naji El Hachem, Onur Çelebi, Paco Guzmán, Paden Tomasello, Pengwei Li, Pierre Andrews, Ruslan Mavlyutov, Russ Howes, Safiyyah Saleem, Skyler Wang, Somya Jain, Sravya Popuri, Tuan Tran, Vish Vogeti, Xutai Ma, and Yilin Yang. 2023.  SeamlessM4T—Massively Multilingual & Multimodal Machine Translation.  _Meta AI_.
  * Barry et al. (2022) James Barry, Joachim Wagner, Lauren Cassidy, Alan Cowap, Teresa Lynn, Abigail Walsh, Mícheál J Ó Meachair, and Jennifer Foster. 2022.  gaBERT — an Irish Language Model.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 4774–4788, Marseille, France. European Language Resources Association.
  * Bogoychev and Sennrich (2019) Nikolay Bogoychev and Rico Sennrich. 2019.  Domain, Translationese and Noise in Synthetic Data for Neural Machine Translation.
  * Dowling et al. (2019) Meghan Dowling, Teresa Lynn, and Andy Way. 2019.  Leveraging backtranslation to improve machine translation for Gaelic languages.  In _Proceedings of the Celtic Language Technology Workshop_ , pages 58–62, Dublin, Ireland. European Association for Machine Translation.
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding Back-Translation at Scale.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 489–500, Brussels, Belgium. Association for Computational Linguistics.
  * Guo et al. (2023) Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chloé Clavel. 2023.  The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text.  In _Proceedings or the Annual Conference of the North American Chapter of the Association for Computational Linguistics_ , Mexico City, Mexico.
  * Haddow et al. (2022) Barry Haddow, Rachel Bawden, Antonio Valerio Miceli Barone, Jindřich Helcl, and Alexandra Birch. 2022.  Survey of Low-Resource Machine Translation.  _Computational Linguistics_ , 06:1–67.
  * Haque et al. (2020) Rejwanul Haque, Yasmin Moslem, and Andy Way. 2020.  Terminology-Aware Sentence Mining for NMT Domain Adaptation: ADAPT’s Submission to the Adap-MT 2020 English-to-Hindi AI Translation Shared Task.  In _Proceedings of the 17th International Conference on Natural Language Processing (ICON): Adap-MT 2020 Shared Task_ , pages 17–23, Patna, India. NLP Association of India (NLPAI).
  * Joulin et al. (2017) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017.  Bag of Tricks for Efficient Text Classification.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 427–431, Valencia, Spain. Association for Computational Linguistics.
  * Klein et al. (2020) Guillaume Klein, François Hernandez, Vincent Nguyen, and Jean Senellart. 2020.  The OpenNMT Neural Machine Translation Toolkit: 2020 Edition.  In _Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)_ , pages 102–109, Virtual. Association for Machine Translation in the Americas.
  * Lankford et al. (2021) Seamus Lankford, Haithem Alfi, and Andy Way. 2021.  Transformers for Low-Resource Languages: Is Féidir Linn! In _Proceedings of Machine Translation Summit XVIII: Research Track_ , pages 48–60, Virtual. Association for Machine Translation in the Americas.
  * Lee et al. (2023) Jihyun Lee, Yejin Jeon, Wonjun Lee, Yunsu Kim, and Gary Geunbae Lee. 2023.  Exploring the Viability of Synthetic Audio Data for Audio-Based Dialogue State Tracking.  In _Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding_ , Taipei, Taiwan.
  * Lovenia et al. (2024) Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester James V Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno P Kampman, Joel Ruben Antony Moniz, Muhammad Ravi Shulthan Habibi, Frederikus Hudi, Railey Montalan, Ryan Ignatius, Joanito Agili Lopo, William Nixon, Börje F Karlsson, James Jaya, Ryandito Diandaru, Yuze Gao, Patrick Amadeus, Bin Wang, Jan Christian Blaise Cruz, Chenxi Whitehouse, Ivan Halim Parmonangan, Maria Khelli, Wenyu Zhang, Lucky Susanto, Reynard Adha Ryanda, Sonny Lazuardi Hermawan, Dan John Velasco, Muhammad Dehan Al Kautsar, Willy Fitra Hendria, Yasmin Moslem, Noah Flynn, Muhammad Farid Adilazuarda, Haochen Li, Johanes Lee, R Damanhuri, Shuo Sun, Muhammad Reza Qorib, Amirbek Djanibekov, Wei Qi Leong, Quyet V Do, Niklas Muennighoff, Tanrada Pansuwan, Ilham Firdausi Putra, Yan Xu, Ngee Chia Tai, Ayu Purwarianti, Sebastian Ruder, William Tjhi, Peerat Limkonchotiwat, Alham Fikri Aji, Sedrick Keh, Genta Indra Winata, Ruochen Zhang, Fajri Koto, Zheng-Xin Yong, and Samuel Cahyawijaya. 2024.  SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages.
  * Lynn (2022) Teresa Lynn. 2022.  Report on the Irish Language.  Technical report, European Language Equality.
  * Moslem et al. (2022) Yasmin Moslem, Rejwanul Haque, John Kelleher, and Andy Way. 2022.  Domain-Specific Text Generation for Machine Translation.  In _Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)_ , pages 14–30, Orlando, USA. Association for Machine Translation in the Americas.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a Method for Automatic Evaluation of Machine Translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Poncelas et al. (2019) Alberto Poncelas, Gideon Maillette de Buy Wenniger, and Andy Way. 2019.  Adaptation of Machine Translation Models with Back-Translated Data Using Transductive Data Selection Methods.  In _Proceedings of the 20th International Conference on Computational Linguistics and Intelligent Text Processing CICLing 2019: Computational Linguistics and Intelligent Text Processing_ , pages 567–579, La Rochelle, France. Springer Nature Switzerland.
  * Popović (2017) Maja Popović. 2017.  chrF++: words helping character n-grams.  In _Proceedings of the Second Conference on Machine Translation_ , pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A Call for Clarity in Reporting BLEU Scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Radford et al. (2022) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022.  Robust Speech Recognition via Large-Scale Weak Supervision.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Improving Neural Machine Translation Models with Monolingual Data.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 86–96, Berlin, Germany. Association for Computational Linguistics.
  * Snover et al. (2006) Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006.  A Study of Translation Edit Rate with Targeted Human Annotation.  In _Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers_ , pages 223–231, Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel Data, Tools and Interfaces in OPUS.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  MINILM: deep self-attention distillation for task-agnostic compression of pre-trained transformers.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , number Article 485 in NIPS’20, pages 5776–5788, Red Hook, NY, USA. Curran Associates Inc.

##  Appendix A Appendix: Arguments

###  A.1 Inference VAD

Argument | Type | Value |  | Argument | Type | Value
---|---|---|---|---|---|---
threshold | float | 0.5 |  | min_silence_duration_ms | int | 2000
min_speech_duration_ms | int | 250 |  | window_size_samples | int | 1024
max_speech_duration_s | float | float("inf") |  | speech_pad_ms | int | 400
Table 3: Default VAD values of Faster-Whisper.

###  A.2 Training Warm-up Ratio

Whisper | Model | Datasets | Data Size | Warm-up | BLEU | chrF++ | WER | Semantic 1 | Semantic 2
---|---|---|---|---|---|---|---|---|---
Small | A | authentic | 29,663 | 0.00 | 31.49 | 45.59 | 59.66 | 58.23 | 60.35
0.01 | 30.97 | 46.19 | 59.57 | 59.69 | 61.09
0.03 | 31.43 | 46.71 | 61.14 | 60.48 | 61.59
B | A + synthetic | 48,719 | 0.00 | 34.09 | 50.79 | 55.47 | 65.64 | 66.66
0.01 | 31.92 | 47.32 | 58.31 | 62.56 | 63.57
0.03 | 34.15 | 49.81 | 56.87 | 65.09 | 66.43
C | B + augmented | 146,157 | 0.00 | 30.75 | 45.87 | 61.37 | 60.51 | 61.98
0.01 | 32.82 | 48.31 | 57.95 | 63.26 | 64.72
0.03 | 35.07 | 50.23 | 56.73 | 63.33 | 64.80
Table 4: Comparing diverse values of warm-up ratio at training time. Ratios are out of 3000 steps. Hence, 0.01 and 0.03 correspond to 30 steps and 90 steps, respectively. The results here are with VAD at inference time, using the default VAD arguments of Faster-Whisper. The highest score in each group is displayed in a bold font. Whisper | Model | Datasets | Data Size | Warm-up | BLEU ↑ | chrF++ ↑ | WER ↓ | Semantic 1 ↑ | Semantic 2 ↑
---|---|---|---|---|---|---|---|---|---
Small | A | authentic | 29,663 | 0.00 | 29.14 | 43.34 | 60.51 | 56.96 | 58.14
0.01 | 30.66 | 45.41 | 62.09 | 58.69 | 59.79
0.03 | 30.68 | 45.36 | 62.09 | 57.82 | 59.29
B | A + synthetic | 48,719 | 0.00 | 32.05 | 48.32 | 58.44 | 62.51 | 63.72
0.01 | 31.94 | 46.81 | 59.93 | 61.57 | 62.36
0.03 | 31.61 | 47.74 | 59.16 | 62.49 | 64.09
C | B + augmented | 146,157 | 0.00 | 30.51 | 44.52 | 63.48 | 59.6 | 60.71
0.01 | 32.58 | 47.65 | 59.39 | 62.86 | 63.72
0.03 | 31.89 | 48.83 | 59.84 | 62.32 | 63.17
Table 5: Comparing diverse values of warm-up ratio at training time. Ratios
are out of 3000 steps. Hence, 0.01 and 0.03 correspond to 30 steps and 90
steps, respectively. The results here are without VAD at inference time. The
highest score in each group is displayed in a bold font.

###  A.3 Training Epochs

Whisper | Model | Datasets | Data Size | Warm-up | Steps | Epoch | Best Epoch | BLEU ↑ | chrF++ ↑ | WER ↓ | Semantic 1 ↑ | Semantic 2 ↑
---|---|---|---|---|---|---|---|---|---|---|---|---
Medium | A | authentic | 29,663 | 0.03 | 2,000 | 1.08 | 1.02 | 29.14 | 47.03 | 63.17 | 60.78 | 62.11
cont. | 4,000 | 2.16 | 1.83 |  32.38 |  48.95 |  58.85 |  62.09 |  63.28
B | A + synthetic (2d) | 48,719 | 0.03 | 4,000 | 1.31 | 1.22 | 36.02 | 53.73 |  53.26 | 66.86 | 68.16
cont. | 7,000 | 2.30 | 2.27 |  36.34 |  54.08 | 53.35 |  68.31 |  69.93
B++ | A + synthetic (3d) | 115,987 | 0.03 | 4,000 | 0.55 | 0.55 | 38.41 | 57.18 | 51.10 | 69.72 | 71.13
cont. | 8,000 | 1.10 | 0.55 | $\sim$ | $\sim$ | $\sim$ | $\sim$ | $\sim$
cont. | 15,000 | 2.07 | 0.55 | $\sim$ | $\sim$ | $\sim$ | $\sim$ | $\sim$
C | B + augmented | 146,157 | 0.03 | 4,000 | 0.44 | 0.38 | 33.46 | 50.72 | 57.59 | 63.01 | 64.56
cont. | 10,000 | 1.09 | 1.05 |  34.09 |  51.4 |  55.83 |  64.26 |  65.56
cont. | 19,000 | 2.08 | 1.05 | $\sim$ | $\sim$ | $\sim$ | $\sim$ | $\sim$
Table 6: Investigating the effect of training for 1-2 epoch(s). It seems that
smaller amounts of training data can benefit from training for 2+ while larger
amounts of data can benefit from training for only 1 epoch or less. The first
row of each section starts the training with warm-up ratio 0.03, then the next
1 or 2 row(s) continues training for more steps without changing any training
arguments. The reported scores are for the best step, based on training
validation with 100-step intervals. That is why some rows are marked with the
“$\sim$" sign, as the best step was still the same as the one reported in the
previous row.