# Multi-Agent Reinforcement Learning with Common Policy for Antenna Tilt
Optimization

Adriano Mendo  Jose Outes-Carnero  Yak Ng-Molina and Juan Ramiro-Moreno
Adriano Mendo, Jose Outes-Carnero, Yak Ng-Molina and Juan Ramiro-Moreno are
with Cloud and Software Services at Ericsson, 29590 Malaga, Spain (e-mail:
{adriano.mendo, jose.outes, yak.ng.molina, juan.ramiro}@ericsson.com).

###### Abstract

This paper presents a method for optimizing wireless networks by adjusting
cell parameters that affect both the performance of the cell being optimized
and the surrounding cells. The method uses multiple reinforcement learning
agents that share a common policy and take into account information from
neighboring cells to determine the state and reward. In order to avoid
impairing network performance during the initial stages of learning, agents
are pre-trained in an earlier phase of offline learning. During this phase, an
initial policy is obtained using feedback from a static network simulator and
considering a wide variety of scenarios. Finally, agents can intelligently
tune the cell parameters of a test network by suggesting small incremental
changes, slowly guiding the network toward an optimal configuration. The
agents propose optimal changes using the experience gained with the simulator
in the pre-training phase, but they can also continue to learn from current
network readings after each change. The results show how the proposed approach
significantly improves the performance gains already provided by expert
system-based methods when applied to remote antenna tilt optimization. The
significant gains of this approach have truly been observed when compared with
a similar method in which the state and reward do not incorporate information
from neighboring cells.

Reinforcement learning, tilt optimization, AI, deep neural network.

##  1 Introduction

Wireless networks are complex systems, where modification of certain cell
parameters may not only affect the performance in that specific cell but also
the surrounding cells. Finding an optimal configuration of this kind of
parameter might be therefore considered a challenging optimization problem.
Examples of these parameters are:

  * •

Remote Electrical Tilt (RET): Defines the angular elevation of a cell antenna
and allows for remote modification through a terminal, eliminating the need
for physical access to the antenna. Adjusting the RET value may improve the
Downlink (DL) Signal to Interference plus Noise Ratio (SINR) in the cell being
modified, but at the same time might degrade the SINR of surrounding cells and
vice versa.

  * •

P0 nominal Physical Uplink Shared Channel (PUSCH): Defines the target power
per Physical Resource Block (PRB) that the cell expects in the Uplink (UL),
i.e., the communication from the User Equipment (UE) to the Base Station (BS).
By increasing it, the UL SINR in the cell under modification increases but, at
the same time, the UL SINR in the surrounding cells may decrease, and vice
versa.

Therefore, there is a clear trade-off between the performance of the modified
cell and that of the surrounding cells. This trade-off is not easy to
estimate, since it varies on a case-by-case basis. The objective is to
optimize the global performance of the network by modifying the RET on a per-
cell basis. In computational complexity theory, this kind of problem is
considered Non-Polynomial Hard (NP-hard) to resolve. Numerous scholarly
articles and studies have been published on this topic. One of the most common
approaches to solving this problem is the use of a control system based on
rules defined by an expert. In [1] a fuzzy rule-based solution is described
for RET optimization. With the explosion of artificial intelligence (AI),
reinforcement learning (RL) has become a very popular method for solving
problems in diverse fields, e.g., autonomous vehicle driving [2], gaming and
puzzle resolution [3], and stock trading [4]. Wireless network optimization
has also become an attractive field of application, especially in addressing
the particular problem of antenna tilt optimization. In [5] a single RL agent
for the entire network is proposed, which is no longer valid if new cells are
added to the cluster or if some cells become temporarily unavailable. Multi-
agent RL (MARL) systems, such as those described by [6] and [7], in which each
agent acts upon a single cell, perform better in terms of knowledge transfer.
In [8] a combination between multi-agent systems and single distributed agents
is introduced. However, MARL scenarios are difficult to train because it is
necessary to also handle the interaction between individual agents. In [9] a
fuzzy system is included as a continuous/discrete converter before an RL agent
based on tabular records. Nowadays, there are more efficient ways to handle
continuous states, e.g., using neural networks. In some cases, as in [8], the
action of the agent produces the final parameter value to apply. However, RL
techniques tend to work better in an incremental fashion, in which the
parameter is changed iteratively in small steps, limiting the negative impact
of inaccurate reward estimations. A formulation for learning a policy for RET
optimization completely offline from real-world network data is successfully
applied in [10]. However, the performance of off-policy learning is highly
sensitive to the quality and variability of data. In [11] a method is proposed
based on fuzzy logic combined with a neural network that considers the impact
on neighboring cells. In this study, the observation space is narrow, limiting
the proposal to the ideal homogeneous network where it is trained. This paper
introduces an alternative RL approach that addresses all the issues mentioned
above.

The rest of the paper is organized as follows. Section 2 describes the basic
concepts of RL. Section 3 reveals the proposed approach for parameter
optimization based on RL. Section 4 describes the methodology used to generate
the results. Results are presented and discussed in Section 5. Concluding
remarks are summarized in Section 6.

##  2 Reinforcement Learning Overview

RL is a machine learning discipline in which an agent accumulates knowledge
about the dynamics of an environment through interactions, with the overall
goal of maximizing some notion of cumulative reward. The focus is on finding a
balance between the exploration of uncharted territory and the exploitation of
current knowledge. Basic RL problems are modeled as a Markov decision process,
in which an RL agent (e.g., a network optimizer) interacts with its
environment (e.g., the wireless network) in discrete time steps. At each time
$t$, the agent receives an observation, which includes the reward $R_{t}$ and
the state $S_{t}$ [12]. Depending on the current state $S_{t}$, the agent
chooses an action $A_{t}$ from the set of available actions (e.g., parameter
changes), which is subsequently applied to the environment. The environment
then transitions to a new state $S_{t+1}$ and provides a reward $R_{t+1}$
associated with the transition. The goal of an RL agent is to maximize current
and future rewards. The selection of actions by the agent is modeled as a map
called policy. The policy map gives the probability of selecting an action
$A_{t}$ when the environment is in state $S_{t}$. An example of a policy is
epsilon-greedy [12], where the action with the highest expected reward is
selected with probability $1-\epsilon$, and a random action is selected with
probability $\epsilon$, which is the exploration rate. The concept of discount
rate is introduced in many fora to consider the relative impact of future
rewards. A discount rate of zero is assumed in this paper, due to the nature
of the problem to resolve, in which higher future rewards at the expense of
negative or sub-optimal immediate rewards are not acceptable. This research
uses an off-policy RL method called Q-learning [12].

Figure 1: Expected performance evolution of an RL agent and an expert system.

##  3 Proposed Optimization Method Based on Reinforcement Learning

###  3.1 Problem Formulation

This paper focuses on cellular network optimization problems in which
modifying a network parameter in a single cell affects not only the
performance of that particular cell but also the performance of the
surrounding cells. The study focuses on the case of the RET parameter.

###  3.2 Proposed solution

Policies remain constant in expert systems, which means policies are not
improved as expert systems interact with the environment. In contrast, while
an RL agent needs to explore the environment to learn, it will eventually
surpass the performance of any agent that is defined by an expert, as shown in
Fig. 1. An offline agent initialization phase can avoid both the initial
training phase in the network to optimize, and the corresponding risk of
network degradation.

This paper proposes a MARL approach with one agent per cell, where all agents
share a common policy. As a consequence, any lessons learned based on rewards
obtained from one cell are immediately available in the common policy for the
rest of the cells. The agents implement Q-learning with a Deep Neural Network
(DNN) and Experience Replay (ER) [13]. A common policy is obtained by making
the agents share the same DNN, which is trained with the experiences collected
by all cells. The proposed approach can also be perceived as using multiple
instances (one per cell) of a unique RL agent. This conceptual vision is
illustrated in Fig. 2, where the single RL agent periodically updates the
policy based on the observations collected by the multiple instances. The
multiple instances receive the new policy after every update.

Figure 2: Multiple instances of a unique RL agent.

A static network simulator is used to pre-train the DNN during the
initialization phase, as proposed in [14]. A simulator can provide enough
training data variability by just sweeping the required ranges of network
configuration parameters. Agents with this same DNN are used to interact with
the network to optimize once pre-trained.

The agents steer the cell parameters toward the optimal global solution thanks
to suggesting small incremental changes. The use of small incremental changes
limits the negative impact caused by errors in reward estimations learned from
the training network simulations.

In order to control the interaction between agents, they have visibility of
the cell and its surrounding cells in the definition of state and reward.
Below, the proposed definitions of state, reward, and action are presented for
the particular case of RET optimization of non-active antennas.

####  3.2.1 State

In this study, the following configuration parameters and Key Performance
Indicators (KPI) have been selected to determine the state of a cell in the RL
scheme at every iteration period $t$:

  * •

Configuration parameters: antenna height, RET, mechanical tilt, carrier
frequency, and average distance to the five closest eNodeBs.

  * •

KPIs synthesized from Cell Traffic Recordings (CTR) as defined in [1]: cell
overshooting, useless high-level cell overlapping, and bad coverage. The state
also includes congestion level, average congestion at the closest cells
weighted by their overlapping factor with respect to the studied cell, and an
interference indicator, computed as the ratio of traffic in which the RSRP of
the second strongest neighbor is higher than a threshold with respect to the
best server. The overlapping factor between a first cell and a second cell can
be obtained as the periodicity in which both cells are reported simultaneously
by the same UE in CTR, provided that the first cell is the serving cell.

####  3.2.2 Reward

The reward at instant $t+1$ after a cell parameter update is defined as the
relative performance gain after the change, i.e.,

$R_{t+1}=1000\cdot\frac{RM_{t+1}-RM_{t}}{RM_{t}},$ (1)

where $RM_{t+1}$ is the reward metric at instant $t+1$ (right after the
parameter update), and $RM_{t}$ is the reward metric at instant $t$ (right
before the parameter update). This relative definition of the reward improves
learning compared to a definition based on absolute values since positive
rewards imply a gain, while negative rewards imply a loss. The factor 1000
moves the reward values into a scale that facilitates convergence when using a
learning rate of 0.001. The reward metric at instant $t$ is computed as

$RM_{t}=1+0.5\cdot\left(GT_{t}+GT_{t}^{neigh}-CR_{t}-CR_{t}^{neigh}\right),$
(2)

where $GT_{t}$ and $CR_{t}$ are the good traffic and the congestion rate at
the cell at instant $t$, respectively, and $GT_{t}^{neigh}$ and
$CR_{t}^{neigh}$ are the average good traffic and congestion rate measured at
instant $t$ at the closest neighboring cells, weighted by their overlapping
factor with respect to the studied cell. Good traffic is defined as the ratio
of traffic with good coverage and good quality with respect to the total
traffic. Good coverage means having reference signal received power (RSRP)
over a predefined threshold. Good quality means having DL SINR over a
predefined threshold.

Figure 3: Topology of the scenario used during the training phase.

####  3.2.3 Action

This optimization process follows an iterative approach, in which the
parameters are not updated with the final value in one single step, but only
with a relative increment with respect to the current value. Three actions are
possible: keep the current value, increase it by a fixed amount, or decrease
it by a fixed amount. A fixed increment of one degree is used in this study. A
small increment limits the reward estimation error and the negative impact in
case of a wrong decision.

##  4 Validation Methodology

###  4.1 Simulator

A proprietary static Monte Carlo simulator has been used to train the agents
during the pre-training phase. A large variety of scenarios is considered
during this initial phase. In this phase, changes are applied to the training
networks, avoiding impacting the test network, which plays the role of the
network to optimize. In order to test the performance of the pre-trained
agents, the test network has also been simulated, although its topology is
different from the one used in the training phase. The regular topology with
two cell rings shown in Fig. 3 has been used during the training phase,
consisting of 19 sites with three cells per site (i.e., 57 cells). Users are
generated following a uniform spatial distribution in the network area. All
cells operate on a common single-carrier frequency. The test phase has been
carried out in a similar but larger regular scenario, which consists of five
rings containing 91 sites (i.e., 273 cells), as displayed in Fig. 4. Table 1
lists the most relevant network simulation parameters used during the training
and test phases. Note that U(A, B) is a discrete uniform distribution in the
range [A, B] with step 1. No exploration is carried out during the test phase.

Figure 4: Topology of the scenario used during the test phase. Table 1: Main Simulation Parameters at the Training and Test Phases. Parameter | Value
---|---
Electrical tilt for optimized cells | U(0, 15) deg
Electrical tilt for non-optimized cells | U(4, 6) deg
Mechanical tilt range | U(0, 4) deg
Antenna height | U(16, 30) m
Inter-site distance | U(1000, 2500) m
Carrier frequencies | 0.7, 1.8, 2.1 & 2.6 GHz
Offered traffic (average per cell) | U(4, 11) Mbps
Bandwidth | 20 MHz
Good coverage threshold | -108 dBm
Good quality threshold | 3 dB

###  4.2 Optimization Process

The agents are initially pre-trained offline with 500 independent episodes.
Each episode simulates a 20-step optimization campaign, in which the RET of
the 21 cells in the inner ring are considered for optimization (see Fig. 3).
The remaining 36 cells in the outer ring keep constant RETs throughout the
same episode. These parameters are randomly reset at the beginning of every
episode: inter-site distance, mechanical tilt, RET, antenna height, frequency,
and offered traffic volume. Mechanical tilt, RET, and antenna height can be
different for different cells. During each step of each episode, one of these
three possible actions is applied to the RETs of the optimized cells: keep the
same value, increase by one degree, or decrease by one degree. Fig. 5 shows
the evolution of the loss and the reward during the training phase averaged
over a 100-step window so that it is possible to appreciate the long-term
trend. The loss function is the mean squared reward estimation error. After
the initial learning phase, the agent is tested on 300 additional independent
episodes of a larger scenario with a regular topology where 183 out of 273
cells are optimized (see Fig. 4). To quantify the benefit of including
neighbor information, a modified version of the RL agents that excludes the
neighbor information from reward and state has also been implemented, pre-
trained, and tested on the same 300 episodes. Additionally, the algorithm in
[1], henceforth referred to as the expert system, has also been implemented to
provide results on the same 300 episodes for benchmarking purposes.

Figure 5: Average loss and reward during the pre-training phase.

###  4.3 Evaluation Metric

The metrics to measure the gain per episode are good traffic improvement, good
coverage traffic improvement, good quality traffic improvement, and congestion
improvement. The gains per episode are calculated as the relative improvement
at the end of an episode compared to the baseline value at the beginning of
the same episode.

##  5 Simulation Results

Figure 6: Evolution of average good traffic improvement in a single episode.
Figure 7: Evolution of average good coverage traffic improvement in a single
episode. Figure 8: Evolution of average good quality traffic improvement in a
single episode.

###  5.1 Single Episode

Fig. 6 compares the progress of the average good traffic improvement per step
when optimizing the same particular episode with pre-trained RL agents with
and without neighbor information, and with the expert system. The solid lines
represent the average values using static agents, i.e., agents that do not
continue training after the initial pre-training phase. The blue dotted line
represents the values obtained with the RL scheme that includes neighbor
information and an agent that continues training while optimizing the test
network. Step zero represents the baseline configuration. Notice how the RL
agents enable greater good traffic improvement from the first step to the last
step of the episode, resulting in significantly better traffic improvements
than the expert system, particularly when the RL agents utilize neighbor
information. This indicates that offline pre-training mitigated the effects of
the initial unstable behavior of RL systems. An additional gain is obtained
when the agent is configured to continue learning from the network to
optimize. This comparison repeats in Figures 7 and 8 for the coverage and
quality improvements, respectively. A higher quality improvement is also
appreciable from the first step when using the RL agents as compared with the
expert system. However, the consideration of neighbor information is not so
significant as when evaluating the coverage improvement. Fig. 9 shows how the
congestion improvement reaches 100% at the third step with all methods. A
congestion improvement of 100% means that the cells have reduced the
congestion to zero. Note that these results have been obtained from a single
episode and are presented solely as an illustrative example. Consequently,
they are insufficient to draw definitive conclusions. To address this, the
experiment has been repeated with a greater number of episodes to ensure
statistical significance.

Figure 9: Evolution of average congestion improvement in a single episode.
Figure 10: Evolution of good traffic improvement over 300 independent
episodes. Figure 11: Evolution of good coverage traffic improvement over 300
independent episodes. Figure 12: Evolution of good quality traffic improvement
over 300 independent episodes. Table 2: Average Final Gains Over 300
Independent Episodes.

KPI |  ES |  RLEN |  RLIN |  RLIN+
---|---|---|---|---
Good traffic improvement (%) |  16.3 |  27.6 |  31.6 |  32.4
Good coverage traffic improvement (%) |  13.3 |  14.7 |  17.6 |  18.4
Good quality traffic improvement (%) |  14.0 |  24.8 |  27.4 |  27.7

ES: Expert system
RLEN: RL scheme excluding neighbor information
RLIN: RL scheme including neighbor information
RLIN+: RL scheme including neighbor information and continuing to train

###  5.2 Multiple Episodes

The pre-trained RL agents and the expert system have been tested on 300
additional independent episodes, as described in Section 4.2. The results for
the multiple episode case are shown in Figures 10 to 12 as the evolution of
good traffic improvement, coverage improvement, and quality improvement,
respectively, with the number of steps. In this case, the improvements per
step are computed as the average values over the 300 episodes, but additional
information is graphically provided about the distribution of the gain in
terms of the first and third quartiles. The solid lines represent the average
values and the shaded areas are the confidence intervals delimited by the
first and third quartiles using static agents. The blue dotted line represents
the average values obtained with the RL scheme that includes neighbor
information and an agent that continues training. The findings are similar to
the single-episode case:

  * •

Our RL agent provides significantly higher good traffic improvement than the
expert system.

  * •

An additional gain is provided by the RL agent when the state and reward
definitions include information from neighboring cells, which is especially
appreciated in the coverage improvement.

  * •

Even higher gain can be achieved if the agent continues to learn from the
optimized network.

A comparison of the average gains provided by the different studied schemes
after the 20 steps of every episode is available in Table 2. On average, the
proposed RL approach with static agents yields a 94.5% higher gain than the
expert system in terms of good traffic improvement and a 70.0% higher gain
than the same approach when excluding neighbor information. Another relevant
finding is that the proposed RL approach with neighbor information yields a
19.4% higher coverage gain than the same approach without neighbor
information, although the quality gain is only 11.6% higher. A possible reason
for the high quality improvement of the RL scheme that excludes neighbor
information is that the quality KPI depends on the SINR, which includes
information on interference from neighboring cells. The RL approach including
neighbor information and configured for continuous learning during network
optimization achieves the highest performance, resulting in an additional 2.3%
improvement in good traffic compared to using a static agent. As illustrated
in the dotted line of Fig. 10, the optimal performance is not achieved at the
last step, but at the 14th step, with a 34.0% good traffic improvement, i.e.,
7.3% higher gain than using a static agent. This behavior implies that the
learning process eventually starts to deteriorate over time. On the one hand,
the good coverage traffic is still increasing (dotted line in Fig. 11), but on
the other hand, the good quality is still decreasing (dotted line in Fig. 12).
By the 15th step, the majority of cells have already achieved optimal RET
values, and going forward, the expected behavior of the agent is to keep the
configuration unchanged. However, agents may propose oscillations around the
optimal RET values. This phase can distort the training of the model,
potentially causing it to forget previously acquired knowledge. One potential
solution is to exclude training data from cells that have already reached an
optimal value. This can be accomplished by detecting consecutive ’keep’
actions, oscillating actions, or a decline in reward growth.

Figure 13: Box plots representing the distribution of the number of steps to reach congestion mitigation over 300 independent episodes. Table 3: Average Steps to Mitigate Congestion. Considered episodes | ES | RLEN | RLIN |  RLIN+
---|---|---|---|---
All 300 episodes | 5.1 | 3.1 | 3.0 |  3.0
Episodes with initial congestion | 7.7 | 4.7 | 4.6 |  4.5

Congestion improvement consistently reaches 100% with all methods, which means
that all agents can mitigate the congestion. For this reason, rather than
looking at the evolution of the average congestion gain, we consider it more
insightful to compare the number of steps each method needs to achieve
congestion mitigation. Fig. 13 displays the distribution of the required steps
to reach congestion mitigation for each optimization approach using box plots.
The median, represented by the red line, is consistent at two steps across all
methods. Table 3 provides the average number of steps needed to achieve
congestion mitigation for the different methods. It is noteworthy that the RL-
based methods not only deliver significantly improved traffic performance but
also exhibit faster congestion mitigation. On average, the expert system
requires two more steps compared to the RL methods. However, when focusing on
episodes with initial congestion, the RL methods require three fewer steps on
average compared to the expert system.

##  6 Conclusion

A new approach has been proposed to optimize cell parameters in a wireless
network based on MARL, which considers the impact on the cell under study, but
also on its neighboring cells. A unique common policy facilitates the
immediate sharing of exploration outcomes with the rest of the cells, thereby
accelerating the learning phase. It also facilitates knowledge sharing when
adding new cells. By employing a simulator for offline pre-training, the
network being optimized is protected from potential damage during the initial
stages of the learning process, where an untrained agent would exhibit erratic
behavior. Simulation results show a significantly higher gain, as compared to
an existing expert system in terms of good traffic, which implies coverage
increase and interference reduction. Although both methods can mitigate the
congestion, the proposed approach does it in fewer steps. Considering neighbor
information in the state and reward guarantees significant extra gain,
especially in terms of coverage. Finally, after connecting the RL agent to the
test network, it can continue learning while optimizing, eventually achieving
even higher performance levels. Finding a method to detect the most suitable
step to stop learning from a cell is recommended to avoid overfitting, which
leads the model to forget previously acquired knowledge.

## References

  * [1] V. Buenestado, M. Toril, S. Luna-Ramírez, J. M. Ruiz-Avilés, and A. Mendo, “Self-tuning of remote electrical tilts based on call traces for coverage and capacity optimization in LTE,” _IEEE Transactions on Vehicular Technology_ , vol. 66, no. 5, pp. 4315–4326, 2017.
  * [2] M. Rhazzaf and T. Masrour, “Smart autonomous vehicles in high dimensional warehouses using deep reinforcement learning approach.” _Engineering Letters_ , vol. 29, no. 1, 2021.
  * [3] K. Hukmani, S. Kolekar, and S. Vobugari, “Solving twisty puzzles using parallel q-learning.” _Engineering Letters_ , vol. 29, no. 4, 2021.
  * [4] Y. Li and Y. Chen, “Enhancing a stock timing strategy by reinforcement learning.” _IAENG International Journal of Computer Science_ , vol. 48, no. 4, 2021.
  * [5] F. B. Mismar, J. Choi, and B. L. Evans, “A framework for automated cellular network tuning with reinforcement learning,” _IEEE Transactions on Communications_ , vol. 67, no. 10, pp. 7152–7167, 2019.
  * [6] W. Guo, S. Wang, Y. Wu, J. Rigelsford, X. Chu, and T. O’Farrell, “Spectral- and energy-efficient antenna tilting in a hetnet using reinforcement learning,” in _2013 IEEE Wireless Communications and Networking Conference (WCNC)_ , 2013, pp. 767–772.
  * [7] G. Koudouridis, “Method and apparatus of communications, International Patent WO2012072445A1,” World Patent WO2 012 072 445A1, Jun. 7, 2012. [Online]. Available: https://patents.google.com/patent/WO2012072445A1
  * [8] E. Balevi and J. G. Andrews, “Online antenna tuning in heterogeneous cellular networks with deep reinforcement learning,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 5, no. 4, pp. 1113–1124, 2019\.
  * [9] R. Razavi, S. Klein, and H. Claussen, “Self-optimization of capacity and coverage in LTE networks using a fuzzy reinforcement learning approach,” in _21st Annual IEEE International Symposium on Personal, Indoor and Mobile Radio Communications_ , 2010, pp. 1865–1870.
  * [10] F. Vannella, J. Jeong, and A. Proutiere, “Off-policy learning for remote electrical tilt optimization,” in _2020 IEEE 92nd Vehicular Technology Conference (VTC2020-Fall)_ , 2020, pp. 1–5.
  * [11] S. Fan, H. Tian, and C. Sengul, “Self-optimization of coverage and capacity based on a fuzzy neural network with cooperative reinforcement learning,” _EURASIP Journal on Wireless Communications and Networking_ , vol. 2014, no. 1, pp. 1–14, 2014.
  * [12] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_ , 2nd ed. The MIT Press, 2018.
  * [13] L.-J. Lin, “Self-improving reactive agents based on reinforcement learning, planning and teaching,” _Machine Learning_ , vol. 8, no. 3-4, pp. 293–321, 1992.
  * [14] Ericsson AB, “AI: Enhancing customer experience in a complex 5G world,” pp. 26–28, June 2021, [Online]. Available: ericsson.com/4a03c2/assets/local/mobility-report/documents/2021/june-2021-ericsson-mobility-report.pdf.

Adriano Mendo received his M.Sc. in Telecommunication Engineering from Malaga
University, Spain, in 2004. Since 2004, he has been a Researcher with Optimi
Corporation and joined Ericsson in 2010. He has authored a few publications
and patents. His current research interests include self-organizing networks,
radio resource management, and AI. Jose Outes Carnero is a Research Specialist
at Ericsson in Malaga, Spain. He received his M.Sc. in Telecommunication
Engineering from Malaga University, Spain, in 2000. He received his Ph.D. in
Electrical and Electronic Engineering from Aalborg University, Denmark, in
2004. His current research interests include network design and optimization,
and AI. Yak Ng Molina is a Program Manager at Ericsson in Malaga, Spain. He
received his M.Sc. in Telecommunication Engineering and his Ph.D. in
Electrical and Electronic Engineering from Malaga University, Spain, in 2008
and 2013, respectively. His current research interests include network design
and optimization, digital twins, energy management, and AI. Juan Ramiro Moreno
is heading the Network Design and Optimization Innovation team at Ericsson. He
holds a Telecom Engineering degree from Malaga University, a Ph.D. in
Electrical and Electronic Engineering from Aalborg University, an MBA from San
Telmo Business School, and an Executive Degree in Big Data & Business
Analytics from EOI. He is also an Honorary Professor at Malaga University and
co-author of a book on Self-Organizing Networks.


# Pseudo Label Is Better Than Human Label

###### Abstract

State-of-the-art automatic speech recognition (ASR) systems are trained with
tens of thousands of hours of labeled speech data. Human transcription is
expensive and time consuming. Factors such as the quality and consistency of
the transcription can greatly affect the performance of the ASR models trained
with these data. In this paper, we show that we can train a strong teacher
model to produce high quality pseudo labels by utilizing recent self-
supervised and semi-supervised learning techniques. Specifically, we use JUST
(Joint Unsupervised/Supervised Training) and iterative noisy student teacher
training to train a 600 million parameter bi-directional teacher model. This
model achieved 4.0% word error rate (WER) on a voice search task, 11.1%
relatively better than a baseline. We further show that by using this strong
teacher model to generate high-quality pseudo labels for training, we can
achieve 13.6% relative WER reduction (5.9% to 5.1%) for a streaming model
compared to using human labels.

Index Terms: speech recognition, pseudo label, semi-supervised, self-
supervised

##  1 Introduction

The success of end-to-end (E2E) speech recognition models [1, 2, 3] are highly
dependent on having a large amount of high-quality transcribed speech data.
However, it is expensive and difficult to obtain and maintain the high-quality
human transcriptions, restricting the development of automatic speech
recognition (ASR). To relieve the dependence on large amounts of human-labeled
data, self-supervised and semi-supervised learning techniques are heavily
explored recently by leveraging on large-scale unlabeled speech data.

Self-supervised learning techniques [4, 5, 6, 7, 8, 9] have been shown to be
effective for pre-training ASR models and achieved impressive performance for
speech recognition tasks by leveraging large-scale unlabeled speech data. In
general, training is done in a 2-stage pretrain-finetune fashion where the
models are first pre-trained using self-supervised learning using a large
amount of unlabeled data and then fine-tuned using supervised data. For
example, it was shown that Wav2vec 2.0 [5] pre-training can greatly reduce the
amount of labeled data to just 100 hours and still achieve a reasonable word
error rate (WER) performance of $4.0\%$ on the LibriSpeech test-other set
[10]. However, this is still worse than the model trained with all the
training data. On the other hand, W2v-BERT [6] pre-training can achieve 2.8%
WER on the testother set with using the full 960 hours of training data.

Another recent direction of research to leverage large-scale unlabeled data is
semi-supervised learning, including noisy student training (NST) [11, 12] and
consistency-based methods [13, 14]. NST [12] achieves the state-of-the-art
(SoTA) performance on LibriSpeech when there are limited supervised speech
data with transcriptions. YouTube model mostly relies on pseudo labels by NST
[15]. To improve the performance further, [16, 17] utilize a combination of
self-supervised and semi-supervised learning as well as adding a third NST
stage after the 2-stage pretrain-finetune scheme. Our previous paper [17]
demonstrates that the new approach reduces the requirement of human labels
significantly, and matches or improves the SoTA performance with only $3\%$
human labels on the Google voice search task. However, it is still unclear
whether a better performance can be achieved when more human labels are
available. Otherwise, how much human labels should be mixed with pseudo labels
to achieve the best performance?

In this paper, we will describe a multi-stage training strategy to train a
strong teacher model to produce high-quality pseudo labels using a combination
of self-supervised and semi-supervised learning, which has SoTA performance on
voice search. We compare the effectiveness of human labels and pseudo labels
by training the same model with corresponding transcriptions respectively. Our
results show that the model trained with pseudo labels has significantly
better WER of $5.1\%$ on voice search compared to that using human labels
($5.9\%$). This shows that the pseudo labels from a strong teacher have better
quality than human labels for large-scale ASR model training. We also
demonstrate that once we have a strong teacher model, subsequent training can
be done entirely using only the pseudo labels.

The remainder of the paper is organized as follows. Section 2 describes the
network architecture and the training methods used to train our strong teacher
models for pseudo labeling. Section 3 shows the experimental results on the
voice search and medium-form tasks. Section 4 presents some ablation studies
and discussions.

##  2 Methods

###  2.1 Model Architecture

We consider two types of models, one for the bi-directional teacher model and
another for the streaming student model. Both models use the input feature
vector of size $528$, consisting of $4$ contiguous frames of $128$-dimension
log-mel features [3] sub-sampled by a factor of $3$ and a one-hot domain-id
vector of size $16$. The outputs correspond to $4,096$ word pieces [18].

The architecture and training procedure of the teacher model is similar to
Conformer XL model ($0.6$B) in [19]. The audio encoder has $24$ Conformer
blocks [20] with model dimension $1024$. The convolution kernel size is $15$
and the self-attention layer consists of $8$ heads with $64$ left and right
context length. The decoder consists of a 2-layer LSTM [21] label encoder with
$2048$ units projected down to $640$ output units, and a joint network with a
single feed-forward layer with $640$ units. The total number of weights is
$0.6$B. The model is trained by minimizing the RNN-T loss [22].

The student model uses cascaded encoders for unifying streaming and non-
streaming ASR [23]. The 1st-pass causal encoders has $7$ Conformer blocks with
model dimension $512$. The 2nd-pass cascaded encoder has $10$ additional non-
causal conformer layers that process a total of 900 milliseconds of future
audio. Both causal and non-causal encoders feed into a shared hybrid
autoregressive transducer (HAT) decoder [24]. This model has a total of number
$155$M weights.

We use a combination of self-supervised and semi-supervised learning to train
a strong teacher model. Specifically, we combine the RNN-T [22] and W2v-BERT
[6] losses using a hydra architecture to improve the teacher performance. We
further improve the teacher model by applying multiple rounds of noisy student
training [11]. We will describe these teacher training improvements in the
following sub-sections.

###  2.2 JUST Hydra

Previous work has shown that combining supervised and self-supervised learning
in a multi-task learning fashion can be beneficial [17, 25]. In this work, we
use joint unsupervised/supervised training (JUST) [25] to combine the
supervised RNNT loss and the self-supervised W2v-BERT loss. JUST simplifies
the 2-stage pretrain-finetune scheme into a 1-stage joint training. The
representations from the last encoding layers are shared by both the RNNT
joint network and W2v-BERT prediction layer. A recent study [26] shows that
the last 3 layers in Wav2vec2.0 [5] are specialized to reconstructing masked
inputs. This suggests that the last 3 layers should not be shared in JUST.
Motivated by this work, we modify JUST to have separate layers for the for the
last 3 Conformer layers, while most of lower layers are still shared, as shown
in Fig. 1.

Figure 1: JUST Hydra – a hydra architecture with separate layers for the last
3 Conformer layers for joint unsupervised and supervised training.

We call it JUST hydra (JUST with Hydra heads). JUST hydra uses both labeled
and unlabeled data. The RNNT loss uses only the labeled data while the
W2v-BERT self-supervised loss uses both the labeled and unlabeled data.

###  2.3 Noisy student training (NST)

In this study, we use noisy student training (NST) [11] in the same way as our
previous study [17]. When generating pseudo labels, we use a confidence
estimation module (CEM) [27] to filter out low confidence utterances to avoid
training with erroneous labels.

Figure 2: Noisy student training (NST).

The filtered pseudo labels are then mixed with human-labeled data to form the
training data to learn the student model using the RNNT loss, as shown in Fig.
2.

We combine NST and JUST hydra as follows to progressively improve the teacher
model:

  1. 1.

Use JUST hydra to train a large ASR model using both the labeled and unlabeled
data.

  2. 2.

Using the above model as both the teacher and student, perform self-training
to produce the 1st generation NST model.

  3. 3.

Using the previous NST model as the teacher and initial JUST hydra model as
the student, perform self-training to produce the 2nd generation NST.

  4. 4.

Repeat NST until convergence.

Once we obtain a strong teacher model, the student models are trained in a
semi-supervised learning fashion using the pseudo labels generated by the
teacher model.

##  3 Experiments

###  3.1 Data

We use a large multi-domain (MD) English dataset [28] for training, which
consists of utterances from multiple domains, such as voice search (VS),
medium-form (MF) and YouTube (YT). All the data are anonymized and hand-
transcribed following Google AI principles [29]. VS is mostly voice command.
MF is mostly natural conversation. The YT labels are generated from YouTube
video transcription with confidence filter [30]. This forms a total of $575$k
hours of data. In addition, we also use $1.15$M hours of unsupervised data for
voice search (VSunsup) and medium-form (MFunsup). We also used another set of
unsupervised voice data (VSunsup-new) with 150k hours of data collected at a
later time to study the generalization of our teacher models to new data. A
summary of the training data is given in Table 1.

JUST hydra uses all the data. NST uses supervised data or all the data. The
average utterance lengths for the VS, MF and YT domains are 4.6 seconds, 10.4
seconds and 5 seconds, respectively. For evaluation, we report the WER
performance of the voice search (VS) and medium-form (MF) tasks.

Table 1: Overview of training data sets. MD denotes all the supervised data (VS \+ MF \+ en_x \+ YT, $575$k hrs). MDunsup denotes the unsupervised data (VSunsup \+ MFunsup, $1.15$M hrs). en_x includes both VS and MF. Data set | Label | Hours
---|---|---
Voice search (VS) | human |  $27$k
Medium-form (MF) | human |  $26$k
Dialect (en_x) | human |  $80$k
Youtube (YT) | semi |  $440$k
Voice search (VSunsup) | pseudo |  $900$k
Medium-form (MFunsup) | pseudo |  $250$k
Voice search (VSunsup-new) | pseudo |  $150$k

###  3.2 Results

####  3.2.1 Bi-directional teacher model

The teacher model is a $0.6$B-parameter bi-directional RNNT model. $B0$ in
Table 2 is the supervised baseline which is trained with MD using the RNNT
loss [31]. E1 is trained using JUST Hydra, where the RNNT loss uses MD, and
the W2v-BERT loss uses both MD and MDunsup. It improves the VS WER from $4.5$
to $4.3$. On top of E1, we conducted multiple-generation NST to train E2, E3
and E4 using only the MD data with pseudo labels generated by a teacher model.
E2 is self-training, where both the teacher and student are based on E1.
Subsequent training uses the previous iteration model as teacher and the
student model is initialized with E1. It is clear from Table 2 that multiple-
generation NST progressively improved the quality of the pseudo labels to
yield better teacher models. We obtained the best teacher model after two NST
iterations, achieving 4.0 WER on VS and 2.0 on MF. The best performance
obtained using human labels is 4.3 and 2.3 on VS and MF, respectively. This
shows that using pseudo labels is better than using human labels. In the
following section, we will use this teacher model to train production-style
streaming models.

Table 2: WERs of large models for supervised baseline, JUST hydra and NST. Algorithms | Data | WER ($\%$)
---|---|---
VS | MF
B0 (Supervised) | MD | ${4.5}$ | ${2.4}$
E1 (JUST hydra) |  MD \+ MDunsup | ${4.3}$ | $2.3$
E2 (1st Gen NST, teacher=E1) | MD | ${4.1}$ | ${2.1}$
E3 (2nd Gen NST, teacher=E2) | MD | 4.0 | 2.0
E4 (3rd Gen NST, teacher=E3) | MD | 4.0 | 2.0

####  3.2.2 Streaming student model

The student model is a $155$M-parameter cascaded RNNT model. In Table 3, we
compare two baseline models trained with MD data using human labels. The
B0stream baseline trained with the RNNT loss achieved 5.9% and 3.2% WER on VS
and MD, respectively. Edit-based minimum Bayes risk (EMBR) [32] further
improved the performance to 5.5% and 2.9%. When we perform NST on top of the
B0stream model, using pseudo labels generated by the strong teacher, E2, from
the previous section, we can improve the SoTA performance on VS and MF by a
significant margin (5.1% and 2.6%). Note that B0stream and E1stream have the
same architecture and training hyper-parameters. The only difference is that
the former is trained with human labels while the latter with pseudo labels.
This shows that using pseudo labels is better than using human labels again.
We conducted user A/B tests for B0stream and E1stream. E1stream won by 30:8
with ${<.1\%}$ p-value (${99.9\%}$ confidence).

EMBR training on top of E1stream (E2stream) did not yield further improvement.
In fact, we observed degradation on VS.

Table 3: WERs comparison for streaming models using supervised and NST training. Algorithms | WER
---|---
VS | MF
B0stream (Supervised) | ${5.9}$ | ${3.2}$
B1stream (B0stream \+ EMBR) | ${5.5}$ | ${2.9}$
E1stream (B0stream \+ NST) | 5.1 | 2.6
E2stream (E1stream \+ EMBR) | ${5.5}$ | ${2.6}$

##  4 Discussion

###  4.1 JUST hydra vs W2v-BERT

Table 4: WERs for supervised baseline and W2v-BERT learning. Algorithms | Data | WER ($\%$)
---|---|---
VS | MF
B0 (Supervised) | MD | ${4.5}$ | ${2.4}$
E5 (W2v-BERT) |  MD \+ MDunsup | ${4.5}$ | ${2.5}$
E6 (JUST) |  MD \+ MDunsup | ${4.5}$ | ${2.5}$
E1 (JUST hydra) |  MD \+ MDunsup | 4.3 | 2.3

W2v-BERT [6] for self-supervised learning has been shown to achieve state-of-
the-art performance on LibriSpeech [10]. However, we found that the vanilla
W2v-BERT (E5) is worse than the baseline (B0), as shown in Table 4. To improve
the performance, we used JUST hydra, as described in Section 2.2. Instead of
using W2v-BERT self-supervised learning for pre-training, we use JUST [25] to
combine RNNT and W2v-BERT losses in a multi-task learning setting. However,
using JUST alone (E6) did not improve the performance. With a hydra
architecture, we use a separate 3-layer Conformer block to learn the
representation for W2v-BERT. As a result, we were able to improve the WER on
VS from $4.5$ to $4.3$ (E1).

As previously described in Section 2.1, the input log-mel features are
augmented with a 16-dimensional one-hot domain embedding vector to better
handle multiple domains. Since all the feature frames within an utterance are
appended with the same domain embedding vector, it does not provide useful
information for the W2v-BERT contrastive loss when the negative samples are
extracted from the same utterance (used in this paper). When the negative
samples come from different utterances within a batch, the domain embedding is
too easy to predict. In both cases, the domain information makes W2v-BERT
training less effective. JUST hydra (E1) helps to alleviate this problem
because jointly training with the RNNT loss can guide the model to learn
meaningful domain embedding from data. The Hydra architecture is optimal
multi-task architecture fully utilizing both W2v-BERT and RNNT losses.

###  4.2 Human label vs Pseudo label

Table 5: WERs with different amounts of human labels and pseudo labels. All models are trained with MD. Model | Human labels | WER ($\%$)
---|---|---
VS | MF
E2 (Teacher) | All | ${4.1}$ | ${2.1}$
E1 (Student) | ${4.3}$ | ${2.3}$
E8 (NST) | None | ${4.1}$ | ${2.1}$
E9 (NST) |  $330$ hrs | ${4.1}$ | ${2.1}$
E3 (NST, from Table 2) |  $1$k hrs | 4.0 | 2.0
E10 (NST) |  $2$k hrs | ${4.1}$ | ${2.1}$
E11 (NST) |  $4$k hrs | ${4.2}$ | ${2.1}$
E12 (NST) | All | ${4.3}$ | ${2.3}$

In Sections 3.2.1 and 3.2.2, we have shown that using pseudo labels for
training is better than using human labels. Here, we investigate whether
mixing a small fraction of human labels with pseudo labels could be
beneficial. In Table 5, we compared mixing $330$, $1$k, $2$k and $4$k hours of
human labels when training the teacher model using NST. We observe that using
$1$k hours of human labels achieve consistent improvements on both the VS and
MF tasks. WERs improved from 4.1% to 4.0% on VS, and from 2.1% to 2.0% on MF.
The last mixing experiment (E12) uses only human labels, so WERs are same to
the student (E1).

On the other hands, mixing either none or $1$k hours labels with pseudo labels
to train the streaming model achieved the same performance of 5.1% and 2.6%,
as shown in Table 3 (E1stream). Therefore, it is sufficient to rely 100% on
pseudo labels once we have a strong teacher model. However, in practice, it
may still be necessary to include a small amount of human labeled data.
Completely relying on pseudo labels in multi-generation NST leads to
deterioration of pseudo label quality as the teacher may reinforce
confirmation bias in the pseudo labels.

###  4.3 Effect of initial student model

We investigated how student models with different WER performance affect the
final performance. In Table 6, we compared NST training starting from three
different initial student models. B1 is a poor starting model since it is
trained with only YT (out-of-domain) data. B0 is trained using MD data with
human labels. E1 is trained with JUST hydra. In these experiments, we used
100% pseudo labels for NST. As shown in Table 6, regardless of the initial
student model, they all converged to the same performance of 4.1% and 2.1%. We
found that with a strong teacher model, NST is not sensitive the initial
student model. As shown in Section 4.1, it is important to apply self-
supervised learning to train an initial teacher model. Once we have a high-
quality teacher, NST with pseudo labels is sufficient.

Table 6: WERs of NST with $100\%$ pseudo labels based on various students. Model | Data | WER ($\%$)
---|---|---
VS | MF
E2 (Teacher) |  | ${4.1}$ | ${2.1}$
B1 (Supervised) | YT | ${5.3}$ | ${4.0}$
B0 (Supervised) | MD | ${4.5}$ | ${2.4}$
E1 (JUST hydra) |  MD \+ MDunsup | ${4.3}$ | $2.3$
E13 (NST, teacher=B1) |  MD w/o labels | 4.1 | 2.1
E14 (NST, teacher=B0) |  MD w/o labels | 4.1 | 2.1
E15 (NST, teacher=E1) |  MD w/o labels | 4.1 | 2.1

###  4.4 NST on out-of-domain data

So far, we have conducted NST experiments by replacing human labels with
pseudo labels for the MD (VS \+ MF \+ en_x \+ YT). We show the teacher to be
able to generate better pseudo labels for the MD data. We investigated the
pseudo label quality on out-of-domain (OOD) data by the teacher model trained
on MD. We conducted ablation studies by performing NST on different OOD
unsupervised data, as shown in Table 7. We considered two unsupervised voice
search datasets (VSunsup and VSunsup-new). The former has about 900k hours of
data while the latter is a smaller (150k hours) but more recently collected
data to capture temporal domain shift. Finally, we also created another VS +
MTR dataset by artificially corrupting the clean utterances in VS using a room
simulator, adding varying degrees of noise and reverberation with an average
SNR of 12dB [33], a data augmentation technique used for multi-style training
(MTR).

We use NST to train both the large $0.6$B-parameter models (Large) and
$155$M-parameter streaming models (Stream) with $100\%$ pseudo labels. From
Table 7, we observe that NST training for the Large model is fairly robust to
the unsupervised data used. The E3 teacher model (trained on VS) works well
for other OOD data (VSunsup, VSunsup-new, VS + MTR) and achieve similar
performance of 4.1–4.2%. On the other hand, the Stream models performed
slightly worse on OOD unsupervised data (5.4–5.5%). This means that the Stream
models are sensitive to domain shift. Large bi-directional model is better for
unseen domain generalization. Nevertheless, the OOD streaming models are still
better than B0stream, the baseline model trained with human labels (5.9%).

Table 7: WERs of NST with various data for large and production models. Data | VS WER ($\%$)
---|---
Large | Stream
E3 (Teacher) | ${4.0}$
B0, B0stream (Student) | ${4.5}$ | ${5.9}$
VS | 4.1 | 5.1
VSunsup | 4.1 | $5.4$
VSunsup-new | ${4.2}$ | $5.5$
VS \+ MTR | 4.1 | ${5.5}$

###  4.5 Limitations and future work

This study demonstrates that a strong teacher can produce high-quality pseudo
labels for NST. Once we have a strong teacher, we need only $1$k hrs human
labels to produce the next generation best teacher model. However, human
labels are still needed to train a good initial teacher model, as JUST hydra
uses all human labels (MD) in Table 4. For languages that do not have
sufficient human labeled data, it remains a challenge and open problem to
train high quality initial teacher models using as little data as possible.

##  5 Conclusions

In this paper, we show that by using joint unsupervised and supervised
training with W2v-BERT and noisy student training, we can train a strong
teacher model that achieved 11.1% relative word error rate improvement
compared to a baseline model trained with human labeled data only. This
teacher can generate high-quality pseudo labels that are more consistent than
the human labels. Semi-supervised learning with the pseudo labels can achieve
13.6% relative word error rate improvement, compared to our best streaming
model for the voice search task.

##  6 Acknowledgements

We thank Françoise Beaufays, Zhehuai Chen, Chung-Cheng Chiu, Shefali Garg,
Mohammadreza Ghodsi, Bo Li, Pedro Moreno Mengibar, Ananya Misra, Bhuvana
Ramabhadran, Andrew Rosenberg, David Qiu, Tara Sainath, Hasim Sak, Nikhil
Siddhartha, Yonghui Wu, Yu Zhang for helpful discussions.

## References

  * [1] T. N. Sainath, Y. He, B. Li, A. Narayanan, R. Pang, A. Bruguier, S.-y. Chang, W. Li, R. Alvarez, Z. Chen _et al._ , “A streaming on-device end-to-end model surpassing server-side conventional model quality and latency,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 6059–6063.
  * [2] B. Li, A. Gulati, J. Yu, T. N. Sainath, C.-C. Chiu, A. Narayanan, S.-Y. Chang, R. Pang, Y. He, J. Qin _et al._ , “A better and faster end-to-end model for streaming asr,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 5634–5638.
  * [3] A. Narayanan, R. Prabhavalkar, C.-C. Chiu, D. Rybach, T. N. Sainath, and T. Strohman, “Recognizing long-form speech using streaming end-to-end models,” in _IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2019, pp. 920–927.
  * [4] S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” _arXiv preprint arXiv:1904.05862_ , 2019.
  * [5] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [6] Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu, “W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” _arXiv preprint arXiv:2108.06209_ , 2021.
  * [7] Z. Chen, Y. Zhang, A. Rosenberg, B. Ramabhadran, G. Wang, and P. Moreno, “Injecting text in self-supervised speech pretraining,” _arXiv preprint arXiv:2108.12226_ , 2021.
  * [8] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [9] C.-C. Chiu, J. Qin, Y. Zhang, J. Yu, and Y. Wu, “Self-supervised learning with random-projection quantizer for speech recognition,” _arXiv preprint arXiv:2202.01855_ , 2022.
  * [10] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in _IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2015, pp. 5206–5210.
  * [11] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le, “Self-training with noisy student improves imagenet classification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 10 687–10 698.
  * [12] D. S. Park, Y. Zhang, Y. Jia, W. Han, C.-C. Chiu, B. Li, Y. Wu, and Q. V. Le, “Improved noisy student training for automatic speech recognition,” _arXiv preprint arXiv:2005.09629_ , 2020.
  * [13] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 596–608, 2020.
  * [14] Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le, “Unsupervised data augmentation for consistency training,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 6256–6268, 2020.
  * [15] T. Doutre, W. Han, M. Ma, Z. Lu, C.-C. Chiu, R. Pang, A. Narayanan, A. Misra, Y. Zhang, and L. Cao, “Improving streaming automatic speech recognition with non-streaming model distillation on unsupervised data,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6558–6562.
  * [16] Y. Zhang, D. S. Park, W. Han, J. Qin, A. Gulati, J. Shor, A. Jansen, Y. Xu, Y. Huang, S. Wang _et al._ , “Bigssl: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition,” _arXiv preprint arXiv:2109.13226_ , 2021.
  * [17] D. Hwang, A. Misra, Z. Huo, N. Siddhartha, S. Garg, D. Qiu, K. C. Sim, T. Strohman, F. Beaufays, and Y. He, “Large-scale asr domain adaptation using self-and semi-supervised learning,” _arXiv preprint arXiv:2110.00165_ , 2021.
  * [18] M. Schuster and K. Nakajima, “Japanese and korean voice search,” in _2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2012, pp. 5149–5152.
  * [19] Y. Zhang, J. Qin, D. S. Park, W. Han, C.-C. Chiu, R. Pang, Q. V. Le, and Y. Wu, “Pushing the limits of semi-supervised learning for automatic speech recognition,” _arXiv preprint arXiv:2010.10504_ , 2020.
  * [20] A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” in _Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020_ , H. Meng, B. Xu, and T. F. Zheng, Eds. ISCA, 2020, pp. 5036–5040. [Online]. Available: https://doi.org/10.21437/Interspeech.2020-3015
  * [21] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [22] A. Graves, “Sequence transduction with recurrent neural networks,” in _International Conference on Machine Learning: Representation Learning Workshop_ , 2012.
  * [23] A. Narayanan, T. N. Sainath, R. Pang, J. Yu, C.-C. Chiu, R. Prabhavalkar, E. Variani, and T. Strohman, “Cascaded encoders for unifying streaming and non-streaming asr,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 5629–5633.
  * [24] E. Variani, D. Rybach, C. Allauzen, and M. Riley, “Hybrid autoregressive transducer (hat),” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 6139–6143.
  * [25] J. Bai, B. Li, Y. Zhang, A. Bapna, N. Siddhartha, K. C. Sim, and T. N. Sainath, “Joint unsupervised and supervised training for multilingual asr,” _arXiv preprint arXiv:2111.08137_ , 2021.
  * [26] A. Pasad, J.-C. Chou, and K. Livescu, “Layer-wise analysis of a self-supervised speech representation model,” _arXiv preprint arXiv:2107.04734_ , 2021.
  * [27] D. Qiu, Q. Li, Y. He, Y. Zhang, B. Li, L. Cao, R. Prabhavalkar, D. Bhatia, W. Li, K. Hu _et al._ , “Learning word-level confidence for subword end-to-end asr,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 6393–6397.
  * [28] A. Narayanan, A. Misra, K. C. Sim, G. Pundak, A. Tripathi, M. Elfeky, P. Haghani, T. Strohman, and M. Bacchiani, “Toward domain-invariant speech recognition via large scale training,” in _2018 IEEE Spoken Language Technology Workshop (SLT)_ , 2018, pp. 441–447.
  * [29] “Artificial intelligence at google: Our principles,” https://ai.google/principles/.
  * [30] H. Liao, E. McDermott, and A. Senior, “Large scale deep neural network acoustic modeling with semi-supervised training data for youtube video transcription,” in _IEEE Workshop on Automatic Speech Recognition and Understanding_ , 2013, pp. 368–373.
  * [31] T. Bagby, K. Rao, and K. C. Sim, “Efficient implementation of recurrent neural network transducer in TensorFlow,” in _2018 IEEE Spoken Language Technology Workshop_ , 2018, pp. 506–512.
  * [32] R. Prabhavalkar, T. N. Sainath, Y. Wu, P. Nguyen, Z. Chen, C.-C. Chiu, and A. Kannan, “Minimum word error rate training for attention-based sequence-to-sequence models,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 4839–4843.
  * [33] C. Kim, A. Misra, K. Chin, T. Hughes, A. Narayanan, T. Sainath, and M. Bacchiani, “Generation of large-scale simulated utterances in virtual rooms to train deep-neural networks for far-field speech recognition in google home,” 2017.
  * [34] G. Ghiasi, B. Zoph, E. D. Cubuk, Q. V. Le, and T.-Y. Lin, “Multi-task self-training for learning general representations,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 8856–8865.

##  Appendix A Appendix

###  A.1 NST based on supervised teacher

When we start NST based on poor supervised baseline (B0), we couldn’t make the
SoTA teacher (E3 in Table 2) by multi generation NST. Self-sup is needed only
once.

Table 8: WERs of teacher models for supervised baseline, JUST hydra and NST. Algorithms | Data | WER ($\%$)
---|---|---
|  | VS | MF
B0 (Supervised) | Sup | ${4.5}$ | ${2.4}$
E16 (B0, 1st Gen NST) | Sup | ${4.4}$ | ${2.3}$
E17 (E16, 2nd Gen NST) | Sup | ${4.3}$ | ${2.3}$
E18 (E17, 3rd Gen NST) | Sup | ${4.3}$ | ${2.2}$

###  A.2 W2v-BERT with the fraction of data

In Table 9, we train both supervised baseline and W2v-BERT with none, $1$k hrs
and full labeled data. In low resource regime (none and $1$k hrs labels),
W2v-BERT enhances WERs compared to the baselines respectively. However, in
high resource regime ($27$k hrs full labels), W2v-BERT contributes nothing.

Self-sup contribution has diminishing returns as the amount of labels is
growing. The self-sup scale issue was found in computer vision domain as well
[34]. Self-sup outperforms to the supervised baseline with ImageNet (1.3M
images), but self-sup has worse performance than the supervised baseline with
JFT (300M images).

Table 9: WERs of W2v-BERT with the fraction of data. Algorithms | Amount of | WER ($\%$)
---|---|---
| VS/MF Data | VS | MF
Supervised | None | ${5.5}$ | ${4.8}$
$1$k hrs | ${5.2}$ | ${2.9}$
All | ${4.5}$ | ${2.4}$
W2v-BERT | None | ${5.2}$ | ${4.0}$
$1$k hrs | ${4.9}$ | ${2.7}$
All | ${4.5}$ | ${2.5}$

###  A.3 Confidence filter

Our previous study [17] reported NST with confidence filter enhances WER
further, because it filters out not-confident utterances whose pseudo label
maybe wrong. It was true for small teacher ($120$M). However, confidence
filter on the large teacher doesn’t improve WERs. We searched the confidence
threshold in range (1e-6, 1e-4, 1e-2, 0.1, 1, 5, 9, 9.9). None of thresholds
gave better WERs. It actually often degraded the performance. Strong teacher
doesn’t need confidence filter.

###  A.4 Hypothesis comparison

In Section 4.2, the model trained with pseudo labels has better WERs. In Table
10, we compare hypothesises from the models trained with human labels and
pseudo labels respectively. The pseudo label model produces hypothesises
respecting the transcription policy of Google, persistently. On the other
hands, the human label model sometimes produces hypothesises which does not
comply our normalization policy. It is because our human labels includes this
kind of incorrect labels, while pseudo labels are more consistent as those are
produced by a statistical model, which provides the most likely labels
consistently. The second group in Table 10 shows the pseudo label model is
generally better as well. We assume the human label model wastes model
capacity by inconsistent normalization, so the general performance of the
model is worse than the pseudo label model.

Table 10: Hypothesis comparison between models trained with human labels and pseudo labels. Model with human label | Model with pseudo label
---|---
OOO km per hour | OOO km/h
OOO 8 and 1/2 | OOO 8 1/2
9 + 6/7 + 7 + 5/12 | 9 6/7 + 7 5/12
OOO AC DC | OOO AC/DC
OOO under water? | OOO underwater?
OOO a County? | OOO accounting?