# Learning with Noisy Labels via Sparse Regularization

Xiong Zhou1,2 Xianming Liu1,2 Chenyang Wang1 Deming Zhai1 Junjun Jiang1,2
Xiangyang Ji3
1Harbin Institute of Technology 2Peng Cheng Laboratory 3Tsinghua University
<EMAIL_ADDRESS><EMAIL_ADDRESS>Correspondence to: Xianming Liu<EMAIL_ADDRESS>
###### Abstract

Learning with noisy labels is an important and challenging task for training
accurate deep neural networks. Some commonly-used loss functions, such as
Cross Entropy (CE), suffer from severe overfitting to noisy labels. Robust
loss functions that satisfy the symmetric condition were tailored to remedy
this problem, which however encounter the underfitting effect. In this paper,
we theoretically prove that any loss can be made robust to noisy labels by
restricting the network output to the set of permutations over a fixed vector.
When the fixed vector is one-hot, we only need to constrain the output to be
one-hot, which however produces zero gradients almost everywhere and thus
makes gradient-based optimization difficult. In this work, we introduce the
sparse regularization strategy to approximate the one-hot constraint, which is
composed of network output sharpening operation that enforces the output
distribution of a network to be sharp and the $\ell_{p}$-norm ($p\leq 1$)
regularization that promotes the network output to be sparse. This simple
approach guarantees the robustness of arbitrary loss functions while not
hindering the fitting ability. Experimental results demonstrate that our
method can significantly improve the performance of commonly-used loss
functions in the presence of noisy labels and class imbalance, and outperform
the state-of-the-art methods. The code is available at
https://github.com/hitcszx/lnl_sr.

##  1 Introduction

Deep neural networks (DNNs) have achieved remarkable success on various
computer vision tasks, such as image classification, segmentation, and object
detection [7]. The most widely used paradigm for DNN training is the end-to-
end supervised manner, whose performance largely relies on massive high-
quality annotated data. However, collecting large-scale datasets with fully
precise annotations (or called clean labels) is usually expensive and time-
consuming, and sometimes even impossible. Noisy labels, which are
systematically corrupted from ground-truth labels, are ubiquitous in many
real-world applications, such as online queries [19], crowdsourcing [1],
adversarial attacks [30], and medical images analysis [12]. On the other hand,
it is well-known that over-parameterized neural networks have enough capacity
to memorize large-scale data with even completely random labels, leading to
poor performance in generalization [28, 1, 11]. Therefore, robust learning
with noisy labels has become an important and challenging task in computer
vision [25, 8, 12, 31].

(a) FL

(b) FL+SR

(c) GCE

(d) GCE+SR

Figure 1: Visualization of learned representations on MNIST with 0.8 symmetric
label noise. The representations learned by the proposed sparse regularization
(SR)-enhanced methods are significantly discriminative than that learned by
original losses, which are with more separated and clearly bound margin.

To prevent over-fitting to mislabeled data, many strategies have been
presented in the literature, among which robust loss function design is one of
the most popular approaches since it enjoys simplicity and universality. Ghosh
et al. [6] theoretically proved that a loss function would be inherently
tolerant to symmetric label noise as long as it satisfies the symmetric
condition. However, the derived loss functions according to this design
principle, such as MAE [6] and Reverse Cross Entropy (RCE) [27], suffer from
the underfitting effect on complicated datasets [3, 20]. As demonstrated in
[29], the robustness of MAE can concurrently cause increased difficulty in
training, leading to performance drops. On the other hand, the commonly-used
CE and focal loss (FL) [18] enjoy the advantage of sufficient learning where
the optimizer puts more emphasis on ambiguous samples, but they tend to
overfit on noisy labels.

How to achieve robustness and learning sufficiency simultaneously? This
question has motivated a large amount of work to design new loss functions
that are robust to noisy labels as well as easy to fit clean ones. For
instance, Zhang et al. proposed a generalization of cross entropy (GCE) [29],
which behaves like a generalized mixture of MAE and CE. Wang et al. proposed
the Symmetric Cross Entropy (SCE) loss [27], which combines RCE with CE.
However, both GCE and SCE just perform the trade-off between symmetric loss
and CE, which are only partially robust to noisy labels. Ma et al. [20]
theoretically proved that by applying a simple normalization, any loss can be
made robust to noisy labels. However, the normalization operation actually
changes the form of loss functions, which no longer preserve the original
fitting ability. The authors further proposed the Active Passive Loss (APL) to
remedy this problem, which is the combination of two symmetric losses.

As reviewed above, all existing methods attempt to design new loss functions
that satisfy the symmetric condition [6] to achieve robustness and achieve
learning sufficiency meanwhile by combining multiple forms of loss function,
such as symmetric loss and CE in [29, 27] and two symmetric losses in [20]. In
this work, we propose a novel perspective to understand the symmetric
condition, and prove in theory that any loss can be made robust through
restricting the hypothesis class. Specifically, we demonstrate that label
noise under risk minimization can be mitigated by restricting the network
output to a permutation set of a fixed one-hot vector instead of modifying the
loss function. This discrete process, however would result in many zero
gradients, making the gradient-based optimization difficult. We then propose a
sparse regularization strategy to approximate the one-hot constraint, which
includes network output sharpening operation and $\ell_{p}$-norm ($p\leq 1$)
regularization in risk minimization. Experimental results on synthesis and
real-world datasets demonstrate that our method can significantly improve the
performance of commonly-used loss functions in the presence of noisy labels,
as illustrated in Fig. 1, and outperform the state-of-the-art robust loss
functions. Moreover, we evaluate the sparse regularization strategy on the
long-tailed and step-imbalanced image classification, which demonstrates that
SR can also mitigate class imbalance well.

The main contributions of our work are highlighted as follows:

  * •

To the best of our knowledge, we are the first work in the literature to meet
the symmetric condition by restricting the hypothesis class. It offers a novel
perspective of understanding the symmetric condition and an alternative
approach for robust learning.

  * •

We theoretically prove that any loss function can be made robust to noisy
labels by restricting the network output to the set of permutations over a
fixed vector.

  * •

We propose a simple but effective approach for robust training through sparse
regularization.

  * •

We provide a principled approach to simultaneously achieve robust training and
preserve the fitting ability of commonly-used losses such as CE.

##  2 The Proposed Method

In this section, we first introduce some preliminaries about robust learning.
Subsequently, we present our finding in theory that any loss function can
achieve noise tolerance via network output permutation. Furthermore, we offer
a simple but effective approach for robust learning by introducing sparse
regularization in network training. Finally, we provide analysis about the
merit of our scheme—achieve better tradeoff between robustness and sufficient
learning.

###  2.1 Preliminaries

Risk Minimization. Assume $\mathcal{X}\subset\mathbb{R}^{d}$ is the feature
space from which the examples are drawn, and $\mathcal{Y}=[k]=\\{1,...,k\\}$
is the class label space, i.e., we consider a $k$-classification problem. In a
typical classifier learning problem, we are given a training set,
$\mathcal{S}=\\{(\mathbf{x}_{1},y_{1}),...,(\mathbf{x}_{N},y_{N})\\}$, where
$(\mathbf{x}_{i},y_{i})$ is drawn i.i.d. according to an unknown distribution,
$\mathcal{D}$, over $\mathcal{X}\times\mathcal{Y}$. The classifier is a
mapping function from feature space to label space
$h(\mathbf{x})=\mathop{\text{arg}}\max_{i}f(\mathbf{x})_{i}$, where
$f:\mathcal{X}\rightarrow\mathcal{C}$ denotes an approximation of
$p(\cdot|\mathbf{x})$, and $\mathcal{C}\subset[0,1]^{k}$,
$\forall\mathbf{c}\in\mathcal{C}$, $\mathbf{1}^{T}\mathbf{c}=1$. In deep
learning, $f(\mathbf{x})$ is usually modeled by a neural network ending with a
softmax layer.

The loss function is defined as a mapping
$L:\mathcal{C}\times\mathcal{Y}\rightarrow\mathbb{R}^{+}$, where
$\arg\min_{\mathbf{u}\in\mathcal{C}}L(\mathbf{u},y)=\mathbf{e}_{y}$, and
$\mathbf{e}_{y}$ denotes the one-hot vector. In this work, we consider the
loss functional $\mathcal{L}$, where $\forall L\in\mathcal{L}$,
$L(\mathbf{u},i)=\ell_{1}(u_{i})+\sum_{j\not=i}\ell_{2}(u_{j})$, and
$\ell_{1}$ as well as $\ell_{2}$ are two basic functions. For example, CE can
be expressed by $\ell_{1}(u_{i})=-\log u_{i}$ and $\ell_{2}(u_{j})=0$. Given
any loss function $L$, and a classifier $f$, the $L$-risk of $f$ is

$R_{L}(f)=\mathbb{E}_{\mathcal{D}}[L(f(\mathbf{x}),y)]=\mathbb{E}_{\mathbf{x},y}[L(f(\mathbf{x}),y)],$
(1)

where $\mathbb{E}$ denotes expectation. Under the risk minimization framework,
the objective is to learn an optimal classifier, $f^{*}$, which is a global
minimum of $R_{L}(f)$.

Noise Tolerance. We define the noise corruption process as that a clean label
$y$ is flipped into a noisy version $\tilde{y}$ with probability
$\eta_{\mathbf{x},\tilde{y}}=p(\tilde{y}|y,\mathbf{x})$ (more label noise
settings can be found in [6]). The corresponding noisy $L$-risk is

$R_{L}^{\eta}(f)=\mathbb{E}_{\mathcal{D}}[(1-\eta_{\mathbf{x}})L(f(\mathbf{x}),y)+\sum_{i\not=y}\eta_{\mathbf{x},i}L(f(\mathbf{x}),i)],$

where $\eta_{\mathbf{x}}=\sum_{i\not=y}\eta_{\mathbf{x},i}$ denotes the noise
rate. Risk minimization under a given loss function is noise-tolerant if
$R_{L}^{\eta}(f)$ shares the same global minimum as $R_{L}(f)$.

Symmetric Loss Functions. A symmetric loss function [22, 6] is proved to be
noise-tolerant for a $k$-class classification under symmetric noise if the
noise rate $\eta<\frac{k-1}{k}$ and the loss function satisfies

$\sum_{i=1}^{k}L(f(\mathbf{x}),i)=C,\ \forall\mathbf{x}\in\mathcal{X},\forall
f\in\mathcal{H},$ (2)

where $C$ is a constant, and $\mathcal{H}$ is the hypothesis class.

The symmetric condition (2) stated above guarantees the noise tolerance by
risk minimization on a symmetric loss function, i.e., the classifier trained
in noisy case has the same misclassification probability as that trained in
noise-free case under the specified assumption. Moreover, if $R_{L}(f^{*})=0$,
$L$ is also noise-tolerant under an asymmetric noise, where $f^{*}$ is a
global risk minimum of $R_{L}$.

###  2.2 Noise Tolerance via Output Permutation

The symmetric condition (2) theoretically guarantees that a symmetric loss can
lead to robust training. However, the derived loss functions according to this
design principle usually suffer from underfitting [27, 3, 20]. On the other
hand, the existing methods all pay attention to designing new loss functions,
but never attempt to restrict the hypothesis class $\mathcal{H}$ to satisfy
the symmetric condition. In this work, we propose to restrict the hypothesis
class $\mathcal{H}$ such that any losses satisfy the symmetric condition (2)
and thus become robust to label noise. Furthermore, we provide a theoretical
analysis to demonstrate the noise tolerance of our scheme. The proofs can be
seen in the supplementary materials.

First of all, we provide the definition of permutation operation, which plays
an important role in our derivation.

######  Definition 1.

For a vector $\mathbf{v}\in\mathbb{R}^{k}$, the permutation operation on it is
defined as [24]:

$\mathbf{v}_{\pi}=P_{\pi}\mathbf{v},$ (3)

where
$P_{\pi}=[\mathbf{e}_{\pi_{1}},\mathbf{e}_{\pi_{2}},...,\mathbf{e}_{\pi_{k}}]^{T}$
is the permutation matrix, and $\\{\pi_{1},\pi_{2},...,\pi_{k}\\}=[k]$.

According to this definition, it is easy to find that $\mathbf{v}$ and
$\mathbf{v}_{\pi}$ share the same space, i.e.,
$\\{v_{\pi_{1}},v_{\pi_{2}},...,v_{\pi_{k}}\\}=\\{v_{1},v_{2},...,v_{k}\\}$.
For instance, when $k=3$, $\mathbf{v}=\\{v_{1},v_{2},v_{3}\\}$ and
$\pi=[3,1,2]$, then
$P_{\pi}=[\mathbf{e}_{3},\mathbf{e}_{1},\mathbf{e}_{2}]^{T}$, and the vector
after permutation operation is $\mathbf{v}_{\pi}=(v_{3},v_{1},v_{2})^{T}$.

More generally, let $\mathcal{P}_{\mathbf{v}}$ denote the permutation set over
$\mathbf{v}$, we have

$\sum_{i=1}^{k}\ell(u_{i})=\sum_{i=1}^{k}\ell(v_{i}),\forall\
\mathbf{u}\in\mathcal{P}_{\mathbf{v}}.$ (4)

According to the above definition and derivation, we arrive at Lemma 1:

######  Lemma 1.

Given a vector $\mathbf{v}$, $\forall L\in\mathcal{L}$, we have

$\sum_{i=1}^{k}L(\mathbf{u},i)=C,\quad\forall\
\mathbf{u}\in\mathcal{P}_{\mathbf{v}},$ (5)

where $C=\sum_{i=1}^{k}L(\mathbf{v},i)$ is a constant when $\mathbf{v}$ is
fixed.

Lemma 1 indicates that, when the network output $\mathbf{u}$ is restricted to
belong to a permutation set $\mathcal{P}_{\mathbf{v}}$ of a fixed vector
$\mathbf{v}$, any loss functions in $\mathcal{L}$ satisfy the symmetric
condition. We further have the following theorems for symmetric and asymmetric
noise that can be proved similarly as [6]:

######  Theorem 1 (Noise tolerance under symmetric noise).

In a multi-class classification problem, $\forall L\in\mathcal{L}$, $L$ is
noise-tolerant under symmetric label noise if $\eta<1-\frac{1}{k}$ and
$f:\mathcal{X}\rightarrow\mathcal{P}_{\mathbf{v}}$, i.e.,

$\mathop{\arg\min}_{f:\mathcal{X}\rightarrow\mathcal{P}_{\mathbf{v}}}R_{L}(f)=\mathop{\arg\min}_{f:\mathcal{X}\rightarrow\mathcal{P}_{\mathbf{v}}}R^{\eta}_{L}(f),$
(6)

where $\mathbf{v}$ is a fixed vector.

######  Theorem 2 (Noise tolerance under asymmetric noise).

In a multi-class classification problem, let
$f:\mathcal{X}\rightarrow\mathcal{P}_{\mathbf{v}}$, where $\mathbf{v}$ is a
fixed vector, and suppose $L\in\mathcal{L}$ satisfy $0\leq
L(f(\mathbf{x}),i)\leq\frac{C}{k-1}$, $\forall i\in[k]$. If $R_{L}(f^{*})=0$,
then $L$ is noise-tolerant under asymmetric or class-conditional noise when
$\eta_{y,i}<1-\eta_{y}$ with $\sum_{k\not=y}\eta_{y,i}=\eta_{y}$,
$\forall\mathbf{x}$.

Theorem 1 and 2 inspire us that label noise under risk minimization can be
mitigated by restricting the network output to a permutation set instead of
changing the loss function. This offers an alternative principle approach to
achieve robust learning. However, the optimization is non-trivial when using
gradient-based strategy because the constraint that $\mathcal{P}_{\mathbf{v}}$
is a discrete mapping produces many zero gradients. Instead, we turn to
approximate the constraint by relaxing the output restriction of the
hypothesis class with an error bound $\varepsilon$, i.e.,
$\mathcal{H}_{\mathbf{v},\varepsilon}=\\{f:\min_{\mathbf{u}\in\mathcal{P}_{\mathbf{v}}}\|f(\mathbf{x})-\mathbf{u}\|_{2}\leq\varepsilon,\forall\mathbf{x}\\}$.
We can derive the risk bound as follows:

######  Theorem 3.

In a multi-class classification problem, if the loss function
$L\in\mathcal{L}$ satisfies
$|\sum_{i=1}^{k}(L(\mathbf{u}_{1},i)-L(\mathbf{u}_{2},i))|\leq\delta$ when
$\|\mathbf{u}_{1}-\mathbf{u}_{2}\|_{2}\leq\varepsilon$, and $\delta\rightarrow
0$ as $\varepsilon\rightarrow 0$, then for symmetric label noise satisfying
$\eta<1-\frac{1}{k}$, the risk bound for
$f\in\mathcal{H}_{\mathbf{v},\varepsilon}$ can be expressed as

$R_{L}(f^{*}_{\eta})-R_{L}(f^{*})\leq 2c\delta,$

where $c=\frac{\eta}{(1-\eta)k-1}$, $f^{*}_{\eta}$ and $f^{*}$ denote the
global minimum of $R_{L}^{\eta}(f)$ and $R_{L}(f)$, respectively.

Theorem 3 indicates that when restricting the output of the network to belong
to $\mathcal{H}_{\mathbf{v},\varepsilon}$, the noisy minimum $f^{*}_{\eta}$,
compared to the clean minimum $f^{*}$, has a risk error bound $2c\delta$. And
when $\varepsilon\rightarrow 0$, the bound also tends to $0$. This implies
that, by shrinking $\mathcal{H}_{\mathbf{v},\varepsilon}$,
$f^{*}_{\eta}=\arg\min_{f\in\mathcal{H}_{\mathbf{v},\varepsilon}}R_{L}^{\eta}(f)$
converges to $f^{*}$.

###  2.3 Robust Learning via Sparse Regularization

Upon the above theoretical analysis, we propose a simple but effective
approach for robust learning, which can make any losses robust to noisy labels
by introducing sparse regularization on network output. Specifically, we
consider the fixed vector $\mathbf{v}$ as a one-hot vector, i.e., we restrict
the network output to one-hot vectors $\mathcal{P}_{0-1}$. This discrete
process would result in many zero gradients, making the optimization
difficult. To approximate the one-hot constraint, we propose the sparse
regularization strategy, which is composed of two modules: network output
sharpening and $\ell_{p}$-norm ($p\leq 1$) regularization in risk
minimization.

Network Output Sharpening. The output sharpening module is to make the network
output closer to a one-hot vector. One popular way to approximate a one-hot
vector by the continuous mapping is to use a temperature-dependent softmax
function, i.e.,

$\sigma_{\tau}(\mathbf{z})_{i}=\frac{\exp(z_{i}/\tau)}{\sum_{j=1}^{k}\exp(z_{j}/\tau)},$
(7)

where $0\leq\tau\leq 1$, and $\sigma_{\tau}(\mathbf{z})_{i}$ is a point in the
probability simplex. Note that, in the limit situation where $\tau\rightarrow
0$, $\sigma_{\tau}(\mathbf{z})$ converges to a one-hot vector. In other words,
with low temperatures, the distribution spends essentially all of its
probability mass in the most probable state. Meanwhile, we may limit the value
of $z_{i}$ to the range $[-1,1]$ by performing $\ell_{2}$ normalization before
the output sharpening to prevent it from trivial scaling solution.

$\ell_{p}$-norm Regularization. We further introduce $\ell_{p}$-norm
regularization into risk minimization to promote the sparsity of network
output. Specifically, we perform the following constrained risk minimization
in network training:

$\min_{f\in\mathcal{H}}R_{L}(f)\quad\text{s.t.}\quad\|f\|_{p}\leq\gamma,$ (8)

where $0\leq p\leq 1$, and $\gamma$ is an appropriately selected parameter. In
practice, we can convert to train a neural network by minimizing the following
form:

$\sum_{i=1}^{N}L(f(\mathbf{x}_{i}),\tilde{y}_{i})+\lambda\|f(\mathbf{x}_{i})\|_{p}^{p}.$
(9)

It is worth noting that, if we define $L(\mathbf{u},i)=-\alpha\log
u_{i}-\lambda u_{i}$ and $p=1$, then Eq. (9) is equivalent to the Symmetric
Cross Entropy (SCE) loss [27]. Thus, SCE can be regarded as a special case of
a loss function with the $\ell_{p}$-norm regularization.

For effective learning, the regularization parameter $\lambda$ in Eq. (9)
cannot be set too large, since the network would tend to minimize
$\lambda\|f(\mathbf{x}_{i})\|_{p}^{p}$ rather than
$L(f(\mathbf{x}_{i}),\tilde{y}_{i})$. As shown in Fig. 2(a), when $\lambda=25$
and 30, although the curves look robust, they suffer from the underfitting
effect. On the other hand, $\lambda$ should not be set too small, otherwise
robustness cannot be guaranteed (see $\lambda\leq 7$ in Fig. 2(a)). We need a
large enough $\lambda$ to maintain robustness. An effective strategy in
practical implementation is to gradually increase the value of $\lambda$
during training, i.e., $\lambda_{t}=\lambda_{0}\cdot\rho^{\lfloor t/r\rfloor}$
($\rho\geq 1$), where $t$ denotes the training epoch, and $r$ denotes the
updating rate of $\lambda$.

(a) CE+SR

(b) FL+SR

Figure 2: Test accuracy curve of different $\lambda$ on CIFAR-10 with 0.6
symmetric label noise. Robustness is improved when increasing the weight of
$\ell_{p}$-norm for both CE and FL.

###  2.4 On the Robustness and Learning Sufficiency

In the following, we provide analysis about the robustness and learning
sufficiency of the proposed scheme.

To obtain enough robustness, we restrict the output of the network to be one-
hot, which naturally satisfies the symmetric condition. As for the output
sharpening process, the derivative of $\sigma_{\tau}(\mathbf{z})_{j}$ with
respect to $z_{i}$ can be derived as

$\frac{\partial\sigma_{\tau}(\mathbf{z})_{j}}{\partial
z_{i}}=\frac{1}{\tau}\sigma_{\tau}(\mathbf{z})_{i}(\delta_{ij}-\sigma_{\tau}(\mathbf{z})_{j}),$
(10)

where $\delta_{ij}=\mathbb{I}(i=j)$, and $\mathbb{I}(\cdot)$ is the identity
function. We can see that the derivative is a scaled derivative of the
original softmax function, so it would not change the optimization direction
but change the step size. A larger step size in Eq. (10) would speed up the
convergence to one-hot vectors. We can achieve this purpose by choosing an
appropriate value for $\tau$. On the other hand, we have
$\lim_{\tau\rightarrow
0^{+}}\frac{\partial\sigma_{\tau}(\mathbf{z})_{j}}{\partial z_{i}}=0$, which
shows that the gradient would disappear if $\tau$ is small, so $\tau$ cannot
be overly small to prevent underfitting. We fix $\tau$ in our implementation
for simplicity, but we suggest to gradually decay $\tau$ in training, which
can be regarded as an early-stopping strategy [16].

Moreover, consider the loss
$L(\sigma_{\tau}(\mathbf{z}),y)=-\log\sigma_{\tau}(\mathbf{z})_{y}$, we have
the derivative of
$-\log\sigma_{\tau}(\mathbf{z})_{y}+\lambda\|\sigma_{\tau}(\mathbf{z})\|_{p}^{p}$
with respect to $\mathbf{z}$ as follows

$\displaystyle\frac{\partial-\log\sigma_{\tau}(\mathbf{z})_{y}}{\partial\mathbf{z}}+\lambda\frac{\partial\|\sigma_{\tau}(\mathbf{z})\|_{p}^{p}}{\partial\mathbf{z}}$
$\displaystyle=$
$\displaystyle-\frac{1}{\sigma_{\tau}(\mathbf{z})_{y}}\cdot\frac{\partial\sigma_{\tau}(\mathbf{z})_{y}}{\partial\mathbf{z}}+\lambda
p\sum_{i=1}^{k}\frac{1}{[\sigma_{\tau}(\mathbf{z})_{i}]^{1-p}}\cdot\frac{\partial\sigma_{\tau}(\mathbf{z})_{i}}{\partial\mathbf{z}}$
$\displaystyle=$
$\displaystyle\underbrace{-(\frac{1}{\sigma_{\tau}(\mathbf{z})_{y}}-\frac{\lambda
p}{[\sigma_{\tau}(\mathbf{z})_{y}]^{1-p}})\cdot\frac{\partial\sigma_{\tau}(\mathbf{z})_{y}}{\partial\mathbf{z}}}_{\text{fitting
term}}$ $\displaystyle+\underbrace{\lambda p\sum_{i\neq
y}\frac{1}{[\sigma_{\tau}(\mathbf{z})_{i}]^{1-p}}\cdot\frac{\partial\sigma_{\tau}(\mathbf{z})_{i}}{\partial\mathbf{z}}}_{\text{complementary
term}}$

The fitting term denotes the gradient of learning towards the target $y$,
while the complementary term limits the increase of
$\sigma_{\tau}(\mathbf{z})_{i}$, $\forall i\neq y$. In the early phase of
training, we guarantee enough fitting power by setting $\lambda p<1$. As
$\lambda$ increases, the fitting term becomes weaker to mitigate label noise,
but the complementary term still maintains a certain amount of fitting power
through minimizing $\sigma_{\tau}(\mathbf{z})_{i}$, $\forall i\neq y$ to
passively maximize $\sigma_{\tau}(\mathbf{z})_{y}$.

(a) Sparse Rate

(b) Test Accuracy

Figure 3: Sparse rate and test accuracy of different methods on MNIST with 0.8
symmetric label noise. One-hot constraint significantly improves the
robustness while not hindering fitting ability.

On the other hand, we can regard the $\ell_{p}$-norm regularized loss function
$L(\mathbf{u},i)+\lambda\|\mathbf{u}\|_{p}^{p}$ as a new loss function. If
there exists $\alpha>0$, such that $L(\mathbf{u},i)+\alpha u_{i}^{p}$ is
monotonically decreasing on $u_{i}$, then the new loss can be divided into the
active loss $L(\mathbf{u},i)+\alpha u_{i}^{p}$ and the passive loss
$(\lambda-\alpha)u_{i}^{p}+\lambda\sum_{j\not=i}u_{j}^{p}$. In fact, $\alpha$
always exists for commonly-used loss functions, for example, when
$L(\mathbf{u},i)=-\log u_{i}$, we have $\alpha<\frac{1}{p}$ such that $-\log
u_{i}+\alpha u_{i}^{p}$ keeps monotonically decreasing on $u_{i}$. Therefore,
our proposed $\ell_{p}$-norm regularization coincides with the Active Passive
Loss proposed in [20]. The analysis demonstrates that our scheme achieves the
best of both worlds of robustness and sufficient learning.

##  3 Experiments

In this section, we empirically investigate the effectiveness of sparse
regularization on synthetic datasets, including MNIST [15], CIFAR-10/-100
[14], and a real-world noisy dataset WebVision [17].

###  3.1 Empirical Analysis

One-hot Constraint Means Robustness. We first run a set of experiments on
MNIST with 0.8 symmetric label noise to analyse the sparse rate and test
accuracy during training, where the sparse rate is formulated as
$\frac{\sum_{i=1}^{N}\mathbb{I}(\max_{j}f(\mathbf{x}_{i})_{j}>1-0.01)}{N}$,
and $f(\mathbf{x}_{i})$ is performed by the output sharpening with $\tau=0.1$.
If $\max_{j}f(\mathbf{x}_{i})_{j}>1-0.01$, then
$\min_{\mathbf{u}\in\mathcal{P}_{0-1}}\|f(\mathbf{x}_{i})-\mathbf{u}\|_{2}\leq\sqrt{2}\cdot
0.01=\varepsilon$. We add the sparse regularization strategy to CE, FL and
GCE, the results are shown in Fig. 3. As we can see, with the role of SR, the
sparse rate usually maintains a high value after several epochs, and the test
accuracy curves of CE+SR, FL+SR and GCE+SR show enough robustness and learning
efficiency for model to mitigate label noise, while the original losses have
low sparse rate and poor accuracy. This validates that the noise tolerance can
be obtained by restricting the output of a network to one-hot vectors.

(a) CE with different $p$

(b) CE with different $\tau$

Figure 4: Test accuracy curve of different parameters on CIFAR-10 with 0.6
symmetric label noise.

(a) CE with $\eta=0.0$

(b) CE with $\eta=0.2$

(c) CE with $\eta=0.4$

(d) CE with $\eta=0.6$

(e) CE with $\eta=0.8$

(f) CE+SR with $\eta=0.0$

(g) CE+SR with $\eta=0.2$

(h) CE+SR with $\eta=0.4$

(i) CE+SR with $\eta=0.6$

(j) CE+SR with $\eta=0.8$

Figure 5: Visualization of learned representations on MNIST with different
symmetric label noise ($\eta\in[0.0,0.2,0.4,0.6,0.8]$).

Sparse Regularization can Mitigate Label Noise. As shown in Fig. 2, when we
add SR to enhance the performance of CE and FL, the training process is
increasingly robust as $\lambda$ increases while not hindering the fitting
ability ($\lambda\leq 20$). This demonstrates that learning with sparse
regularization can be both robust and effective when mitigating label noise.
The projected representations on MNIST are illustrated in Fig. 1. Under both
settings, the representations learned by the SR-enhanced method are of
significantly better quality than those learned by the original losses with
more separated and clearly bounded clusters.

Parameter Analysis. We choose the different parameters $\tau$, $\lambda$ and
$p$ for sparse regularization to CE. The experiments are conducted on CIFAR-10
with 0.6 symmetric noise. We first tested
$\tau\in\\{0.01,0.03,0.1,0.3,0.5,0.7,0.9,1\\}$ while $\lambda=0$, and the
results can be show in Fig. 4(b). When $\tau$ is small ($\leq 0.003$), the
curve is very robust, but it also suffer from significant underfitting
problem. As $\tau$ increases, the curve falls into overfitting. Then we tested
$\lambda\in\\{0.1,0.3,1.0,3.0,5.0,7.0,10,15,20,25,30\\}$ while $p=0.1$ and
$\tau=1$. As shown in Fig. 2, curve gets more robustness as $\lambda$
increases, but when $\lambda\geq 7$, it encounters severe underfitting since
the optimization pays more attention on minimizing
$\lambda\|f(\mathbf{x})\|_{p}^{p}$. Moreover, we tuned
$p\in\\{0.01,0.05,0.1,0.3,0.5,0.7,0.9,1\\}$ while $\tau=1$ and $\lambda=5$.
The results in Fig. 4(a) show that both small $p$ and large $p$ tend to cause
more overfitting to label noise, so we need choose an appropriate value.

Remark. As for parameter tunings, a simple principled approach for parameter
setting is that parameters with strong regularization are selected for simple
datasets, and parameters with weak regularization are selected otherwise.
Specifically, this can be achieved by setting proper $\lambda$: the larger
$\lambda$, the stronger regularization effect. However, as shown in Fig. 2,
the large initial $\lambda$ leads to underfitting since much attention is paid
on minimizing $\lambda\|f(\mathbf{x})\|_{p}^{p}$, especially when
$\lambda\cdot p>1$. Instead, we turn to gradually increase $\lambda$, i.e.,
$\lambda_{t}=\lambda_{0}\cdot\rho^{\lfloor t/r\rfloor}$, where $t$ is the
iteration number and $r=1$; $p=0.1$ and $0.01$, $\rho=50^{1/120}\approx 1.03$
and $50^{1/200}\approx 1.02$ for CIFAR-10 and CIFAR-100 (where 120 and 200
represent the training epochs). The only parameter that requires careful
tuning is $\lambda_{0}$, which is set as $\lambda_{0}<\frac{1}{p}$. We use the
similar parameter setting strategy for CIFAR-10, CIFAR-100, WebVision, and all
achieve satisfactory results.

###  3.2 Evaluation on Benchmark Datasets

Experimental Details. The benchmark datasets, noise generation, networks,
training details, parameter settings, more comparisons, and more experimental
results can be found in the supplementary materials.

Table 1: Test accuracies (%) of different methods on benchmark datasets with clean or symmetric label noise ($\eta\in[0.2,0.4,0.6,0.8]$). The results (mean$\pm$std) are reported over 3 random runs and the top 3 best results are boldfaced. Datasets | Methods | Clean ($\eta=0.0$) |  Symmetric Noise Rate ($\eta$)
---|---|---|---
0.2 | 0.4 | 0.6 | 0.8
MNIST | CE |  99.15 $\pm$ 0.05 |  91.62 $\pm$ 0.39 |  73.98 $\pm$ 0.27 |  49.36 $\pm$ 0.43 |  22.66 $\pm$ 0.61
FL |  99.13 $\pm$ 0.09 |  91.68 $\pm$ 0.14 |  74.54 $\pm$ 0.06 |  50.39 $\pm$ 0.28 |  22.65 $\pm$ 0.26
GCE |  99.27 $\pm$ 0.05 |  98.86 $\pm$ 0.07 |  97.16 $\pm$ 0.03 |  81.53 $\pm$ 0.58 |  33.95 $\pm$ 0.82
SCE |  99.23 $\pm$ 0.10 |  98.92 $\pm$ 0.12 |  97.38 $\pm$ 0.15 |  88.83 $\pm$ 0.55 |  48.75 $\pm$ 1.54
NLNL |  98.85 $\pm$ 0.05 |  98.33 $\pm$ 0.03 |  97.80 $\pm$ 0.07 |  96.18 $\pm$ 0.11 |  86.34 $\pm$ 1.43
APL |  99.34 $\pm$ 0.02 | 99.14 $\pm$ 0.05 |  98.42 $\pm$ 0.09 |  95.65 $\pm$ 0.13 |  72.97 $\pm$ 0.34
CE+SR |  99.33 $\pm$ 0.02 | 99.22 $\pm$ 0.06 | 99.16 $\pm$ 0.04 | 98.85 $\pm$ 0.02 | 98.06 $\pm$ 0.86
FL+SR |  99.35 $\pm$ 0.05 | 99.25 $\pm$ 0.01 | 99.10 $\pm$ 0.10 | 98.81 $\pm$ 0.06 | 97.00 $\pm$ 1.28
GCE+SR |  99.27 $\pm$ 0.06 |  99.13 $\pm$ 0.07 | 99.06 $\pm$ 0.02 | 98.84 $\pm$ 0.09 | 98.37 $\pm$ 0.26
CIFAR-10 | CE |  90.48 $\pm$ 0.11 |  74.68 $\pm$ 0.25 |  58.26 $\pm$ 0.21 |  38.70 $\pm$ 0.53 |  19.55 $\pm$ 0.49
FL |  89.82 $\pm$ 0.20 |  73.72 $\pm$ 0.08 |  57.90 $\pm$ 0.45 |  38.86 $\pm$ 0.07 |  19.13 $\pm$ 0.28
GCE |  89.59 $\pm$ 0.26 |  87.03 $\pm$ 0.35 |  82.66 $\pm$ 0.17 |  67.70 $\pm$ 0.45 |  26.67 $\pm$ 0.59
SCE |  91.61 $\pm$ 0.19 |  87.10 $\pm$ 0.25 |  79.67 $\pm$ 0.37 |  61.35 $\pm$ 0.56 |  28.66 $\pm$ 0.27
NLNL |  90.73 $\pm$ 0.20 |  73.70 $\pm$ 0.05 |  63.90 $\pm$ 0.44 |  50.68 $\pm$ 0.47 |  29.53 $\pm$ 1.55
APL |  89.17 $\pm$ 0.09 |  86.98 $\pm$ 0.07 |  83.74 $\pm$ 0.10 |  76.02 $\pm$ 0.16 |  46.69 $\pm$ 0.31
CE+SR |  90.06 $\pm$ 0.02 | 87.93 $\pm$ 0.07 | 84.86 $\pm$ 0.18 | 78.18 $\pm$ 0.36 | 51.13 $\pm$ 0.51
FL+SR |  89.86 $\pm$ 0.11 | 87.94 $\pm$ 0.19 | 84.65 $\pm$ 0.05 | 77.85 $\pm$ 0.74 | 52.42 $\pm$ 0.76
GCE+SR |  90.02 $\pm$ 0.40 | 87.93 $\pm$ 0.27 | 84.82 $\pm$ 0.06 | 77.65 $\pm$ 0.05 | 51.97 $\pm$ 1.13
CIFAR-100 | CE |  71.33 $\pm$ 0.43 |  56.51 $\pm$ 0.39 |  39.92 $\pm$ 0.10 |  21.39 $\pm$ 1.17 |  7.59 $\pm$ 0.20
FL |  70.06 $\pm$ 0.70 |  55.78 $\pm$ 1.55 |  39.83 $\pm$ 0.43 |  21.91 $\pm$ 0.89 |  7.51 $\pm$ 0.09
GCE |  63.09 $\pm$ 1.39 |  61.57 $\pm$ 1.06 |  56.11 $\pm$ 1.35 | 45.28 $\pm$ 0.61 | 17.42 $\pm$ 0.06
SCE |  70.64 $\pm$ 0.05 |  56.07 $\pm$ 0.26 |  39.88 $\pm$ 0.67 |  21.16 $\pm$ 0.65 |  7.63 $\pm$ 0.15
NLNL |  68.72 $\pm$ 0.60 |  46.99 $\pm$ 0.91 |  30.29 $\pm$ 1.64 |  16.60 $\pm$ 0.90 |  11.01 $\pm$ 2.48
APL |  67.95 $\pm$ 0.21 |  64.21 $\pm$ 0.24 |  57.70 $\pm$ 0.64 | 45.20 $\pm$ 0.75 | 24.91 $\pm$ 0.42
CE+SR |  72.19 $\pm$ 0.06 | 67.51 $\pm$ 0.29 | 60.70 $\pm$ 0.25 | 44.95 $\pm$ 0.65 | 17.35 $\pm$ 0.13
FL+SR |  72.08 $\pm$ 0.31 | 67.64 $\pm$ 0.10 | 60.67 $\pm$ 0.48 |  44.76 $\pm$ 0.08 |  17.16 $\pm$ 0.24
GCE+SR |  72.11 $\pm$ 0.26 | 67.03 $\pm$ 0.46 | 60.68 $\pm$ 0.90 |  44.66 $\pm$ 0.84 |  17.35 $\pm$ 0.42

Baselines. We experiment with the state-of-the art methods GCE [29], SCE [27],
NLNL [13], APL [20], and two effective loss functions CE and Focal Loss (FL)
[18] for classification. Moreover, we add the proposed sparse regularization
mechanism to CE, FL and GCE, i.e., CE+SR, FL+SR and GCE+SR. All the
implementations and experiments are based on PyTorch.

Table 2: Test accuracies (%) of different methods on benchmark datasets with clean or asymmetric label noise ($\eta\in[0.1,0.2,0.3,0.4]$). The results (mean$\pm$std) are reported over 3 random runs and the top 3 best results are boldfaced. Datasets | Methods |  Asymmetric Noise Rate ($\eta$)
---|---|---
0.2 | 0.3 | 0.4
MNIST | CE |  94.56 $\pm$ 0.22 |  88.81 $\pm$ 0.10 |  82.27 $\pm$ 0.40
FL |  94.25 $\pm$ 0.15 |  89.09 $\pm$ 0.25 |  82.13 $\pm$ 0.49
GCE |  96.69 $\pm$ 0.12 |  89.12 $\pm$ 0.24 |  81.51 $\pm$ 0.19
SCE |  98.03 $\pm$ 0.05 |  93.68 $\pm$ 0.43 |  85.36 $\pm$ 0.17
NLNL |  98.35 $\pm$ 0.01 |  97.51 $\pm$ 0.15 |  95.84 $\pm$ 0.26
APL |  98.89 $\pm$ 0.04 |  96.93 $\pm$ 0.17 |  91.45 $\pm$ 0.40
CE+SR | 99.27 $\pm$ 0.06 | 99.24 $\pm$ 0.08 | 99.23 $\pm$ 0.07
FL+SR | 99.31 $\pm$ 0.02 | 99.23 $\pm$ 0.02 | 99.36 $\pm$ 0.05
GCE+SR | 99.22 $\pm$ 0.02 | 99.13 $\pm$ 0.05 | 99.09 $\pm$ 0.02
CIFAR-10 | CE |  83.32 $\pm$ 0.12 |  79.32 $\pm$ 0.59 |  74.67 $\pm$ 0.38
FL |  83.37 $\pm$ 0.07 |  79.33 $\pm$ 0.08 |  74.28 $\pm$ 0.44
GCE |  85.93 $\pm$ 0.23 |  80.88 $\pm$ 0.38 |  74.29 $\pm$ 0.43
SCE |  86.20 $\pm$ 0.37 |  81.38 $\pm$ 0.35 |  75.16 $\pm$ 0.39
NLNL |  84.74 $\pm$ 0.08 |  81.26$\pm$ 0.43 |  76.97 $\pm$ 0.52
APL |  86.50 $\pm$ 0.31 |  83.34 $\pm$ 0.39 |  77.14 $\pm$ 0.33
CE+SR | 87.70 $\pm$ 0.19 | 85.63 $\pm$ 0.07 | 79.29 $\pm$ 0.20
FL+SR | 87.56 $\pm$ 0.29 | 85.10 $\pm$ 0.23 | 79.07 $\pm$ 0.50
GCE+SR | 87.55 $\pm$ 0.08 | 84.69 $\pm$ 0.46 | 79.01 $\pm$ 0.18
CIFAR-100 | CE |  58.11 $\pm$ 0.32 |  50.68 $\pm$ 0.55 |  40.17 $\pm$ 1.31
FL |  58.05 $\pm$ 0.42 |  51.15 $\pm$ 0.84 |  41.18 $\pm$ 0.68
GCE |  59.35 $\pm$ 1.10 |  53.83 $\pm$ 0.64 |  40.91 $\pm$ 0.57
NLNL |  50.19 $\pm$ 0.56 |  42.81 $\pm$ 1.13 |  35.10 $\pm$ 0.20
SCE |  58.16 $\pm$ 0.73 |  50.98 $\pm$ 0.33 |  41.54 $\pm$ 0.52
APL |  62.80 $\pm$ 0.05 |  56.74 $\pm$ 0.53 |  42.61 $\pm$ 0.24
CE+SR | 64.79 $\pm$ 0.01 | 59.09 $\pm$ 2.10 | 49.51 $\pm$ 0.59
FL+SR | 64.61 $\pm$ 0.67 | 58.94 $\pm$ 0.33 | 46.94 $\pm$ 1.68
GCE+SR | 64.35 $\pm$ 0.78 | 57.22 $\pm$ 0.80 | 49.51 $\pm$ 1.31

Results. The test accuracies (mean $\pm$ std) under symmetric label noise are
reported in Table 1. As we can see, our proposed SR mechanism significantly
improves the robustness of CE, FL, and GCE, which achieve the top 3 best
results in most test cases across all datasets. In the scenarios of serious
noise, our CE+SR, FL+SR have a very obvious improvement over the original
losses. For example, on MNIST with 0.8 symmetric noise, CE+SR outperforms CE
by more than 52%. On CIFAR-10 with 0.8 symmetric noise, CE+SR outperforms CE
by more than 31%. As for CIFAR-100, GCE and APL outperform our method by a
small gap for $\eta=0.6$ and $0.8$, but they failed in the cases with small
noise rates. The reason is that the fitting ability is not enough, which can
be derived according to the experiments in the clean case. When APL and GCE
meet a complicated dataset CIFAR-100 in a clean setting, their test accuracies
are worse than the commonly used losses CE and FL, while the SR-enhanced
methods outperform the original losses and achieve an improvement of $\geq
0.86$. Therefore, our methods not only have good robustness, but also
guarantee and even improve the fitting ability.

Results for asymmetric noise are reported in Table 2. Again, our methods
significantly improve the robustness of the original version across all
datasets and achieve top 3 best results in most cases. On MNIST, CE+SR and
FL+SR outperform all the state-of-the-art methods over all asymmetric noise by
a clear margin. More Surprisingly, the test accuracy (99.36 $\pm$ 0.05) of
FL+SR under 0.4 asymmetric noise is higher than the clean case. On CIFAR-10
with 0.1 asymmetric noise, SCE has the best accuracy, but it loses the
superiority in the other three cases where our method works much better than
all other baselines with at least 1% increase. On CIFAR-100, the enhanced loss
functions show particularly superior performance in all cases.

Representations. We further investigate the representations learned by CE+SR
compared to those learned by CE. We extract the high-dimensional features at
the second last full-connected layer, then project all test samples’ features
into 2D embeddings by t-SNE [26]. The projected representations on MNIST with
different symmetric noise are illustrated in Fig. 5. As can be observed, CE
encounters severe overfitting on label noise, and the embeddings look
completely mixed when $\eta=0.8$. On the contrary, CE+SR learns good
representations with more separated and clearly bounded clusters in all noisy
cases.

###  3.3 Evaluation on Real-world Noisy Dataset

Here, we evaluate our sparse regularization method on large-scale real-world
noisy dataset WebVision 1.0 [17]. It contains 2.4 million images with real-
world noisy labels, which crawled from the web using 1,000 concepts in
ImageNet ILSVRC12 [5]. Since the dataset is very big, for quick experiments,
we follow the training setting in [11] that only takes the first 50 classes of
the Google resized image subset. We evaluate the trained network on the same
50 classes of WebVision 1.0 validation set, which can be considered as a clean
validation set. We add sparse regularization to CE and GCE. The training
details follow [20], where for each loss , we train a ResNet-50 [10] by using
SGD for 250 epochs with initial learning rate 0.4, nesterov momentum 0.9,
weight decay $3\times 10^{-5}$, and batch size 512. The learning rate is
multiplied by 0.97 after every epoch of training. All the images are resized
to $224\times 224$. Typical data agumentations including random width/height
shift, color jittering, and random horizontal flip are applied. As shown in in
Table 3, our proposed SR mechanism obviously enhances the performance of CE
and FL, which outperform the existing loss functions SCE and APL with a clear
margin ($\geq 2.2\%$). This verifies the effectiveness of SR against real-
world label noise.

Table 3: Top-1 validation accuracies (%) on WebVision validatoin set of ResNet-50 models trained on WebVision using different loss functions, under the Mini setting in [11]. Loss | CE | FL | SCE | APL | CE+SR | FL+SR
---|---|---|---|---|---|---
Acc | 66.96 | 63.80 | 66.92 | 66.32 | 69.12 | 70.28

More Comparison. We also compare with Co-teaching[9], which is the
representative work of sample selection, and PHuber-CE [23], which is a simple
variant of gradient clipping. As shown in Table 4, our method works better
than Co-teaching and PHuber-CE.

Table 4: A comparison with other methods  Dataset | Method | Label Noise Type
---|---|---
sy 0.6 | sy 0.8 | asy 0.3 | asy 0.4
CIFAR-10 | CE | 38.70 | 19.55 | 79.32 | 74.67
Co-teaching | 65.74 | 38.01 | 64.01 | 51.26
PHuber-CE | 75.44 | 41.18 | 76.06 | 55.78
CE+SR | 78.18 | 51.13 | 85.63 | 79.29
CIFAR-100 | CE | 21.39 | 7.59 | 50.68 | 40.17
Co-teaching | 34.28 | 7.94 | 42.82 | 33.67
PHuber-CE | 21.54 | 9.33 | 26.91 | 23.43
CE+SR | 44.95 | 17.35 | 59.09 | 49.51

###  3.4 Additional Experiments

Imbalanced Classification. As shown in Table 3, our method achieves the best
result on WebVision that has a certain class imbalance. To better show the
performance on class imbalance, we additionally test the ability of SR on the
pure imbalanced classification task. We first follow the controllable data
imbalance in [21] to create the imbalanced CIFAR-10/-100 by reducing the
number of training examples and keeping the validation set unchanged. We also
consider two imbalance types: long-tailed imbalance[4] and step imbalance [2].
The results shown in Table 5 are encouraging, where CE with sparse
regularization achieves a very significant improvements compared with CE in
all cases. Albeit simple, these additional experiments demonstrate that SR can
also mitigate class imbalance well.

Table 5: Validation accuracy on imbalanced CIFAR-10/-100. Dataset | Method | Imbalanced Type
---|---|---
lt-0.01 | lt-0.1 | step-0.01 | step-0.1
CIFAR-10 | CE | 64.16 | 81.81 | 57.44 | 79.35
CE+SR | 69.78 | 84.49 | 61.03 | 82.11
CIFAR-100 | CE | 35.17 | 51.43 | 37.92 | 53.43
CE+SR | 41.24 | 59.51 | 40.21 | 58.42

##  4 Conclusion and Future Work

In this paper, we presented a novel method for learning with noisy labels. We
first provided a theoretical conclusion that any loss can be made robust to
noisy labels by restricting the output of a network to a permutation set of
any fixed vector. According to this principle, subsequently, we proposed a
simple but effective strategy for robust learning through sparse
regularization, which is the approximation of the constraint of one-hot
permutation. The meanings of sparse regularization are two-fold: the network
output sharpening operation is designed to enforce the output distribution of
a network to be sharp, and the $\ell_{p}$-norm ($p\leq 1$) regularization is
tailored to promote the network output to be sparse. Experimental results
demonstrated the superior performance of the proposed method over the SOTA
methods on both synthetic and real-world datasets. Moreover, we additionally
experiment with imbalanced classification, and the results are encouraging,
which demonstrates that sparse regularization can also mitigate class
imbalance well.

Overall, this paper has investigated the one-hot constraint, i.e., restricting
the output to a permutation set over a one-hot vector. In future research, a
promising direction is replacing the one-hot vector with a fixed smoothing
vector.

## Acknowledgments

This work was supported by National Natural Science Foundation of China under
Grants 61922027, 62071155 and 61932022.

## References

  * [1] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al.  A closer look at memorization in deep networks.  In International Conference on Machine Learning, pages 233–242. PMLR, 2017.
  * [2] Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski.  A systematic study of the class imbalance problem in convolutional neural networks.  Neural Networks, 106:249–259, 2018.
  * [3] Nontawat Charoenphakdee, Jongyeong Lee, and Masashi Sugiyama.  On symmetric losses for learning from corrupted labels.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 961–970, Long Beach, California, USA, 09–15 Jun 2019. PMLR.
  * [4] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie.  Class-balanced loss based on effective number of samples.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9268–9277, 2019.
  * [5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [6] Aritra Ghosh, H. Kumar, and P. S. Sastry.  Robust loss functions under label noise for deep neural networks.  In AAAI, 2017.
  * [7] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio.  Deep learning, volume 1.  MIT press Cambridge, 2016.
  * [8] B. Han, Quanming Yao, T. Liu, Gang Niu, I. Tsang, James T. Kwok, and M. Sugiyama.  A survey of label-noise representation learning: Past, present and future.  ArXiv, abs/2011.04406, 2020.
  * [9] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W Tsang, and Masashi Sugiyama.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  In NeurIPS, 2018.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [11] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei.  Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels.  In International Conference on Machine Learning, pages 2304–2313. PMLR, 2018.
  * [12] Davood Karimi, Haoran Dou, Simon K Warfield, and Ali Gholipour.  Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis.  Medical Image Analysis, 65:101759, 2020.
  * [13] Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim.  Nlnl: Negative learning for noisy labels.  In Proceedings of the IEEE International Conference on Computer Vision, pages 101–110, 2019.
  * [14] A. Krizhevsky and G. Hinton.  Learning multiple layers of features from tiny images.  Computer Science Department, University of Toronto, Tech. Rep, 1, 01 2009.
  * [15] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [16] Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak.  Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks.  In International Conference on Artificial Intelligence and Statistics, pages 4313–4324. PMLR, 2020.
  * [17] Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool.  Webvision database: Visual learning and understanding from web data.  08 2017.
  * [18] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [19] Wei Liu, Yu-Gang Jiang, Jiebo Luo, and Shih-Fu Chang.  Noise resistant graph ranking for improved web image search.  In CVPR 2011, pages 849–856. IEEE, 2011.
  * [20] Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey.  Normalized loss functions for deep learning with noisy labels.  In ICML, 2020.
  * [21] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts.  Learning word vectors for sentiment analysis.  In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142–150, 2011\.
  * [22] N. Manwani and P. S. Sastry.  Noise tolerance under risk minimization.  IEEE Transactions on Cybernetics, 43(3):1146–1151, 2013.
  * [23] Aditya Krishna Menon, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar.  Can gradient clipping mitigate label noise?  In International Conference on Learning Representations, 2020.
  * [24] Sebastian Prillo and Julian Eisenschlos.  SoftSort: A continuous relaxation for the argsort operator.  In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7793–7802. PMLR, 13–18 Jul 2020\.
  * [25] Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee.  Learning from noisy labels with deep neural networks: A survey.  arXiv preprint arXiv:2007.08199, 2020.
  * [26] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [27] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey.  Symmetric cross entropy for robust learning with noisy labels.  In IEEE International Conference on Computer Vision, 2019.
  * [28] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning requires rethinking generalization.  In International Conference on Learning Representation, 2017.
  * [29] Zhilu Zhang and Mert R Sabuncu.  Generalized cross entropy loss for training deep neural networks with noisy labels.  In NeurIPS, 2018.
  * [30] Mingyi Zhou, Jing Wu, Yipeng Liu, Shuaicheng Liu, and Ce Zhu.  Dast: Data-free substitute training for adversarial attacks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 234–243, 2020.
  * [31] Xiong Zhou, Xianming Liu, Junjun Jiang, Xin Gao, and Xiangyang Ji.  Asymmetric loss functions for learning with noisy labels.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12846–12856. PMLR, 18–24 Jul 2021.

Learning with Noisy Labels via Sparse Regularization:
Supplementary Materials

##  Appendix A Proof of Theorems

######  Theorem 1.

In a multi-class classification problem, $\forall L\in\mathcal{L}$, $L$ is
noise-tolerant under symmetric label noise if $\eta<1-\frac{1}{k}$ and
$f:\mathcal{X}\rightarrow\mathcal{P}_{\mathbf{v}}$, where $\mathbf{v}$ is a
fixed vector, i.e.,

$\mathop{\arg\min}_{f:\mathcal{X}\rightarrow\mathcal{P}_{\mathbf{v}}}R_{L}(f)=\mathop{\arg\min}_{f:\mathcal{X}\rightarrow\mathcal{P}_{\mathbf{v}}}R^{\eta}_{L}(f).$
(11)

###### Proof.

For symmetric label noise, we have

$\displaystyle R_{L}^{\eta}(f)$
$\displaystyle=\mathbb{E}_{\mathbf{x},y}\left[(1-\eta)L(f(\mathbf{x}),y)+\frac{\eta}{k-1}\sum_{i\not=y}L(f(\mathbf{x}),i)\right]$
$\displaystyle=\mathbb{E}_{\mathbf{x},y}\left[(1-\eta)L(f(\mathbf{x}),y)+\frac{\eta}{k-1}(\sum_{i=1}^{k}L(f(\mathbf{x}),i)-L(f(\mathbf{x}),y)\right]$
$\displaystyle=(1-\eta)R_{L}(f)+\frac{\eta}{k-1}(\sum_{i=1}^{k}L(\mathbf{v},i)-R_{L}(f))$
$\displaystyle=(1-\frac{\eta
k}{k-1})R_{L}(f)+\frac{\eta}{k-1}\sum_{i=1}^{k}L(\mathbf{v},i)$

since $1-\frac{\eta k}{k-1}>0$ and $\sum_{i=1}^{k}L(\mathbf{v},i)$ is a
constant, then $f^{*}$ minimizes $R^{\eta}(f)$ if and only if $f^{*}$
minimizes $R(f)$. ∎

######  Theorem 2.

In a multi-class classification problem, we let
$f:\mathcal{X}\rightarrow\mathcal{P}_{\mathbf{v}}$, where $\mathbf{v}$ is a
fixed vector. If $R_{L}(f^{*})=0$, $\forall
f:\mathcal{X}\rightarrow\mathcal{P}_{\mathbf{v}}$, $\forall L\in\mathcal{L}$
and $0\leq L\leq\frac{C}{k-1}$, $L$ is noise-tolerant under asymmetric or
class-conditional noise when $\eta_{\mathbf{y},i}<1-\eta_{y}$ with
$\sum_{k\not=y}\eta_{y,i}=\eta_{y}$, $\forall\mathbf{x}$.

###### Proof.

For asymmetric or class-conditional noise, we have

$\displaystyle R_{L}^{\eta}(f)$
$\displaystyle=\mathbb{E}_{\mathbf{x},y}(1-\eta_{y})L(f(\mathbf{x}),y)+\mathbb{E}_{\mathbf{x},y}\sum_{i\not=y}\eta_{y,i}L(f(\mathbf{x}),i)$
(12)
$\displaystyle=\mathbb{E}_{\mathbf{x},y}(1-\eta_{y})(C-\sum_{i\not=y}L(f(\mathbf{x}),i))+\mathbb{E}_{\mathbf{x},y}\sum_{i\not=y}\eta_{y,i}L(f(\mathbf{x}),i)$
$\displaystyle=C\mathbb{E}_{\mathbf{x},y}(1-\eta_{y})-\mathbb{E}_{\mathbf{x},y}\sum_{i\not=y}(1-\eta_{y}-\eta_{y,i})L(f(\mathbf{x}),i)$

Let $f^{*}_{\eta}$ and $f^{*}$ be the minimizer of $R_{L}^{\eta}(f)$ and
$R_{L}(f)$ when $f:\mathcal{X}\rightarrow\mathcal{P}_{\mathbf{v}}$,
respectively. We have $R_{L}^{\eta}(f^{*}_{\eta})-R_{L}^{\eta}(f^{*})\leq 0$
and hence derive that

$\mathbb{E}_{\mathbf{x},y}\sum_{i\not=y}(1-\eta_{y}-\eta_{y,i})(L(f(\mathbf{x}),i)-L(f(\mathbf{x}),i))\leq
0$ (13)

Since we are given $R_{L}(f^{*})=0$, we have $L(f^{*}(\mathbf{x}),y)=0$. Given
the condition on $L$ in the theorem, this implies
$L(f^{*}(\mathbf{x}),i)=C/(k-1)$, $i\not=y$. As per the assumption on noise in
the theorem, $1-\eta_{y}-\eta_{y,i}>0$. Moreover, $L$ has to satisfy
$L(f^{*}_{\eta}(\mathbf{x}),i)\leq C/(k-1)$, $\forall i$. Thus for Eq. 13 to
hold, it must be the case that $L(f^{*}_{\eta}(\mathbf{x}),i)=C/(k-1)$, which
implies $L(f^{*}_{\eta}(\mathbf{x}),y)=0$. Thus, the minimizer of true risk is
also a minimizer of risk under noisy case. ∎

######  Theorem 3.

In a multi-class classification problem, if the loss function $L$ satisfies
$|\sum_{i=1}^{k}(L(\mathbf{u}_{1},i)-L(\mathbf{u}_{2},i))|\leq\delta$ when
$\|\mathbf{u}_{1}-\mathbf{u}_{2}\|_{2}\leq\varepsilon$, and $\delta\rightarrow
0$ as $\varepsilon\rightarrow 0$, then for symmetric label noise satisfying
$\eta<1-\frac{1}{k}$, the risk bound can be expressed as

$R_{L}(f^{*}_{\eta})-R_{L}(f^{*})\leq 2c\delta,$

where $c=\frac{\eta}{(1-\eta)k-1}$, $f^{*}_{\eta}$ and $f^{*}$ denote the
minimizer of $R_{L}^{\eta}(f)$ and $R_{L}(f)$ when
$f\in\mathcal{H}_{\mathbf{v},\varepsilon}$, respectively.

###### Proof.

For symmetric label noise, we have

$\displaystyle R_{L}^{\eta}(f^{*})$
$\displaystyle=\mathbb{E}_{\mathbf{x},y}\big{[}(1-\eta)L(f^{*}(\mathbf{x}),y)+\frac{\eta}{k-1}\sum_{i\not=y}L(f^{*}(\mathbf{x}),i)\big{]}$
(14) $\displaystyle=(1-\frac{\eta
k}{k-1})R_{L}(f^{*})+\frac{\eta}{k-1}\mathbb{E}_{\mathbf{x},y}\left[\sum_{i=1}^{k}L(f^{*}(\mathbf{x}),i)\right]$
$\displaystyle=(1-\frac{\eta
k}{k-1})R_{L}(f^{*})+\frac{\eta}{k-1}\sum_{i=1}^{k}L(\mathbf{v},i)+\frac{\eta}{k-1}\delta_{1}$

where
$\delta_{1}=\mathbb{E}_{\mathbf{x},y}[\sum_{i=1}^{k}L(f(\mathbf{x}),i)-\sum_{i=1}^{k}L(\mathbf{v},i)]$.
On the other hand, $f^{*}\in\mathcal{H}_{\mathbf{v},\varepsilon}$, i.e.,
$\|f^{*}(\mathbf{x})-\mathbf{v}\|_{2}\leq\varepsilon$, so we have
$|\sum_{i=1}^{k}L(f(\mathbf{x}),i)-\sum_{i=1}^{k}L(\mathbf{v},i)|\leq\delta$.
This means that $\delta_{1}\in[-\delta,\delta]$. Similarly, we can obtain

$R_{L}^{\eta}(f^{*}_{\eta})=(1-\frac{\eta
k}{k-1})R_{L}(f^{*}_{\eta})+\frac{\eta}{k-1}\sum_{i=1}^{k}L(\mathbf{v},i)+\frac{\eta}{k-1}\delta_{2}$
(15)

Since
$f^{*}_{\eta}=\mathop{\arg\min}_{f\in\mathcal{H}_{\mathbf{v},\varepsilon}}R_{L}^{\eta}(f)$,
and
$f^{*}=\mathop{\arg\min}_{f\in\mathcal{H}_{\mathbf{v},\varepsilon}}R_{L}(f)$,
we have

$\displaystyle 0\geq R_{L}^{\eta}(f^{*}_{\eta})-R_{L}^{\eta}(f^{*})$ (16)
$\displaystyle=(1-\frac{\eta
k}{k-1})(R_{L}(f^{*}_{\eta})-R_{L}(f^{*}))+\frac{\eta}{k-1}(\delta_{2}-\delta_{1})$
$\displaystyle\Rightarrow$ $\displaystyle
R_{L}(f^{*}_{\eta})-R_{L}(f^{*})\leq\frac{\eta}{(1-\eta)k-1}(\delta_{1}-\delta_{2})\leq\frac{2\eta\delta}{(1-\eta)k-1}$

where we have used the fact that $1-\frac{\eta k}{k-1}>0$, and
$\delta_{2}-\delta_{1}\leq 2\delta$ holds for
$\delta_{1},\delta_{2}\in[-\delta,\delta]$. ∎

##  Appendix B Experiments

In this section, we provide the experimental details.

Datasets. We verify the effectiveness of our method on benchmark datasets,
including MNIST [15], CIFAR-10/-100 [14] with synthetic label noise.

Since MNIST, CIFAR-10, and CIFAR-100 are clean, following previous works [27,
20], we experiment with two types of label noise: symmetric (uniform) noise
and asymmetric (class-conditional) noise. For symmetric noise, we corrupt the
training labels by flipping the labels in each class randomly to incorrect
labels in other classes with flip probability $\eta\in\\{0.2,0.4,0.6,0.8\\}$.
For asymmetric noise, we flip the labels within a specific set of classes, for
example, for MNIST, flipping 2 $\rightarrow$ 7, 7 $\rightarrow$ 1, 5
$\leftrightarrow$ 6, and 3 $\rightarrow$ 8; for CIFAR-10, flipping TRUCK
$\rightarrow$ AUTOMOBILE, BIRD $\rightarrow$ AIRPLANE, DEER $\rightarrow$
HORSE, and CAT $\leftrightarrow$ DOG; for CIFAR-100, the 100 classes are
grouped into 20 super-classes with each has 5 sub-classes, and each class are
flipped within the same super-classes into the next.

Baselines. We experiment with the following state-of-the art methods, and two
effective loss functions CE and Focal Loss (FL) [18] for classification.
Moreover, we add the proposed sparse regularization mechanism to CE, FL and
GCE, i.e., CE+SR, FL+SR and GCE+SR. All the implementations and experiments
are based on PyTorch.

  * •

GCE [29]. The Generalized Cross Entropy (GCE) is defined as
$L_{GCE}(\mathbf{u},i)=(1-u_{i}^{q})/q$ ($0<q\leq 1$).

  * •

SCE [27]. The Symmetric Cross Entropy (SCE) can be regarded as a weighted loss
of CE and RCE (scaled MAE): $L_{SCE}(\mathbf{u},i)=\alpha
L_{CE}(\mathbf{u},i)+\beta L_{RCE}(\mathbf{u},i)$.

  * •

NLNL [13]. NLNL improves robustness with a complementary label.

  * •

APL [20]. The Active Passive Loss (APL) was proposed to combine a robust
active loss and a robust passive loss, i.e., $L_{APL}=\alpha L_{Active}+\beta
L_{Passive}$.

Network Structure and Training Details. Following the setting in [20], we use
a 4-layer CNN for MNIST, an 8-layer CNN for CIFAR-10 and a ResNet-34 for
CIFAR-100. The networks are trained for 50, 120, 200 epochs for MNIST,
CIFAR-10, CIFAR-100, respectively. For all the training, we use SGD optimizer
with momentum 0.9 and cosine learning rate annealing. Weight decay is set as
$1\times 10^{-3}$, $1\times 10^{-4}$, $1\times 10^{-5}$ for MNIST, CIFAR-10,
CIFAR-100, respectively. The initial learning rate is set to 0.01 for CIFAR-10
and 0.1 for CIFAR-100. Batch size is set to 128. Typical data augmentations
including random width/height shift and horizontal flip are applied.

Parameters Setting. We set the parameters which match their original papers
for all baseline methods. Specifically, for FL, we set $\gamma=0.3$. For GCE,
we set $q=0.7$. For SCE, we set $A=-3$, and $\alpha=0.01$, $\beta=1$ for
MNIST, $\alpha=0.1$, $\beta=1$ for CIFAR-10, $\alpha=6$, $\beta=0.1$ for
CIFAR-100. For APL (NCE+MAE), we set $\alpha=1$, $\beta=100$ for MNIST,
$\alpha,\beta=1$ for CIFAR-10, and $\alpha=10$, $\beta=0.1$ for CIFAR-100. For
our sparse regularization, we set
$(\tau,p,\lambda_{0},\rho,r)=(0.1,0.1,4,2,5)$ for MNIST,
$(0.5,0.1,1.1,1.03,1)$ for CIFAR-10, and $(0.5,0.01,\cdot,1.02,1)$ for
CIFAR-100. Otherwise, on CIFAR-100, we set $\lambda_{0}$ to $10$ and $4$ for
symmetric and asymmetric label noise, respectively.

As for the parameter setting for Webvision, we use the suggested $q=0.7$ for
GCE, $A=-4$, $\alpha=10$, $\beta=1$ for SCE, while for APL, we set
$\alpha=50$, $\eta=0.1$. For our CE+SR and FL+SR, we set $\tau=0.5$, $p=0.01$,
$\lambda_{0}=2$, $\rho=1.02$ and $f=1$.

More experiments about hyperparameter selection. We offer more experimental
results on selecting different $\lambda$ on CIFAR-10 with 0.6 symmetric label
noise. We adjust $\tau$ from 1.0 to 0.5. The results are shown in Fig. 2. We
found that the output sharpening can benefit the sparse regularization. We can
achieve the similar robustness result of $\lambda=20$ ($\tau=1$) by setting
$\lambda=5$ and $\tau=0.5$, which demonstrates that the output sharpening also
plays the role of sparse regularization when using $\ell_{p}$-norm. Moreover,
smaller $\lambda$ can help maintain the fitting ability of the model with
classification loss $L(f(\mathbf{x}),y)$ (i.e., learning efficiently while
keeping robustness). As a evidence, the eventual accuracy ($\tau=0.5$,
$\lambda=5$) is higher than the experiments with $\tau=1.0$.

More results of Comparison study. Fig. 7 shows test accuracy vs. epochs on
MNIST. As can be observed, the commonly-used loss functions CE and FL suffer
from significant overfitting in all noisy cases. The state-of-the-art methods
GCE, SCE and APL show non-trival effectiveness of mitigating label noise, but
the effects are crippled when meeting hard label noise. On the contrary, our
proposed SR-enhanced methods CE+SR, FL+SR and GCE+SR perform better robustness
and more efficiency . Fig. 8 shows test accuracy vs. epochs on CIFAR-10. The
results are similar to MNIST, our SR-enhanced methods keep robust and achieve
the best accuracy in most cases. Fig. 9 shows test accuracy vs. epochs on
CIFAR-100. Our methods are of better fitting ability than commonly-used losses
in the clean case, while the state-of-the-art GCE, SCE and APL encounter
little underfitting. For 0.2 and 0.4 symmetric label noise, our methods
perform the best test accuracy. Interestingly, for all asymmetric label noise,
our methods perform overfitting at the beginning, but they later mitigate
label noise and outperform other methods.

Table 6: Test accuracies (%) of different methods on benchmark datasets with clean or asymmetric label noise ($\eta\in[0.1,0.2,0.3,0.4]$). The results (mean$\pm$std) are reported over 3 random runs and the top 3 best results are boldfaced. Datasets | Methods |  Asymmetric Noise Rate ($\eta$)
---|---|---
0.1 | 0.2 | 0.3 | 0.4
MNIST | CE |  97.57 $\pm$ 0.22 |  94.56 $\pm$ 0.22 |  88.81 $\pm$ 0.10 |  82.27 $\pm$ 0.40
FL |  97.58 $\pm$ 0.09 |  94.25 $\pm$ 0.15 |  89.09 $\pm$ 0.25 |  82.13 $\pm$ 0.49
GCE |  99.01 $\pm$ 0.04 |  96.69 $\pm$ 0.12 |  89.12 $\pm$ 0.24 |  81.51 $\pm$ 0.19
SCE |  99.14 $\pm$ 0.04 |  98.03 $\pm$ 0.05 |  93.68 $\pm$ 0.43 |  85.36 $\pm$ 0.17
NLNL |  98.63 $\pm$ 0.06 |  98.35 $\pm$ 0.01 |  97.51 $\pm$ 0.15 |  95.84 $\pm$ 0.26
APL | 99.32 $\pm$ 0.09 |  98.89 $\pm$ 0.04 |  96.93 $\pm$ 0.17 |  91.45 $\pm$ 0.40
CE+SR | 99.42 $\pm$ 0.02 | 99.27 $\pm$ 0.06 | 99.24 $\pm$ 0.08 | 99.23 $\pm$ 0.07
FL+SR | 99.34 $\pm$ 0.05 | 99.31 $\pm$ 0.02 | 99.23 $\pm$ 0.02 | 99.36 $\pm$ 0.05
GCE+SR |  99.28 $\pm$ 0.06 | 99.22 $\pm$ 0.02 | 99.13 $\pm$ 0.05 | 99.09 $\pm$ 0.02
CIFAR-10 | CE |  87.55 $\pm$ 0.14 |  83.32 $\pm$ 0.12 |  79.32 $\pm$ 0.59 |  74.67 $\pm$ 0.38
FL |  86.43 $\pm$ 0.30 |  83.37 $\pm$ 0.07 |  79.33 $\pm$ 0.08 |  74.28 $\pm$ 0.44
GCE |  88.33 $\pm$ 0.05 |  85.93 $\pm$ 0.23 |  80.88 $\pm$ 0.38 |  74.29 $\pm$ 0.43
SCE | 89.77 $\pm$ 0.11 |  86.20 $\pm$ 0.37 |  81.38 $\pm$ 0.35 |  75.16 $\pm$ 0.39
NLNL |  88.54 $\pm$ 0.25 |  84.74 $\pm$ 0.08 |  81.26$\pm$ 0.43 |  76.97 $\pm$ 0.52
APL |  88.31 $\pm$ 0.20 |  86.50 $\pm$ 0.31 |  83.34 $\pm$ 0.39 |  77.14 $\pm$ 0.33
CE+SR | 89.08 $\pm$ 0.08 | 87.70 $\pm$ 0.19 | 85.63 $\pm$ 0.07 | 79.29 $\pm$ 0.20
FL+SR |  88.68 $\pm$ 0.23 | 87.56 $\pm$ 0.29 | 85.10 $\pm$ 0.23 | 79.07 $\pm$ 0.50
GCE+SR | 89.20 $\pm$ 0.23 | 87.55 $\pm$ 0.08 | 84.69 $\pm$ 0.46 | 79.01 $\pm$ 0.18
CIFAR-100 | CE |  64.85 $\pm$ 0.37 |  58.11 $\pm$ 0.32 |  50.68 $\pm$ 0.55 |  40.17 $\pm$ 1.31
FL |  64.78 $\pm$ 0.50 |  58.05 $\pm$ 0.42 |  51.15 $\pm$ 0.84 |  41.18 $\pm$ 0.68
GCE |  63.01 $\pm$ 1.01 |  59.35 $\pm$ 1.10 |  53.83 $\pm$ 0.64 |  40.91 $\pm$ 0.57
NLNL |  59.55 $\pm$ 1.22 |  50.19 $\pm$ 0.56 |  42.81 $\pm$ 1.13 |  35.10 $\pm$ 0.20
SCE |  64.26 $\pm$ 0.43 |  58.16 $\pm$ 0.73 |  50.98 $\pm$ 0.33 |  41.54 $\pm$ 0.52
APL |  66.48 $\pm$ 0.12 |  62.80 $\pm$ 0.05 |  56.74 $\pm$ 0.53 |  42.61 $\pm$ 0.24
CE+SR | 68.96 $\pm$ 0.22 | 64.79 $\pm$ 0.01 | 59.09 $\pm$ 2.10 | 49.51 $\pm$ 0.59
FL+SR | 68.96 $\pm$ 0.17 | 64.61 $\pm$ 0.67 | 58.94 $\pm$ 0.33 | 46.94 $\pm$ 1.68
GCE+SR | 69.27 $\pm$ 0.31 | 64.35 $\pm$ 0.78 | 57.22 $\pm$ 0.80 | 49.51 $\pm$ 1.31

(a) CE+SR ($\tau=1.0$)

(b) FL+SR ($\tau=1.0$)

(c) CE+SR ($\tau=0.5$)

(d) FL+SR ($\tau=0.5$)

Figure 6: Test accuracy curve of different $\lambda$ on CIFAR-10 with 0.6
symmetric label noise.

(a) $\eta=0.0$

(b) $\eta=0.2$ (c) $\eta=0.4$ (d) $\eta=0.6$

(e) $\eta=0.8$ (f) $\eta=0.1$ (g) $\eta=0.2$

(h) $\eta=0.3$ (i) $\eta=0.4$

Figure 7: Test accuracies of different methods on MNIST with different label
noise, where (a) denotes the clean case, (b-e) denote the symmetric label
noise, and (f-i) denote the asymmetric label noise.

(a) $\eta=0.0$

(b) $\eta=0.2$ (c) $\eta=0.4$ (d) $\eta=0.6$

(e) $\eta=0.8$ (f) $\eta=0.1$ (g) $\eta=0.2$

(h) $\eta=0.3$ (i) $\eta=0.4$

Figure 8: Test accuracies of different methods on CIFAR-10 with different
label noise, where (a) denotes the clean case, (b-e) denote the symmetric
label noise, and (f-i) denote the asymmetric label noise.

(a) $\eta=0.0$

(b) $\eta=0.2$ (c) $\eta=0.4$ (d) $\eta=0.6$

(e) $\eta=0.8$ (f) $\eta=0.1$ (g) $\eta=0.2$

(h) $\eta=0.3$ (i) $\eta=0.4$

Figure 9: Test accuracies of different methods on CIFAR-100 with different
label noise, where (a) denotes the clean case, (b-e) denote the symmetric
label noise, and (f-i) denote the asymmetric label noise.

More results of visualizations. More visualizations of representations on
different datasets are shown in Fig. 10, 11 and 12. As can be seen, the
representations learned by the proposed sparse regularization (SR)-enhanced
methods are more discriminative than those learned by original losses, which
are with a more separated and clearly bound margin.

(a) CE+SR with $\eta=0.1$

(b) CE+SR with $\eta=0.2$

(c) CE+SR with $\eta=0.3$

(d) CE+SR with $\eta=0.4$

(e) FL+SR with $\eta=0.1$

(f) FL+SR with $\eta=0.2$

(g) FL+SR with $\eta=0.3$

(h) FL+SR with $\eta=0.4$

(i) GCE+SR with $\eta=0.1$

(j) GCE+SR with $\eta=0.2$

(k) GCE+SR with $\eta=0.3$

(l) GCE+SR with $\eta=0.4$

Figure 10: Features visualization for CE+SR (top) and FL+SR (bottom) on MNIST
with different asymmetric label noise ($\eta\in[0.1,0.2,0.3,0.4]$) by t-SNE
[26] 2D embeddings at the last second full-connected layer.

(a) CE with $\eta=0.2$

(b) FL with $\eta=0.2$

(c) GCE with $\eta=0.2$

(d) CE+SR with $\eta=0.2$

(e) FL+SR with $\eta=0.2$

(f) GCE+SR with $\eta=0.2$

Figure 11: Features visualization for CE (top) and CE+SR (bottom) on CIFAR10
with 0.2 symmetric label noise by t-SNE [26] 2D embeddings at the last second
full-connected layer.

(a) CE with $\eta=0.8$

(b) FL with $\eta=0.8$

(c) GCE with $\eta=0.8$

(d) CE+SR with $\eta=0.8$

(e) FL+SR with $\eta=0.8$

(f) GCE+SR with $\eta=0.8$

Figure 12: Features visualization for CE (top) and CE+SR (bottom) on CIFAR10
with 0.8 symmetric label noise by t-SNE [26] 2D embeddings at the last second
full-connected layer.


# Personalized Automatic Speech Recognition Trained on Small Disordered Speech
Datasets

###### Abstract

This study investigates the performance of personalized automatic speech
recognition (ASR) for recognizing disordered speech using small amounts of
per-speaker adaptation data. We trained personalized models for 195
individuals with different types and severities of speech impairment with
training sets ranging in size from $<$1 minute to 18-20 minutes of speech
data. Word error rate (WER) thresholds were selected to determine Success
Percentage (the percentage of personalized models reaching the target WER) in
different application scenarios. For the home automation scenario, 79% of
speakers reached the target WER with 18-20 minutes of speech; but even with
only 3-4 minutes of speech, 63% of speakers reached the target WER. Further
evaluation found similar improvement on test sets with conversational and out-
of-domain, unprompted phrases. Our results demonstrate that with only a few
minutes of recordings, individuals with disordered speech could benefit from
personalized ASR.

Index Terms—  automatic speech recognition, speech disorders, personalized
models

##  1 Introduction

Voice controlled home automation technology offers convenience for many;
however, often the automatic speech recognition (ASR) systems that power these
technologies do not work for the millions of individuals with speech
impairments [1]. This population could arguably benefit the most from home
automation and other voice controlled assistive technology. People who have
speech impairments often also have mobility impairments, due to their
conditions. While ASR accuracy for typical speech may be as high as 95% [2,
3], accuracy drops off significantly for disordered speech with varying levels
of impairment severity and intelligibility [4].

Adaptation of speaker independent ASR systems with dysarthric speech in [5]
has shown promising results, but they cite the dearth of speech data as a
major hurdle. Recent work has shown the potential of personalizing ASR models
for recognizing disordered speech [6, 7, 8, 9, 10]. However, often-times these
promising results are based on relatively large amounts of speech recordings
per speaker (often in the range of hours). For people with speech impairment,
recording so many speech samples is often impractically difficult.

For example, [6] reports average WER improvements of 75% through model
personalization on the Euphonia corpus [11], a large corpus with speech
recordings from people with speech impairment. While these WER improvements
are very promising, it should be noted that this comes with significant
recording times per speaker: The median number of utterances per speaker in
that study is 1529, which on average results in about 2 hours of speech
recordings. Recording speech data (e.g. triggering recording start/stops and
flipping to the next prompt) is a significant investment of time and effort,
especially for individuals with motor or cognitive impairments. It has been
reported that 4-7 hours of recording time were required to contribute 1500
phrases [11]. Two recent studies [7, 12] suggest that large WER improvements
may be achieved with much smaller amounts of data. However, both studies don’t
analyze in greater detail how much data is actually needed, how different
levels of severity of speech impairment may impact the amount of data required
or implications for practical applications.

This paper aims to close this gap. We analyze how much speech data from people
with disordered speech is required for model personalization to achieve low
enough Word Error Rates (WER) to be usable for various voice technology
applications and human-in-the-loop conversations. Our ultimate goal is to make
technology usable with minimal possible effort demanded of the speaker.
Moreover, understanding how much speech data is required for model
personalization can help accurately set expectations for ASR performance and
better utilize the limited time and energy of people with speech impairments.

##  2 Methods

###  2.1 Dataset

We use a subset of the Euphonia corpus [11]. This corpus consists of over 1
million samples (over 1300 hours) of more than 1000 anonymized speakers with
different types and severity levels of speech impairments. All our experiments
are performed on a subset of 195 speakers who have each recorded more than
1000 utterances. The resulting subset is diverse in terms of severity of
speech impairment (7.7% typical, 35.4% mild, 31.8% moderate, 25.1% severe) and
covered etiologies (39.2% with amyotrophic lateral sclerosis (ALS), 13.4%
cerebral palsy, 10.8% Parkinson’s Disease, 5.2% hearing impairment, 3.1%
Ataxia etc). We use the predefined per-speaker train, dev, and test splits
(80%/10%/10%) of the Euphonia corpus, which ensures that there is no phrase
overlap between these splits. The median number of utterances per speaker
amounts to 1788 for these 195 speakers, but variance here is high (see Figure
1). For our study, subsets of the training data of various sizes (10, 30, 50,
100, 250 utterances) were created by randomly sampling from all utterances
available per speaker. For each size, 10 subsets were created by repeated
random sampling. Subsets can overlap due to sampling with replacement between
sets. Table 1 shows the resulting subsample sizes along with the average
duration in minutes.

Fig. 1: Total number of training utterances across all 195 speakers. Median: 1788 utterances (2.1 hours).  Number of | Average duration
---|---
utterances | in minutes (std dev)
250 | 18.1 (6.6)
100 | 7.3 ( 2.7)
50 | 3.6 (1.3)
30 | 2.2 (0.8)
10 | 0.7 (0.3)
Table 1: Number of utterances and average duration of recordings averaged
across speakers.

###  2.2 ASR Model Personalization

For all our experiments, we employ the Recurrent Neural Network Transducers
(RNN-T) architecture [13, 14, 15] which allows deployment on mobile devices,
supports streaming, and has demonstrated high performance. We use 8 LSTM
encoder layers and 2 LSTM layers for the language model component. Input
features are 128-dimensional log Mel features computed every 10 milliseconds.
4 consecutive features are stacked with a stride of 3 frames to yield a
512-dimensional input to the encoder every 30 milliseconds. There are 4096
output units that correspond to the word piece tokens.

We follow the same fine-tuning recipe as described in [6]: We start from a
speaker-independent base model pre-trained on 162k hours of typical speech.
This base model has been optimized to (a) be robust across various application
domains and acoustic conditions, and (b) generalize well to unseen conditions
[16]. We only update the first 5 encoder layers. SpecAugment [17] is used for
data augmentation. Models were trained with a maximum number of steps based on
training set size: 5000 steps for sets with $<$100 utterances, 10000 steps for
100 utterances, and 15000 steps for 250 utterances. The best checkpoint was
picked using each speaker’s dev set.

###  2.3 Evaluation

For all our experiments, we use Euphonia’s default test set splits, which for
the selected speakers consist of at least 100 utterances. We report WER on two
specific domains of the Euphonia corpus: phrases associated with a) home
automation111 These are short phrases of 3.2 words on average, eg, ”Turn on
the radio” and b) human conversation222Longer phrases of 7.4 words on average
with open vocabulary.. 93 of the 195 speakers recorded a conversational test
set, while all speakers recorded a home automation test set. Speaker WER
reported at each subsample size is an average across the 10 models
personalized with the 10 random subsamples of the same size.

We chose different WER thresholds as success criteria for our two application
scenarios: Home automation and conversational domains. Voice assistants –
Apple’s Siri, Amazon’s Alexa, Microsoft’s Cortana, and Google’s Assistant
being the most popular ones – typically utilize Natural Language Understanding
capabilities to infer user intent allowing them to function even without
perfect transcriptions from the ASR system. We found that on Google Assistant,
a WER of 15% corresponds to a success rate of around 80% for home automation
commands. The conversational domain consists of less structured, longer and
more complex phrases. Accordingly, we selected a higher WER target of 20%,
assuming that this renders transcriptions useful in a human conversation.

##  3 Results

|  | target WER
---|---|---
domain | speakers | 5 | 10 | 15 | 20
home automation | 195 | 59% | 81% | 90% | 94%
conversational | 93 | 18% | 59% | 74% | 82%
Table 2: Success percentage (portion of speakers reaching target WER) when
personalizing with all their data.

We aim to understand the expected portion of users for which a personalized
model trained on a specific amount of data will work in a satisfactory manner.
We hence report Success Percentage – the percentage of speakers to reach a
domain specific target WER threshold.

Table 2 shows the Success Percentage for four WER thresholds when using all
the data available per speaker (as per Figure 1). The conversational domain is
more challenging and percentages are significantly lower than for home
automation. Yet, when using all data per speaker we reach a target WER of 15
for 90% of the speakers on home automation, and a target WER of 20 for 82% of
the speakers on the conversational domain. These numbers serve as baselines.

###  3.1 Percentage of speakers to reach a target WER

Fig. 2: Success percentage on home automation domain, different target WERs
(dashed lines: 95% confidence interval). Fig. 3: Success Percentage on
conversational domain, different target WERs (dashed lines: 95% confidence
interval).

Figures 2 and 3 show Success Percentages for both domains for several
different target WERs. In both cases, the choice of target WER significantly
impacts the results. On the home automation domain for a target WER of 15, 250
utterances yield a Success Percentage of 79%; with as little as 50 utterances,
the Success Percentage is 63%. This marks a significant improvement compared
to the unadapted, out-of-the-box model, which reaches this target WER for only
35% of the speakers with speech impairment. On the conversational domain,
Success Percentages are significantly lower and we restrict our analysis to
two higher target WER scenarios. For a target WER of 20, the Success
Percentage is about 73% for speakers’ models personalized on 250 utterances.
This is a clear improvement compared to the base model which yields the same
target WER only for 29% of speakers.

Overall, this analysis shows that many speakers need only a small amount of
data to achieve satisfactory model performance. Many speakers will not need to
record more than 250 utterances, with the biggest improvements coming from the
first 50. Some users may still want to record more than 250 utterances in
order to get sufficient improvement, albeit with marginal gains at a
relatively high recording price (e.g. By recording an order of magnitude more
data than 250 utterances, a median of 1788 utterances, the Success Percentage
in the home automation domain increases by 11% more speakers and 9% more
speakers in the conversational domain).

###  3.2 Break-down by severity of speech impairment

| num | number of utterances
---|---|---
severity | spkrs | 0 | 10 | 50 | 250 | All
Typical | 15 | 100% | 100% | 100% | 100% | 100%
Mild | 69 | 45% | 78% | 93% | 96% | 99%
Mod | 62 | 10% | 32% | 60% | 81% | 94%
Severe | 49 | 10% | 8% | 27% | 47% | 69%
Table 3: Percentage of speakers reaching target WER of 15 on home automation domain, split by severity. | num | number of utterances
---|---|---
severity | spkrs | 0 | 10 | 50 | 250 | All
Typical | 4 | 100% | 100% | 100% | 100% | 100%
Mild | 29 | 41% | 62% | 86% | 93% | 97%
Mod | 37 | 16% | 35% | 46% | 70% | 86%
Severe | 23 | 17% | 22% | 26% | 48% | 52%
Table 4: Percentage of speakers reaching target WER of 20% on conversational
domain, split by severity.

To further understand how much personalization data is really necessary for a
given speaker, we here report Success Percentages for the different levels of
severity of speech impairment. For home automation, we chose a target WER of
15 (Table 3), for conversational we chose a target WER of 20 (Table 4). For
all speakers which are rated as ”typical” (i.e. no speech impairment), out-of-
the-box ASR models work well with a maximum WER of 9 (home automation) and 12
(conversational). As severity increases, we observe that the maximum possible
Success Percentage using all data decreases: only 52% of speakers with a
severe speech impairment on conversational and 69% of those speakers on home
automation. With even fewer training data (especially $<$100 utterances),
success percentages here sharply decline.

On the other hand, for speakers with a mild speech impairment, small amounts
of training data yield big Success Percentages, getting close to the rates
achievable with all data: With only 50 utterances (i.e. about 3-4 minutes of
recorded speech, Figure 1) we can expect 90% of these speakers to yield the
target WER of 15%. This is not far off the 99% Success Percentage of the ”all
data” scenario which clocks in with a median duration of 2.1 hours of speech
recordings (1788 utterances) per speaker. These are large gains when
contrasted with the 45% success percentage of the out-of-the-box model
(results on conversational domain analogous).

This analysis shows that the answer to the question of how much data is needed
to achieve well working, personalized models for speakers with speech
impairment clearly depends on the severity of impairment. Personalized models
for speakers with moderate and severe impairment benefit significantly from
more training data beyond the maximum subsampling size of 250 utterance of
this study. For mildly impaired speakers, however, the effort of recording
more samples for personalization has only marginal benefit.

###  3.3 Implications for spontaneous free-form speech

Fig. 4: Average WER and 95% confidence intervals on organic test set of 5
selected speakers. Last point is WER when training with all data of given
speaker.

In the previous sections, both the training and the test data were taken from
the Euphonia corpus where all recordings are based on prompted speech [11].
Our experiments clearly show that useful personalized models can be obtained
with relatively small amounts of training data per speaker. However, an
important, yet to answer question is whether such models also work in a
scenario of spontaneous speech of an unknown domain. In other words, how well
do models personalized on small amounts of data generalize to other domains,
acoustic conditions, and scripted vs unscripted speech.

For 15 of the speakers included in our study, we recorded a so-called ”organic
test set”. In direct interactions with these individuals, these speakers’
spontaneous speech on a topic of their choice was recorded, broken into
utterances and then transcribed by speech professionals. Speakers recorded an
average of 140 test utterances. Figure 4 shows the WER curves for five
selected speakers.333Due to the small amount of speakers and the fact that the
organic test sets of all speakers have different characteristics in domain and
phrase complexity we refrain from showing Success Percentage for a target WER.

Unsurprisingly, WERs on the organic test set are higher because spontaneous
speech is more unlike the training set.

However, just as with the prompted home automation and conversational domains,
we can see a knee in the WER curves within the first 100 utterances – the
biggest improvements are even gained within the first 50 utterances. Table 5
shows the relative WER improvement for different training set sizes averaged
over all speakers. Note, that there is significant variance between the 15
speakers and we can see ongoing relevant gains beyond 250 utterances. Overall
we observe the same trends as on prompted data, showing that training with
small amounts of out-of-domain data does indeed generalize and leads to large
model quality improvements on spontaneous speech.

utterances | average improvement (std dev)
---|---
avg = 3239 (std dev = 1717) | 56% (15%)
250 | 45% (16%)
50 | 34% (12%)
10 | 17% (12%)
Table 5: Average improvement over out-of-the-box model on organic test set
across 15 speakers.

##  4 Conclusions

Our study shows that small amounts of training data can indeed be sufficient
to provide useful personalized ASR models for a large percentage of speakers
with speech impairments, especially for use cases like home automation with
limited vocabulary and less complex phrases. In our scenario, with as little
as 17.8 minutes of recorded speech samples on average per speaker, we were
able to produce personalized models with a pre-defined target WER for 96% of
speakers with mild, and 81% of speakers with moderate speech impairment across
a variety of etiologies and types of speech disorders.

This study also emphasizes how severity of speech impairment and amount of
training data needed for useful models are correlated. For speakers with more
severe speech impairment, we see larger gradual model improvements as the
amount of training data is increased. We also show that even with small
amounts of training data from narrow domains and prompted speech, personalized
speech models show big improvements in recognition quality on out-of-domain
phrases and spontaneous speech, suggesting good generalization properties of
personalized models.

In future work, we aim to study how we can further reduce the amount of data
needed for speech model personalization based on task-specific utterance
selection.

## References

  * [1] Neil Bhattacharyya,  “The prevalence of voice problems among adults in the united states,”  The Laryngoscope, vol. 124, no. 10, pp. 2359–2362, 2014.

  * [2] Robert Stonjic, Ross Taylor, Marcin Kardas, Viktor Kerkez, Ludovic Viaud, Elvis Saravia, and Guillem Cucurull,  “Papers with code - speech recognition,” urlpaperswithcode.com/task/speech-recognition, 2021,  Accessed: 2021-09-25.
  * [3] Gabriel Synnaeve,  “Wer are we?,” urlgithub.com/syhw/wer_are_we, 2021,  Accessed: 2021-09-25.
  * [4] Meredith Moore, Hemanth Venkateswara, and Sethuraman Panchanathan,  “Whistle-blowing ASRs: Evaluating the Need for More Inclusive Speech Recognition Systems,”  in Proc. Interspeech 2018, 2018, pp. 466–470.
  * [5] Mumtaz Begum Mustafa, Siti Salwah Salim, Noraini Mohamed, Bassam Al-Qatab, and Chng Eng Siong,  “Severity-based adaptation with limited data for asr to aid dysarthric speakers,”  PLOS ONE, vol. 9, no. 1, pp. 1–11, 01 2014.
  * [6] Jordan E. Green, Robert L. MacDonald, Pan-Pan Jiang, Julie Cattiau, Rus Heywood, Richard Cave, Katie Seaver, Marilyn A. Ladewig, Jimmy Tobin, Michael P. Brenner, Philip C. Nelson, and Katrin Tomanek,  “Automatic speech recognition of disordered speech: Personalized models now outperforming human listeners on short phrases,”  in Proc. Interspeech 2021, 2021.
  * [7] Joel Shor, Dotan Emanuel, Oran Lang, Omry Tuval, Michael Brenner, Julie Cattiau, Fernando Vieira, Maeve McNally, Taylor Charbonneau, Melissa Nollstadt, Avinatan Hassidim, and Yossi Matias,  “Personalizing ASR for Dysarthric and Accented Speech with Limited Data,”  in Proc. Interspeech 2019, 2019, pp. 784–788.
  * [8] Han Zhu, Li Wang, Pengyuan Zhang, and Yonghong Yan,  “Multi-Accent Adaptation Based on Gate Mechanism,”  in Proc. Interspeech 2019, 2019, pp. 744–748.
  * [9] Robert Gale, Liu Chen, Jill Dolata, Jan van Santen, and Meysam Asgari,  “Improving ASR Systems for Children with Autism and Language Impairment Using Domain-Focused DNN Transfer Techniques,”  in Proc. Interspeech 2019, 2019, pp. 11–15.
  * [10] F. Biadsy, R. J. Weiss, P. J. Moreno, D. Kanvesky, and Y. Jia,  “Parrotron: An end-to-end speech-to-speech conversion model and its applications to hearing-impaired speech and speech separation,”  in Proc. Interspeech 2019, 2019, pp. 4115–4119.
  * [11] Robert L. MacDonald, Pan-Pan Jiang, Julie Cattiau, Rus Heywood, Richard Cave, Katie Seaver, Marilyn Ladewig, Jimmy Tobin, Michael P. Brenner, Philip Q. Nelson, Jordan R. Green, and Katrin Tomanek,  “Disordered speech data collection: Lessons learned at 1 million utterances from project euphonia,”  in Proc. Interspeech 2021, 2021.
  * [12] Katrin Tomanek, Françoise Beaufays, Julie Cattiau, Angad Chandorkar, and Khe Chai Sim,  “On-device personalization of automatic speech recognition models for disordered speech,” 2021.
  * [13] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton,  “Speech recognition with deep recurrent neural networks,”  in Proc. ICASSP 2013, 03 2013.
  * [14] Yanzhang He, Tara N Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, et al.,  “Streaming end-to-end speech recognition for mobile devices,”  in Proc. ICASSP 2019, 2019, pp. 6381–6385.
  * [15] Tara N Sainath, Yanzhang He, Bo Li, Arun Narayanan, Ruoming Pang, Antoine Bruguier, Shuo-yiin Chang, Wei Li, Raziel Alvarez, Zhifeng Chen, et al.,  “A streaming on-device end-to-end model surpassing server-side conventional model quality and latency,”  in Proc. ICASSP 2020, 2020, pp. 6059–6063.
  * [16] Arun Narayanan, Rohit Prabhavalkar, Chung-Cheng Chiu, David Rybach, Tara N. Sainath, and Trevor Strohman,  “Recognizing long-form speech using streaming end-to-end models,”  in IEEE Automatic Speech Recognition and Understanding Workshop, 2019, pp. 920–927.
  * [17] Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le,  “SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,”  in Proc. Interspeech 2019, 2019, pp. 2613–2617.