# A Curriculum View of Robust Loss Functions

Zebin Ou1 & Yue Zhang2
1Zhihu Inc., 2School of Engineering, Westlake University
<EMAIL_ADDRESS><EMAIL_ADDRESS>Work mainly done at Westlake
University.

###### Abstract

Robust loss functions are designed to combat the adverse impacts of label
noise, whose robustness is typically supported by theoretical bounds agnostic
to the training dynamics. However, these bounds may fail to characterize the
empirical performance as it remains unclear why robust loss functions can
underfit. We show that most loss functions can be rewritten into a form with
the same class-score margin and different sample-weighting functions. The
resulting curriculum view provides a straightforward analysis of the training
dynamics, which helps attribute underfitting to diminished average sample
weights and noise robustness to larger weights for clean samples. We show that
simple fixes to the curriculums can make underfitting robust loss functions
competitive with the state-of-the-art, and training schedules can
substantially affect the noise robustness even with robust loss functions.
Code is available at github.

##  1 Introduction

Label noise is non-negligible in automatic annotation (Liu et al., 2021),
crowd-sourcing (Russakovsky et al., 2015) and expert annotation (Kato &
Matsubara, 2010). Their adverse impacts can be mitigated with loss functions
that are theoretically robust against label noise (Song et al., 2020), whose
robustness is supported by bounding the difference between expected risk
minimizers obtained with clean or noisy labels (Ghosh et al., 2017; Zhou et
al., 2021b). However, these bounds do not consider how minimizers are
approached or whether they apply to empirically obtained local optimums,
leaving open questions on the empirical performance. For example, robust loss
functions can underfit (Zhang & Sabuncu, 2018; Wang et al., 2019c) difficult
tasks, but the underlying reason cannot be derived from these bounds.

To address such limitations, we analyze training dynamics towards the risk
minimizers to characterize the empirical performance of different loss
functions. We rewrite loss functions into a form with equivalent gradients,
which consists of _the same_ class-score margin and _different_ sample-
weighting functions. The former is a lower bound of the score difference
between the labeled class and any other classes, which determines the
_direction_ of the sample gradient. The latter _weight_ samples based on their
class-score margins. Interactions between distributions of class-score margins
and sample weighting functions thus reveal aspects of the training dynamics.
Notably, loss functions with our derived form _implicitly_ define different
_sample-weighting curriculums_ (section 4). Here a curriculum, by definition
(Wang et al., 2020), specifies a sequence of re-weighting for the distribution
of training samples, e.g., sample weighting (Chang et al., 2017) or sample
selection (Zhou et al., 2021a), based on a metric for sample difficulty.

We first attribute the underfitting issue to diminished average sample weights
during training (section 5.1). In particular, classification with more classes
lead to smaller class-score margins at initialization, which can lead to
minimal sample weights given some robust loss functions. Robust loss functions
that severely underfit can become competitive with the best-performing ones by
adapting the sample-weighting functions to the number of classes. We then
attribute the noise robustness of loss functions to larger weights for clean
samples than noise samples (section 5.2). We find that dynamics of SGD
suppress the learning of noise samples or even get them unlearned. The sample-
weighting functions of robust loss functions then magnify differences in the
learning pace between clean and noise samples and neglect the unlearned noise
samples. Finally, to support our understanding of the training dynamics, we
present surprising results that deviate from existing theoretical bounds: by
simply modifying the learning rate schedule, (1) robust loss functions can
become vulnerable to label noise, and (2) cross entropy can appear robust.

##  2 Related Work

Most existing studies on robust loss functions (Ghosh et al., 2017; Zhang &
Sabuncu, 2018; Wang et al., 2019c; Feng et al., 2020; Liu & Guo, 2020; Cheng
et al., 2021; Zhou et al., 2021b) focus on deriving bounds of the difference
between risk minimizers obtained with noisy and clean labels, which are
agnostic to the training dynamics. We instead analyze the training dynamics of
robust loss functions for reasons behind their underfitting and noise
robustness. Although the underfitting issue has been heuristically mitigated
with loss combination (Zhang & Sabuncu, 2018; Wang et al., 2019c; Ma et al.,
2020), we aim to explicitly identify the cause and support it with our fixes.

Our curriculum view connects existing robust loss functions to the seemingly
distinct (Song et al., 2020) curriculum learning. To mitigate the adverse
impacts of noisy labels, curriculum-based approaches use either sample
selection (Chen et al., 2019; Huang et al., 2019; Zhou et al., 2021a) or
sample weighting (Chang et al., 2017; Ren et al., 2018; Wang et al., 2019a;
b). Our work is related to studies of sample-weighting curriculums but differs
in four perspectives. First, the sample weights analyzed in our work are
_implicitly_ defined by robust loss functions rather than _explicitly_
designed (Chang et al., 2017; Wang et al., 2019a; b) or predicted by a model
(Jiang et al., 2018; Ren et al., 2018). Second, the sample difficulty metric
of the implicit sample-weighting curriculums is the class-score margin we
derived rather than common ones based on loss values (Kumar et al., 2010;
Loshchilov & Hutter, 2015) or gradient magnitudes (Gopal, 2016). Third,
instead of designing better sample-weighting curriculums (Chang et al., 2017;
Wang et al., 2019a; b), we focus on characterizing the performance of
_existing_ robust loss functions by analyzing their implicit sample-weighting
curriculums. Finally, although existing work (Jiang et al., 2018) provides a
robust-loss-function view of some sample-weighting curriculums by identifying
their effective loss functions, we focus on the curriculum view of _existing_
robust loss functions.

Our work is also related to the ongoing debate (Hacohen & Weinshall, 2019;
Wang et al., 2020) on the strategies to select or weight samples in curriculum
learning: either easier first (Bengio et al., 2009; Kumar et al., 2010) or
harder first (Loshchilov & Hutter, 2015; Zhang et al., 2018). In contrast, the
sample-weighting functions we identified in robust loss functions can be
viewed as a combination of both strategies, emphasizing samples with moderate
difficulty.

##  3 Background

We formulate classification with label noise and noise robustness and briefly
review existing research on robust loss functions before diving into our
curriculum view.

Classification $k$-ary classification with input $\bm{x}\in\mathbb{R}^{d}$ can
be solved with classifier $\operatorname*{arg\,max}_{i}s_{i}$, where $s_{i}$
is the score of the $i$-th class in scoring function
$\bm{s}_{\bm{\theta}}:\mathbb{R}^{d}\to\mathbb{R}^{k}$ parameterized by
$\bm{\theta}$. $s_{i}$ can be converted into probability $p_{i}$ with the
softmax function $p_{i}=e^{s_{i}}/\sum_{j}e^{s_{j}}$. Given the ground truth
label $y^{*}\in\\{1..k\\}$ for $\bm{x}$ and a loss function
$L(\bm{s}_{\bm{\theta}}(\bm{x}),y^{*})$, we can estimate $\bm{\theta}$ with
risk minimization
$\operatorname*{arg\,min}_{\bm{\theta}}\mathbb{E}_{\bm{x},y^{*}}[L(\bm{s}_{\bm{\theta}}(\bm{x}),y^{*})]$.

Noise robustness  Labeling errors can corrupt the ground truth label $y^{*}$
into a noisy one,

$y=\left\\{\begin{array}[]{l l}y^{*},&\textrm{with probability
}P(y=y^{*}|\bm{x},y^{*})\\\ i,i\neq y^{*}&\textrm{with probability
}P(y=i|\bm{x},y^{*})\end{array}\right.$

Samples $(\bm{x},y)$ with noisy label $y$ are clean samples if $y=y^{*}$ and
noise samples otherwise. Following Ghosh et al. (2017), label noise is
_symmetric_ (or uniform) if $P(\tilde{y}=i|\bm{x},y)=\eta/(k-1),\forall i\neq
y$ and _asymmetric_ (or class-conditional) when
$P(\tilde{y}=i|\bm{x},y)=P(\tilde{y}=i|y)$, where $\eta=P(\tilde{y}\neq y)$ is
the noise rate. Given a noisy label $\tilde{y}$ for $\bm{x}$, a loss function
$L$ is robust against label noise if

$\operatorname*{arg\,min}\nolimits_{\bm{\theta}}\mathbb{E}_{\bm{x},\tilde{y}}L(\bm{s}_{\bm{\theta}}(\bm{x}),\tilde{y})\approx\operatorname*{arg\,min}\nolimits_{\bm{\theta}}\mathbb{E}_{\bm{x},y}L(\bm{s}_{\bm{\theta}}(\bm{x}),y)$

approximately holds. We rewrite $\bm{s}_{\bm{\theta}}(\bm{x})$ into $\bm{s}$
for notation simplicity hereafter.

###  3.1 Typical Robust Loss Functions

We briefly review typical robust loss functions besides cross entropy (CE)
that is vulnerable (Ghosh et al., 2017) to label noise. See Table 1 for the
formulae and Appendix A for the derivations and extra loss functions.

Symmetric  A loss function $L$ is _symmetric_ (Ghosh et al., 2017) if

$\sum_{i=1}^{k}L(\bm{s},i)=C,\ \forall\bm{s}\in\mathbb{R}^{k},$

with a constant $C$. It is robust against _symmetric_ label noise when
$\eta<(k-1)/k$. Mean absolute error (MAE; Ghosh et al., 2017) and the
equivalent reverse cross entropy (RCE; Wang et al., 2019c) are both symmetric.
Ma et al. (2020) make loss functions with $L(\bm{s},i)>0,\forall
i\in\\{1..k\\}$ symmetric by normalizing them,
$L^{\prime}(\bm{s},y)=L(\bm{s},y)/\sum_{i}L(\bm{s},i)$. We include normalized
cross entropy (NCE; Ma et al. 2020) as an example.

Asymmetric  $L$ as a function of the softmax probability $p_{i}$,
$L(\bm{s},i)=l(p_{i})$, is _asymmetric_ (Zhou et al., 2021b) if

$\tilde{r}=\max_{i\neq
y}\frac{P(\tilde{y}=i|\bm{x},y)}{P(\tilde{y}=y|\bm{x},y)}\leq\inf_{\begin{subarray}{c}0\leq
p_{i},p_{j}\leq 1\\\ p_{i}+p_{j}\leq
1\end{subarray}}\frac{l(p_{i})-l(p_{i}+p_{j})}{l(0)-l(p_{j})},$

where $p_{j}$ is a valid increment of $p_{i}$. An asymmetric loss function is
robust against _generic_ label noise when $\tilde{r}<1$, i.e., there are more
clean samples than noisy samples. We include asymmetric generalized cross
entropy (AGCE) and asymmetric unhinged loss (AUL) from Zhou et al. (2021b).

Combined  Robust loss functions that underfit (Zhang & Sabuncu, 2018) can be
combined with loss functions like CE to balance robustness and sufficient
learning. For example, generalized cross entropy (GCE; Zhang & Sabuncu, 2018)
is a smooth interpolation between CE and MAE. Alternatively, symmetric cross
entropy (SCE; Wang et al., 2019c) is a weighted average of CE and RCE (MAE).
Ma et al. (2020) propose to combine active and passive loss functions with
weighted average. We include NCE+MAE as an example.

###  3.2 Explaining Underfitting of Robust Loss Functions

Despite the theoretical bounds for noise robustness, robust loss functions
such as MAE can underfit difficult tasks (Zhang & Sabuncu, 2018). Existing
explanations can be limited. Zhang & Sabuncu (2018) attribute underfitting of
MAE to the lack of the $1/p_{y}$ term in the sample gradient compared to CE,
thus “treating every sample equally” and hampering learning. However, we show
that MAE instead emphasizes samples with moderate class-score margins after
factoring sample gradients into weights and directions. Ma et al. (2020)
attribute underfitting to failure in balancing the active-passive components.
They rewrite loss functions into $L(\bm{s},y)=\sum_{i}l(\bm{s},i)$ where
$i\in\\{1..k\\}$ is an arbitrary class, and define active loss functions with
$\forall i\neq y,\ l(\bm{s},i)=0$, which emphasizes learning the labeled class
$y$. In contrast, passive loss functions defined with $\exists i\neq y,\
l(\bm{s},i)\neq 0$ can be improved by unlearning other classes $i\neq y$.
However, since there is no canonical guideline to specify $l(\bm{s},i)$,
different specifications can lead to ambiguities. Given

$L_{\mathrm{MAE}}(\bm{s},y)\propto\sum\nolimits_{i}|\mathbb{I}(i=y)-p_{i}|\propto\sum\nolimits_{i}\mathbb{I}(i=y)(1-p_{i})$

with indicator function $\mathbb{I}(\cdot)$, MAE is active if
$l(\bm{s},i)=\mathbb{I}(i=y)(1-p_{i})$ but passive if
$l(\bm{s},i)=|\mathbb{I}(i=y)-p_{i}|$. Finally, Wang et al. (2019a) view
$\|\nabla_{\bm{s}}L(\bm{s},y)\|_{1}$ as weights for sample gradients and
attribute underfitting to their low variance, making clean and noise samples
less distinguishable. However, as shown in section 5.1, MAE also underfits
data with clean labels. We provide an alternative explanation with our
curriculum view in section 5.1, which leads to effective fixes for the issue.

##  4 Implicit Curriculums of Loss Functions

We derive the main results of our curriculum view for later analysis. The
softmax probability $p_{y}$ of the (noisily) labeled class $y$ can be written
into a sigmoid form,

$p_{y}=\frac{e^{s_{y}}}{\sum_{i}e^{s_{i}}}=\frac{1}{e^{-(s_{y}-\log\sum_{i\neq
y}e^{s_{i}})}+1}=\frac{1}{e^{-\Delta_{y}}+1},$

where

$\Delta_{y}=s_{y}-\log\sum_{i\neq y}e^{s_{i}}\leq s_{y}-\max_{i\neq y}s_{i}$

is the soft score margin between the labeled class $y$ and any other classes.
A large $\Delta_{y}$ indicates a well-learned sample as $\Delta_{y}\geq 0$
leads to successful classification with $y=\operatorname*{arg\,max}_{i}s_{i}$.
Note that loss functions in Table 1 except NCE and NCE+MAE are functions of
$p_{y}$, $L(\bm{s},y)=l(p_{y})$. Given

$\displaystyle\nabla_{\bm{s}}L(\bm{s},y)$
$\displaystyle=\nabla_{\bm{s}}l(p_{y})=\frac{\mathrm{d}l}{\mathrm{d}p_{y}}\frac{\mathrm{d}p_{y}}{\mathrm{d}\Delta_{y}}\cdot\nabla_{\bm{s}}\Delta_{y}=\nabla_{\bm{s}}\left[\rho\left(\frac{\mathrm{d}l}{\mathrm{d}p_{y}}\frac{\mathrm{d}p_{y}}{\mathrm{d}\Delta_{y}}\right)\cdot\Delta_{y}\right]=\nabla_{\bm{s}}\left[w(\Delta_{y})\cdot\Delta_{y}\right],$

where $\rho(\cdot)$ is the stop-gradient operator, we can rewrite $L$ into a
form with equivalent gradient,

$\tilde{L}(\bm{s},y)=w(\Delta_{y})\cdot\Delta_{y},$ (1)

where
$w(\Delta_{y})=\rho(\frac{\mathrm{d}l}{\mathrm{d}p_{y}}\frac{\mathrm{d}p_{y}}{\mathrm{d}\Delta_{y}})\leq
0$ is the _sample-weighting function_. A larger $|w(\Delta_{y})|$ emphasizes
more on increasing $\Delta_{y}$ of the sample. Since
$\|\nabla_{\bm{s}}\Delta_{y}\|_{1}=2$, $\Delta_{y}$ determines the direction
of sample gradients. Thus Eq. 1 essentially factorizes the weight and
direction of sample gradients. Loss functions in the form of Eq. 1 differ only
in the sample-weighting functions $w(\Delta_{y})$, each implicitly defines a
_sample-weighting curriculum_ based on $\Delta_{y}$ that reflects sample
difficulty. Compared to existing sample-difficulty metrics like loss value
(Kumar et al., 2010) or gradient magnitude (Gopal, 2016), $\Delta_{y}$ factors
out the nonlinear preference of $w(\Delta_{y})$ and can be a better drop-in
replacement in curriculum-based approaches. The interactions between
$w(\Delta_{y})$ and distributions of $\Delta_{y}$ further reveal training
dynamics with different loss functions, which facilitate our analysis in
section 5. See Table 1 for $w(\Delta_{y})$ of the reviewed loss functions, and
Appendix A for how hyperparameters affect $w(\Delta_{y})$.

Type | Name | Function | Sample Weight $w$ | Constraints
---|---|---|---|---
/ | CE | $-\log p_{y}$ | $1-p_{y}$ | /
Sym. | MAE/RCE | $1-p_{y}$ | $p_{y}(1-p_{y})$ | /
NCE | $-\log p_{y}/\left(\sum_{i=1}^{k}-\log p_{i}\right)$ | / | /
Asym. | AUL | $[(a-p_{y})^{q}-(a-1)^{q}]/q$ | $p_{y}(1-p_{y})(a-p_{y})^{q-1}$ |  $a>1$, $q>0$
AGCE | $[(a+1)-(a+p_{y})^{q}]/q$ | $p_{y}(a+p_{y})^{q-1}(1-p_{y})$ |  $a>0$, $q>0$
Comb. | GCE | $(1-p_{y}^{q})/q$ | $p_{y}^{q}(1-p_{y})$ | $0<q\leq 1$
SCE | $(1-q)\cdot L_{\mathrm{CE}}+q\cdot L_{\mathrm{MAE}}$ | $(1-q+q\cdot p_{y})(1-p_{y})$ | $0<q<1$
NCE+MAE | $(1-q)\cdot L_{\mathrm{NCE}}+q\cdot L_{\mathrm{MAE}}$ | / | $0<q<1$
Table 1: Expressions, constraints and sample-weighting functions (section 4)
for loss functions in section 3.1.

###  4.1 The Additional Regularizer of NCE

NCE does not exactly follow Eq. 1 as it additionally depends on $p_{i}$ with
class $i\neq y$. However, with equivalent gradients, it can be rewritten into

$\tilde{L}_{\mathrm{NCE}}(\bm{s},y)=\gamma\cdot
L_{\mathrm{CE}}(\bm{s},y)+\gamma\cdot\epsilon\cdot R_{\mathrm{NCE}}(\bm{s}),$
(2)

with $\gamma=\rho(-1/\sum_{i}\log p_{i})$ and $\epsilon=\rho(k\log
p_{y}/\sum_{i}\log p_{i})$ the weights and $\rho(\cdot)$ the stop-gradient
operator. Both $\gamma$ and $\epsilon$ decrease as $\Delta_{y}$ increases. The
first additive term in Eq. 2 is a _primary loss function_ following Eq. 1,
which defines a sample-weighting curriculum. The second is a _regularizer_

$R_{\mathrm{NCE}}(\bm{s})=\sum_{i=1}^{k}\frac{1}{k}\log p_{i}$

that reduces the entropy of softmax outputs and decreases $\gamma$ and
$\epsilon$. Although training dynamics of NCE are complicated by the extra
regularizer, the L1 norm of sample gradients can be bounded with

$\|\nabla_{\bm{s}}L_{\mathrm{NCE}}(\bm{s},y)\|_{1}\leq
2\gamma\cdot\left(1+\epsilon\right)\cdot w_{\mathrm{CE}}$ (3)

which helps explain why NCE can underfit in section 5.1. See section A.5 for
discussions on similar loss functions with an additional regularizer and
detailed derivations.

##  5 Loss Functions with the Curriculum View

We examine the interaction between $w(\Delta_{y})$ and $\Delta_{y}$
distributions to address questions in section 3.2. Results are reported on
MNIST (Lecun et al., 1998) and CIFAR10/100 (Krizhevsky, 2009) with synthetic
symmetric and asymmetric label noise following Ma et al. (2020); Zhou et al.
(2021b). For real-world scenarios, we include CIFAR10/100 with human label
noise (Wei et al., 2022) and the large-scale noisy dataset WebVision (Li et
al., 2017), which exhibit more complex noise patterns than symmetric and
asymmetric label noise. Unlike standard settings, we scale $w(\Delta_{y})$ to
unit maximum to avoid complications, since hyperparameters of loss functions
can change the scale of $w(\Delta_{y})$, essentially adjusting the learning
rate of SGD. See Appendix B for more experimental details.

###  5.1 Understanding Underfitting of Robust Loss Functions

We reproduce the underfitting issue without label noise in Table 2. The
hyperparameters of loss functions are tuned on CIFAR100 and listed in Table 8
of section C.1. CE outperforms NCE, AGCE, AUL and MAE by a nontrivial margin
on CIFAR100. The less performant loss functions has smaller gap between
training and testing performance, suggesting the issue of underfitting. In
contrast, all loss functions performs equally well on CIFAR10.

Marginal sample weights explains underfitting. Since the same model fits each
dataset well with CE in Table 2, underfitting should result from insufficient
parameter updates with the altered loss functions rather than inadequate model
size. Based on our derivation, the average scale of parameter update up to
$t$-th step can be estimated with

$\alpha_{t}=\frac{\sum_{i=1}^{t}\sum_{\bm{s}\in\mathcal{B}_{i}}\eta_{i}\cdot\|\nabla_{\bm{s}}\Delta_{y}\|_{1}}{\sum_{i=1}^{t}\eta_{i}\cdot|\mathcal{B}_{i}|}$

where $\eta_{i}$ is the learning rate and $\mathcal{B}_{i}$ the sampled batch
at training step $i$. As shown in Table 2, at the final training step, a small
$\alpha_{t}$ highly correlates with underfitting.

Fast diminishing sample weights lead to underfitting. In Fig. 1(a),
$\alpha_{t}$ of NCE peaks at initialization similar to CE. However, it
decreases much faster than CE since both $\gamma$ and $\epsilon$ decrease with
improved $\Delta_{y}$. The regularizer $R_{\mathrm{NCE}}(\bm{s})$ further
reduces the entropy of softmax output and thus $\gamma$. As a result, the fast
decreasing $\alpha_{t}$ hampers the learning of training samples and leads to
underfitting.

| CIFAR10 | CIFAR100
---|---|---
Loss | Train | Test | $\alpha_{t}$ | Train | Test | $\alpha_{t}$
CE | 99.98 | 92.96 | 45.74 | 99.97 | 71.02 | 86.79
SCE | 99.99 | 93.20 | 19.72 | 99.97 | 71.11 | 30.63
GCE | 99.97 | 92.78 | 23.72 | 99.90 | 70.14 | 30.86
NCE+MAE | 99.68 | 92.35 | / | 92.30 | 68.28 | /
AUL | 99.93 | 91.80 | 4.49 | 88.19 | 59.62 | 2.11
AGCE | 99.83 | 92.88 | 17.20 | 66.99 | 51.33 | 9.24
NCE | 99.92 | 91.17 | / | 31.64 | 29.67 | /
MAE | 98.67 | 91.93 | 4.05 | 10.32 | 9.85 | 0.28
AUL† | 98.80 | 92.06 | 3.25 | 9.22 | 8.67 | 0.21
AGCE† | 91.60 | 86.30 | 3.98 | 3.82 | 3.82 | 0.12
Table 2: With clean labels, robust loss functions can underfit CIFAR100 but
CIFAR10. We report the average accuracies and $\alpha_{t}$ (scaled by
$10^{4}$) at the final training step with learning rate $\eta=0.1$ from 3
different runs. See section C.1 for hyperparameters of loss functions tuned on
CIFAR100 in Table 8. Settings with inferior hyperparameters are denoted with
†.

(a) NCE with estimated weight upperbound.

(b) AUL with inferior/superior hyperparameters.

Figure 1: Different explanations for underfitting: (a) fast diminishing sample
weights; (b) marginal initial sample weights. We plot the variation of
$\alpha_{t}$ with training step $t$ on CIFAR100 without label noise for each
loss function. $\alpha_{t}$ in (a) is normalized with its maximum to emphasize
its variation during training.

Marginal initial sample weights lead to underfitting. Unlike NCE, loss
functions that severely underfit in Table 2 assign marginal initial weights
(Fig. 2(c)) to samples in CIFAR100, which leads to marginal initial
$\alpha_{t}$. $\Delta_{y}$ of these samples can barely improve before the
learning rate vanishes, thus leading to underfitting. In contrast, loss
functions with non-trivial initial sample weights (Fig. 2(b) and 2(a)) result
in moderate or no underfitting. As further corroboration, we plot $\alpha_{t}$
of AUL with superior and inferior hyperparameters (AUL and AUL† in Table 2) in
Fig. 1(b). $\alpha_{t}$ stays marginal with AUL†, but quickly increases to a
non-negligible value before gradually decreasing with AUL.

Loss combination can mitigate underfitting. As $\alpha_{t}$ of NCE peaks at
initialization but quickly diminishes while $\alpha_{t}$ of MAE is marginal at
initialization but peaks later during training, combining NCE with MAE can
mitigate the underfitting issue of each other. In Table 2, combining NCE and
MAE suffers less from underfitting compared to both individuals.

Increased number of classes leads to marginal initial sample weights. Unlike
CIFAR100, loss functions in Table 2 perform equally well on CIFAR10. The
difference has been vaguely attributed to the increased task difficulty of
CIFAR100 (Zhang & Sabuncu, 2018; Song et al., 2020). Intuitively, the more
classes, the more subtle differences to be distinguished. In addition, the
number of classes $k$ determines the initial distribution of $\Delta_{y}$.
Assume that class scores $s_{i}$ at _initialization_ are i.i.d. normal
variables $s_{i}\sim\mathcal{N}(\mu,\sigma)$. In particular, $\mu=0$ and
$\sigma=1$ for most neural networks with standard initializations (Glorot &
Bengio, 2010; He et al., 2015) and normalization layers (Ioffe & Szegedy,
2015; Ba et al., 2016). The expected $\Delta_{y}$ can be approximated with

$\mathbb{E}[\Delta_{y}]\approx-\log(k-1)-\sigma^{2}/2+\frac{e^{\sigma^{2}}-1}{2(k-1)}$
(4)

We leave derivations and comparisons between our assumptions and real settings
to section C.1. A large $k$ results in small initial $\Delta_{y}$; with
sample-weighting functions in Fig. 2(c) it further leads to marginal initial
sample weights, which results in underfitting on CIFAR100 as discussed above.

(a) No underfitting

(b) Mild underfitting

(c) Severe underfitting

Figure 2: Sample-weighting functions $w(\Delta_{y})$ of loss functions in Table 2 with hyperparameters in Table 8. We include the initial $\Delta_{y}$ distributions of training samples on CIFAR10 and CIFAR100 for reference, which are extracted with a randomly initialized model. | Clean | Symmetric | Asymmetric | Human
---|---|---|---|---
Loss | $\eta=0$ | $\eta=0.4$ | $\eta=0.8$ | $\eta=0.4$ | $\eta=0.4$
CE‡ | 71.33 $\pm$ 0.43 | 39.92 $\pm$ 0.10 | 7.59 $\pm$ 0.20 | 40.17 $\pm$ 1.31 | /
GCE‡ | 63.09 $\pm$ 1.39 | 56.11 $\pm$ 1.35 | 17.42 $\pm$ 0.06 | 40.91 $\pm$ 0.57 | /
NCE‡ | 29.96 $\pm$ 0.73 | 19.54 $\pm$ 0.52 | 8.55 $\pm$ 0.37 | 20.64 $\pm$ 0.40 | /
NCE+AUL‡ | 68.96 $\pm$ 0.16 | 59.25 $\pm$ 0.23 | 23.03 $\pm$ 0.64 | 38.59 $\pm$ 0.48 | /
AGCE | 49.27 $\pm$ 1.03 | 47.76 $\pm$ 1.75 | 16.03 $\pm$ 0.59 | 33.40 $\pm$ 1.57 | 30.45 $\pm$ 1.50
AGCE shift | 69.39 $\pm$ 0.84 | 48.21 $\pm$ 1.06 | 14.49 $\pm$ 0.17 | 40.76 $\pm$ 0.74 | 48.71 $\pm$ 0.45
AGCE scale | 70.57 $\pm$ 0.62 | 56.69 $\pm$ 0.33 | 14.64 $\pm$ 0.79 | 39.71 $\pm$ 0.17 | 50.85 $\pm$ 0.11
MAE | 3.69 $\pm$ 0.59 | 1.29 $\pm$ 0.50 | 1.00 $\pm$ 0.00 | 2.53 $\pm$ 1.34 | 2.09 $\pm$ 0.55
MAE shift | 68.57 $\pm$ 0.54 | 49.95 $\pm$ 0.16 | 13.10 $\pm$ 0.41 | 39.83 $\pm$ 0.18 | 47.91 $\pm$ 0.36
MAE scale | 70.97 $\pm$ 0.41 | 60.57 $\pm$ 1.04 | 24.44 $\pm$ 0.73 | 44.48 $\pm$ 1.05 | 54.70 $\pm$ 0.48
Table 3: Shifting or scaling $w(\Delta_{y})$ mitigates underfitting on
CIFAR100 under different label noise. We report test accuracies with 3
different runs. Results from Zhou et al. (2021b) are included as context
(denoted with ‡). See section C.1 for hyperparameter $\tau$ and results with
more noise rates.

####  5.1.1 Addressing Underfitting from Marginal Initial Sample Weights

Our analysis suggests that the fixed sample-weighting function $w(\Delta_{y})$
is to blame for underfitting. Assuming $\mathbb{E}[\Delta_{y}]<0$ at
initialization, to address underfitting from marginal initial sample weights,
we can simply scale

$w^{*}(\Delta_{y})=w(\Delta_{y}^{*})=w(\Delta_{y}/|\mathbb{E}[\Delta_{y}]|\cdot\tau)$

or shift

$w^{+}(\Delta_{y})=w(\Delta_{y}^{+})=w(\Delta_{y}+|\mathbb{E}[\Delta_{y}]|-\tau)$

the sample-weighting functions, where $\tau$ is a hyperparameter. Intuitively,
both approaches cancel the effect of large $k$ on the weight of
$\mathbb{E}[\Delta_{y}]$ at initialization. A small $\tau$ thus leads to high
initial sample weights regardless of $k$. See section C.1.1 for visualizations
of the scaled and shifted sample-weighting functions and discussions on the
robustness of loss functions they induce.

We report results on CIFAR100 with different label noise in Table 3, and
results on the noisy large-scale dataset WebVision in Table 4. In summary,
shifting and scaling alleviate underfitting, making MAE and AGCE comparable to
the previous state-of-the-art (NCE+AUL; Zhou et al. 2021b). Notably,
$w^{*}(\Delta_{y})$ leads to dramatic improvements for MAE under all settings.
Interestingly, although $w^{*}(\Delta_{y})$ and $w^{+}(\Delta_{y})$ are both
agnostic to the number of classes at initialization, their performances differ
significantly. Intuitively, $w^{+}(\Delta_{y})$ diminishes much faster than
$w^{*}(\Delta_{y})$ with increased $\Delta_{y}$, which can lead to
insufficient training of clean samples and thus inferior performance.

###  5.2 Understanding Noise Robustness of Loss Functions

We show that robust loss functions following Eq. 1 _implicitly_ assign larger
weights to clean samples. The underlying reasons are explored by examining how
$\Delta_{y}$ distributions change during training. Notably, similar sample-
weighting rules are _explicitly_ adopted by curriculums for noise robust
training (Ren et al., 2018). We leave NCE to future work as it involves an
additional regularizer.

Robust loss functions assign larger weights to clean samples. We use the ratio
between the average weights of clean ($\bar{w}_{\mathrm{clean}}$) and noisy
($\bar{w}_{\mathrm{noise}}$) samples,
$\mathrm{snr}=\bar{w}_{\mathrm{clean}}/\bar{w}_{\mathrm{noise}}$, to
characterize their relative contribution during training. See section C.2 for
the exact formulas. Noise robustness is characterized by differences in test
accuracy compared to results with clean labels ($\mathrm{diff}$). We report
$\mathrm{diff}$ and $\mathrm{snr}$ under different label noise on CIFAR10 in
Table 5. Loss functions with higher $\mathrm{snr}$ have less performance drop
with label noise in general, thus being more robust.

| $k=50$ | $k=200$ | $k=400$
---|---|---|---
Settings | $\tau=2.0$ | $\tau=1.8$ | $\tau=1.6$
CE | 66.40 | 70.26 | 70.16
MAE |  3.68 |  0.50 |  0.25
MAE shift | 60.76 | 59.31 | 47.32
MAE scale | 66.72 | 71.92 | 71.87
Table 4: Shifting or scaling $w(\Delta_{y})$ mitigates underfitting on WebVision subsampled with different numbers of classes. $k=50$ is the standard “mini” setting in previous work (Ma et al., 2020; Zhou et al., 2021b). We report test accuracy with a single run due to a limited computation budget. | Clean | Asymmetric | Symmetric | Human
---|---|---|---|---
|  | $\eta=0.2$ | $\eta=0.2$ | $\eta=0.4$ | $\eta=0.8$ | $\eta=0.4$
Loss | Acc | $\mathrm{diff}$ | $\mathrm{snr}$ | $\mathrm{diff}$ | $\mathrm{snr}$ | $\mathrm{diff}$ | $\mathrm{snr}$ | $\mathrm{diff}$ | $\mathrm{snr}$ | $\mathrm{diff}$ | $\mathrm{snr}$
CE | 90.64 | -7.06 | 0.32 | -15.47 | 0.39 | -31.95 | 0.57 | -50.87 | 0.77 | -28.51 | 0.53
SCE | 89.87 | -5.39 | 0.51 | -3.84 | 0.99 | -10.47 | 1.27 | -27.25 | 1.51 | -15.84 | 0.86
GCE | 90.44 | -7.42 | 0.36 | -6.80 | 0.96 | -23.23 | 0.89 | -45.32 | 1.04 | -21.94 | 0.72
AUL | 89.90 | -2.51 | 0.81 | -2.07 | 3.10 | -5.87 | 2.96 | -13.90 | 2.83 | -12.08 | 1.11
MAE | 89.29 | -2.21 | 1.00 | -1.92 | 3.56 | -4.36 | 3.33 | -11.53 | 3.22 | -10.35 | 1.32
AGCE | 82.62 | -9.42 | 0.92 | -1.55 | 3.02 | -19.90 | 2.16 | -41.11 | 1.83 | -21.73 | 1.28
Table 5: Robust loss functions assign larger weights to clean samples. We
report $\mathrm{snr}$ and $\mathrm{diff}$ from the best of 5 runs on CIFAR10
under each noise setting, as inferior initialization can heavily degrade the
performance. Hyperparameters listed in Table 12 are selected to cover more
variants of sample-weighting functions (plotted in Fig. 8), which are not
necessarily optimal.

To explain what leads to a large $\mathrm{snr}$, we plot changes of
$\Delta_{y}$ distributions during training on CIFAR10 with symmetric label
noise in Fig. 3. When trained with loss functions that are more robust against
label noise (Fig. 3(b) and 3(c)), $\Delta_{y}$ distributions of noisy and
clean samples spread wider and get better separated. In addition, the
consistent decrease of $\Delta_{y}$ for noisy samples suggests that they can
be _unlearned_. In contrast, training with CE (Fig. 3(a)) results in more
compact and less separated $\Delta_{y}$ distributions. Furthermore,
$\Delta_{y}$ of noisy samples consistently increases.

Dynamics of SGD suppress learning of noisy samples. As shown in Fig. 3(a),
noisy samples are learned slower than clean samples as measured by
improvements of $\Delta_{y}$, which can be explained by more coherent
gradients among clean samples (Chatterjee & Zielinski, 2022). Similar results
have been reported (Zhang et al., 2017; Arpit et al., 2017) and utilized in
curriculum-based robust training (Yao et al., 2019; Han et al., 2018). In
addition, noisy samples can be unlearned as shown in Fig. 3(b) and 3(c), which
can stem from generalization with clean samples. Both dynamics suppress the
learning of noisy samples but clean ones, thus leading to robustness against
label noise.

Robust $w(\Delta_{y})$ synergizes with SGD dynamics for noise robustness. In
Fig. 2, the bell-shaped $w(\Delta_{y})$ of robust loss functions only assigns
large weights to samples with moderate $\Delta_{y}$. Since $\Delta_{y}$
distributions initially concentrate at the monotonically increasing interval
of $w(\Delta_{y})$, (1) samples with faster improving $\Delta_{y}$, due to
either larger initial weights or faster learning as clean samples, are
weighted more during early training and learned faster. The magnified learning
pace difference explains the widely spread distributions in Fig. 3(b) and
3(c). In addition, (2) the unlearned samples with small $\Delta_{y}$ receive
diminishing weights from $w(\Delta_{y})$, which hampers their pace of
learning. Noisy samples in Fig. 3(b) and 3(c) are consistently unlearned and
ignored with marginal sample weights, leading to a consistent decrease in
$\Delta_{y}$. In addition to the SGD dynamics, (1) and (2) further suppress
the learning of noisy samples and enhance that of clean samples, thus leading
to increased robustness against label noise. In contrast, the monotonically
decreasing $w_{\mathrm{CE}}(\Delta_{y})$ emphasizes samples with smaller
$\Delta_{y}$, essentially acting against the SGD dynamics for noise
robustness. Thus training with CE results in increased vulnerability to label
noise as shown in Table 5.

(a) CE: 57.74

(b) SCE: 79.56

(c) MAE: 83.88

Figure 3: How $\Delta_{y}$ distributions of noisy (green, left) and clean
(orange, right) samples change on CIFAR10 during training with symmetric label
noise and $\eta=0.4$. Vertical axes denoting probability density are scaled to
the peak of histograms for readability, with epoch number (axis scaling
factor) denoted on the right of each subplot. We plot $w(\Delta_{y})$ and
report the test accuracy of each setting for reference. See section C.2 for
results with additional types of label noise and loss functions.

####  5.2.1 Training Schedules Affect Noise Robustness

Although the learning pace of noisy samples gets initially suppressed, the
expected gradient will eventually be dominated by noisy samples, since well-
learned clean samples receive marginal sample weights thanks to the
monotonically decreasing interval of $w(\Delta_{y})$. Models with extended
training111Enough training steps without early stopping or diminishing
learning rates for a small training loss. thus risk overfitting noisy samples
during the late training stage. Adjusting the training schedules to enable or
avoid such overfitting can therefore affect the noise robustness of models.
Based on this intuition, we present two surprising examples that deviate from
existing theoretical results:

(a) $\alpha=0.01$ with different ($\eta$, loss).

(b) CE with different ($\eta$, $\alpha$).

Figure 4: Learning curves with fixed learning rate and extended training
epochs on MNIST under symmetric label noise, where $\alpha$ is the learning
rate and $\eta$ the noise rate.

Extended training can make robust loss functions vulnerable to label noise.
The learning curves of CE and MAE with _constant_ learning rates on MNIST are
shown in Fig. 4(a). Despite the theoretically guaranteed noise robustness
(Ghosh et al., 2017), similar to CE, with extended training, MAE eventually
overfits noisy samples, resulting in vulnerability to label noise.

CE can become robust by adjusting the learning rate schedule. To avoid
overfitting noisy samples, we can avoid learning when noisy samples dominate
the expected gradient. It can be achieved with either early stopping (Song et
al., 2019), or a constrained learning pace that prevents sufficient learning
of clean samples, which avoids diminishing weights for them. We show the
learning curve of CE using fixed learning rates under symmetric label noise on
MNIST in Fig. 4(b). By simply increasing or decreasing the learning rate,
which strengthens the implicit regularization of SGD (Smith et al., 2021) or
directly slows down the learning pace, CE can become robust against label
noise.

##  6 Conclusion and Discussions

We extend the understanding of robust loss functions by considering the
training dynamics to reach the risk minimizers. By rewriting numerous loss
functions into the same class-score margin and varied sample-weighting
functions, we explicitly connect the design of loss functions to the design of
sample-weighting curriculums and unify a broad array of loss functions. Based
on the curriculum view, we gain more insights into how robust loss functions
work and propose effective fixes to address the underfitting issue of robust
loss functions.

## References

  * Arpit et al. (2017) Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien.  A closer look at memorization in deep networks.  In Doina Precup and Yee Whye Teh (eds.), _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , volume 70 of _Proceedings of Machine Learning Research_ , pp. 233–242. PMLR, 2017.  URL http://proceedings.mlr.press/v70/arpit17a.html.
  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer normalization, 2016.  URL https://arxiv.org/abs/1607.06450.
  * Bengio et al. (2009) Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston.  Curriculum learning.  In Andrea Pohoreckyj Danyluk, Léon Bottou, and Michael L. Littman (eds.), _Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, Montreal, Quebec, Canada, June 14-18, 2009_ , volume 382 of _ACM International Conference Proceeding Series_ , pp. 41–48. ACM, 2009.  doi: 10.1145/1553374.1553380.  URL https://doi.org/10.1145/1553374.1553380.
  * Chang et al. (2017) Haw-Shiuan Chang, Erik G. Learned-Miller, and Andrew McCallum.  Active bias: Training more accurate neural networks by emphasizing high variance samples.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pp. 1002–1012, 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/2f37d10131f2a483a8dd005b3d14b0d9-Abstract.html.
  * Chatterjee & Zielinski (2022) Satrajit Chatterjee and Piotr Zielinski.  On the generalization mystery in deep learning.  _ArXiv preprint_ , abs/2203.10036, 2022.  URL https://arxiv.org/abs/2203.10036.
  * Chen et al. (2019) Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang.  Understanding and utilizing deep neural networks trained with noisy labels.  In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pp. 1062–1070. PMLR, 2019\.  URL http://proceedings.mlr.press/v97/chen19g.html.
  * Cheng et al. (2021) Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu.  Learning with instance-dependent label noise: A sample sieve approach.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.  URL https://openreview.net/forum?id=2VXyy9mIyU3.
  * Cobb et al. (2012) Barry Cobb, Rafael Rumí, and Antonio Salmerón.  Approximating the distribution of a sum of log-normal random variables.  2012\.
  * Feng et al. (2020) Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An.  Can cross entropy loss be robust to label noise?  In Christian Bessiere (ed.), _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020_ , pp. 2206–2212. ijcai.org, 2020.  doi: 10.24963/ijcai.2020/305.  URL https://doi.org/10.24963/ijcai.2020/305.
  * Ghosh et al. (2017) Aritra Ghosh, Himanshu Kumar, and P. S. Sastry.  Robust loss functions under label noise for deep neural networks.  In Satinder P. Singh and Shaul Markovitch (eds.), _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA_ , pp. 1919–1925. AAAI Press, 2017\.  URL http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14759.
  * Glorot & Bengio (2010) Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , pp. 249–256. JMLR Workshop and Conference Proceedings, 2010.  URL https://proceedings.mlr.press/v9/glorot10a.html.
  * Gopal (2016) Siddharth Gopal.  Adaptive sampling for SGD by exploiting side information.  In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), _Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016_ , volume 48 of _JMLR Workshop and Conference Proceedings_ , pp. 364–372. JMLR.org, 2016\.  URL http://proceedings.mlr.press/v48/gopal16.html.
  * Hacohen & Weinshall (2019) Guy Hacohen and Daphna Weinshall.  On the power of curriculum learning in training deep networks.  In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pp. 2535–2544. PMLR, 2019\.  URL http://proceedings.mlr.press/v97/hacohen19a.html.
  * Han et al. (2018) Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , pp. 8536–8546, 2018.  URL https://proceedings.neurips.cc/paper/2018/hash/a19744e268754fb0148b017647355b7b-Abstract.html.
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In _2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015_ , pp. 1026–1034. IEEE Computer Society, 2015.  doi: 10.1109/ICCV.2015.123.  URL https://doi.org/10.1109/ICCV.2015.123.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_ , pp. 770–778. IEEE Computer Society, 2016.  doi: 10.1109/CVPR.2016.90.  URL https://doi.org/10.1109/CVPR.2016.90.
  * Huang et al. (2019) Jinchi Huang, Lie Qu, Rongfei Jia, and Binqiang Zhao.  O2u-net: A simple noisy label detection approach for deep neural networks.  In _2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019_ , pp. 3325–3333. IEEE, 2019.  doi: 10.1109/ICCV.2019.00342.  URL https://doi.org/10.1109/ICCV.2019.00342.
  * Ioffe & Szegedy (2015) Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Francis R. Bach and David M. Blei (eds.), _Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015_ , volume 37 of _JMLR Workshop and Conference Proceedings_ , pp. 448–456. JMLR.org, 2015.  URL http://proceedings.mlr.press/v37/ioffe15.html.
  * Jiang et al. (2018) Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei.  Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels.  In Jennifer G. Dy and Andreas Krause (eds.), _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , volume 80 of _Proceedings of Machine Learning Research_ , pp. 2309–2318. PMLR, 2018\.  URL http://proceedings.mlr.press/v80/jiang18c.html.
  * Kato & Matsubara (2010) Yoshihide Kato and Shigeki Matsubara.  Correcting errors in a treebank based on synchronous tree substitution grammar.  In _Proceedings of the ACL 2010 Conference Short Papers_ , pp. 74–79, Uppsala, Sweden, 2010. Association for Computational Linguistics.  URL https://aclanthology.org/P10-2014.
  * Krizhevsky (2009) Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, 2009.
  * Kumar et al. (2010) M. Pawan Kumar, Benjamin Packer, and Daphne Koller.  Self-paced learning for latent variable models.  In John D. Lafferty, Christopher K. I. Williams, John Shawe-Taylor, Richard S. Zemel, and Aron Culotta (eds.), _Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of a meeting held 6-9 December 2010, Vancouver, British Columbia, Canada_ , pp. 1189–1197. Curran Associates, Inc., 2010.  URL https://proceedings.neurips.cc/paper/2010/hash/e57c6b956a6521b28495f2886ca0977a-Abstract.html.
  * Lecun et al. (1998) Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.  doi: 10.1109/5.726791.
  * Li et al. (2017) Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool.  Webvision database: Visual learning and understanding from web data, 2017\.  URL https://arxiv.org/abs/1708.02862.
  * Lin et al. (2017) Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_ , pp. 2999–3007. IEEE Computer Society, 2017.  doi: 10.1109/ICCV.2017.324.  URL https://doi.org/10.1109/ICCV.2017.324.
  * Liu et al. (2021) Kun Liu, Yao Fu, Chuanqi Tan, Mosha Chen, Ningyu Zhang, Songfang Huang, and Sheng Gao.  Noisy-labeled NER with confidence estimation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 3437–3445, Online, 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.269.  URL https://aclanthology.org/2021.naacl-main.269.
  * Liu & Guo (2020) Yang Liu and Hongyi Guo.  Peer loss functions: Learning from noisy labels without knowing noise rates.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pp. 6226–6236. PMLR, 2020\.  URL http://proceedings.mlr.press/v119/liu20e.html.
  * Loshchilov & Hutter (2015) Ilya Loshchilov and Frank Hutter.  Online batch selection for faster training of neural networks.  _ArXiv preprint_ , abs/1511.06343, 2015.  URL https://arxiv.org/abs/1511.06343.
  * Lukasik et al. (2020) Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar.  Does label smoothing mitigate label noise?  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pp. 6448–6458. PMLR, 2020\.  URL http://proceedings.mlr.press/v119/lukasik20a.html.
  * Ma et al. (2020) Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah M. Erfani, and James Bailey.  Normalized loss functions for deep learning with noisy labels.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pp. 6543–6553. PMLR, 2020\.  URL http://proceedings.mlr.press/v119/ma20c.html.
  * Patrini et al. (2017) Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu.  Making deep neural networks robust to label noise: A loss correction approach.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pp. 2233–2241. IEEE Computer Society, 2017.  doi: 10.1109/CVPR.2017.240.  URL https://doi.org/10.1109/CVPR.2017.240.
  * Ren et al. (2018) Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun.  Learning to reweight examples for robust deep learning.  In Jennifer G. Dy and Andreas Krause (eds.), _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , volume 80 of _Proceedings of Machine Learning Research_ , pp. 4331–4340. PMLR, 2018\.  URL http://proceedings.mlr.press/v80/ren18a.html.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  Imagenet large scale visual recognition challenge.  _arXiv:1409.0575 [cs]_ , 2015.  URL http://arxiv.org/abs/1409.0575.  arXiv: 1409.0575.
  * Smith et al. (2021) Samuel L. Smith, Benoit Dherin, David G. T. Barrett, and Soham De.  On the origin of implicit regularization in stochastic gradient descent.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.  URL https://openreview.net/forum?id=rq_Qr0c1Hyo.
  * Song et al. (2019) Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee.  How does early stopping help generalization against label noise?, 2019\.  URL https://arxiv.org/abs/1911.08059.
  * Song et al. (2020) Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee.  Learning from noisy labels with deep neural networks: A survey.  _ArXiv preprint_ , abs/2007.08199, 2020.  URL https://arxiv.org/abs/2007.08199.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_ , pp. 2818–2826. IEEE Computer Society, 2016.  doi: 10.1109/CVPR.2016.308.  URL https://doi.org/10.1109/CVPR.2016.308.
  * Teh et al. (2006) Yee Whye Teh, David Newman, and Max Welling.  A collapsed variational bayesian inference algorithm for latent dirichlet allocation.  In Bernhard Schölkopf, John C. Platt, and Thomas Hofmann (eds.), _Advances in Neural Information Processing Systems 19, Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 4-7, 2006_ , pp. 1353–1360. MIT Press, 2006.  URL https://proceedings.neurips.cc/paper/2006/hash/532b7cbe070a3579f424988a040752f2-Abstract.html.
  * Wang et al. (2020) Xin Wang, Yudong Chen, and Wenwu Zhu.  A survey on curriculum learning.  _ArXiv preprint_ , abs/2010.13166, 2020.  URL https://arxiv.org/abs/2010.13166.
  * Wang et al. (2019a) Xinshao Wang, Yang Hua, Elyor Kodirov, and Neil M. Robertson.  Imae for noise-robust learning: Mean absolute error does not treat examples equally and gradient magnitude’s variance matters.  _ArXiv preprint_ , abs/1903.12141, 2019a.  URL https://arxiv.org/abs/1903.12141.
  * Wang et al. (2019b) Xinshao Wang, Elyor Kodirov, Yang Hua, and Neil M. Robertson.  Derivative manipulation for general example weighting.  _ArXiv preprint_ , abs/1905.11233, 2019b.  URL https://arxiv.org/abs/1905.11233.
  * Wang et al. (2019c) Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey.  Symmetric cross entropy for robust learning with noisy labels.  In _2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019_ , pp. 322–330. IEEE, 2019c.  doi: 10.1109/ICCV.2019.00041.  URL https://doi.org/10.1109/ICCV.2019.00041.
  * Wei et al. (2021) Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu.  To smooth or not? when label smoothing meets noisy labels.  _ArXiv preprint_ , abs/2106.04149, 2021.  URL https://arxiv.org/abs/2106.04149.
  * Wei et al. (2022) Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu.  Learning with noisy labels revisited: A study using real-world human annotations.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=TBWA6PLJZQm.
  * Yao et al. (2019) Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James Kwok.  Searching to exploit memorization effect in learning from corrupted labels, 2019.  URL https://arxiv.org/abs/1911.02377.
  * Zhang et al. (2017) Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning requires rethinking generalization.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=Sy8gdB9xx.
  * Zhang et al. (2018) Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy Gwinnup, Marianna J. Martindale, Paul McNamee, Kevin Duh, and Marine Carpuat.  An empirical exploration of curriculum learning for neural machine translation.  _ArXiv preprint_ , abs/1811.00739, 2018.  URL https://arxiv.org/abs/1811.00739.
  * Zhang & Sabuncu (2018) Zhilu Zhang and Mert R. Sabuncu.  Generalized cross entropy loss for training deep neural networks with noisy labels.  In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , pp. 8792–8802, 2018.  URL https://proceedings.neurips.cc/paper/2018/hash/f2925f97bc13ad2852a7a551802feea0-Abstract.html.
  * Zhou et al. (2021a) Tianyi Zhou, Shengjie Wang, and Jeff A. Bilmes.  Robust curriculum learning: from clean label detection to noisy label self-correction.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021a.  URL https://openreview.net/forum?id=lmTWnm3coJJ.
  * Zhou et al. (2021b) Xiong Zhou, Xianming Liu, Junjun Jiang, Xin Gao, and Xiangyang Ji.  Asymmetric loss functions for learning with noisy labels.  In Marina Meila and Tong Zhang (eds.), _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 12846–12856. PMLR, 2021b.  URL http://proceedings.mlr.press/v139/zhou21f.html.

##  Appendix A Extended Review of Loss Functions

Due to limited space, we only briefly describe typical robust loss functions
in section 3.1. As a general reference, here we provide a comprehensive review
of loss functions related to the standard form Eq. 1. Similar to section 3.1,
we ignore the differences in constant scaling factors and additive bias. Loss
functions and their sample-weighting functions are summarized in Table 6. We
examine how hyperparameters affect their sample-weighting functions in Fig. 5.

###  A.1 Loss Functions without Robustness Guarantees

Cross Entropy (CE)

$L_{\mathrm{CE}}(\bm{s},y)=-\log p_{y}$

is the standard loss function for classification.

Focal Loss (FL; Lin et al. 2017)

$L_{\mathrm{FL}}(\bm{s},y)=-(1-p_{y})^{q}\log p_{y}$

aims to address label imbalance when training object detection models. Both CE
and FL are neither symmetric (Ma et al., 2020) nor asymmetric (Zhou et al.,
2021b).

###  A.2 Symmetric Loss Functions

Mean Absolute Error (MAE; Ghosh et al. 2017)

$L_{\mathrm{MAE}}(\bm{s},y)=\sum_{i=1}^{k}|\mathbb{I}(i=y)-p_{i}|=2-2p_{y}\propto
1-p_{y}$

is a classic symmetric loss function, where $\mathbb{I}(i=y)$ is the indicator
function.

Reverse Cross Entropy (RCE; Wang et al. 2019c)

$L_{\mathrm{RCE}}(\bm{s},y)=\sum_{i=1}^{k}p_{i}\log\mathbf{1}(i=y)=\sum_{i\neq
y}p_{i}A=(1-p_{y})A\propto 1-p_{y}=L_{\mathrm{MAE}}(\bm{s},y)$

is equivalent to MAE in implementation, where $\log 0$ is truncated to a
negative constant $A$ to avoid numerical overflow.

Ma et al. (2020) argued that any generic loss functions with
$L(\bm{s},i)>0,\forall i\in\\{1..k\\}$ can become symmetric by simply
normalizing them. As an example,

Normalized Cross Entropy (NCE; Ma et al. 2020)

$L_{\mathrm{NCE}}(\bm{s},y)=\frac{L_{\mathrm{CE}}(\bm{s},y)}{\sum_{i=1}^{k}L_{\mathrm{CE}}(\bm{s},i)}=\frac{-\log
p_{y}}{\sum_{i=1}^{k}-\log p_{i}}$

is a symmetric loss function. However, NCE does not follow the standard form
of Eq. 1 as it additionally depends on $p_{i}$, $i\neq y$. It involves an
additional regularizer, thus being more relevant to discussions in section
A.5.

Name | Function | Sample Weight $w$ | Constraints
---|---|---|---
CE | $-\log p_{y}$ | $1-p_{y}$ |
FL | $-(1-p_{y})^{q}\log p_{y}$ | $(1-p_{y})^{q}(1-p_{y}-qp_{y}\log p_{y})$ | $q>0$
MAE/RCE | $1-p_{y}$ | $p_{y}(1-p_{y})$ |
AUL | $\frac{(a+1)-(a+p_{y})^{q}}{q}$ | $p_{y}(1-p_{y})(a-p_{y})^{q-1}$ |  $a>1$, $q>0$
AGCE | $\frac{(a-p_{y})^{q}-(a-1)^{q}}{q}$ | $p_{y}(a+p_{y})^{q-1}(1-p_{y})$ |  $a>0$, $q>0$
AEL | $e^{-p_{y}/q}$ | $\frac{1}{q}p_{y}(1-p_{y})e^{-p_{y}/q}$ | $q>0$
GCE | $(1-p_{y}^{q})/q$ | $p_{y}^{q}(1-p_{y})$ | $0<q\leq 1$
SCE | $-(1-q)\log p_{y}+q(1-p_{y})$ | $(1-q+q\cdot p_{y})(1-p_{y})$ | $0<q<1$
TCE | $\sum_{i=1}^{q}(1-p_{y})^{i}/i$ | $p_{y}\sum_{i=1}^{q}(1-p_{y})^{i}$ | $q\geq 1$
Table 6: Expressions, constraints of hyperparameters and sample-weighting
functions of loss functions reviewed in Appendix A that follow the standard
form Eq. 1.

###  A.3 Asymmetric Loss Functions

Zhou et al. (2021b) derived the asymmetric condition for noise robustness and
propose numerous asymmetric loss functions:

Asymmetric Generalized Cross Entropy (AGCE)

$L_{\mathrm{AGCE}}(\bm{s},y)=\frac{(a+1)-(a+p_{y})^{q}}{q}$

where $a>0$ and $q>0$. It is asymmetric when $\mathbb{I}(q\leq
1)(\frac{a+1}{a})^{1-q}+\mathbb{I}(q>1)\leq 1/\tilde{r}$.

Asymmetric Unhinged Loss (AUL)

$L_{\mathrm{AUL}}(\bm{s},y)=\frac{(a-p_{y})^{q}-(a-1)^{q}}{q}$

where $a>1$ and $q>0$. It is asymmetric when $\mathbb{I}(q\leq
1)(\frac{a}{a-1})^{q-1}+\mathbb{I}(q\leq 1)\leq 1/\tilde{r}$.

Asymmetric Exponential Loss (AEL)

$L_{\mathrm{AEL}}(\bm{s},y)=e^{-p_{y}/q}$

where $q>0$. It is asymmetric when $e^{1/q}\leq 1/\tilde{r}$.

###  A.4 Combined Loss Functions

Generalized Cross Entropy (GCE; Zhang & Sabuncu 2018)

$L_{\mathrm{GCE}}(\bm{s},y)=\frac{1-p_{y}^{q}}{q}$

can be viewed as a smooth interpolation between CE and MAE, where $0<q\leq 1$.
CE or MAE can be recovered by setting $q\to 0$ or $q=1$.

(a) FL

(b) AUL, $a=2.0$

(c) AUL, $q=0.1$

(d) AGCE, $a=1.0$

(e) AGCE, $q=0.5$

(f) AEL

(g) GCE

(h) SCE

(i) TCE

Figure 5: How hyperparameters affect the sample-weighting functions in Table
6. The initial $\Delta_{y}$ distributions of CIFAR100 extracted with a
randomly initialized model are included as reference.

Symmetric Cross Entropy (SCE; Wang et al. 2019c)

$\displaystyle L_{\mathrm{SCE}}(\bm{s},y)$ $\displaystyle=a\cdot
L_{\mathrm{CE}}(\bm{s},y)+b\cdot L_{\mathrm{RCE}}(\bm{s},y)$
$\displaystyle\propto(1-q)\cdot(-\log p_{i})+q\cdot(1-p_{i})$

is a weighted average of CE and RCE (MAE), where $a>0$, $b>0$, and $0<q<1$.

Taylor Cross Entropy (TCE; Feng et al. 2020)

$L_{\mathrm{TCE}}(\bm{s},y)=\sum_{i=1}^{q}\frac{(1-p_{y})^{i}}{i}$

is derived from Taylor series of the $\log$ function. It reduces to MAE when
$q=1$. Interestingly, the summand of TCE $(1-p_{y})^{i}/i$ with $i>2$ is
proportional to AUL with $a=1$ and $q=i$. Thus TCE can be viewed as a
combination of symmetric and asymmetric loss functions.

Active-Passive Loss (APL; Ma et al. 2020)

Ma et al. (2020) propose weighted combinations of active and passive loss
functions. We include NCE+MAE as an example:

$\displaystyle L_{\mathrm{NCE+MAE}}(\bm{s},y)$ $\displaystyle=a\cdot
L_{\mathrm{NCE}}(\bm{s},y)+b\cdot L_{\mathrm{MAE}}(\bm{s},y)$
$\displaystyle\propto(1-q)\cdot\frac{-\log p_{y}}{\sum_{i=1}^{k}-\log
p_{i}}+q\cdot(1-p_{y})$

where $a>0$, $b>0$, and $0<q<1$.

###  A.5 Loss Functions with Additional Regularizers

We additionally review loss functions that implicitly involve a regularizer
and a primary loss function following the standard form Eq. 1. See Table 7 for
a summary. In addition to the sample-weighting curriculums implicitly defined
by the primary loss function, the additional regularizer complicates the
analysis of the training dynamics. We leave investigations on how these
regularizers affect noise robustness for future work.

Name | Original | Primary Loss | Regularizer
---|---|---|---
MSE | $1-2p_{y}+\sum_{i=1}^{k}p_{i}^{2}$ | $1-p_{y}$ | $\sum_{i=1}^{k}p_{i}^{2}$
PL(CR) | $-\log p_{y}+\log p_{y_{n}|\bm{x}_{m}}$ | $-\log p_{y}$ | $\sum_{i=1}^{k}P(\tilde{y}=i)\log p_{i}$
CE+GLS | $-\sum_{i=1}^{k}[\mathbb{I}(i=y)(1-\alpha)+\frac{\alpha}{k}]\log p_{i}$ | $-\log p_{y}$ | $\pm\sum_{i=1}^{k}\frac{1}{k}\log p_{i}$
NCE | $-\log p_{y}/(\sum_{i=1}^{k}-\log p_{i})$ | $-\gamma_{\mathrm{NCE}}\cdot\log p_{i}$ | $\sum_{i=1}^{k}\frac{1}{k}\log p_{i}$
Table 7: Original expressions, primary loss functions in the standard form Eq.
1 and regularizers for loss functions reviewed in section A.5. We view PL in
its expectation to derive its regularizer. $p_{y_{n}|\bm{x}_{m}}$ is the
softmax probability of a random label $y_{n}$ with a random input $\bm{x}_{m}$
sampled from the noisy data. $\gamma_{\mathrm{NCE}}=1/(\sum_{i=1}^{k}-\log
p_{i})$ is a scalar wrapped with the stop-gradient operator.

Mean Square Error (MSE; Ghosh et al. 2017)

$\displaystyle L_{\mathrm{MSE}}(\bm{s},y)$
$\displaystyle=\sum_{i=1}^{k}(\mathbb{I}(i=y)-p_{i})^{2}=1-2p_{y}+\sum_{i=1}^{k}p_{i}^{2}$
$\displaystyle\propto
1-p_{y}+\frac{1}{2}\cdot\sum_{i=1}^{k}p_{i}^{2}=L_{\mathrm{MAE}}(\bm{s},y)+\alpha\cdot
R_{\mathrm{MSE}}(\bm{s})$

is more robust than CE (Ghosh et al., 2017), where $\alpha=0.5$ and the
regularizer

$R_{\mathrm{MSE}}(\bm{s})=\sum_{i=1}^{k}p_{i}^{2}$ (5)

increases the entropy of the softmax output. We can generalize $\alpha$ to a
hyperparamter, making MSE a combination of MAE and an entropy regularizer
$R_{\mathrm{MSE}}$.

Peer Loss (PL; Liu & Guo 2020)

$L_{\mathrm{PL}}(\bm{s},y)=L(\bm{s},y)-L(\bm{s}_{n},y_{m})$

makes a generic loss function $L(\bm{s},y)$ robust against label noise, where
$\bm{s}_{n}$ denotes the score of an input $\bm{x}_{n}$ and $y_{m}$ a label,
both randomly sampled from the noisy data. Its noise robustness is
theoretically established for binary classification and extended to multi-
class setting (Liu & Guo, 2020).

Confidence Regularizer (CR; Cheng et al. 2021)

$R_{\mathrm{CR}}(\bm{s})=-\mathbb{E}_{\tilde{y}}[L(\bm{s},\tilde{y})]$

is shown (Cheng et al., 2021) to be the regularizer induced by PL in
expectation. Substituting $L$ with cross entropy leads to

$R_{\mathrm{CR}}(\bm{s})=-\mathbb{E}_{\tilde{y}}[-\log
p_{\tilde{y}}]=\sum_{i=1}^{k}P(\tilde{y}=i)\log p_{i}$ (6)

Minimizing $R_{\mathrm{CR}}(\bm{s})$ thus makes the softmax output
distribution $\bm{p}$ deviate from the prior label distribution of the noisy
dataset $P(\tilde{y}=i)$, reducing the entropy of the softmax output.

Generalized Label Smoothing (GLS; Wei et al. 2021)

Lukasik et al. (2020) show that label smoothing (LS; Szegedy et al. 2016) can
mitigate overfitting with label noise, which is later extended to GSL. Cross
entropy with GLS is

$\displaystyle L_{\mathrm{CE+GLS}}(\bm{s},y)$
$\displaystyle=\sum_{i=1}^{k}-[\mathbb{I}(i=y)(1-\alpha)+\frac{\alpha}{k}]\log
p_{i}$ $\displaystyle=-(1-\alpha)\log
p_{y}-\alpha\cdot\frac{1}{k}\sum_{i=1}^{k}\log p_{i}$
$\displaystyle\propto-\log
p_{y}-\frac{\alpha}{1-\alpha}\cdot\frac{1}{k}\sum_{i=1}^{k}\log
p_{i}=L_{\mathrm{CE}}(\bm{s},y)+\alpha^{\prime}\cdot R_{\textrm{GLS}}(\bm{s})$

where $\alpha^{\prime}=\alpha/(1-\alpha)$, has regularizer $R_{\textrm{GLS}}$

$R_{\mathrm{GLS}}(\bm{s})=-\sum_{i=1}^{k}\frac{1}{k}\log p_{i}$ (7)

With $\alpha^{\prime}>0$, $R_{\mathrm{GLS}}$ corresponds to the original label
smoothing, which increases the entropy of softmax outputs. In contrast,
$\alpha^{\prime}<0$ corresponding to negative label smoothing (Wei et al.,
2021), which decreases the output entropy similar to $R_{\mathrm{CR}}$.

####  A.5.1 Derivations for NCE

Deriving Eq. 2 With equivalent derivatives, since

$\displaystyle\nabla_{\bm{s}}L_{\mathrm{NCE}}(\bm{s},y)$
$\displaystyle=\frac{\nabla_{\bm{s}}L_{\mathrm{CE}}(\bm{s},y)\cdot\left[\sum_{i=1}^{k}L_{\mathrm{CE}}(\bm{s},i)\right]-\nabla_{\bm{s}}\left[\sum_{i=1}^{k}L_{\mathrm{CE}}(\bm{s},i)\right]\cdot
L_{\mathrm{CE}}(\bm{s},y)}{\left[\sum_{i=1}^{k}L_{\mathrm{CE}}(\bm{s},i)\right]^{2}}$
$\displaystyle=\frac{1}{\sum_{i=1}^{k}L_{\mathrm{CE}}(\bm{s},i)}\left\\{\nabla_{\bm{s}}L_{\mathrm{CE}}(\bm{s},y)+\frac{kL_{\mathrm{CE}}(\bm{s},y)}{\sum_{i=1}^{k}L_{\mathrm{CE}}(\bm{s},i)}\cdot\nabla_{\bm{s}}\left[\sum_{i=1}^{k}-\frac{1}{k}L_{\mathrm{CE}}(\bm{s},i)\right]\right\\}$
$\displaystyle=\gamma_{\mathrm{NCE}}\cdot\left[\nabla_{\bm{s}}L_{\mathrm{CE}}(\bm{s},y)+\epsilon_{\mathrm{NCE}}\cdot\nabla_{\bm{s}}R_{\mathrm{NCE}}(\bm{s})\right],$

NCE can be rewritten as

$L_{\mathrm{NCE}}(\bm{s},y)=\gamma_{\mathrm{NCE}}\cdot
L_{\mathrm{CE}}(\bm{s},y)+\gamma_{\mathrm{NCE}}\cdot\epsilon_{\mathrm{NCE}}\cdot
R_{\mathrm{NCE}}(\bm{s})$

where $\gamma_{\mathrm{NCE}}=1/(\sum_{i=1}^{k}-\log p_{i})$ and
$\epsilon_{\mathrm{NCE}}=k(-\log p_{y})/(\sum_{i=1}^{k}-\log p_{i})$ are
scalar weights wrapped with the stop-gradient operator as discussed in section
4.1. The regularizer

$R_{\mathrm{NCE}}(\bm{s})=\sum_{i=1}^{k}\frac{1}{k}\log p_{i}$

has a form similar to $R_{\mathrm{GLS}}$.

Deriving $\hat{w}_{\mathrm{NCE}}$ of Eq. 3 Here we derive the upperbound of
$\|\nabla_{\bm{s}}L_{\mathrm{NCE}}(\bm{s},y)\|_{1}$ discussed in section 4.1:

$\displaystyle\|\nabla_{\bm{s}}L_{\mathrm{NCE}}(\bm{s},y)\|_{1}$
$\displaystyle\leq\gamma_{\mathrm{NCE}}\cdot\left(\|\nabla_{\bm{s}}L_{\mathrm{CE}}(\bm{s},y)\|_{1}+\epsilon_{\mathrm{NCE}}\cdot\|\nabla_{\bm{s}}R_{\mathrm{NCE}}(\bm{s})\|_{1}\right)$
$\displaystyle\leq\gamma_{\mathrm{NCE}}\cdot\left(\|\nabla_{\bm{s}}L_{\mathrm{CE}}(\bm{s},y)\|_{1}+\epsilon_{\mathrm{NCE}}\cdot\frac{1}{k}\sum_{i=1}^{k}\|\nabla_{\bm{s}}L_{\mathrm{CE}}(\bm{s},i)\|_{1}\right)$
$\displaystyle=\gamma_{\mathrm{NCE}}\cdot\left(w_{\mathrm{CE}}\cdot\|\nabla_{\bm{s}}\Delta_{y}\|_{1}+\epsilon_{\mathrm{NCE}}\cdot\frac{1}{k}\sum_{i=1}^{k}w_{\mathrm{CE}}\cdot\|\nabla_{\bm{s}}\Delta_{i}\|_{1}\right)$
$\displaystyle=2\gamma_{\mathrm{NCE}}\cdot
w_{\mathrm{CE}}\left(1+\epsilon_{\mathrm{NCE}}\right)$
$\displaystyle=\hat{w}_{\mathrm{NCE}}$

The derivation is based on the inequality $|x\pm y|\leq|x|+|y|$ and the fact
that $\|\nabla_{\bm{s}}\Delta_{i}\|_{1}=2$. The latter can be proved by
straightforward calculations. Given

$\frac{\partial\Delta_{i}}{\partial
s_{j}}=\left\\{\begin{array}[]{lr}1,&j=i\\\ -\frac{e^{s_{j}}}{\sum_{k\neq
i}e^{s_{k}}}=-\frac{p_{j}}{1-p_{i}},&j\neq i\end{array}\right.$

we then have

$\|\nabla_{\bm{s}}\Delta_{i}\|_{1}=\sum_{j}\left|\frac{\partial\Delta_{i}}{\partial
s_{j}}\right|=1+\sum_{j\neq i}\frac{p_{j}}{1-p_{i}}=1+1=2$

##  Appendix B Detailed Experimental Settings

Label noise  The synthetic noisy labels are generated following (Ma et al.,
2020; Zhou et al., 2021b; Patrini et al., 2017). For symmetric label noise,
the training labels are randomly flipped to a different class with
probabilities $\eta\in\\{0.2,0.4,0.6,0.8\\}$. Asymmetric label noise is
generated from a class-dependent flipping pattern. On CIFAR100, the 100
classes are grouped into 20 super-classes, each having 5 sub-classes. Each
class is flipped within the same super-class into the next in a circular
fashion. The flip probabilities are $\eta\in\\{0.1,0.2,0.3,0.4\\}$. Human
label noise for CIFAR10/100 are adopted from Wei et al. (2022). We use the
“worst” labels of CIFAR10-N and the “fine” labels of CIFAR100-N, both leading
to $\eta=0.4$.

Models and hyperparameters  We use a 4-layer CNN for MNIST, an 8-layer CNN for
CIFAR10, a ResNet-34 (He et al., 2016) for CIFAR100, and a ResNet-50 (He et
al., 2016) for WebVision, all with batch normalization (Ioffe & Szegedy,
2015). Data augmentation on CIFAR10/100 include random width/height shift and
horizontal flip. On WebVision, we additionally include random cropping and
color jittering. Without further specifications, all models are trained using
SGD with momentum 0.9 and batch size 128 for 50, 120, 200 and 250 epochs on
MNIST, CIFAR10, CIFAR100 and WebVision, respectively. Learning rates with
cosine annealing are 0.01 on MNIST and CIFAR10, 0.1 on CIFAR100, and 0.2 on
WebVision. Weight decays are $10^{-3}$ on MNIST, $10^{-4}$ on CIFAR10,
$10^{-5}$ on CIFAR100 and $3\times 10^{-5}$ on WebVision. All loss functions
are normalized to have unit maximum in sample weights, which is different from
(Ma et al., 2020). Hyperparameters of loss functions are listed in Tables 8
and 12 for different experiments.

##  Appendix C Additional Results to Understand Robust Loss Functions

We complement section 5 in the main text with detailed derivations and
additional results. section C.1 extends discussions in section 5.1 while
section C.2 extends section 5.2

| SCE | GCE | NCE+MAE | AUL | AGCE | AUL† | AGCE†
---|---|---|---|---|---|---|---
$a$ | / | / | / | 1.1 | 0.1 | 3.0 | 1.6
$q$ | 0.7 | 0.3 | 0.3 | 5.0 | 0.1 | 0.7 | 2.0
Table 8: Hyperparameters of different loss functions for results in sections
5.1 and C.1. They are tuned on CIFAR100 without label noise. Settings with
inferior hyperparameters are denoted with †.

###  C.1 Understanding Underfitting of Robust Loss Functions

Hyperparameters  We list the hyperparameters tuned on CIFAR100 without label
noise in Table 8 for experiments in sections 5.1 and C.1.

Simulated $\Delta_{y}$ approximate real settings. We compare the simulated
$\Delta_{y}$ distributions based on our assumptions in section 5.1 to
distributions of real datasets at initialization in Fig. 6. The expectations
of simulated $\Delta_{y}$ follow real settings, which supports the analysis in
section 5.1.

(a) MNIST

(b) CIFAR10

(c) CIFAR100

(d) WebVision50

(e) WebVision200

(f) WebVision400

Figure 6: Comparisons between simulated and real $\Delta_{y}$ distributions at
initialization. The simulations are based on the assumption that class scores
follow normal distribution $s_{i}\sim\mathcal{N}(0,1)$ at initialization and
plotted as curves. Real distributions are extracted with randomly initialized
models and plotted as histograms. The vertical axis denotes probability
density $f(\Delta_{y})$.

Derivation of Eq. 4 Assume that class scores $s_{i}$ at _initialization_ are
i.i.d. normal variables, i.e., $s_{i}\sim\mathcal{N}(\mu,\sigma)$,

$\displaystyle\mathbb{E}(\Delta_{y})$
$\displaystyle=\mathbb{E}[s_{y}-\log\sum_{i\neq
y}e^{s_{i}}]=\mu-\mathbb{E}[\log\sum_{i\neq y}e^{s_{i}}]$
$\displaystyle\approx_{1}\mu-\log\mathbb{E}[\sum_{i\neq
y}e^{s_{i}}]+\frac{\mathbb{V}[\sum_{i\neq
y}e^{s_{i}}]}{2\mathbb{E}[\sum_{i\neq y}e^{s_{i}}]^{2}}$
$\displaystyle=_{2}\mu-\log\\{(k-1)\mathbb{E}[e^{s_{y}}]\\}+\frac{(k-1)\mathbb{V}[e^{s_{y}}]}{2\\{(k-1)\mathbb{E}[e^{s_{y}}]\\}^{2}}$
$\displaystyle=_{3}\mu-\log[(k-1)e^{\mu+\sigma^{2}/2}]+\frac{(k-1)(e^{\sigma^{2}}-1)e^{2\mu+\sigma^{2}}}{2[(k-1)e^{\mu+\sigma^{2}/2}]^{2}}$
$\displaystyle=-\log(k-1)-\sigma^{2}/2+\frac{e^{\sigma^{2}}-1}{2(k-1)}$

where $\approx_{1}$ follows the approximation with Taylor expansion
$\mathbb{E}[\log
X]\approx\log\mathbb{E}[X]-\mathbb{V}[X]/(2\mathbb{E}[X]^{2})$ (Teh et al.,
2006), $=_{2}$ utilizes properties of sum of log-normal variables (Cobb et
al., 2012), and $=_{3}$ substitutes $\mathbb{E}[e^{s_{y}}]$ and
$\mathbb{V}[e^{s_{y}}]$ with expressions for log-normal distributions.

| Clean | Symmetric | Asymmetric | Human
---|---|---|---|---
Settings | 0.0 | 0.2 | 0.4 | 0.6 | 0.8 | 0.1 | 0.2 | 0.3 | 0.4 | 0.4
MAE shift | 2.0 | 2.6 | 3.0 | 4.0 | 4.0 | 2.6 | 2.6 | 3.0 | 3.0 | 2.6
MAE scale | 2.0 | 2.6 | 2.6 | 3.0 | 4.0 | 2.6 | 2.6 | 3.0 | 3.0 | 2.6
AGCE shift | 4.0 | 4.0 | 4.0 | 5.0 | 5.0 | 4.0 | 4.0 | 4.0 | 4.0 | 4.0
AGCE scale | 2.6 | 3.0 | 4.0 | 5.0 | 5.0 | 3.0 | 3.0 | 3.0 | 3.0 | 4.0
Table 9: Hyperparameter $\tau$ of $w^{*}(\Delta_{y})$ and $w^{+}(\Delta_{y})$ for different noise settings on CIFAR100. For reference, $\mathbb{E}[\Delta_{y}]=5.097$ on CIFAR100 and $\mathbb{E}[\Delta_{y}]=2.717$ on CIFAR10. Better performance can be achieved with a more thorough hyperparameter search.  | Clean | Symmetric Noise (Noise Rate $\eta$)
---|---|---
Loss | $\eta=0$ | $\eta=0.2$ | $\eta=0.4$ | $\eta=0.6$ | $\eta=0.8$
CE‡ | 71.33 $\pm$ 0.43 | 56.51 $\pm$ 0.39 | 39.92 $\pm$ 0.10 | 21.39 $\pm$ 1.17 | 7.59 $\pm$ 0.20
GCE‡ | 63.09 $\pm$ 1.39 | 61.57 $\pm$ 1.06 | 56.11 $\pm$ 1.35 | 45.28 $\pm$ 0.61 | 17.42 $\pm$ 0.06
NCE‡ | 29.96 $\pm$ 0.73 | 25.27 $\pm$ 0.32 | 19.54 $\pm$ 0.52 | 13.51 $\pm$ 0.65 | 8.55 $\pm$ 0.37
NCE+AUL‡ | 68.96 $\pm$ 0.16 | 65.36 $\pm$ 0.20 | 59.25 $\pm$ 0.23 | 46.34 $\pm$ 0.21 | 23.03 $\pm$ 0.64
AGCE | 49.27 $\pm$ 1.03 | 49.17 $\pm$ 2.15 | 47.76 $\pm$ 1.75 | 38.17 $\pm$ 1.43 | 16.03 $\pm$ 0.59
AGCE shift | 69.39 $\pm$ 0.84 | 62.81 $\pm$ 0.42 | 48.21 $\pm$ 1.06 | 36.70 $\pm$ 2.89 | 14.49 $\pm$ 0.17
AGCE scale | 70.57 $\pm$ 0.62 | 64.73 $\pm$ 0.98 | 56.69 $\pm$ 0.33 | 39.02 $\pm$ 1.20 | 14.64 $\pm$ 0.79
MAE | 3.69 $\pm$ 0.59 | 2.92 $\pm$ 0.46 | 1.29 $\pm$ 0.50 | 2.27 $\pm$ 1.24 | 1.00 $\pm$ 0.00
MAE shift | 69.02 $\pm$ 0.78 | 59.75 $\pm$ 0.84 | 44.60 $\pm$ 0.24 | 24.27 $\pm$ 0.26 | 8.08 $\pm$ 0.26
MAE scale | 70.97 $\pm$ 0.41 | 66.83 $\pm$ 0.84 | 60.57 $\pm$ 1.04 | 49.23 $\pm$ 1.22 | 24.44 $\pm$ 0.73
Table 10: Addition results to Table 3 with more symmetric label noise rates on CIFAR100. | Clean | Asymmetric Noise (Noise Rate $\eta$)
---|---|---
Loss | $\eta=0$ | $\eta=0.1$ | $\eta=0.2$ | $\eta=0.3$ | $\eta=0.4$
CE‡ | 71.33 $\pm$ 0.43 | 64.85 $\pm$ 0.37 | 58.11 $\pm$ 0.32 | 50.68 $\pm$ 0.55 | 40.17 $\pm$ 1.31
GCE‡ | 63.09 $\pm$ 1.39 | 63.01 $\pm$ 1.01 | 59.35 $\pm$ 1.10 | 53.83 $\pm$ 0.64 | 40.91 $\pm$ 0.57
NCE‡ | 29.96 $\pm$ 0.73 | 27.59 $\pm$ 0.54 | 25.75 $\pm$ 0.50 | 24.28 $\pm$ 0.80 | 20.64 $\pm$ 0.40
NCE+AUL‡ | 68.96 $\pm$ 0.16 | 66.62 $\pm$ 0.09 | 63.86 $\pm$ 0.18 | 50.38 $\pm$ 0.32 | 38.59 $\pm$ 0.48
AGCE | 49.27 $\pm$ 1.03 | 47.53 $\pm$ 0.73 | 46.77 $\pm$ 2.37 | 39.82 $\pm$ 2.70 | 33.40 $\pm$ 1.57
AGCE shift | 69.39 $\pm$ 0.84 | 63.03 $\pm$ 0.42 | 55.84 $\pm$ 0.78 | 49.05 $\pm$ 0.81 | 40.76 $\pm$ 0.74
AGCE scale | 70.57 $\pm$ 0.62 | 67.13 $\pm$ 0.60 | 59.71 $\pm$ 0.10 | 48.23 $\pm$ 0.29 | 39.71 $\pm$ 0.17
MAE | 3.69 $\pm$ 0.59 | 3.59 $\pm$ 0.56 | 3.19 $\pm$ 0.98 | 2.11 $\pm$ 1.93 | 2.53 $\pm$ 1.34
MAE shift | 68.57 $\pm$ 0.54 | 63.44 $\pm$ 0.32 | 56.47 $\pm$ 0.48 | 48.79 $\pm$ 1.22 | 39.83 $\pm$ 0.18
MAE scale | 70.97 $\pm$ 0.41 | 69.50 $\pm$ 0.24 | 64.80 $\pm$ 0.49 | 59.04 $\pm$ 1.52 | 44.48 $\pm$ 1.05
Table 11: Addition results to Table 3 with more asymmetric label noise rates
on CIFAR100.

####  C.1.1 Addressing Underfitting from Marginal Initial Sample Weights

Hyperparamter $\tau$ for different settings  The hyperparameter $\tau$
controlling the shape of modified sample-weighting functions
$w^{+}(\Delta_{y})$ and $w^{*}(\Delta_{y})$ can affect the noise robustness.
Thus we tune $\tau$ for the best performance under different noise types and
noise rates, which are listed in Table 9.

Additional results with $w^{*}(\Delta_{y})$ and $w^{+}(\Delta_{y})$. We report
additional results under symmetric and asymmetric label noise with diverse
noise rates $\eta$ in Table 10 and Table 11, respectively. Performance of MAE
and AGCE gets substantially improved with $w^{*}(\Delta_{y})$ and
$w^{+}(\Delta_{y})$.

Visualization of $w^{*}(\Delta_{y})$ and $w^{+}(\Delta_{y})$. In Fig. 7 we
visualize the shifted and scaled sample-weighting functions of MAE on
CIFAR100. Although both achieve the same initial sample weights at
$|\mathbb{E}[\Delta_{y}]|$ of CIFAR100, $w^{+}(\Delta_{y})$ diminishes much
faster as $\Delta_{y}$ increases, leading to insufficient learning of training
samples, which can explain its inferior performance in Tables 3, 4, 10 and 11.

Robustness of loss functions from $w^{*}(\Delta_{y})$ and $w^{+}(\Delta_{y})$.
Our proposed $w^{*}(\Delta_{y})$ and $w^{+}(\Delta_{y})$ aim to address the
underfitting issue of robust loss functions with marginal initial sample
weights. They modify $p_{y}$ into

$p_{y}^{*}=\frac{1}{e^{-(\Delta_{y}/|\mathbb{E}[\Delta_{y}]|\cdot\tau)}+1}=\frac{1}{e^{-(\Delta_{y}\cdot\alpha)}+1}$

and

$p_{y}^{*}=\frac{1}{e^{-(\Delta_{y}+|\mathbb{E}[\Delta_{y}]|-\tau)}+1}=\frac{1}{e^{-(\Delta_{y}+\beta)}+1},$

where $\alpha=\tau/|\mathbb{E}[\Delta_{y}]|$ and
$\beta=|\mathbb{E}[\Delta_{y}]|-\tau$, which induces new loss functions
$L^{*}(\bm{s},y)=l(p_{y}^{*})$ and $L^{+}(\bm{s},y)=l(p_{y}^{+})$,
respectively. Commonly $\alpha<1$ and $\beta>0$ since a small $\tau$ leads to
large initial sample weights and underfitting results from small
$\mathbb{E}[\Delta_{y}]$. Notably, $\tau$ can determine the robustness of the
induced loss functions. As shown in Table 9, a larger noise rate $\eta$
requires a larger $\tau$ for better performance, which assigns less weights to
samples with small $\Delta_{y}$ in general. However, our preliminary
exploration find no straightforward derivation from $L(\bm{s},y)$ being
symmetric/asymmetric to $L^{*}(\bm{s},y)$ and $L^{+}(\bm{s},y)$ being
symmetric/asymmetric. We leave the theoretical discussions to future work.

Figure 7: Shifted, scaled and the vanilla sample-weighting functions of MAE on
CIFAR100. $\tau$ equals $|\mathbb{E}[\Delta_{y}]|$ on CIFAR10. We include the
initial $\Delta_{y}$ distributions of CIFAR10/100 extracted with a randomly
initialized model as reference.

###  C.2 Noise Robustness of Loss Functions

Computation of $\bar{w}_{\mathrm{clean}}$ and $\bar{w}_{\mathrm{noise}}$ for
$\mathrm{snr}$ in Table 5 The average weight for clean samples, adjusted by
the learning rate at each step $\alpha_{t}$, can be

$\bar{w}_{\mathrm{clean}}=\frac{\sum_{i,t}\alpha_{t}\cdot\mathbb{I}(\tilde{y}_{i,t}=y_{i,t})w_{i,t}}{\sum_{i,t}\alpha_{t}\cdot\mathbb{I}(\tilde{y}_{i,t}=y_{i,t})}$

where $w_{i,t}$ denotes the weight of $i$-th sample of the batch at step $t$,
$\tilde{y}_{i,t}$ is the potentially corrupted noisy label and $y_{i,t}$ the
uncorrupted label. Similarly, for noisy samples,

$\bar{w}_{\mathrm{noise}}=\frac{\sum_{i,t}\alpha_{t}\cdot\mathbb{I}(\tilde{y}_{i,t}\neq
y_{i,t})w_{i,t}}{\sum_{i,t}\alpha_{t}\cdot\mathbb{I}(\tilde{y}_{i,t}\neq
y_{i,t})}$

Hyperparameters  We list the hyperparameters for different loss functions in
Table 12 for results in sections 5.2 and C.2. In Fig. 8, we plot the sample-
weighting functions of different loss functions.

Changes of $\Delta_{y}$ distributions with different label noise and loss
functions  Complementing Fig. 3, in Fig. 9 we plot how distributions of
$\Delta_{y}$ change during training on CIFAR10 with additional types of label
noise using hyperparameters in Table 12. They follow similar trends as in Fig.
3, thus supporting analysis in section 5.2. As MAE is not robust against
asymmetric label noise with high $\eta$ (Ghosh et al., 2017), it results in
inferior performance. We also include results with additional loss functions
in Fig. 10. Since optimal hyperparameters will result in similar sample-
weighting functions, we choose hyperparameters for broad coverage of
$w(\Delta_{y})$ to better understand how they affect robustness.

| AUL | AGCE | GCE | SCE
---|---|---|---|---
$a$ | 2.0 | 3.0 | / | /
$q$ | 2.0 | 4.0 | 0.4 | 0.95
Table 12: Hyperparameters of different loss functions for results in sections
5.2 and C.2. They are selected for broad coverage of shapes, scales and
horizontal locations of sample-weighting functions instead of optimal
performance on CIFAR10. Figure 8: Plots of sample-weighting functions of loss
functions used in Table 5 with hyperparameters in Table 8.

(a) CE, Sym., 0.2: 74.49

(b) SCE, Sym., 0.2: 85.31

(c) MAE, Sym., 0.2: 86.71

(d) CE, Human, 0.4: 61.12

(e) SCE, Human, 0.4: 72.56

(f) MAE, Human, 0.4: 78.99

(g) CE, Asym., 0.4: 73.75

(h) SCE, Asym., 0.4: 72.44

(i) MAE, Asym., 0.4: 61.11

Figure 9: Additional results to Fig. 3 with different label noise: (a-c)
symmetric label noise with $\eta=0.2$; (d-f) human label noise with
$\eta=0.4$; (g-i) asymmetric label noise with $\eta=0.4$. Noisy samples are
colored green (on the left) and clean samples are orange (on the right). Test
accuracies are included in the caption for reference.

(a) AGCE, Sym., 0.4: 44.52

(b) GCE, Sym., 0.4: 66.60

(c) AUL, Sym., 0.4: 84.12

(d) AGCE, Human, 0.4: 35.73

(e) GCE, Human, 0.4: 68.22

(f) AUL, Human, 0.4: 77.22

(g) AGCE, Asym., 0.4: 46.75

(h) GCE, Asym., 0.4: 73.23

(i) AUL, Asym., 0.4: 67.72

Figure 10: Additional results to Fig. 3 with more robust loss functions under
different label noise: (a-c) symmetric label noise with $\eta=0.4$; (d-f)
human label noise with $\eta=0.4$; (g-i) asymmetric label noise with
$\eta=0.4$. Test accuracies are included in the caption for reference. Noisy
samples are colored green (on the left) and clean samples are orange (on the
right). Hyperparameters of these loss functions are selected for broad
coverage rather than optimal performance.