# Deep learning with noisy labels in medical prediction problems: a scoping
review

Yishu Wei  Department of Population Health Sciences, Weill Cornell Medicine,
New York, 10065, US  Reddit Inc., San Francisco, 16093, US  Ph.D  Department
of Population Health Sciences, Weill Cornell Medicine, New York, 10065, US
Reddit Inc., San Francisco, 16093, US  Yu Deng  Statistical Innovation Group,
Data & Statistical Science, AbbVie Inc., Chicago, 60064, US  Ph.D  Statistical
Innovation Group, Data & Statistical Science, AbbVie Inc., Chicago, 60064, US
Cong Sun  Department of Population Health Sciences, Weill Cornell Medicine,
New York, 10065, US  Ph.D  Department of Population Health Sciences, Weill
Cornell Medicine, New York, 10065, US  Mingquan Lin  Department of Population
Health Sciences, Weill Cornell Medicine, New York, 10065, US  Ph.D  Department
of Population Health Sciences, Weill Cornell Medicine, New York, 10065, US
Hongmei Jiang  Department of Statistics and Data Science, Northwestern
University, Evanston, 60208, US  Ph.D  Department of Statistics and Data
Science, Northwestern University, Evanston, 60208, US  Yifan Peng  Department
of Population Health Sciences, Weill Cornell Medicine, New York, 10065, US
Corresponding author(s<EMAIL_ADDRESS>Ph.D  Department of
Population Health Sciences, Weill Cornell Medicine, New York, 10065, US
Corresponding author(s<EMAIL_ADDRESS>
###### Abstract

Objectives: Medical research faces substantial challenges from noisy labels
attributed to factors like inter-expert variability and machine-extracted
labels. Despite this, the adoption of label noise management remains limited,
and label noise is largely ignored. To this end, there is a critical need to
conduct a scoping review focusing on the problem space. This scoping review
aims to comprehensively review label noise management in deep learning-based
medical prediction problems, which includes label noise detection, label noise
handling, and evaluation. Research involving label uncertainty is also
included.

Methods: Our scoping review follows the Preferred Reporting Items for
Systematic Reviews and Meta-Analyses (PRISMA) guidelines. We searched 4
databases, including PubMed, IEEE Xplore, Google Scholar, and Semantic
Scholar. Our search terms include “noisy label AND medical / healthcare /
clinical”, “uncertainty AND medical / healthcare / clinical”, and “noise AND
medical / healthcare / clinical”.

Results: A total of 60 papers met inclusion criteria between 2016 and 2023. A
series of practical questions in medical research are investigated. These
include the sources of label noise, the impact of label noise, the detection
of label noise, label noise handling techniques, and their evaluation.
Categorization of both label noise detection methods and handling techniques
are provided.

Discussion: From a methodological perspective, we observe that the medical
community has been up to date with the broader deep-learning community, given
that most techniques have been evaluated on medical data. We recommend
considering label noise as a standard element in medical research, even if it
is not dedicated to handling noisy labels. Initial experiments can start with
easy-to-implement methods, such as noise-robust loss functions, weighting, and
curriculum learning.

##  1 Introduction

Numerous studies have shown the extensive use of deep learning (DL) across
various medical domains. Remarkably, DL has achieved performance on par with
domain experts in specific applications. However, the performance of DL
training dramatically relies upon the quality of the underlying data, which
can be heavily impacted by data noise. This noise can be categorized into two
categories: feature and label noise. In this review, we focus on the latter.

The medical research landscape presents unique characteristics related to
label noise. Firstly, many disease diagnoses are challenging even for experts,
leading to diverse outcomes [1]. Moreover, there is a prevalent issue with the
scarcity of expert-annotated labels, owing to the costly and time-consuming
nature of the annotation process. To mitigate this, many researchers use
labels generated through natural language processing (NLP), crowd-sourcing, or
semi-supervised pseudo-label approaches. These alternatives, however, often
lead to considerable label noises. Despite the significant presence of noisy
labels in medical data, this issue hasn’t received much attention in studies
focused on medical machine learning.

To bridge this gap, this scoping review is intended to address several
pertinent questions: the sources of label noise, the consequential effects of
label noise, methods for identifying and handling label noise, as well as
their evaluation. Our target readers are researchers and practitioners engaged
in applying deep learning within the medical domain. Through examining these
questions, we aim to highlight the challenge posed by noisy labels and
encourage the adoption of suitable measures to mitigate this issue.

There are several reviews regarding learning from noisy labels in the general
machine-learning domain. For example, Frenay et al. covered studies before the
advent of deep learning [2]. Song et al. focused on model architecture change
to handle label noise [3]. Algan et al. comprehensively reviewed noisy label
handling methodology [4]. Liang et al. summarized methods regarding sample
selection and loss correction [5]. Among all the reviews currently available,
we only came across one survey in the medical domain [6].

Our review offers several extensions to previous studies. First, previous
studies focused on methodologies before 2018 within the broader deep-learning
community and cited limited references in the medical domain [6]. Our study
offers a comprehensive review of the recent advancements in this field.
Second, instead of focusing only on modeling techniques to handle label noise,
our approach adopts a problem-centric perspective and takes a deeper look into
the entire process of noisy label management, including noise detection, noise
handling, and evaluation. Typical scenarios where label noise may happen are
also audited. Through our survey, we hope to draw greater attention to this
underexplored challenge and suggest potential avenues for future research.

##  2 Methods

Our scoping review follows the Preferred Reporting Items for Systematic
Reviews and Meta-Analyses (PRISMA) guidelines [7].

###  2.1 Eligibility criteria

We overview research on noisy label detection and handling in disease
diagnosis and prognosis using deep learning methods. Segmentation problems are
excluded from the scope of this study. The inclusion criteria for our review
consisted of English-language articles published between 2016 and 2023,
including both conference papers and journal articles. We chose this time
frame to capture the most up-to-date research in this rapidly evolving field.
Additionally, we refer to relevant preprint articles to ensure we consider
cutting-edge research that has yet to be published in peer-reviewed venues.

###  2.2 Information sources

A search of multiple databases was carried out, including PubMed
(https://pubmed.ncbi.nlm.nih.gov), the Institute of Electrical and Electronics
Engineers (IEEE) Xplore Digital Library (https://ieeexplore.ieee.org), Google
Scholar (https://scholar.google.com), and Semantic Scholar
(https://www.semanticscholar.org). Important references are also tracked
accordingly. The most recent search was executed on Dec 15, 2023.

###  2.3 Search strategy

All the studies collected in this research were confined to the medical field.
Our search terms include “noisy label AND medical”, “uncertainty AND medical”,
and “noise AND medical”.

###  2.4 Study selection

Each article was reviewed and summarized by the source of label noise, problem
space (e.g., computer vision and Natural Language Processing (NLP)), medical
problem (e.g., skin cancer and chest image), label noise detection method,
label noise handling method, whether a clean dataset is needed, backbone
model, evaluation method and publishing year. During the screening and the
full-text review stages, we excluded review articles, non-medical articles,
image segmentation-related problems, data denoising, and feature denoising
problems.

##  3 Results

###  3.1 Included studies and datasets

####  3.1.1 Summary of included literature

A total of 172 articles were retrieved from the 4 databases, of which 60 met
the inclusion criteria for our review (Figure 1). Additionally, 21
supplementary articles partially addressed the problems of interest and were
selectively referenced in the relevant sections. Details and a summary of the
included papers are listed in Table LABEL:tab:summary_all_papers.

Figure 1: Flowchart of the search strategy. Table 1: Summary of all articles. Clean - the training dataset is noisy and the evaluation dataset is verified to be high quality. Inject - noise injected into training data. Same - training and testing are done on the same dataset. Inherent - the underlying problem is very hard to label. Disagreement - disagreement among experts or inter-expert variability. Curriculum - curriculum learning. OCT - Optical coherence tomography. WSI-Whole slide images Ref. |  ML scope |  Modality |  Disease |  Noise source |  Detection method |  Handling method |  Evaluation method |  Need clean data |  Backbone NN |  Year |  Dataset (size) |  Is dataset public
---|---|---|---|---|---|---|---|---|---|---|---|---
[8] |  CV |  MRI |  Brain disease |  Inherent |  Loss |  Reweight, robust loss |  Inject |  N |  MLP |  2016 |  Yale-B (2,048), COIL-100 (7,200), Scene-15 (4,485) |  Y
[9] |  CV |  X-ray |  Mammography |  Inherent |  – |  Noise adaption layer |  Inject |  N |  MLP |  2018 |  DDMS (55,890) |  Y
[10] |  CV |  Dermoscopy |  Melanoma |  Inherent |  Loss |  Reweight, curriculum |  Inject |  N |  ResNet |  2,019 |  ISIC-2019 (3,582) |  Y
[11] |  CV |  Dermoscopy, WSI, fundus photographs |  Skin lesion, prostate cancer, retinal disease |  Inherent, disagreement |  Disagreement, uncertainty |  Reweight, filter, curriculum |  Clean |  Y |  ResNet |  2021 |  ISIC-2019 (3,582), Gleason 2019 (331), Retinal disease Kaggle (88,702 ) |  Y
[12] |  CV |  Dermoscopy, WSI, X-ray |  Melanoma, lymph node, lung disease, prostate cancer |  Inherent |  Co-training |  Contrastive loss |  Inject, clean |  N |  ResNet |  2022 |  ISIC-2017 (2,600), ISIC archive (1,582), Kaggle histopathologic (220,025), Gleason 2019 (331) |  Y
[13] |  CV |  OCT, WSI |  Retinal disease, blood cell, colon pathology |  Inherent |  – |  Contrastive loss, mixup |  Inject |  N |  ResNet |  2023 |  OCT (2,100), Blood cell (4,200), Colon pathology (10,250) |  Y
[14] |  CV |  CT |  Covid |  Inherent |  – |  Robust loss |  Same |  N |  VGG |  2021 |  6982 |  N
[15] |  CV |  OCT |  Fbrocalcifc plaques |  Disagreement |  F1 score |  Active learning, label refinement |  – |  N |  CNN |  2020 |  6,556 |  Need Request
[16] |  CV |  WSI |  Cervical cancer |  Pseudo label |  Known |  Label refinement, regularization |  Clean |  Y |  Efficient Net |  2023 |  56,996 |  N
[17] |  CV |  MRI |  Aortic valve malformations |  Pseudo label |  Explicit |  Robust noise |  Clean |  N |  CNN-LSTM |  2019 |  UKbiobank (4,000) |  Need request
[18] |  CV |  WSI |  AD |  Inherent, disagreement |  Disagreement |  Label refinement |  Clean |  N |  CNN |  2022 |  29 WSI (153,050) |  Y
[19] |  CV |  WSI |  Stomach cancer |  Inherent |  Loss |  Label removal |  Inject |  N |  Dense Net |  2022 |  905 WSI (139,650) |  Y
[20] |  CV |  X-ray |  Covid |  Inherent |  – |  Label refinement, weighting |  Inject |  N |  CNN |  2023 |  Tawsifur (931), Skytells (1,071), CRs (818) |  Y
[21] |  CV |  WSI |  Tumor, stroma, and immune infiltrates |  Crowdsourcing |  – |  Label refinement |  Clean |  N |  Gaussian process |  2021 |  151 WSI |  N
[22] |  CV |  MRI |  Brain lesion |  Inherent |  Uncertainty |  Label refinement |  Clean |  N |  CNN |  2020 |  165 |  N
[23] |  CV |  MRI |  Brain metastases |  Pseudo label |  Known |  Label refinement |  Clean |  Y |  Cropnet |  2022 |  217 |  N
[24] |  CV |  Xray |  Lung disease |  Inherent |  – |  Label refinement |  – |  N |  AlexNet, ResNet |  2021 |  1,248 |  Y
[25] |  CV |  Dermoscopy, X-ray |  Skin lesion, lung disease |  Inherent, disagreement, machine label |  – |  Robustness |  – |  N |  ResNet |  2022 |  Skin (25,331), NIH Chest X-ray (112,120) |  Y
[26] |  CV |  WSI |  Histology |  Pseudo label |  Known |  Model robustness |  Clean |  Y |  ResNet |  2022 |  387 |  N
[27] |  CV |  X-ray |  Lung disease |  Machine label |  Known |  Label refinement |  Clean |  N |  MVSE |  2021 |  NIH-900 (11,014), Open-i (508,531), PMC (103), CheXpert (4,236) |  Y
[28] |  CV |  WSI |  Pathology |  Pseudo label |  Loss |  Label refinement , curriculum |  Clean |  Y |  Efficient Net |  2022 |  Kaggle PANDA (11,000) |  Y
[29] |  CV |  Endomicroscopy |  Gastrointestinal disease |  Pseudo label |  Uncertainty |  Active learning, label refinement |  Inject |  Y |  ResNet |  2019 |  CLE (1,366) |  Y
[30] |  CV |  WSI |  Pathology |  Pseudo label |  Loss |  Label refinement, reweight |  Inject |  N |  Large NN |  2021 |  MICCAI DigestPath2019 (455) |  Y
[31] |  CV |  X-ray |  Lung disease |  Machine label |  Model performance |  Reweight |  Clean |  Y |  DenseNet |  2021 |  29,7541 |  Y
[32] |  CV |  Retina image |  Retinopathy of Prematurity |  Disagreement |  Meta-learning |  Label refinement |  Clean |  Y |  ResNet |  2021 |  1,947 |  N
[33] |  CV |  X-ray |  Lung disease |  Inherent |  Uncertainty |  Reweight, label filter |  Clean |  N |  DenseNet |  2019 |  NIH Chest X-ray (112,120), PLCO (185,421) |  Y
[34] |  CV |  X-ray |  Lung disease |  Explicit |  – |  Label smoothing |  Clean |  Y |  DenseNet |  2021 |  CheXpert (224,316) |  Y
[35] |  CV |  X-ray |  Lung disease |  Explicit |  – |  Label filter, label refinement, model as separate class |  Clean |  N |  DenseNet, ResNet |  2019 |  CheXpert (224,316) |  Y
[36] |  CV |  Dermoscopy,WSI |  Skin lesion, histopathology |  Disagreement |  – |  Label filter |  Inject |  N |  ResNet |  2023 |  Skin (10,000), histopathology (3,000) |  Y
[37] |  CV |  CT |  Otosclerosis |  Inherent, disagreement |  – |  Robust loss |  Clean |  N |  W-Net |  2022 |  902 |  N
[38] |  CV |  WSI |  Prostate cancer |  Disagreement |  Loss |  Filter label |  Clean |  N |  ResNet |  2023 |  PANDA (110,000) |  Y
[39] |  CV |  X-ray |  Lung disease |  Expertise |  Explicit |  Robust architecture |  Clean |  N |  VGG |  2023 |  271 |  N
[40] |  CV |  X-ray |  Lung disease |  Noisy dataset |  Loss, uncertainty |  - |  Inject, clean |  N |  ResNet |  2019 |  NIH Chest X-ray (112,120) |  Y
[41] |  CV |  MRI |  Lumbar spine stenosis |  Disagreement |  Influence score |  Reweight |  Clean |  N |  MLP |  2022 |  26,728 |  N
[42] |  CV |  X-ray |  Proximal femur fractures |  Expertise, disagreement |  Known, uncertainty |  Reweight, curriculum |  Same |  N |  CNN |  2022 |  780 |  N
[43] |  CV |  Dermoscopy, WSI |  Skin cancer, lymph node |  Crowdsourcing |  Loss |  Label refinement, curriculum, gradient based, label update |  Inject |  N |  ResNet |  2021 |  ISIC-Archive (23,906), PatchCamelyon (327,680) |  Y
[44] |  CV |  Fundus image |  Fundus disease |  Inherent |  Loss |  Label filter, robust loss |  Inject |  N |  EfficientNet |  2023 |  private data (8),999, ODIR (9,144) |  Y/N
[45] |  CV |  OCT |  Vulnerable plaque |  Pseudo label |  Known |  Gradient based, label update |  Clean |  N |  ResNet |  2021 |  IVOCT-2017 (3,000) |  Y
[46] |  CV |  WSI |  Thyroid nodule |  Disagreement |  Model performance |  Reweight |  Clean |  N |  Small CNN |  2023 |  5232 |  N
[47] |  CV |  OCT |  Retina disease |  Inherent |  Loss |  Label refinement |  Inject |  N |  VGG, inception, ResNet |  2022 |  OCT (4,000), Messidor (1,187), ANIMAL-10N (55,000) |  Y
[48] |  CV |  Dermoscopy |  Skin lesion |  Pseudo label |  Explicit |  Label smoothing, regularization |  Clean |  Y |  DenseNet |  2023 |  ISIC-2018 (10,015), ISIC-2019 (25,531) |  Y
[49] |  CV |  Dermoscopy |  Skin cancer |  Disagreement |  Loss |  Label refinement |  Clean |  N |  ResNet |  2023 |  |
[50] |  CV |  Ultra sound |  Prostate cancer |  Inherent |  Uncertainty |  Label refinement |  Same |  N |  DNN |  2022 |  353 |  N
[51] |  CV |  Dermoscopy |  Skin cancer |  Pseudo label |  Loss |  Robust loss, mixup |  Inject |  N |  ResNet |  2023 |  24,691 |  Y
[52] |  CV |  X-ray |  Lung disease |  Machine label |  Cluster based |  Label refinement |  Inject |  N |  DenseNet |  2023 |  NIH Chest X-ray14 (112,120), CheXpert (224,316), PadChest (160,861), NIH-Google (1,962) |  Y
[53] |  NLP |  EHR |  Preterm birth |  De-identification |  Known |  Loss correction |  Clean |  Y |  LSTM |  2018 |  23,578 |  N
[54] |  NLP |  EHR |  Covid |  Data entry error, system error, diagnose error |  Known |  Robust loss, label smoothing |  Inject |  N |  MLP |  2023 |  37,996 |  Y
[55] |  NLP |  Medical data, clinical notes |  Lupus erythematosus |  Machine label |  Explicit |  Improve robustness |  Clean |  N |  LR |  2019 |  Chest X-ray14 (112,120), CheXpert (224,316), OpenI (7,470), PadChest (160,861) |  Y
[56] |  NLP |  EHR |  PICO entity extraction |  Machine label |  Explicit |  Robust loss |  Clean |  N |  PubMedBERT |  2023 |  23,172 |  N
[57] |  NLP |  EHR |  Mix of disease |  Disagreement |  Others |  Adversarial training |  Same |  N |  BiLSTM |  2021 |  2,500 |  Y
[58] |  Serial |  ECG |  Cardiac abnormalities |  Experience level |  Cluster based |  Label correction |  Clean |  N |  1-d CNN |  2022 |  CURIAL (37,896) |  Need request
[59] |  Serial |  ECG |  Cardiac abnormalities |  Disagreement |  Co-training |  Co-training |  Clean |  N |  Small CNN |  2023 |  UCSF EHR (499,855) |  N
[60] |  Serial |  PCG |  Cardiac diseases |  Inherent |  – |  Label refinement |  Inject |  N |  CNN |  2020 |  PICO (4,081) |  Y
[61] |  Serial |  ECG |  Cardiac diseases |  Machine label |  Cluster based |  Label refinement |  – |  N |  1-d CNN |  2021 |  CCKS 2020 (2,050) |  Y
[62] |  Serial |  PPG |  Atrial Fibrillation |  Pseudo label |  Cluster based |  Robust loss |  Clean |  Y |  ResNet |  2022 |  CINC 2021 (10,1149) |  Y
[63] |  Survival |  EHR |  Lung cancer, covid |  Pseudo label |  Explicit |  Label refinement |  Clean |  Y |  ML |  2023 |  PTB-XL (21,837), MRI Sunnybrook challenge (153) |  Y
[64] |  Sequential model |  Genome |  Chromatin region detection |  Pseudo label |  Loss function, uncertainty |  Robust loss |  Clean |  N |  CNN-LSTM |  2023 |  1,000 |  Y
[65] |  ML |  Regular data |  Acute respiratory failure, shock in ICU |  Inherent |  NN to estimate, transition prob |  Reweight |  Inject |  Y |  MLP |  2023 |  CINC 2021 (101,149) |  Y
[66] |  ML |  Regular data |  Diabete |  Inherent |  – |  Label refinement |  Clean |  Y |  MLP |  2022 |  24,100 |  N
[67] |  ML |  Regular data |  Mix of disease |  Inherent |  Cluster based |  Label filtering |  Inject |  N |  ML |  2020 |  Mix of datasets, each $\leq$ 1,000 |  Y

####  3.1.2 Overview of problem space

Most of the prediction problems in the included studies are disease diagnosis
problems. Due to the wide availability of data, Chest X-ray and skin cancer
are among the top medical domains that have been extensively explored. Some
medical specialties are hard to diagnose by nature, which in turn causes the
prevalence of label noise. For instance, Xue et al. demonstrate images
indicating that the malignant histology lymph node images present quite
similar colors and structures to benign ones [12]. The same situation can be
observed for skin lesion images as well.

Table LABEL:tab:summary_all_papers lists “disease” and “noise source” for
detailed information. Since most of the problems are diagnosis problems,
“labels” are usually the disease of interest. The “Noise source” column
explains the reason that causes label noise. “Modality”, “ML scope,” and
“Backbone NN” columns summarize the problem space from a deep learning
perspective. The selection of the backbone model usually depends on the domain
and data size. Since most problems are image-based, ResNet and DenseNet are
widely adopted. If the dataset size is small, customized small CNN can be an
alternative. The rest of the columns will be elaborated in the following
sections.

###  3.2 Source, impact, and detection of label noise

####  3.2.1 Sources of label noise

The primary source of label noise originates from the inherent complexity of
the medical problem being examined. Multiple factors such as annotators’
fatigue or distraction [68, 69] or varying levels of expertise among
annotators may contribute to label noise [69, 19, 25, 37, 39, 42].
Furthermore, there are inherent challenges in specific medical domains, such
as the similarity between skin cancer and benign conditions [36, 11, 12, 25,
10, 70]. Such complexity of medical diagnosis is usually reflected by intra-
observer and inter-observer variability, seen across multiple areas from
Alzheimer’s Disease (AD) to retinopathy [15, 11, 32, 58, 18, 25, 37, 59, 36,
38, 46, 49]. Some studies have attempted to explicitly quantify the proportion
of intra- and inter-observer variability, finding a range of 5% to 40%,
depending on the specific specialties and datasets [71, 1, 36, 37, 41, 15].

Another significant contributor to label noise is the generation of labels by
non-experts or non-humans. These labels may be extracted from an automated
system [55, 56], developed as pseudo labels in semi-supervised learning [64,
17], stemmed from crowdsourcing [21, 43], or be self-reported or non-verified
[70, 72, 73]. It is worth noting that although crowdsourcing is proven to be
highly influential within the traditional computer vision (CV) community, its
efficacy needs to improve in medical research due to the specialized expertise
required [21].

Besides the above-mentioned two types of contributors, label noise can also be
attributed to other factors such as the de-identification process [53], data
entry errors [10, 74], incomplete information [10], as well as data
acquisition, transmission, and storage errors [60, 61, 20]. Notably, some
popular datasets, like ChestX-Ray14, are widely known to be noisy [33, 35].

####  3.2.2 Impact of label noise

The impact of label noise is reflected by its detriment to classification
performance, which is usually measured by accuracy, AUC, PRAUC, and F1. For
multi-class prediction problems, the macro average of metrics is usually used.
Many studies aim to evaluate the impact of label noise on the performance of
deep learning models. While some studies argued that deep learning is
resilient to corrupted labels [75], they failed to assess the scenario where
noisy labels were properly handled, which could potentially yield even better
outcomes. In contrast, most studies have demonstrated a substantial
detrimental effect when label noise is not addressed appropriately [49, 70,
73, 76, 77, 78, 79]. For example, research by Samala et al. highlights that as
little as 10% of injected label noise can greatly increase DNN’s
generalization error [77]. Jang et al. found that even a 2% random flip of
labels significantly affects chest X-ray prediction accuracy [79].
Collectively, these findings suggest that the impact of label noise is indeed
remarkable, particularly in cases with relatively small sample sizes.

Apart from its effect on prediction accuracy, label noise could also influence
fairness, especially when the noise varies across different patient subgroups
[80, 65]. For instance, Tjandra et al. pointed out that females are more
frequently underdiagnosed for cardiovascular diseases [65].

####  3.2.3 Detection of noisy labels

Many strategies adopt a two-step or iterative approach: label noise detection
and handling. In this section, we summarize the detection methods (Figure 2).
Not all label noise requires detection; some are evident by nature. In semi-
supervised learning, pseudo-labels are known to be noisy. In supervised
learning, expert disagreements on disease diagnosis also indicate label noise,
which can be quantified by Cohen’s Kappa coefficient [39, 11, 18].

for tree= calign=child edge, calign child=1, grow’=east, growth parent
anchor=west, parent anchor=east, child anchor=west, edge=line width=1pt, edge
path=[edge,¡-, ¿=latex] (.child anchor) – +(-10pt,0pt) —- (!u.parent anchor)
edge label;, s sep=2pt, , before typesetting nodes= for tree= content/.wrap
value=#1, [Detection of label noise, root, [Model based detection, first,
[High loss function, second, [[46, 10, 19, 12, 64, 30, 8, 49, 38, 43, 40, 44,
51], leaf] ] [High model uncertainty, second, [[11, 42, 64, 29, 33, 40, 22,
50], leaf] ] [Clustering based method, second, [[58, 20, 61, 52, 67], leaf] ]
[Voting among classifiers, second, [[59, 43], leaf] ] [Other criteria, second,
[[15, 65, 32, 41, 31, 17, 28, 57], leaf] ] ] [External information, first,
[Machine extracted label, second, [[55, 56, 27, 35, 34, 54, 24], leaf] ]
[Pseudo labels, second, [[16, 63, 23, 26], leaf] ] [Expert agreement, second,
[[39, 11, 42, 18], leaf] ] ] [No explicit detection, first, [[13, 25, 37, 14,
62, 48, 21, 74, 60, 66, 9], leaf] ] ]

Figure 2: Summary of noisy label detection methods. Terminologies in the
figure are explained in the glossary.

When the degree of label noisiness is unknown, machine learning based
detection methods can be applied. The most widely used criteria are the first-
order information (e.g., prediction magnitude or loss function) and the
second-order information (model or epistemic uncertainty). The study by Calli
et al. shows that they are both valid in assessing label noisiness [40]. The
difficulty of using them is the selection of threshold when the noise rate is
still being determined [43, 64, 38, 49, 81]. Despite the popularity of these
two criteria, other studies indicate that a high loss function could also
reflect a correct but hard-to-learn instance. Therefore, they try to
disentangle the effects of noisy labels, hard instances, and class imbalance
[28, 36].

Besides the loss function and model uncertainty, other machine-learning
approaches are used to detect noisy labels. Ying et al. have used the
clustering-based method to identify prototypes or clusters within the feature
space and assume labels in the same cluster are similar [20]. The same
strategy can also be used in label refinement (Section 3.3.1). Braun et al.
have developed another method to detect noise by evaluating the self-influence
of a sample. This is defined by the change in the test sample loss function
due to the inclusion or exclusion of the candidate instance to the training
set [41]. The underlying logic is that a noisy sample can decrease its own
training error, but not the error of a test sample. Algan et al. have employed
a meta-learning criterion to search for optimal soft-labels that contribute to
the most noise-robust training [32].

###  3.3 Deep learning methodologies

Techniques for addressing noisy labels in medical research can be organized
into four main categories (Figure 3). Some studies only utilize one technique,
while others leverage multiple techniques in the same study. Further
categorization of these methods can be done based on whether they require a
clean dataset for training or if they are applied iteratively.

for tree= calign=child edge, calign child=1, grow’=east, growth parent
anchor=west, parent anchor=east, child anchor=west, edge=line width=1pt, edge
path=[edge,¡-, ¿=latex] (.child anchor) – +(-10pt,0pt) —- (!u.parent anchor)
edge label;, s sep=2pt, , before typesetting nodes= for tree= content/.wrap
value=#1, [Methods to handle label noise, root, [Label refinement and
filtering, first, [Co-teaching, second, [[43, 59, 49, 64, 50], leaf] ]
[Teacher-student, second, [[16], leaf] ] [Label correction module, second,
[[30], leaf] ] [Active learning, second, [[15, 69, 29], leaf] ] [Cluster
based, second [[58, 20, 61, 66, 52], leaf] ] [Gradient based, second, [[45,
43], leaf] ] [Other label refinement, second, [[32, 27, 21], leaf] ] [Removing
noisy labels, second, [[36, 11, 19, 38, 28, 33, 67], leaf] ] ] [Loss and
architecture change, first, [Architecture change, second, [[30, 9], leaf] ]
[Loss correction, second, [[53, 65], leaf] ] ] [Loss and model robustness
improvement, first, [Noise robust loss function, second, [[37, 14, 8, 62, 64,
54, 44, 51], leaf] ] [Mixup / mixmatch, second, [[54, 34, 26, 13, 36, 51],
leaf] ] [Label smoothing, second, [[48, 34, 54], leaf] ] [Contrastive
learning, second, [[13, 12], leaf] ] ] [Training paradigm change, first,
[Weighting, second, [[11, 10, 8, 41, 55, 56, 17, 30, 33], leaf] ] [Curriculum
learning, second, [[11, 10, 28, 43], leaf] ] ] [Others, first, [[35, 74, 24,
18, 32, 25, 39, 15, 57], leaf] ] ]

Figure 3: Methods applied to handle label noise. Terminologies in the figure
are explained in the glossary.

####  3.3.1 Label filtering and refinement

The label filtering process starts with applying noise detection methods
(Section 3.2.3), followed by a straightforward approach to discarding noisy
instances. One disadvantage of discarding noisy instances is the potential
removal of genuinely complex but valuable data. Hence, the weighting method
(Section 3.3.4) or label refinement are better alternatives [41].

Label refinement is another approach that takes a more conservative standpoint
by revising noisy labels rather than discarding them entirely. One way is
through the use of a dedicated classifier [66]. Another approach involves
utilizing the stochastic gradient descent to update labels [43, 45], where the
gradient is calculated with respect to labels. After the label refinement
step, original labels may still be used in model training [58].

####  3.3.2 Neural network architecture change

Architecture change explicitly models the transition between latent ground
true labels and observed noisy labels through dedicated layers. However, it is
less common in medical research due to the constraint of a limited sample
size. The most common changes to architecture involve employing loss
correction or adaptation layers.

Loss correction explicitly estimates the transition matrix between the
observed label and the latent ground truth label, denoted as $T$
($T_{i,j}=P(\tilde{y}=e^{j}|y=e^{i})$) [82], where $\tilde{y}$ and $y$ are the
noisy and true labels. The estimated transition matrix is then used to weight
the loss function $l_{corrected}=T^{-1}l\left(\hat{p}(y|x)\right)$. It is
proved that the corrected loss function is unbiased.

Differently, the noisy adaptation layer integrates this noise transition
process directly into the neural network by adding an extra softmax layer
after the original one [83]. However, this additional layer makes the neural
network non-identifiable, which poses challenges to optimization, and a
careful initialization is needed.

####  3.3.3 Enhancing model robustness

Instead of directly addressing noisy labels, an alternative workstream focuses
on enhancing model robustness and reducing overfitting.

Designing new loss functions is the most popular approach to improve
robustness. The issue with the cross-entropy loss in regular classification
applications is that its gradient can sharply increase as the predicted
probability approaches the opposite of the corresponding label. While this
accelerates convergence, it leads to overfitting when the label is wrong. Loss
functions that suppress gradient can mitigate this issue. Interestingly, mean
square error, the default loss function in regression, has demonstrated
resilience to label noise in classification problems [32, 64]. Another
approach to construct robust loss is to apply similar motivations as
contrastive learning, which enforces instances close in latent feature space
to share similar labels [8, 62, 54].

In addition to robust loss functions, regularization methods such as label
smoothing and mix-up can also be employed. Regularization can also be achieved
by customizing the optimization strategy. For instance, Hu et al. use
sharpness-aware minimization to find a “flat minimum” whose neighbors have
uniformly low training loss values [44].

####  3.3.4 Training paradigm change

Reweighting and curriculum learning are the most widely used training paradigm
changes to tackle label noise. The reweighting strategy adjusts the instance
weights by prioritizing clean labels over noisy ones. The concept of
curriculum learning, originating from education, starts with training neural
networks on easy samples (easy curriculum) and gradually progresses to more
complex ones (hard curriculum). It aims to smoothen the warm-up process during
optimization and achieve better local minima. In the context of label
denoising, training can start with clean labels identified as described in
Section 3.2.3.

There are other paradigm changes in addition to curriculum learning and
reweighting. Irvin et al. treated noisy labels as a distinct class rather than
assigning a pseudo label [35]. Algan et al. introduced a meta-learning
strategy to generate soft labels [32]. The goal is to produce optimal soft
labels for noisy instances so that a base classifier trained on such instances
would perform best on a separate clean meta-data.

###  3.4 Evaluation

Evaluation is crucial yet challenging when studying noisy label handling,
primarily due to the necessity of a clean dataset. Two common evaluation
methods were identified from the reviewed studies (Figure 4): (1) using data
with ground truth for evaluation, and (2) injecting noise into training data
(with the original data being used for evaluation).

In the first approach, ground truth data could come from expert annotation, or
the labeled part of data in a semi-supervised learning setting [12]. In the
second approach where noise is simulated, three different underlying
assumptions exist [2]: (1) random noise where the noise probability is
independent of both features and latent true labels, (2) class-dependent noise
where the noise probability depends only on the class labels, and (3)
instance-based noise [3], which assumes the noise mechanism depends on both
labels and features.

We suggest the use of ground truth data as an evaluation criterion whenever
possible. While instance-based label injection is arguably the most compelling
simulation for label noise, we find it is rarely applied in practice.
Moreover, a positive ratio should be maintained the same after noise injection
to mitigate the confounding impact of class imbalance [54], and noise may be
simulated across various severity [47].

for tree= calign=child edge, calign child=1, grow’=east, growth parent
anchor=west, parent anchor=east, child anchor=west, edge=line width=1pt, edge
path=[edge,¡-, ¿=latex] (.child anchor) – +(-10pt,0pt) —- (!u.parent anchor)
edge label;, s sep=2pt, , before typesetting nodes= for tree= content/.wrap
value=#1, [Evaluation method, root, [Train on noisy and eval on clean, first,
[Semi-supervised, second, [[41, 16, 63, 17, 23, 26, 48], leaf] ] [Expert
annotated, second, [[28, 31, 62, 53, 11, 58, 59, 32, 33, 34, 35, 55, 56, 64,
18, 21, 22, 27, 37, 38, 39, 45, 46, 66], leaf] ] ] [Noise injection, first,
[Random, second, [[29, 9, 60, 8, 30, 36, 40, 43, 44, 47, 54, 12, 13, 74, 19,
51, 52, 67], leaf] ] [Class dependent, second, [[73, 20], leaf] ] [Instance
based, second, [[10, 65], leaf] ] ] [Train and eval on the same data, first,
[[14, 15, 24, 25, 61, 42, 49, 50, 57], leaf] ] ]

Figure 4: Evaluation method summary.

##  4 Discussion

Medical research is intrinsically prone to noisy labels stemming from its
inherent ambiguity, variations in label sources relating to data quality,
dependence on labels generated by automated systems, and more. A considerable
number of the papers we have reviewed report substantial deterioration in
model performance due to label noise. In this review, we focus on a
comprehensive examination of different aspects related to label noise
management.

How label noise is handled depends on factors such as the availability of
clean data, sample size, and noise level. Approaches like label refinement and
training paradigm optimization, which heavily rely on noise detection,
typically require a clean dataset. In contrast, architecture change and
robustness improvement usually do not have this requirement. Applying noise-
handling techniques iteratively often leads to improved results, with the
noise level tending to decrease with each iteration. Due to the uncertainty in
noise detection, priority should be given to reweighting or label refinement
over filtering out noisy labels [41]. To evaluate how well the noisy label
issue is addressed, simulations should be conducted using instance-based
noise. In real-world applications where addressing noisy labels is not the
primary objective, the quality of a clean evaluation dataset becomes crucial.

Several directions may deserve attention for future research. The first is to
further distinguish between label noise and genuinely hard examples. A duality
can be applied to instances with a high loss function or model uncertainty.
They can be perceived either as ”noisy instances” in our context or as
valuable ”hard examples” in hard example mining. In the papers we reviewed,
they are deprioritized, whereas in hard example mining literature, they should
be emphasized. While some studies have attempted to address this issue, their
suggested approaches are often difficult to implement and lack broad
applicability. Secondly, learning from noisy labels is closely linked to other
domains, such as semi-supervised learning, active learning, uncertainty-aware
classification, ensemble learning, and regularization. Cross-disciplinary
investigation could provide another promising direction for future research.
Thirdly, few studies compare different backbone models’ sensitivity and
robustness to label noise. This will be helpful in guiding the selection of
model architecture. Fourthly, even in the broader deep learning community,
there are limited studies comparing different method’s performance and how to
more effectively combine them. A systematic review or benchmark study will be
helpful. Lastly, label noise detection methods have not been thoroughly
discussed in existing literature. Detection methods could not only help to
handle noise but also be useful in assessing the quality of evaluation data
and estimating the noise ratio.

##  5 Conclusion

In this scoping review, we systematically examined the current state of
research on noisy label detection and management in the medical domain. Our
findings suggest that the techniques for handling noisy labels in the medical
community are currently up to date with the general deep-learning community.
Despite their utility, the adoption of label noise management in mainstream
medical research remains limited. These methodologies are predominantly
discussed in works specifically targeting the issue of noisy labels. We
recommend considering label noise as a standard element in medical research.
Many of these noisy label-handling methodologies (e.g., reweighting, noise
robust loss, curriculum learning) do not require extensive efforts and can
seamlessly integrate into existing research workflows.

## Glossary

Active learning |  A machine learning algorithm that interactively queries a human (or other information source) to label new data points. Usually, the data points to be queried are selected to maximize the information that can be obtained.
---|---
Architecture change |  Architecture change explicitly models the transition between latent ground true labels and observed noisy labels through dedicated layers.
Cluster-based |  Instances are clustered in the feature space. The label of each cluster is usually decided by some ground truth data or majority voting. If the label of an instance disagrees with the cluster label, it might be corrected.
Clustering based method |  Instances are clustered in the feature space. The label of each cluster is usually decided by some ground truth data or majority voting. Instances whose label is different from their cluster are considered noisy.
Co-teaching |  Two classifiers are trained with different parts of training data and feature sets. The labels are refined iteratively. In each iteration, each classifier’s most confident predictions will be used to refine the label of the training set of the other classifier.
Contrastive learning |  A deep learning approach to extract meaningful representations by contrasting positive and negative pairs of instances. It leverages the assumption that similar instances should be closer together in the learned embedding space, while dissimilar instances should be further apart. Contrastive loss is usually more robust to label noise than cross-entropy loss.
Expert agreement |  Consider a label to be noisy if experts disagree on the annotation.
Gradient based |  Calculate the gradient with respect to the label instead of parameters and use stochastic gradient descent to update the label.
High loss function |  Consider the label to be noisy if the loss function is high.
High model uncertainty |  Consider the label to be noisy if estimated prediction uncertainty is high.
Label correction module |  Add a dedicated component in the neural network to refine labels.
Label smoothing |  A regularization technique that introduces noise to the labels. For a k-class classification problem, labels 0 and 1 are replaced by $\frac{\epsilon}{k-1}$ and 1-$\epsilon$, respectively. Here $\epsilon$ decides the noise level to be introduced.
Loss correction |  Loss correction explicitly estimates the transition matrix between the observed label and the latent ground truth label, denoted as $T$ ($T_{i,j}=P(\tilde{y}=e^{j}|y=e^{i})$), where $\tilde{y}$ and $y$ are the noisy and true labels. The estimated transition matrix is then used to weight the loss function $l_{corrected}=T^{-1}l\left(\hat{p}(y|x)\right)$.
Machine extracted labels |  Labels are derived from automated systems such as the NLP model instead of being annotated by experts. It is considered noisy since the automated system is error-prone.
Mixup/mixmatch |  Mixup is a data augmentation strategy. A new instance is created by taking a weighted average of two existing instances. Here the weighted average is applied to both the features and labels. Mixmatch is a data augmentation strategy built on top of mixup.
Pseudo labels |  In semi-supervised learning, usually the unlabeled part will be assigned a label by a classifier trained on the labeled part of data. This label is called pseudo label and considered noisy.
Removing noisy labels |  Remove the instances whose labels are considered noisy.
Teacher-student |  Use a larger neural network’s prediction to refine the label for the training set of a smaller neural network.
Voting among classifiers |  Consider the label to be noisy if different estimators’ predictions disagree

## Funding Statement

This work was supported by NSF CAREER Award No. 2145640 and the National
Library of Medicine under Award No. R01LM014306.

## Competing Interests Statement

There are no competing interests to declare.

## Data Availability statement

The data underlying this article are available in the article and in its
online supplementary material.

## Contributorship Statement

Study concepts/study design, Y.W., H.J., Y.P.; manuscript drafting or
manuscript revision for important intellectual content, all authors; approval
of the final version of the submitted manuscript, all authors; agrees to
ensure any questions related to the work are appropriately resolved, all
authors; literature research, Y.W., Y.D., C.S., M.L.; data interpretation,
Y.W., Y.D.; and manuscript editing, all authors.

## References

  * [1] Wallace DK, Quinn GE, Freedman SF, Chiang MF.  Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity.  Journal of American Association for Pediatric Ophthalmology and Strabismus. 2008;12(4):352-6.
  * [2] Frénay B, Kabán A, et al.  A comprehensive introduction to label noise.  In: ESANN. Citeseer; 2014. .
  * [3] Song H, Kim M, Park D, Shin Y, Lee JG.  Learning From Noisy Labels With Deep Neural Networks: A Survey.  IEEE Trans Neural Netw Learn Syst. 2023 Nov;34(11):8135-53.
  * [4] Algan G, Ulusoy I.  Image classification with deep learning in the presence of noisy labels: A survey.  Knowledge-Based Systems. 2021;215:106771.
  * [5] Liang X, Liu X, Yao L.  Review–a survey of learning from noisy labels.  ECS Sensors Plus. 2022;1(2):021401.
  * [6] Karimi D, Dou H, Warfield SK, Gholipour A.  Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis.  Medical image analysis. 2020;65:101759.
  * [7] Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al.  PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation.  Annals of internal medicine. 2018;169(7):467-73.
  * [8] Deng Y, Bao F, Deng X, Wang R, Kong Y, Dai Q.  Deep and structured robust information theoretic learning for image analysis.  IEEE Transactions on Image Processing. 2016;25(9):4209-21.
  * [9] Dgani Y, Greenspan H, Goldberger J.  Training a neural network based on unreliable human annotation of medical images.  In: 2018 IEEE 15th International symposium on biomedical imaging (ISBI 2018). IEEE; 2018. p. 39-42.
  * [10] Xue C, Dou Q, Shi X, Chen H, Heng PA.  Robust learning at noisy labeled medical images: Applied to skin lesion classification.  In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). IEEE; 2019. p. 1280-3.
  * [11] Ju L, Wang X, Wang L, Mahapatra D, Zhao X, Zhou Q, et al.  Improving medical images classification with label noise using dual-uncertainty estimation.  IEEE transactions on medical imaging. 2022;41(6):1533-46.
  * [12] Xue C, Yu L, Chen P, Dou Q, Heng PA.  Robust medical image classification from noisy labeled data with global and local representation guided co-training.  IEEE Transactions on Medical Imaging. 2022;41(6):1371-82.
  * [13] Jiang H, Gao M, Hu Y, Ren Q, Xie Z, Liu J.  Label-noise-tolerant medical image classification via self-attention and self-supervised learning.  arXiv preprint arXiv:230609718. 2023.
  * [14] Hu K, Huang Y, Huang W, Tan H, Chen Z, Zhong Z, et al.  Deep supervised learning using self-adaptive auxiliary loss for COVID-19 diagnosis from imbalanced CT images.  Neurocomputing. 2021;458:232-45.
  * [15] Lee J, Prabhu D, Kolluru C, Gharaibeh Y, Zimin VN, Dallan LA, et al.  Fully automated plaque characterization in intravascular OCT images using hybrid convolutional and lumen morphology features.  Scientific reports. 2020;10(1):2596.
  * [16] Kurita Y, Meguro S, Tsuyama N, Kosugi I, Enomoto Y, Kawasaki H, et al.  Accurate deep learning model using semi-supervised learning and Noisy Student for cervical cancer screening in low magnification images.  Plos one. 2023;18(5):e0285996.
  * [17] Fries JA, Varma P, Chen VS, Xiao K, Tejeda H, Saha P, et al.  Weakly supervised classification of aortic valve malformations using unlabeled cardiac MRI sequences.  Nature communications. 2019;10(1):3111.
  * [18] Wong DR, Tang Z, Mew NC, Das S, Athey J, McAleese KE, et al.  Deep learning from multiple experts improves identification of amyloid neuropathologies.  Acta neuropathologica communications. 2022;10(1):66.
  * [19] Ashraf M, Robles WRQ, Kim M, Ko YS, Yi MY.  A loss-based patch label denoising method for improving whole-slide image analysis using a convolutional neural network.  Scientific reports. 2022;12(1):1392.
  * [20] Ying X, Liu H, Huang R.  COVID-19 chest X-ray image classification in the presence of noisy labels.  Displays. 2023;77:102370.
  * [21] López-Pérez M, Amgad M, Morales-Álvarez P, Ruiz P, Cooper LA, Molina R, et al.  Learning from crowds in digital pathology using scalable variational Gaussian processes.  Scientific reports. 2021;11(1):11612.
  * [22] Karimi D, Peters JM, Ouaalam A, Prabhu SP, Sahin M, Krueger DA, et al.  Learning to detect brain lesions from noisy annotations.  In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE; 2020. p. 1910-4.
  * [23] Dikici E, Nguyen XV, Bigelow M, Ryu JL, Prevedello LM.  Advancing Brain Metastases Detection in T1-Weighted Contrast-Enhanced 3D MRI Using Noisy Student-Based Training.  Diagnostics. 2022;12(8):2023.
  * [24] Momeny M, Neshat AA, Hussain MA, Kia S, Marhamati M, Jahanbakhshi A, et al.  Learning-to-augment strategy using noisy and denoised data: Improving generalizability of deep CNN for the detection of COVID-19 in X-ray images.  Computers in Biology and Medicine. 2021;136:104704.
  * [25] Jaiswal A, Ashutosh K, Rousseau JF, Peng Y, Wang Z, Ding Y.  Ros-kd: A robust stochastic knowledge distillation approach for noisy medical imaging.  In: 2022 IEEE International Conference on Data Mining (ICDM). IEEE; 2022. p. 981-6.
  * [26] Pulido JV, Guleria S, Ehsan L, Fasullo M, Lippman R, Mutha P, et al.  Semi-supervised classification of noisy, gigapixel histology images.  In: 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering (BIBE). IEEE; 2020. p. 563-8.
  * [27] Paul A, Shen TC, Lee S, Balachandar N, Peng Y, Lu Z, et al.  Generalized zero-shot chest x-ray diagnosis through trait-guided multi-view semantic embedding with self-training.  IEEE Transactions on Medical Imaging. 2021;40(10):2642-55.
  * [28] Li W, Li J, Wang Z, Polson J, Sisk AE, Sajed DP, et al.  Pathal: An active learning framework for histopathology image analysis.  IEEE Transactions on Medical Imaging. 2021;41(5):1176-87.
  * [29] Gu Y, Shen M, Yang J, Yang GZ.  Reliable label-efficient learning for biomedical image recognition.  IEEE Transactions on Biomedical Engineering. 2018;66(9):2423-32.
  * [30] Zhang S, Yuan Z, Wang Y, Bai Y, Chen B, Wang H.  REUR: a unified deep framework for signet ring cell detection in low-resolution pathological images.  Computers in Biology and Medicine. 2021;136:104711.
  * [31] Gündel S, Setio AA, Ghesu FC, Grbic S, Georgescu B, Maier A, et al.  Robust classification from noisy labels: Integrating additional knowledge for chest radiography abnormality assessment.  Medical Image Analysis. 2021;72:102087.
  * [32] Algan G, Ulusoy I, Gönül Ş, Turgut B, Bakbak B.  Deep learning from small amount of medical data with noisy labels: A meta-learning approach.  arXiv preprint arXiv:201006939. 2020.
  * [33] Ghesu FC, Georgescu B, Gibson E, Guendel S, Kalra MK, Singh R, et al.  Quantifying and leveraging classification uncertainty for chest radiograph assessment.  In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part VI 22. Springer; 2019. p. 676-84.
  * [34] Pham HH, Le TT, Tran DQ, Ngo DT, Nguyen HQ.  Interpreting chest X-rays via CNNs that exploit hierarchical disease dependencies and uncertainty labels.  Neurocomputing. 2021;437:186-94.
  * [35] Irvin J, Rajpurkar P, Ko M, Yu Y, Ciurea-Ilcus S, Chute C, et al.  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In: Proceedings of the AAAI conference on artificial intelligence. vol. 33; 2019. p. 590-7.
  * [36] Li J, Cao H, Wang J, Liu F, Dou Q, Chen G, et al.  Learning Robust Classifier for Imbalanced Medical Image Dataset with Noisy Labels by Minimizing Invariant Risk.  In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer; 2023. p. 306-16.
  * [37] Chen H, Tan W, Li J, Guan P, Wu L, Yan B, et al.  Adaptive Cross Entropy for ultrasmall object detection in Computed Tomography with noisy labels.  Computers in Biology and Medicine. 2022;147:105763.
  * [38] Xiang J, Wang X, Wang X, Zhang J, Yang S, Yang W, et al.  Automatic diagnosis and grading of Prostate Cancer with weakly supervised learning on whole slide images.  Computers in Biology and Medicine. 2023;152:106340.
  * [39] Del Amor R, Silva-Rodríguez J, Naranjo V.  Labeling confidence for uncertainty-aware histology image classification.  Computerized Medical Imaging and Graphics. 2023;107:102231.
  * [40] Calli E, Sogancioglu E, Scholten ET, Murphy K, van Ginneken B.  Handling label noise through model confidence and uncertainty: application to chest radiograph classification.  In: Medical Imaging 2019: Computer-Aided Diagnosis. vol. 10950. SPIE; 2019. p. 289-96.
  * [41] Braun J, Kornreich M, Park J, Pawar J, Browning J, Herzog R, et al.  Influence Based Re-Weighing for Labeling Noise in Medical Imaging.  In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI). IEEE; 2022. p. 1-5.
  * [42] Jiménez-Sánchez A, Mateus D, Kirchhoff S, Kirchhoff C, Biberthaler P, Navab N, et al.  Curriculum learning for improved femur fracture classification: Scheduling data with prior knowledge and uncertainty.  Medical Image Analysis. 2022;75:102273.
  * [43] Liu J, Li R, Sun C.  Co-correcting: noise-tolerant medical image classification via mutual label correction.  IEEE Transactions on Medical Imaging. 2021;40(12):3580-92.
  * [44] Hu T, Yang B, Guo J, Zhang W, Liu H, Wang N, et al.  A fundus image classification framework for learning with noisy labels.  Computerized Medical Imaging and Graphics. 2023;108:102278.
  * [45] Shi P, Xin J, Zheng N.  Correcting Pseudo Labels with Label Distribution for Unsupervised Domain Adaptive Vulnerable Plaque Detection.  In: 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE; 2021. p. 3225-8.
  * [46] Gao Z, Chen Y, Sun P, Liu H, Lu Y.  Clinical knowledge embedded method based on multi-task learning for thyroid nodule classification with ultrasound images.  Physics in Medicine & Biology. 2023;68(4):045018.
  * [47] Gao M, Feng X, Geng M, Jiang Z, Zhu L, Meng X, et al.  Bayesian statistics-guided label refurbishment mechanism: Mitigating label noise in medical image classification.  Medical Physics. 2022;49(9):5899-913.
  * [48] Zhou S, Tian S, Yu L, Wu W, Zhang D, Peng Z, et al.  ReFixMatch-LS: reusing pseudo-labels for semi-supervised skin lesion classification.  Medical & Biological Engineering & Computing. 2023;61(5):1033-45.
  * [49] Zhu M, Zhang L, Wang L, Li D, Zhang J, Yi Z.  Robust co-teaching learning with consistency-based noisy label correction for medical image classification.  International Journal of Computer Assisted Radiology and Surgery. 2023;18(4):675-83.
  * [50] Javadi G, Samadi S, Bayat S, Sojoudi S, Hurtado A, Eshumani W, et al.  Training deep neural networks with noisy clinical labels: toward accurate detection of prostate cancer in US data.  International Journal of Computer Assisted Radiology and Surgery. 2022;17(9):1697-705.
  * [51] Chen B, Ye Z, Liu Y, Zhang Z, Pan J, Zeng B, et al.  Combating Medical Label Noise via Robust Semi-supervised Contrastive Learning.  In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer; 2023. p. 562-72.
  * [52] Chen Y, Liu F, Wang H, Wang C, Liu Y, Tian Y, et al.  Bomd: bag of multi-label descriptors for noisy chest x-ray classification.  In: Proceedings of the IEEE/CVF International Conference on Computer Vision; 2023. p. 21284-95.
  * [53] Boughorbel S, Jarray F, Venugopal N, Elhadi H.  Alternating loss correction for preterm-birth prediction from ehr data with noisy labels.  arXiv preprint arXiv:181109782. 2018.
  * [54] Yang J, Triendl H, Soltan AA, Prakash M, Clifton DA.  Addressing Label Noise for Electronic Health Records: Insights from Computer Vision for Tabular Data.  medRxiv. 2023:2023-10.
  * [55] Murray SG, Avati A, Schmajuk G, Yazdany J.  Automated and flexible identification of complex disease: building a model for systemic lupus erythematosus using noisy labeling.  Journal of the American Medical Informatics Association. 2019;26(1):61-5.
  * [56] Dhrangadhariya A, Müller H.  Not so weak PICO: leveraging weak supervision for participants, interventions, and outcomes recognition for systematic review automation.  JAMIA open. 2023;6(1):ooac107.
  * [57] Li Z, Gan Z, Zhang B, Chen Y, Wan J, Liu K, et al.  Semi-supervised noisy label learning for Chinese clinical named entity recognition.  Data Intelligence. 2021;3(3):389-401.
  * [58] Vázquez CG, Breuss A, Gnarra O, Portmann J, Madaffari A, Da Poian G.  Label noise and self-learning label correction in cardiac abnormalities classification.  Physiological measurement. 2022;43(9):094001.
  * [59] de Vos BD, Jansen GE, Išgum I.  Stochastic co-teaching for training neural networks with unknown levels of label noise.  Scientific reports. 2023;13(1):16875.
  * [60] Baghel N, Dutta MK, Burget R.  Automatic diagnosis of multiple cardiac diseases from PCG signals using convolutional neural network.  Computer Methods and Programs in Biomedicine. 2020;197:105750.
  * [61] Vázquez CG, Breuss A, Gnarra O, Portmann J, Da Poian G.  Two will do: CNN with asymmetric loss, self-learning label correction, and hand-crafted features for imbalanced multi-label ECG data classification.  In: 2021 Computing in Cardiology (CinC). vol. 48. IEEE; 2021. p. 1-4.
  * [62] Ding C, Guo Z, Shah A, Clifford G, Rudin C, Hu X.  Learning From Alarms: A Robust Learning Approach for Accurate Photoplethysmography-Based Atrial Fibrillation Detection using Eight Million Samples Labeled with Imprecise Arrhythmia Alarms.  arXiv preprint arXiv:221103333. 2022.
  * [63] Hong C, Liang L, Yuan Q, Cho K, Liao KP, Pencina MJ, et al.  Semi-Supervised Calibration of Noisy Event Risk (SCANER) with Electronic Health Records.  Journal of Biomedical Informatics. 2023:104425.
  * [64] Ren J, Liu Y, Zhu X, Wang X, Li Y, Liu Y, et al.  OCRFinder: a noise-tolerance machine learning method for accurately estimating open chromatin regions.  Frontiers in Genetics. 2023;14:1184744.
  * [65] Tjandra D, Wiens J.  Leveraging an Alignment Set in Tackling Instance-Dependent Label Noise.  In: Conference on Health, Inference, and Learning. PMLR; 2023. p. 477-97.
  * [66] Vernekar S, Ayyar A, Rajagopalan A, Kumar B, Mishra VK.  Improving Medical Predictions with Label Noise Tolerant Classification.  In: 2022 4th International Conference on Advances in Computing, Communication Control and Networking (ICAC3N). IEEE; 2022. p. 765-9.
  * [67] Xu J, Yang Y, Yang P.  Hybrid label noise correction algorithm for medical auxiliary diagnosis.  In: 2020 IEEE 18th International Conference on Industrial Informatics (INDIN). vol. 1. IEEE; 2020. p. 567-72.
  * [68] Brady AP.  Error and discrepancy in radiology: inevitable or avoidable?  Insights into imaging. 2017;8:171-82.
  * [69] Lu H, Lee J, Ray S, Tanaka K, Bezerra HG, Rollins AM, et al.  Automated stent coverage analysis in intravascular OCT (IVOCT) image volumes using a support vector machine and mesh growing.  Biomedical Optics Express. 2019;10(6):2809-28.
  * [70] Hekler A, Kather JN, Krieghoff-Henning E, Utikal JS, Meier F, Gellrich FF, et al.  Effects of label noise on deep learning-based skin cancer classification.  Frontiers in Medicine. 2020;7:177.
  * [71] Campbell JP, Kalpathy-Cramer J, Erdogmus D, Ostmo S, Swan R, Sonmez K, et al.  Plus disease in rop: why do experts disagree, and how can we improve diagnosis?  Journal of American Association for Pediatric Ophthalmology and Strabismus $\\{$JAAPOS$\\}$. 2017;21(4):e5-6.
  * [72] Cosentino J, Behsaz B, Alipanahi B, McCaw ZR, Hill D, Schwantes-An TH, et al.  Inference of chronic obstructive pulmonary disease with deep learning on raw spirograms identifies new genetic loci and improves risk models.  Nature Genetics. 2023:1-9.
  * [73] Ding C, Pereira T, Xiao R, Lee RJ, Hu X.  Impact of label noise on the learning based models for a binary classification of physiological signal.  Sensors. 2022;22(19):7166.
  * [74] Pechenizkiy M, Tsymbal A, Puuronen S, Pechenizkiy O.  Class noise and supervised learning in medical domains: The effect of feature extraction.  In: 19th IEEE symposium on computer-based medical systems (CBMS’06). IEEE; 2006. p. 708-13.
  * [75] Potapenko I, Kristensen M, Thiesson B, Ilginis T, Lykke Sørensen T, Nouri Hajari J, et al.  Detection of oedema on optical coherence tomography images using deep learning model trained on noisy clinical data.  Acta Ophthalmologica. 2022;100(1):103-10.
  * [76] Khanal B, Hasan SK, Khanal B, Linte CA.  Investigating the impact of class-dependent label noise in medical image classification.  In: Medical Imaging 2023: Image Processing. vol. 12464. SPIE; 2023. p. 728-33.
  * [77] Samala RK, Chan HP, Hadjiiski LM, Helvie MA, Richter CD.  Generalization error analysis for deep convolutional neural network with transfer learning in breast cancer diagnosis.  Physics in Medicine & Biology. 2020;65(10):105002.
  * [78] Büttner M, Schneider L, Krasowski A, Krois J, Feldberg B, Schwendicke F.  Impact of Noisy Labels on Dental Deep Learning—Calculus Detection on Bitewing Radiographs.  Journal of Clinical Medicine. 2023;12(9):3058.
  * [79] Jang R, Kim N, Jang M, Lee KH, Lee SM, Lee KH, et al.  Assessment of the robustness of convolutional neural networks in labeling noise by using chest X-ray images from multiple centers.  JMIR medical informatics. 2020;8(8):e18089.
  * [80] Petersen E, Holm S, Ganz M, Feragen A.  The path toward equal performance in medical machine learning.  Patterns. 2023;4(7).
  * [81] Liu T, Tao D.  Classification with noisy labels by importance reweighting.  IEEE Transactions on pattern analysis and machine intelligence. 2015;38(3):447-61.
  * [82] Patrini G, Rozza A, Krishna Menon A, Nock R, Qu L.  Making deep neural networks robust to label noise: A loss correction approach.  In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2017. p. 1944-52.
  * [83] Goldberger J, Ben-Reuven E.  Training deep neural-networks using a noise adaptation layer.  In: International conference on learning representations; 2016. .