# Augment like there’s no tomorrow: Consistently performing neural networks
for medical imaging

Joona Pohjonen  Research Program in Systems Oncology, Faculty of Medicine,
University of Helsinki  Authors contributed equally  Carolin Stürenberg
Research Program in Systems Oncology, Faculty of Medicine, University of
Helsinki  Authors contributed equally  Atte Föhr  Research Program in Systems
Oncology, Faculty of Medicine, University of Helsinki  Reija Randen-Brady
Department of Pathology, Helsinki University Hospital  Lassi Luomala  Research
Program in Systems Oncology, Faculty of Medicine, University of Helsinki
Department of Pathology, Helsinki University Hospital  Jouni Lohi  Department
of Pathology, Helsinki University Hospital  Esa Pitkänen  Institute for
Molecular Medicine Finland (FIMM), HiLIFE, University of Helsinki  Research
Program in Applied Tumor Genomics, Faculty of Medicine, University of Helsinki
iCAN Digital Precision Cancer Medicine Flagship, Finland  Senior author  Antti
Rannikko  Research Program in Systems Oncology, Faculty of Medicine,
University of Helsinki  iCAN Digital Precision Cancer Medicine Flagship,
Finland  Department of Urology, Helsinki University Hospital  Senior author
Tuomas Mirtti  Research Program in Systems Oncology, Faculty of Medicine,
University of Helsinki  Research Program in Applied Tumor Genomics, Faculty of
Medicine, University of Helsinki  iCAN Digital Precision Cancer Medicine
Flagship, Finland  Senior author

###### Abstract

Deep neural networks have achieved impressive performance in a wide variety of
medical imaging tasks. However, these models often fail on data not used
during training, such as data originating from a different medical centre. How
to recognise models suffering from this fragility, and how to design robust
models are the main obstacles to clinical adoption. Here, we present general
methods to identify causes for model generalisation failures and how to
circumvent them. First, we use distribution-shifted datasets to show that
models trained with current state-of-the-art methods are highly fragile to
variability encountered in clinical practice and then develop a strong
augmentation strategy to address this fragility. Distribution-shifted datasets
allow us to discover this fragility, which can otherwise remain undetected
after validation against multiple external datasets. Strong augmentation
allows us to train robust models achieving consistent performance under shifts
from the training data distribution. Importantly, we demonstrate that strong
augmentation yields biomedical imaging models which retain high performance
when applied to real-world clinical data. Our results pave the way for the
development and evaluation of reliable and robust neural networks in clinical
practice.

11footnotetext:  Correspondence<EMAIL_ADDRESS>
Code: https://github.com/jopo666/StrongAugment

##  1 Introduction

Figure 1: AUROC percentage-point (pp) difference between StrongAugment and the
current state-of-the-art methods on a cancer classification task. Neural
networks trained with strong augmentation retain similar performance on all
distribution-shifted datasets, external datasets and clinical validation
cases, achieving up to 50 percentage points higher discrimination performance
than current state-of-the-art training methods.

Through digitalisation and neural network-based solutions, pathology is
experiencing its third revolution [1]. Neural networks have been applied to
tissue diagnostics with impressive results, often surpassing human
counterparts in consistency, speed and accuracy [2, 3]. Despite promising
results, many recent works have demonstrated neural networks performing
substantially worse on datasets not used during training [4, 5, 6, 7, 8, 9,
10]. The two main challenges facing any neural network intended for clinical
practice, stem from this generalisation problem [11]. How to train neural
networks which perform consistently despite real-world variability, and how to
demonstrate this consistency?

All clinical-grade neural networks should be robust to possible data
distribution shifts encountered in a clinical practice [12]. This robustness
is hard to assess and has been commonly evaluated by performance on external
validation datasets. Although an important part of neural networks’ path to
clinic [11], external validation is limited to exposing a failure to
generalise in the particular dataset, and cannot establish clinical
usefulness, robustness or generalisation ability of a network [13]. Even when
a neural network achieves good performance on multiple external datasets, it
may fail on another dataset or a small subset of samples. Thus, there is a
critical need for evaluating neural networks’ robustness to data distribution
shifts.

A promising solution for evaluating neural networks’ robustness to
distribution shifts is shifted evaluation [14]. With shifted evaluation, a
distribution-shifting function is applied to every sample in a dataset to
create a new distribution-shifted dataset. The used function completely
describes any performance differences between the original and distribution-
shifted datasets and thus provides a measure of robustness as well as guidance
on how to improve the evaluated model in case of failure. For example,
distribution-shifted datasets created with the JPEG-compression algorithm [15]
can be used to measure whether a neural network is robust to artefacts created
during image compression. Shifted evaluation has been used in several recent
works, to demonstrate that neural networks are extremely fragile to even small
distribution shifts from the training data distribution [12, 16, 17, 18].

Large multi-cohort datasets, which are representative of all types of data
encountered in clinical practice, are often seen as the main solution to the
generalisation problem [11]. Although important, larger datasets often have
lower data quality [19] and different cohorts introduce their own biases,
which makes training harder as neural networks overfit these biases easily
[10, 12, 20]. Still, even a large high-quality multi-cohort dataset could not
fully represent all types of data encountered in clinical practice, and other
complementary methods for improving neural networks’ robustness are needed.
Another popular method is stain normalisation, which reduces the variability
(mainly colour) by normalising each image to a well-defined common standard
[6, 21, 22, 23, 24, 25, 26]. Although stain normalisation can be useful, it is
often computationally costly and does not fix the underlying problem of neural
networks’ fragility towards distribution shifts [16].

Only a few solutions exist for increasing the robustness to distribution
shifts from the training data. Recently proposed method of spectral decoupling
[27] can be used to increase robustness to data distribution shifts, as well
as avoid overfitting unwanted biases, for example in multi-cohort datasets
[12]. Another solution is to introduce artificial variability to the images
during neural network training, also known as augmentation. Augmentation has
been shown to explicitly improve robustness to certain data distribution
shifts [12, 25, 28]. For example, the sharpness of the scanned pathology slide
images varies significantly between medical centres, scanning equipment and
slide areas [12, 25]. By randomly adjusting the sharpness of the images during
training, neural networks can be made robust to sharpness variations.
Automatic augmentation strategies [29, 30, 31] have become a standard in
training neural networks [32]. However, augmentation has been traditionally
seen as a way to increase the amount of training data, and the development of
augmentation strategies has not focused explicitly on improving neural
networks’ robustness to data distribution shifts.

In this study, we propose general methods for evaluating and training robust
neural networks intended for clinical practice. First, we demonstrate how
shifted evaluation can be used to evaluate neural networks’ robustness to data
distribution shifts, and thus also its generalisation ability in clinical
practice. We discover that neural networks trained with the current state-of-
the-art methods are extremely fragile to even small shifts from the training
data distribution. We then develop an automatic augmentation strategy,
StrongAugment, to address this fragility. Using large-scale heterogeneous
histopathology data from several studies, we demonstrate that strong
augmentation yields robust biomedical imaging models, while state-of-the-art
methods achieve significantly lower discrimination performance (Figure 1).

##  2 Materials and methods

###  2.1 Shifted evaluation with distribution-shifted datasets

Let $x_{1},\dots,x_{n}\in X$ denote a dataset with an unknown data generating
process $\mathcal{P}$. To create a distribution shifted dataset $f(X)$, the
distribution shifting function $f$ is applied to each image $x_{i}\in X$,
where $i\in 1,\dots,n$. Although the data generating process $\mathcal{P}$ is
unknown, any performance differences between the datasets $f(X)$ and $X$ are
guaranteed to be due to the distribution shift defined by $f$. For example, if
a given model achieves an accuracy of 0.9 for dataset $X$ and 0.5 for dataset
$f_{invert}(X)$, where $f_{invert}(x_{i})=255-x_{i}$, we can say that the
given model is sensitive to the distribution shift caused by colour inversion.

The data generating process $\mathcal{P}$ is inherently unknown for external
datasets, as the image acquisition process cannot be exhaustively described.
Thus, even when a given model passes external validation, it cannot establish
clinical usefulness, robustness or generalisation ability of the model as
there is no way of knowing why the model succeeds. External validation can
only expose a failure to generalise, with no feedback given to the researcher
on reasons for the failure.

####  2.1.1 Image transformations

To evaluate neural networks’ overall robustness to data distribution shifts,
15 image transformations are used to create distribution-shifted datasets.
These datasets are denoted by $t(X,m)$, where $t$ denotes the used
transformation and $m$ its magnitude. By increasing or decreasing magnitude
$m$, the data distribution can be incrementally shifted further from the
original distribution of dataset $X$. Figure 2 presents all selected
transformations with examples of transformed images with different magnitudes.

Figure 2: Examples of image transformations with $t(x_{i},m)$, where
transformation $t$ is denoted by the figure title and magnitude $m$ by the
x-axis value.

The transformations and magnitudes, presented in Figure 2, were selected to
create distribution-shifted datasets to evaluate neural networks’ overall
robustness to data distribution shifts. It should be reasonable to expect a
neural network to perform similarly on all of the images in Figure 2, and any
potential failures will give important insights into the robustness of the
model. Although some of the distribution-shifted datasets may not be directly
encountered in routine clinical practice, they can still be useful in other
ways. For example, if a given network performs consistently for dataset
$\textit{posterize}(X,4)$, $\textit{saturation}(X,0)$ and
$\textit{jpeg}(X,10)$, it could be possible to significantly reduce the size
of large digital slides by reducing the number of bits, dropping colour
channels or compressing images more heavily, without compromising model
performance.

Noise, emboss and jpeg transformations are from the albumentations (1.2.1)
library [33], red, green, blue transformations have been defined by us, and
all other transformations are from the torchvision (1.11.1) library [34].

####  2.1.2 Haematoxylin and eosin stain intensities

Colours of the digital slide images can vary substantially due to differences
in the staining processes and the choice of imaging equipment and its
settings, especially between medical centres [25]. Although several
transformations presented in Figure 2 evaluate robustness to changes in the
colour of the images, none of them explicitly measure robustness to data
distribution shifts caused by differences in the staining process. For this
reason, and to demonstrate the flexibility of distribution-shifted datasets,
distribution-shifted datasets are created by adjusting the intensity of the
haematoxylin and eosin stains.

Figure 3: Examples of haematoxylin and eosin stain intensity adjustments where
haematoxylin and eosin staining intensities are denoted by the y-axis and
x-axis, respectively.

To create a distribution-shifted dataset $\textit{stain}(X,h,e)$, the Macenko
method [21] is used to separate the haematoxylin and eosin stains, and then
concentrations of each stain are multiplied with magnitudes $h$ and $e$,
respectively. For example, $h=0.0,e=1.0$ would remove the eosin stain
completely from the image and $e=1.0,m_{h}=2.0$ would double the haematoxylin
stain intensity. Figure 3 demonstrates example images from the distribution-
shifted datasets. The mean stain vector of the original dataset is used to
separate the stains.

###  2.2 StrongAugment

Table 1: Augmentation spaces of StrongAugment, RandAugment [30], RandAugment-HE [35] and TrivialAugment [31]. No effect column represents the magnitude which does not change the image where applicable. StrongAugment includes 6 additional transformations, Gaussian blurring and wider magnitude ranges for most of the common transformations. Examples for some of the transformations can be seen in Figure 2. Please note, that a hyper-parameter is used to further limit the magnitude ranges for RandAugment and RandAugment-HE. Transformation | StrongAugment | RandAugment | RandAugment-HE | TrivialAugment | No effect
---|---|---|---|---|---
Identity | included | included | included | included |
Shear{X,Y} | [-145, 145] | [-17, 17] | [-51, 51] | [-145, 145] | 1.0
Translate{X,Y} | [-32, 32] | [-72, 72] | [-30, 30] | [-32, 32] | 0
Rotate | [-135, 135] | [-30, 30] | [-90, 90] | [-135, 135] | 0
Saturation | [0.0, 2.0] | [0.1, 1.9] | [0.0, 5.5] | [0.01, 1.99] | 1.0
Brightness | [0.1, 1.9] | [0.1, 1.9] | [0.0, 5.5] | [0.01, 1.99] | 1.0
Contrast | [0.1, 1.9] | [0.1, 1.9] | [0.0, 5.5] | [0.01, 1.99] | 1.0
Sharpness | [1.0, 2.0] | [0.1, 1.9] | [0.0, 5.5] | [0.01, 1.99] | 1.0
Gaussian blur | [0.0, 2.0] | – | – | – | 0.0
Solarize | [0, 255] | [0, 255] | – | [0, 255] | 256
Posterize | [1, 8] | [4, 8] | – | [1, 8] | 8
Equalize | included | included | included | included |
Auto contrast | included | included | included | included |
Grayscale | included | – | – | – |
Gamma | [0.1, 1.9] | – | – | – | 1.0
Hue | [-0.5, 0.5] | – | – | – | 0.0
Red | [0.01, 1.99] | – | – | – | 1.0
Green | [0.01, 1.99] | – | – | – | 1.0
Blue | [0.01, 1.99] | – | – | – | 1.0
HED colour shift | – | – | [-0.9, 0.9] | – | 0.0
HSV colour shift | – | – | [-0.9, 0.9] | – | 0.0
HED: hematoxylin-eosin-DAB, HSV: and hue-saturation-value |

In this study, we create an automatic augmentation method, StrongAugment,
which is designed to increase neural networks’ robustness to data distribution
shifts from the training data. StrongAugment is compared to RandAugment [30],
one of the most commonly used automatic augmentation methods, and
TrivialAugment [31] which represents the current state-of-the-art in automatic
augmentation. Importantly, implementations for both methods are easily
accessible through the torchvision library, and thus likely used by other
researchers in the field. Additionally, we compare StrongAugment to a HE-
tailored variation of RandAugment, RandAugment-HE [35], which exploits prior
knowledge of the presumable variation of histological images such as
differences in the haematoxylin and eosin stains seen in Figure 3. This allows
comparisons between augmentation strategies explicitly designed to increase
robustness for certain data distribution shifts, such as HE-staining
differences, and StrongAugment, which is designed to increase robustness for
all distribution shifts.

StrongAugment applies a varying number of transformations sequentially to an
image $x$. After $x$ has been transformed two times, each consecutive
transformation is applied with probability $p$, or until $x$ has been
transformed five times. Each transformation $t$ and its magnitude $m$ is
sampled uniformly from an augmentation space $A$, which contains a specified
list of transformations and a range of possible magnitudes for each
transformation. After applying transformation $t$, the transformation $t$ is
removed from $A$. Only one affine transformation is allowed per image, to
avoid cases where sequentially applied affine transformations move the image
data out of the image. This allows significantly higher number of
transformations than previous state-of-the-art augmentation methods [30, 31,
35, 36].

Table 1 represents the augmentation spaces of StrongAugment, RandAugment,
RandAugment-HE and TrivialAugment, and Supplementary Figure S1 examples of
augmented images with each method. StrongAugment has both a wider augmentation
space $A$ and a higher number of transformations applied per image. The
augmentation space of StrongAugment contains all the same transformations as
RandAugment and TrivialAugment, with six additional transformations. To enable
stronger blurring of images, the default degenerate method implemented by the
torchvision library was replaced with Gaussian blurring. No neural networks
were trained while building the augmentation space to avoid overfitting the
augmentation space to a specific dataset. Transformations and their magnitude
ranges were simply chosen to be as wide as possible.

RandAugment, TrivialAugment have been implemented by torchvision (1.11.1).
RandAugment-HE has been implemented in a previous study [35]. For RandAugment,
the magnitude $m$ and number of operations $n$ are set to $m=10$ and $n=2$
[32], and for RandAugment-HE to $m=5$ and $n=3$ [35]. For TrivialAugment there
are no tunable hyper-parameters.

###  2.3 Training details

ResNet-RS50 [32] neural networks are used for all the experiments in this
study. Dropout [37] and stochastic depth [38] are set to 0.2 [32] for all
networks. For training, a random area of the image is cropped and resized to
$160\times 160$ pixels with either bi-linear, bi-cubic or nearest
interpolation, and flipped horizontally and/or vertically with a probability
of 0.5. Then the image is transformed with either StrongAugment, RandAugment,
RandAugment-HE or TrivialAugment, a random area of the image is erased with an
0.2 probability, and either Mixup [39] or Cutmix [40] is applied for pair of
images in a batch with an alpha value of 0.2. For testing, the images are
resized to $224\times 224$ pixels with bi-linear interpolation.

Adaptive sharpness aware minimisation [41] is used with $\rho=0.05$ and Adam
[42] with $\beta_{1}=0.9$, $\beta_{2}=0.999$ as the base optimiser. The
learning rate is increased linearly from $10^{-6}$ to
$0.0004\times\texttt{batch\\_size}/256$ during the first 10 epochs and then
reduced back to $10^{-6}$ with a cosine schedule. Label smoothing is set to
0.1 [43]. All networks are trained for 150 epochs with a batch size of 1024.
For each experiment, five neural networks are trained and the mean and
standard deviation of the evaluation metric are reported.

Spectral decoupling [27] is used instead of weight decay, which has been shown
to increase neural networks’ robustness for data distribution shifts [12].
Spectral decoupling coefficient is set to 0.0001 and weight decay to 0. For
experiments, where spectral decoupling is intentionally left out to assess its
effect on neural networks’ robustness, weight decay is set to 0.00004 [32].

PyTorch (1.10.0) ([44]) is used for training the neural networks, and ResNet-
RS50, Mixup and Cutmix implementations are from the timm (0.5.4) library [45].

###  2.4 Evaluation metric

To evaluate the performance of a neural network in a given dataset, the area
under the receiver operating characteristic curve (AUROC) is reported for each
experiment. AUROC measures the ability to discriminate between positive and
negative samples. The AUROC value corresponds to the proportion of positive
and negative sample pairs, where the positive sample has a higher probability
given by the classifier. AUROC values of 0.5 and 1.0 correspond to random and
perfect discrimination, respectively.

###  2.5 Datasets

Table 2: Datasets used in this study. Name | Medical centre | Years | Images | Label noise | Reference
---|---|---|---|---|---
Helsinki30 | Helsinki University Hospital | 2014-2015 | 4.7 mil. | 0.03% | [12]
Helsinki60 | Helsinki University Hospital | 2019-2020 | 13.1 mil. | 17.0% | –
Karolinska | Karolinska Institutet | 2012-2014 | 0.6 mil. | 9.6% | [46]
Radboud | Radboud University Medical Center | 2012-2017 | 0.8 mil. | 3.9% | [46]
PESO | Radboud University Medical Center | 2006-2011 | 5655 | – | [47, 48]
Gleason2019 | Vancouver General Hospital | 1997-2011 | 94230 | 14.7% | [49]
HelsinkiRCC | Helsinki University Hospital | 2006-2013 | 2.0 mil. | 8.1% | –

Seven different datasets from two tissue types, originating from four
countries are used in this study. A basic summary of the datasets is presented
in Table 2. All digital slide images are processed with the HistoPrep library
(1.0.7) [50].

A total of 30 prostate cancer patients’ full glass slide sets from surgical
specimens are annotated for classification into cancerous and benign tissue,
where the cancerous areas were annotated in consensus by two observers (C.S.,
T.M.). All patients have undergone radical prostatectomy at the Helsinki
University Hospital between 2014 and 2015. Each case contains 14 to 21 tissue
section slides. Tissue sections have a thickness of 4 $\mu$m and were stained
with haematoxylin and eosin in a clinical-grade laboratory at the Helsinki
University Hospital Diagnostic Center, Department of Pathology. Two different
scanners are used to obtain images of the tissue section slides at 20x
magnification. Larger macro slides (whole-mount, 2x3 inch slides) are scanned
with an Axio Scan Z.1 scanner (Zeiss, Oberkochen, Germany), and the normal
size slides with a Pannoramic Flash III 250 scanner (3DHistech, Budapest,
Hungary). From the 30 patient cases, seven are set aside for a validation set.
Digital slide images are cut into tiles with $1024\times 1024$ pixels and 20%
overlap, resulting in 4.7 million tiles with 10% containing cancerous tissue.
For training, we selected all cancerous tiles and sample randomly the same
amount of benign tiles to have equal amounts of positive and negative samples.
We denote this dataset as Helsinki30.

Another set of 60 radical prostatectomy slides is also annotated into
cancerous and benign tissue by one of six experienced pathologists as part of
routine clinical diagnostics. All patients have undergone radical
prostatectomy at the Helsinki University Hospital between 2019 and 2020. Each
case contains 10 to 21 normal and macro tissue section slides of the prostate.
Tissue sections have a thickness of 4 $\mu$m and are also stained with
haematoxylin and eosin in a clinical-grade laboratory at the Helsinki
University Hospital Diagnostic Center, Department of Pathology. All slides are
scanned with an Axio Scan Z.1 scanner. From the 60 patient cases, seven are
set aside for a validation set. Digital slide images are cut into $1024\times
1024$ pixel tiles with 20% overlap, resulting in 13.1 million tiles with 16%
containing cancerous tissue. For training, we selected all cancerous tiles and
sample randomly the same amount of benign tiles to have equal amounts of
positive and negative samples. We denote this dataset as Helsinki60.

As external datasets, four publicly available prostate cancer datasets are
used. These datasets yield five external datasets from three different
countries.

The PANDA development dataset contains 10616 prostate biopsy slides from 2113
patients from Radboud University Medical Center between 2012 and 2017 and
Karolinska Institutet (Stockholm, Sweden) between 2012 and 2014 [46]. The
digital slide images are cut into $512\times 512$ pixel tiles with 20% overlap
resulting in 609180 (48% cancer) and 780694 (23% cancer) tiles from Radboud
University Medical Center and Karolinska Institutet, respectively. Tile images
were labelled as cancerous if there was any overlap with the cancerous
annotations. Validation splits are omitted for both datasets as patient
identifiers are missing from the publicly available data. We denote the
datasets from Radboud University Medical Center and Karolinska Institutet as
Radboud and Karolinska, respectively.

The PESO dataset contains tissue section slides from patients who have
undergone a radical prostatectomy at the Radboud University Medical Center
(Nijmegen, the Netherlands) between 2006 and 2011 ([47, 48]). The dataset
contains images with $2500\times 2500$ pixels annotated by a uropathologist as
either cancerous or benign. These images are cut into $512\times 512$ pixel
tiles with 20% overlap, resulting in 5655 tiles with 45% containing cancerous
tissue.

The Gleason2019 dataset contains 333 prostate tissue microarray spots from 231
patients who had undergone radical prostatectomy at Vancouver General Hospital
between 1997 and 2011 [49]. These digital slide images are cut into $224\times
224$ pixel tiles with 20% overlap resulting in 94230 tiles with 88.5%
containing cancerous tissue.

To evaluate augmentation strategies on training data from a different tissue
type, a dataset was compiled from 167 patients with renal cell carcinoma
(clear cell). All patients had undergone radical nephrectomy at the Helsinki
University Hospital between 2006 and 2013. Tissue sections were stained with
haematoxylin and eosin in a clinical-grade laboratory at the Helsinki
University Hospital Diagnostic Center, Department of Pathology. A total of 698
cancerous and 172 benign tissue microarray spots were gathered from the
sections and scanned with Pannoramic Flash III 250 scanner (3DHistech,
Budapest, Hungary). Of the 167 patients, 24 were set aside for a validation
set. Scanned tissue microarray spots were cut into tiles with $384\times 384$
pixels and 20% overlap, resulting in 2.0 million tiles with 21% containing
cancerous tissue. We denote this dataset as HelsinkiRCC.

####  2.5.1 Label noise

Each dataset used in this study has a variable annotation strategy as they
stem from different projects. Thus, each dataset contains different amounts of
label noise. For this reason, the reported performances on different datasets
are not directly comparable. For example, a perfect classifier would achieve
100% accuracy on a clean dataset, but only 80% accuracy on a dataset with 20%
label noise. In reality, there are no perfect classifiers [51] and the
decreased performance on an external dataset may also be caused by the fact
that the classifier cannot generalise to the dataset. For this reason, the
amount of label noise in a dataset should be estimated.

To estimate the amount of label noise in a given dataset, a neural network is
trained on the dataset, and the training discrimination performance is
reported. Due to overfitting, the estimate of label noise in the dataset can
easily be deflated. To minimise overfitting, we use strong regularisation
outlined in Section 2.3 with StrongAugment. To evaluate the accuracy of this
estimation method, label noise is introduced to the Helsinki30 dataset, which
contains the least amount of label noise based on a niche expert evaluation
(T.M.). After flipping 0, 5, 10 and 20% of the labels in the Helsinki30
dataset, the estimated label noise is 0.03, 5.5, 10.21 and 20.16%,
respectively.

The estimated amount of label noise in the Helsinki30, Helsinki60, Karolinska,
Radboud, Gleason2019 and HelsinkiRCC datasets are 0.03%, 17.0%, 9.6%, 3.9%,
14.7% and 8.1%, respectively.

###  2.6 Clinical Validation

To evaluate neural networks’ robustness to distribution shifts encountered in
routine clinical practice, several clinical validation cases were collected.
Three surgical specimen paraffin blocks from patients who have undergone
radical prostatectomy at the Helsinki University Hospital (HUS) in June of
2022, were collected. For each specimen, 36 serial tissue sections were
stained in one of four HUS Diagnostic Centre’s clinical-grade laboratories,
using one of three slide types. Each slide is then scanned with two different
scanners at 20x magnification. The four medical centres are Hyvinkää Hospital
(Hyvinkää, Finland), Jorvi Hospital (Espoo, Finland), Kotka hospital (Kotka,
Finland) and Meilahti Hospital (Helsinki, Finland). The three microscope
slides are Superfrost Plus (SP) (Fisher Scientific, Waltham, MA, United
States), TOMO (Matsunami Glass, Bellingham, WA, United States) and Klinipath
(KP) (VWR International, Radnor, PA, United States). The two scanners are
Pannoramic 1000 (3DHistech, Budapest, Hungary) and Aperio AT2 (Leica
Microsystems, Wetzlar, Germany). This produces 72 scanned slide images, which
are annotated for classification into cancerous and benign tissue, by one
observer (T.M.). There is considerable variability between medical
laboratories, slide glasses and scanners, which is demonstrated in
Supplementary Figure S2.

##  3 Results

Figure 4: Robustness of neural networks, trained on the Helsinki30 dataset
with either StrongAugment, RandAugment, RandAugment-HE or TrivialAugment, to
distribution shifts caused by image transformations. The lines show the mean
AUROC values and shaded regions with one standard deviation around the mean
for the five trained networks. The distribution-shifted datasets
$t(X_{\text{PESO}},m)$ are represented by dots, where the title denotes the
transformation $t$ and the x-axis the magnitude $m$. Networks trained with
StrongAugment retain their discrimination performance despite increasingly
large distribution shifts, whereas the performance of networks trained with
RandAugment or TrivialAugment degrade quickly with even small distribution
shifts. This fragility is not discoverable by simply looking at the
performance on the unmodified PESO dataset, denoted by dotted red lines.
Figure 5: Robustness of neural networks, trained on the Helsinki30 dataset
with either StrongAugment, RandAugment, RandAugment-HE or TrivialAugment, to
distribution shifts caused by differences in the haematoxylin and eosin stain
intensities. Each cell represents a given distribution-shifted dataset
$\textit{stain}(X_{\text{PESO}},h,e)$, where $h$ and $e$ denote the magnitudes
for haematoxylin and eosin stains, respectively. Cells are annotated with mean
AUROC $\times$ 100 value across the five trained networks, rounded to the
closest integer. Neural networks trained with StrongAugment retain performance
on all datasets, even when networks trained with RandAugment or TrivialAugment
have degraded to no better than random discrimination. StrongAugment also
outperforms RandAugment-HE, even though the latter method explicitly controls
for staining differences.

###  3.1 Robustness to data distribution shifts

First, neural networks’ robustness to data distribution shifts is evaluated
with distribution-shifted datasets created from the PESO dataset. For each
experiment, five neural networks are trained on the Helsinki30 dataset with
either StrongAugment, RandAugment, RandAugment-HE or TrivialAugment while
keeping all other hyper-parameters fixed.

First, the general robustness to data distribution shifts is evaluated using
the distribution-shifted datasets, described in Section 2.1.1, with examples
presented in Figure 2. Discrimination performances of the trained neural
networks on these datasets are presented in Figure 4. Although all methods
achieve similar performance on the original dataset, neural networks trained
with either RandAugment, RandAugment-HE or TrivialAugment are highly sensitive
to even small data distribution shifts, which is evident from the large
performance differences between the original and distribution-shifted
datasets. Neural networks trained with StrongAugment retain similar or
significantly higher discrimination performances for all distribution-shifted
datasets, even in cases where networks trained with other methods have
degraded to no better than random discrimination. Notably, neural networks
trained with StrongAugment are robust to distribution shifts not included in
the augmentation space, such as Noise, Emboss and JPEG transformations.
Comparably, neural networks trained with either RandAugment, RandAugment-HE or
TrivialAugment are sensitive to many of the transformations included in their
augmentation space, such as Hue and Brightness transformations.

Second, neural networks’ robustness to data distribution shifts caused by
differences in the haematoxylin and eosin stain intensities is evaluated using
the distribution-shifted datasets described in Section 2.1.2, with example
images presented in Figure 3. Discrimination performances of the trained
neural networks on these datasets are presented in Figure 5. Neural networks
trained with either RandAugment, or TrivialAugment are highly sensitive to
distribution shifts caused by changes in the haematoxylin and eosin stain
intensities, and the discrimination performance degrades to no better than
random with even small distribution shifts. Neural networks trained with
StrongAugment retain their discrimination performance on all distribution-
shifted datasets, even with datasets where networks trained with RandAugment
or TrivialAugment have no better than random discrimination. StrongAugment
also significantly outperforms RandAugment-HE, which explicitly controls for
the distribution shifts caused by haematoxylin and eosin differences [35].

We recommend comparing the discrimination performances in Figures 4 and 5 to
the example images in Figures 2, 3, respectively.

###  3.2 External validation

Figure 6: Discrimination performances of neural networks trained with
StrongAugment, RandAugment, RandAugment-HE or TrivialAugment on four different
training datasets. Each network is evaluated with five external datasets and
the training dataset, denoted by a red box. Evaluation datasets are denoted by
the figure titles and training datasets by the x-axis labels. Neural networks
trained with StrongAugment achieve better or comparable performance on every
evaluation dataset, whereas RandAugment, RandAugment-HE and TrivialAugment
demonstrate a significant lack of generalisation ability with several
evaluation datasets. Please note that the y-axes of Helsinki60, Karolinska and
Gleason2019 evaluation datasets have been truncated to account for the label
noise in these datasets.

To assess, whether better performance on the distribution-shifted datasets
translates to better performance on real-world datasets, we evaluate the
trained neural networks on the Helsinki30, Helsinki60, Karolinska, Radboud
[46], PESO [47] and Gleason2019 [49] datasets. To demonstrate that
StrongAugment has not just been over-fitted to the Helsinki30 dataset used in
Section 3.1, we also train and evaluate neural networks on the Helsinki60,
Karolinska and Radboud datasets. For each experiment, five neural networks are
trained with either StrongAugment, RandAugment, RandAugment-HE or
TrivialAugment while keeping all other hyper-parameters fixed.

The discrimination performances of neural networks trained with StrongAugment,
RandAugment, RandAugment-HE or TrivialAugment are presented in Figure 6.
Compared to the current state-of-the-art methods, neural networks trained with
StrongAugment achieve better or similar results on every external validation
dataset. Networks trained with RandAugment, RandAugment-HE or TrivialAugment
fail to generalise to several of the external datasets, despite the fact that
the same networks achieve good results on other external validation datasets.
For example, networks trained on the Helsinki30 dataset (first columns) with
RandAugment or RandAugment-HE perform comparably to StrongAugment on the
Helsinki60, Karolinska, Radboud and PESO datasets, but then achieve
unacceptably low discrimination performances of 0.6 to 0.7 on the Gleason2019
dataset. Here, shifted evaluation allowed us to easily detect fragility to
distribution shifts, which otherwise would have been missed if Gleason2019
dataset was not available.

As expected, neural networks trained on the datasets with more label noise
achieve noticeably worse performance with all augmentation methods, including
StrongAugment. Nevertheless, networks trained with StrongAugment still
maintain an acceptable performance on all evaluation datasets, whereas the
other methods result in no better than random discrimination performance. This
is most apparent with the Karolinska (third columns) and Radboud (last
columns) training datasets, where StrongAugment improves the mean
discrimination performance over RandAugment and TrivialAugment by 0.116 to
0.196 on the Helsinki30 dataset, and by 0.099 and 0.181 on the Helsinki60
dataset.

In addition to improved performance, there is a significantly lower variance
in the discrimination performances across the five trained networks when using
StrongAugment. When training neural networks with StrongAugment, RandAugment,
RandAugment-HE and TrivialAugment, the maximum difference between the best and
worst-performing networks is 0.032, 0.132, 0.086 and 0.121, respectively.

###  3.3 Clinical validation

Figure 7: Discrimination performances of neural networks trained with
StrongAugment, RandAugment, RandAugment-HE or TrivialAugment on 72 clinical
validation cases. Each case is represented by a $1\times 4$ block of cells,
and due to different amounts of label noise, comparisons are valid only within
each block. Each cell is annotated by AUROC $\times$ 100 value and coloured
based on the percentage point (pp) difference to StrongAugment. Neural
networks trained with StrongAugment perform consistently on all clinical
validation cases, whereas each of the compared methods fails to generalise to
several clinical validation cases.

To assess neural networks’ robustness to actual data distribution shifts
encountered in clinical practice, we evaluate five neural networks trained on
the Helsinki30 dataset with clinical validation cases described in Section
2.6.

The discrimination performances for each of the 72 clinical validation cases
are presented in Figure 7. Neural networks trained with StrongAugment
generalise to all clinical validation cases, whereas each of the compared
methods fails to generalise to several cases. StrongAugment achieves at least
0.03 better AUROC values in 24, 10, and 28 cases when compared to RandAugment,
RandAugment-HE and TrivialAugment, respectively. StrongAugment achieves at
least 0.03 worse AUROC value in only a single case. Without StrongAugment,
neural networks fail to generalise to actual data distribution shifts
encountered in clinical practice.

###  3.4 Effect of spectral decoupling

As spectral decoupling has been shown to increase robustness to data
distribution shifts [12], we evaluate the effect spectral decoupling has on
neural network performance. Five neural networks are trained on the Helsinki30
dataset, using either StrongAugment, RandAugment or TrivialAugment, and weight
decay instead of spectral decoupling. The neural networks are then evaluated
on the distribution-shifted datasets, adjusting haematoxylin and eosin stain
intensities, described in Section 2.1.2.

The discrimination performances of the trained neural networks are presented
in Supplementary Figure S3. Networks trained with StrongAugment achieve
consistent results even without spectral decoupling, although there is a
slight decrease in performance of up to four percentage points. Spectral
decoupling is much more crucial for RandAugment and TrivialAugment, where the
use of weight decay decreases the AUROC values by up to 25 percentage points.
Based on these results, spectral decoupling is complementary with
StrongAugment and crucial for training robust neural networks.

###  3.5 Other tissues

To assess whether the previous results are somehow specific to prostate
tissue, five neural networks are trained with StrongAugment, RandAugment or
TrivialAugment on the HelsinkiRCC dataset training split containing kidney
tissue. The neural networks are then evaluated on distribution-shifted
datasets created from the HelsinkiRCC validation split by adjusting
haematoxylin and eosin stains similarly to Section 2.1.2.

The discrimination performances of the trained neural networks are presented
in Supplementary Figure S4. Neural networks trained with StrongAugment achieve
consistent performance on the distribution-shifted datasets, whereas the
performance of networks trained with RandAugment and TrivialAugment degrades
quickly with even small distribution shifts. These results indicate that
neural network fragility is not specific to prostate tissue, and strong
augmentation is helpful with other types of tissues as well. Additionally, it
is demonstrated here that shifted evaluation can also be useful when extending
the validation split, and not only external datasets.

##  4 Discussion

Robustness evaluation is crucial for understanding the clinical usefulness,
safety and accuracy of machine learning models intended for clinical practice
[11]. Despite showing strong performance on multiple external validation
datasets, state-of-the-art neural networks fail to generalise to multiple
clinical validation cases. With shifted evaluation, this fragility can be
discovered by simply extending the validation split with distribution-shifted
datasets, and addressed with strong augmentation.

Although external validations will be a crucial part of any neural network’s
path to clinic [11], there are serious limitations [13]. As the data-
generating process is unknown, there is no way of knowing why the evaluated
network fails or succeeds in external evaluation. By extending the external
dataset with distribution-shifted datasets, it is possible to have known data
generation functions, which completely describe any performance differences
between the datasets. Thus, it is possible to discover cases where the network
fails, and demonstrate robustness for the selected distribution shifts.

We believe that due to the lack of adequate evaluation methods, the role of
augmentation as a method of increasing neural networks’ robustness to data
distribution shifts has been largely unexplored. In medical imaging, some work
has been done with training stain-invariant neural networks through
augmentation [28, 35]. A comparison of different stain augmentation and
normalisation methods concluded that augmentation is crucial for achieving
good performance, but found slightly better results with light rather than
strong augmentation [25]. As the results were based on external validation
performance and augmentation was coupled with stain normalisation, it is
likely that the fragility of neural networks was simply not discovered. With
natural images, the development of augmentation methods has mostly focused on
improving performance on the validation split of the ImageNet [52] dataset
[29, 30, 31, 36]. Thus, light augmentation has been preferred over strong,
which can reduce performance on in-distribution datasets, although the
reduction in performance is often dwarfed when compared to the performance
decreases on external datasets [19].

Interestingly, even though the augmentation space of RandAugment, RandAugment-
HE or TrivialAugment contains a given transformation, this does not guarantee
that the trained network becomes robust for the distribution shift caused by
the transformation. For example, the solarize transformation is included in
the augmentation spaces of RandAugment and TrivialAugment, but networks
trained with either method are still extremely fragile for this distribution
shift. TrivialAugment even has a wider augmentation space for the brightness
transformation, but StrongAugment achieves 0.461 better AUROC when doubling
the brightness of the images in the dataset. We speculate that the combination
of a wider augmentation space, coupled with a significantly higher number of
transformations per image makes neural networks trained with StrongAugment
robust to distribution shifts included in its augmentation space.
Additionally, the results of this paper suggest that networks trained with
StrongAugment are also robust to other data distribution shifts, not just the
ones included in the augmentation space.

Neural networks trained with StrongAugment achieve consistent results with
distribution-shifted datasets, external validation and clinical validation
cases. This is not the case for current state-of-the-art methods, which are
extremely fragile to distribution shifts and fail to generalise to several
external datasets and clinical validation cases. The performance decreases for
networks trained with RandAugment, RandAugment-HE and TrivialAugment are
unacceptable with most of the datasets, especially when considering that the
decrease does not occur with networks trained with StrongAugment. The
magnitude of differences we show between StrongAugment and current state-of-
the-art methods are rarely reported. For example, TrivialAugment is reported
to increase AUROC by 0.005 over RandAugment, and RandAugment-HE [35] to
increase the AUROC by 0.006 over the compared method [25]. In comparison,
StrongAugment increases AUROC by at least 0.1 in 104, 29 and 102 datasets for
RandAugment, RandAugment-HE and TrivialAugment, respectively.

Although neural networks trained with StrongAugment achieve unprecedented
results, it does not solve the generalisation problem in medical imaging on
its own. Other methods which increase neural networks’ robustness to data
distribution shifts should be used. For example, spectral decoupling has been
shown to increase robustness to data distribution shifts [12], and it is
highly complementary with StrongAugment (Figure S3). Also, high-quality and
representative datasets are crucial for training neural networks intended for
clinical practice. This is demonstrated in Figure 6, where datasets with more
label noise achieve lower discrimination performance on external validation.
With StrongAugment this gap is significantly smaller, emphasising the
usefulness of strong augmentation on lower-quality datasets. With
StrongAugment, the collection of representative training datasets can now
focus on biological representativeness, instead of using multiple different
scanners or staining pipelines.

##  5 Conclusions

In this study, we have exposed the fragility of neural networks using shifted
evaluation. Shifted evaluation can be used to thoroughly evaluate a neural
network while providing a measure of robustness to distribution shifts, and
guidance on how to improve the network in case of failure. To address the
exposed fragility of neural networks, we developed an augmentation strategy
which allows neural networks to be robust to data distribution shifts from the
training data. StrongAugment achieves consistent results on external datasets
and clinical validation cases, even when the current state-of-the-art methods
achieve no better than random performance. The results of this study indicate
that without strong augmentation, neural networks are not likely to generalise
to variation encountered in clinical practice.

Our study provides a baseline for evaluating and training neural networks
intended for clinical practice. Strong augmentation allows networks to perform
consistently despite real-world variability, and shifted evaluation can be
used to demonstrate this consistency. Although we establish the effectiveness
of these methods in medical imaging, both methods are applicable to any
domain.

## Acknowledgements

This work was supported by Cancer Foundation Finland [304667, 191118], Jane
and Aatos Erkko Foundation [290520], Academy of Finland [322675] and Hospital
District of Helsinki and Uusimaa [TYH2018214, TYH2018222, TYH2019235,
TYH2019249]. The authors also wish to acknowledge Harry Nisen and Petrus
Järvinen for Helsinki RCC database management, FIMM Digital Microscopy and
Molecular Pathology Unit supported by HiLIFE and Biocenter Finland for imaging
services, and CSC – IT Center for Science, Finland for generous computational
resources.

## Ethics statement

Ethical approvals for the use of human tissue material and clinicopathologic
data were obtained from Institutional Ethics Committee of Hospital District of
Helsinki and Uusimaa (§70/16.5.2018; HUS/419/2018) and by the National
Supervisory Authority for Welfare and Health (VALVIRA, D:no V/38176/2018).
According to the national and European Union legislation on noninterventional
medical research, the study was conducted without informed individual patient
consents by permission of the Hospital District of Helsinki and Uusimaa
(§105/21.12.2018; HUS/419/2018). The experiments conformed to the principles
set out in the WMA Declaration of Helsinki and the Department of Health and
Human Services Belmont Report.

## Declaration of competing interests

The authors have no interests to declare.

## Credit authorship contribution statement

Joona Pohjonen: Conceptualisation; Formal analysis; Visualisation; Software;
Methodology; Data curation; Roles/Writing - original draft Carolin Stürenberg:
Data curation, Writing - review & editing; Atte Föhr: Data curation, Formal
analysis; Reija Randen-Brady: Data curation; Lassi Luomala: Data curation;
Jouni Lohi: Data curation; Esa Pitkänen: Funding acquisition; Resources;
Supervision; Writing - review & editing; Antti Rannikko: Funding acquisition;
Resources; Supervision; Writing - review & editing; Tuomas Mirtti:
Conceptualisation; Data curation; Funding acquisition; Resources; Supervision;
Writing - review & editing;

## References

  * [1] Manuel Salto-Tellez, Perry Maxwell, and Peter Hamilton.  Artificial intelligence-the third revolution in pathology.  Histopathology, 74(3):372–376, 2019.
  * [2] Xiaoxuan Liu, Livia Faes, Aditya U Kale, Siegfried K Wagner, Dun Jack Fu, Alice Bruynseels, Thushika Mahendiran, Gabriella Moraes, Mohith Shamdas, Christoph Kern, et al.  A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis.  The lancet digital health, 1(6):e271–e297, 2019.
  * [3] Eric J Topol.  High-performance medicine: the convergence of human and artificial intelligence.  Nature medicine, 25(1):44–56, 2019.
  * [4] Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus J Busam, Edi Brogi, Victor E Reuter, David S Klimstra, and Thomas J Fuchs.  Clinical-grade computational pathology using weakly supervised deep learning on whole slide images.  Nature medicine, 25(8):1301–1309, 2019.
  * [5] Kunal Nagpal, Davis Foote, Yun Liu, Po-Hsuan Cameron Chen, Ellery Wulczyn, Fraser Tan, Niels Olson, Jenny L Smith, Arash Mohtashamian, James H Wren, et al.  Development and validation of a deep learning algorithm for improving gleason scoring of prostate cancer.  NPJ digital medicine, 2(1):1–10, 2019.
  * [6] Thomas de Bel, Meyke Hermsen, Jesper Kers, Jeroen van der Laak, and Geert Litjens.  Stain-transforming cycle-consistent generative adversarial networks for improved segmentation of renal histopathology.  In International Conference on Medical Imaging with Deep Learning–Full Paper Track, 2018.
  * [7] Yun Liu, Timo Kohlberger, Mohammad Norouzi, George E Dahl, Jenny L Smith, Arash Mohtashamian, Niels Olson, Lily H Peng, Jason D Hipp, and Martin C Stumpe.  Artificial intelligence-based breast cancer nodal metastasis detection: Insights into the black box for pathologists.  Archives of pathology & laboratory medicine, 143(7):859–868, 2019\.
  * [8] John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann.  Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study.  PLoS medicine, 15(11):e1002683, 2018.
  * [9] Julia K Winkler, Christine Fink, Ferdinand Toberer, Alexander Enk, Teresa Deinlein, Rainer Hofmann-Wellenhof, Luc Thomas, Aimilios Lallas, Andreas Blum, Wilhelm Stolz, et al.  Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition.  JAMA dermatology, 155(10):1135–1141, 2019.
  * [10] Alex J DeGrave, Joseph D Janizek, and Su-In Lee.  Ai for radiographic covid-19 detection selects shortcuts over signal.  Nature Machine Intelligence, pages 1–10, 2021.
  * [11] Jeroen van der Laak, Geert Litjens, and Francesco Ciompi.  Deep learning in histopathology: the path to the clinic.  Nature medicine, 27(5):775–784, 2021.
  * [12] Joona Pohjonen, Carolin Stürenberg, Antti Rannikko, Tuomas Mirtti, and Esa Pitkänen.  Spectral decoupling for training transferable neural networks in medical imaging.  iScience, page 103767, 2022.
  * [13] Myura Nagendran, Yang Chen, Christopher A Lovejoy, Anthony C Gordon, Matthieu Komorowski, Hugh Harvey, Eric J Topol, John PA Ioannidis, Gary S Collins, and Mahiben Maruthappu.  Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies.  Bmj, 368, 2020.
  * [14] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  Advances in neural information processing systems, 32, 2019.
  * [15] Digital compression and coding of continuous-tone still images: Requirements and guidelines, 2 1992.
  * [16] Birgid Schömig-Markiefka, Alexey Pryalukhin, Wolfgang Hulla, Andrey Bychkov, Junya Fukuoka, Anant Madabhushi, Viktor Achter, Lech Nieroda, Reinhard Büttner, Alexander Quaas, et al.  Quality control stress test for deep learning-based diagnostic model in digital pathology.  Modern Pathology, 34(12):2098–2108, 2021.
  * [17] Yijiang Chen, Andrew Janowczyk, and Anant Madabhushi.  Quantitative assessment of the effects of compression on deep learning in digital pathology image analysis.  JCO clinical cancer informatics, 4:221–233, 2020.
  * [18] Timo Kohlberger, Yun Liu, Melissa Moran, Po-Hsuan Cameron Chen, Trissia Brown, Jason D Hipp, Craig H Mermel, and Martin C Stumpe.  Whole-slide image focus quality: Automatic assessment and impact on ai cancer detection.  Journal of pathology informatics, 10(1):39, 2019.
  * [19] Gaël Varoquaux and Veronika Cheplygina.  Machine learning for medical imaging: methodological failures and recommendations for the future.  npj Digital Medicine, 5(1):1–8, 2022.
  * [20] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann.  Shortcut learning in deep neural networks.  Nature Machine Intelligence, 2(11):665–673, 2020.
  * [21] Marc Macenko, Marc Niethammer, J. S. Marron, David Borland, John T. Woosley, Xiaojun Guan, Charles Schmitt, and Nancy E. Thomas.  A method for normalizing histology slides for quantitative analysis.  pages 1107–1110, 2009.
  * [22] Andrew Janowczyk, Ajay Basavanhally, and Anant Madabhushi.  Stain normalization using sparse autoencoders (stanosa): application to digital pathology.  Computerized Medical Imaging and Graphics, 57:50–61, 2017.
  * [23] Hyungjoo Cho, Sungbin Lim, Gunho Choi, and Hyunseok Min.  Neural stain-style transfer learning using gan for histopathological images.  arXiv preprint arXiv:1710.08543, 2017.
  * [24] Yushan Zheng, Zhiguo Jiang, Haopeng Zhang, Fengying Xie, Dingyi Hu, Shujiao Sun, Jun Shi, and Chenghai Xue.  Stain standardization capsule for application-driven histopathological image normalization.  IEEE journal of biomedical and health informatics, 25(2):337–347, 2020.
  * [25] David Tellez, Geert Litjens, Péter Bándi, Wouter Bulten, John-Melle Bokhorst, Francesco Ciompi, and Jeroen van der Laak.  Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology.  Medical image analysis, 58:101544, 2019.
  * [26] Nicola Michielli, Alessandro Caputo, Manuela Scotto, Alessandro Mogetta, Orazio Antonino Maria Pennisi, Filippo Molinari, Davide Balmativola, Martino Bosco, Alessandro Gambella, Jasna Metovic, et al.  Stain normalization in digital pathology: Clinical multi-center evaluation of image quality.  Journal of Pathology Informatics, page 100145, 2022.
  * [27] Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guillaume Lajoie.  Gradient starvation: A learning proclivity in neural networks.  Advances in Neural Information Processing Systems, 34, 2021.
  * [28] David Tellez, Maschenka Balkenhol, Irene Otte-Höller, Rob van de Loo, Rob Vogels, Peter Bult, Carla Wauters, Willem Vreuls, Suzanne Mol, Nico Karssemeijer, Geert Litjens, Jeroen van der Laak, and Francesco Ciompi.  Whole-slide mitosis detection in h amp;e breast histology using phh3 as a reference to train distilled stain-invariant convolutional networks.  IEEE Transactions on Medical Imaging, 37(9):2126–2136, 2018.
  * [29] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.  Autoaugment: Learning augmentation strategies from data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 113–123, 2019.
  * [30] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702–703, 2020.
  * [31] Samuel G Müller and Frank Hutter.  Trivialaugment: Tuning-free yet state-of-the-art data augmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 774–782, 2021.
  * [32] Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, and Barret Zoph.  Revisiting resnets: Improved training and scaling strategies.  Advances in Neural Information Processing Systems, 34, 2021.
  * [33] Alexander Buslaev, Vladimir I Iglovikov, Eugene Khvedchenya, Alex Parinov, Mikhail Druzhinin, and Alexandr A Kalinin.  Albumentations: fast and flexible image augmentations.  Information, 11(2):125, 2020.
  * [34] Sébastien Marcel and Yann Rodriguez.  Torchvision the machine-vision package of torch.  In Proceedings of the 18th ACM international conference on Multimedia, pages 1485–1488, 2010.
  * [35] Khrystyna Faryna, Jeroen van der Laak, and Geert Litjens.  Tailoring automated data augmentation to h&e-stained histopathology.  In Medical Imaging with Deep Learning, 2021.
  * [36] Tom Ching LingChen, Ava Khonsari, Amirreza Lashkari, Mina Rafi Nazari, Jaspreet Singh Sambee, and Mario A Nascimento.  Uniformaugment: A search-free probabilistic data augmentation approach.  arXiv preprint arXiv:2003.14348, 2020.
  * [37] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958, 2014\.
  * [38] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger.  Deep networks with stochastic depth.  In European conference on computer vision, pages 646–661. Springer, 2016.
  * [39] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  arXiv preprint arXiv:1710.09412, 2017.
  * [40] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023–6032, 2019.
  * [41] Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi.  Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks.  In International Conference on Machine Learning, pages 5905–5914. PMLR, 2021.
  * [42] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [43] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  * [44] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [45] Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * [46] Wouter Bulten, Kimmo Kartasalo, Po-Hsuan Cameron Chen, Peter Ström, Hans Pinckaers, Kunal Nagpal, Yuannan Cai, David F Steiner, Hester van Boven, Robert Vink, et al.  Artificial intelligence for diagnosis and gleason grading of prostate cancer: the panda challenge.  Nature medicine, pages 1–10, 2022.
  * [47] Wouter Bulten, Péter Bándi, Jeffrey Hoven, Rob van de Loo, Johannes Lotz, Nick Weiss, Jeroen van der Laak, Bram van Ginneken, Christina Hulsbergen-van de Kaa, and Geert Litjens.  Peso: Prostate epithelium segmentation on h&e-stained prostatectomy whole slide images, 2018.  Accessed: 05.03.2021.
  * [48] Jonathan I Epstein, Lars Egevad, Mahul B Amin, Brett Delahunt, John R Srigley, and Peter A Humphrey.  The 2014 international society of urological pathology (isup) consensus conference on gleason grading of prostatic carcinoma.  The American journal of surgical pathology, 40(2):244–252, 2016\.
  * [49] Davood Karimi, Guy Nir, Ladan Fazli, Peter C Black, Larry Goldenberg, and Septimiu E Salcudean.  Deep learning-based gleason grading of prostate cancer from histopathology images—role of multiscale decision aggregation and data augmentation.  IEEE journal of biomedical and health informatics, 24(5):1413–1426, 2019.
  * [50] Joona Pohjonen and Valeria Ariotta.  Histoprep: Preprocessing large medical images for machine learning made easy!  https://github.com/jopo666/HistoPrep, 2021.
  * [51] David H Wolpert and William G Macready.  No free lunch theorems for optimization.  IEEE transactions on evolutionary computation, 1(1):67–82, 1997\.
  * [52] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.

## Supplementary Figures

Figure S1: Examples of augmented images using StrongAugment with $p=0.4$,
RandAugment with $m=10$ and $n=2$, RandAugment-HE with $m=5$ and $n=3$ and
TrivialAugment. Images augmented with StrongAugment contain significantly more
variability than any of the other methods.

Figure S2: Same region of interest from sample 1 serial sections. There is
considerable variation in the scanned images due to variability between
medical centres, slide glasses and scanners. Nevertheless, neural networks
intended for clinical practice should achieve consistent performance on each
clinical validation case.

Figure S3: Robustness of neural networks, trained on the Helsinki30 dataset
with weight decay instead of spectral decoupling, to distribution shifts
caused by differences in the haematoxylin and eosin stain intensities. Each
cell represents a given distribution-shifted dataset
$\textit{stain}(X_{\text{PESO}},h,e)$, where $h$ and $e$ denote the magnitudes
for haematoxylin and eosin stains, respectively. Cells are annotated with mean
AUROC $\times$ 100 value for the five trained networks, rounded to the closest
integer. Even without spectral decoupling, StrongAugment achieves consistent
results for each dataset. Networks trained with RandAugment or TrivialAugment
are more dependent on spectral decoupling and achieve up to 25 percentage
points lower AUROC than in Figure 5. Figure S4: Robustness of neural networks,
trained on the HelsinkiRCC dataset with either StrongAugment, RandAugment, or
TrivialAugment, to distribution shifts caused by differences in the
haematoxylin and eosin stain intensities. Each cell represents a given
distribution-shifted dataset $\textit{stain}(X_{\text{val}},h,e)$, where $h$
and $e$ denote the magnitudes for haematoxylin and eosin stains, respectively,
and $x_{\text{val}}$ the validation split of HelsinkiRCC dataset. Cells are
annotated with mean AUROC $\times$ 100 value for the five trained networks,
rounded to the closest integer. Neural networks trained with StrongAugment
retain performance on all datasets, even when networks trained with
RandAugment or TrivialAugment quickly degrade to no better than random
discrimination.