# Private, fair and accurate: Training large-scale, privacy-preserving AI
models in medical imaging

Soroosh Tayebi Arasteh  Department of Diagnostic and Interventional Radiology,
University Hospital RWTH Aachen, Aachen, Germany.  These authors contributed
equally to this work Alexander Ziller  Institute of Diagnostic and
Interventional Radiology, Technical University of Munich, Munich, Germany.
Artificial Intelligence in Healthcare and Medicine, Technical University of
Munich, Munich, Germany.  These authors contributed equally to this work
Christiane Kuhl  Department of Diagnostic and Interventional Radiology,
University Hospital RWTH Aachen, Aachen, Germany.  Marcus Makowski  Institute
of Diagnostic and Interventional Radiology, Technical University of Munich,
Munich, Germany.  Sven Nebelung  Department of Diagnostic and Interventional
Radiology, University Hospital RWTH Aachen, Aachen, Germany.  Rickmer Braren
Institute of Diagnostic and Interventional Radiology, Technical University of
Munich, Munich, Germany.  Daniel Rueckert  Artificial Intelligence in
Healthcare and Medicine, Technical University of Munich, Munich, Germany.
Daniel Truhn  Department of Diagnostic and Interventional Radiology,
University Hospital RWTH Aachen, Aachen, Germany.  These authors contributed
equally to this work Georgios Kaissis  Institute of Diagnostic and
Interventional Radiology, Technical University of Munich, Munich, Germany.
Artificial Intelligence in Healthcare and Medicine, Technical University of
Munich, Munich, Germany.  Department of Computing, Imperial College London,
London, United Kingdom.  Institute for Machine Learning in Biomedical Imaging,
Helmholtz-Zentrum Munich, Neuherberg, Germany.  These authors contributed
equally to this work<EMAIL_ADDRESS>
###### Abstract

Artificial intelligence (AI) models are increasingly used in the medical
domain. However, as medical data is highly sensitive, special precautions to
ensure its protection are required. The gold standard for privacy preservation
is the introduction of differential privacy (DP) to model training. Prior work
indicates that DP has negative implications on model accuracy and fairness,
which are unacceptable in medicine and represent a main barrier to the
widespread use of privacy-preserving techniques. In this work, we evaluated
the effect of privacy-preserving training of AI models for chest radiograph
diagnosis regarding accuracy and fairness compared to non-private training.
For this, we used a large dataset ($N=193\,311$) of high quality clinical
chest radiographs, which were retrospectively collected and manually labeled
by experienced radiologists. We then compared non-private deep convolutional
neural networks (CNNs) and privacy-preserving (DP) models with respect to
privacy-utility trade-offs measured as area under the receiver-operator-
characteristic curve (AUROC), and privacy-fairness trade-offs, measured as
Pearson’s r or Statistical Parity Difference. We found that the non-private
CNNs achieved an average AUROC score of $0.90\pm 0.04$ over all labels,
whereas the DP CNNs with a privacy budget of $\varepsilon=7.89$ resulted in an
AUROC of $0.87\pm 0.04$, i.e., a mere $2.6\%$ performance decrease compared to
non-private training. Furthermore, we found the privacy-preserving training
not to amplify discrimination against age, sex or co-morbidity. Our study
shows that –under the challenging realistic circumstances of a real-life
clinical dataset– the privacy-preserving training of diagnostic deep learning
models is possible with excellent diagnostic accuracy and fairness.

##  1 Introduction

The development of artificial intelligence (AI) systems for medical
applications represents a delicate trade-off: On the one hand, diagnostic
models must offer high accuracy and certainty, as well as treat different
patient groups equitably and fairly. On the other hand, clinicians and
researchers are subject to ethical and legal responsibilities towards the
patients whose data is used for model training. In particular, when diagnostic
models are published to third parties whose intentions are impossible to
verify, care must be undertaken to ascertain that patient privacy is not
compromised. Privacy breaches can occur, e.g. through data reconstruction,
attribute inference or membership inference attacks against the shared model
[1]. Federated learning [2, 3, 4] has been proposed as a tool to address some
of these problems. However, it has become evident that training data can be
reverse-engineered piecemeal from federated systems, rendering them just as
vulnerable to the aforementioned attacks as centralized learning [5]. Thus, it
is apparent that formal privacy preservation methods are required to protect
the patients whose data is used to train diagnostic AI models. The gold
standard in this regard is differential privacy (DP) [6].

DP is a formal framework encompassing a collection of techniques to allow
analysts to obtain insights from sensitive datasets while guaranteeing the
protection of individual data points within them. DP thus is a property of a
data processing system which states that the results of a computation over a
sensitive dataset must be approximately identical whether or not any single
individual was included or excluded from the dataset. Formally, a randomised
algorithm (mechanism) $\mathcal{M}:\mathcal{X}\rightarrow\mathcal{Y}$ is said
to satisfy $(\varepsilon,\delta)$-DP if, for all pairs of databases
$D,D^{\prime}\in\mathcal{X}$ which differ in one row and all
$S\subseteq\mathcal{Y}$, the following holds:

$\text{Pr}\left(\mathcal{M}(D)\in S\right)\leq
e^{\varepsilon}\text{Pr}\left(\mathcal{M}(D^{\prime})\in S\right)+\delta,$ (1)

where the guarantee is given over the randomness of $\mathcal{M}$ and holds
equally when $D$ and $D^{\prime}$ are swapped.

Applied to neural network training, the randomization required by DP is
ensured through the addition of calibrated Gaussian noise to the gradients of
the loss function computed for each individual data point after they have been
clipped in $\ell_{2}$-norm to ensure that their magnitude is bounded [7] (see
Figure 1). By specifying the noise variance and the number of training steps,
it is possible to summarize the total privacy expenditure, intuitively, the
amount of information that has flown from the input data to the model in the
form of the $\varepsilon$ and $\delta$-values introduced above, which denote
the so-called privacy budget. Stronger privacy guarantees are denoted by
smaller values of $\varepsilon$ and $\delta$. The fact that quantitative
privacy guarantees can be computed over many iterations (compositions) of
complex algorithms like the ones used to train neural networks is unique to
DP. This process is typically referred to as privacy accounting.

Figure 1: Differences between the private and non-private training process of
a neural network. (A) Images from a dataset are fed to a neural network and
predictions are made. (B) From the predictions and the ground truth labels,
the gradient is calculated via backpropagation. (C, upper panel) In normal
training all gradients are averaged and an update step is performed. (C, lower
panel) In private training, each per-sample gradient is clipped to a
predetermined $\ell_{2}$-norm, averaged and noise proportional to the norm is
added. This ensures that the information about each sample is upper-bounded
and perturbed with sufficient noise.

Although training with DP offers formal (and empirical) protection against
both membership inference and reconstruction attacks [8], whose strength is
directly proportional to the chosen privacy level, the utilization of DP also
creates two fundamental trade-offs. The first is a privacy-utility trade-off,
i.e., a reduction in diagnostic accuracy when stronger privacy guarantees are
required [9, 10]. The other trade-off is between privacy and fairness.
Intuitively, the fact that AI models learn proportionally less about under-
represented patient groups [11] in the training data is amplified by DP (which
further limits how much information flows about them), leading to demographic
disparity in the model’s predictions or diagnoses [12]. Both of these trade-
offs are delicate in sensitive applications, such as medical ones, as it is
not acceptable to have wrong diagnoses or discriminate against a certain
patient group.

The aforementioned considerations outline a fundamental tension between
accuracy, fairness and privacy which exists in the training of differentially
private models for medical applications. So far, these trade-offs have only
been evaluated in benchmark datasets, such as CIFAR-10 or ImageNet. We thus
contend that the widespread use of privacy-preserving machine learning
requires testing under real-life circumstances. In the current study, we
perform the first in-depth investigation into this topic. Concretely, we
utilize a large clinical database of radiologist-labelled radiographic images
which has previously been used to train an expert-level diagnostic AI model,
but otherwise not been curated or pre-processed for private training in any
way. This mirrors the type of datasets available at clinical institutions. In
this setting, we then study the extent of privacy-utility and privacy-fairness
trade-offs in training advanced computer vision architectures. Our main
contributions can be summarized as follows:

  1. 1.

We study the diagnostic accuracy ramifications of differentially private deep
learning in multi-label classification of a large, curated database of
intensive care unit chest radiographs. We find the accuracy reductions to be
negligible compared to non-private training through the utilization of
transfer learning on public datasets and careful choice of architecture.

  2. 2.

We investigate the fairness implications of differentially private learning
with respect to key demographic characteristics such as sex, age and co-
morbidity. We find that – while differentially private learning has a mild
fairness effect– it does not introduce significant discrimination concerns
compared to non-private training.

##### Prior work

The training of deep neural networks on medical data with differential privacy
(DP) has so far not been widely investigated. Pati et al. [13] use privacy-
preserving techniques, most notably federated learning and differential
privacy to comply with privacy legislation and thus allow training on a
dataset of ca. $6\,000$ multi-parametric magnetic resonance imaging scans. The
authors show that –for this use case– privacy preservation incentivises data
sharing and thus makes large datasets available. However, they do not
investigate privacy-utility or privacy-fairness trade-offs. In our previous
work [14] we demonstrated the utilization of a suite of privacy-preserving
techniques for pneumonia classification in pediatric chest x-rays. However,
the focus of this study was not to elucidate privacy-utility or privacy-
fairness trade-offs, but to showcase that federated learning workflows can be
used to train diagnostic AI models with good (yet diminished compared to the
non-private and centralized setting) accuracy on decentralized data while
minimizing data privacy and governance concerns; the authors demonstrate this
using empirical data reconstruction attacks, which are thwarted by the
utilization of differential privacy. Moreover, the work did not consider
differential diagnosis but only coarse-label classification into
normal/bacterial/viral pneumonia.

To the best of our knowledge, our study is the first work to investigate the
use of differential privacy in the training of complex diagnostic AI models on
a real-world dataset of this magnitude (nearly $200\,000$ samples) and to
include an extensive evaluation of privacy-utility and privacy-fairness trade-
offs.

Our results are of interest to medical practitioners, deep learning experts in
the medical field and regulatory bodies such as legislative institutions,
institutional review boards and data protection officers and we undertook
specific care to formulate our main lines of investigation across the
important axes delineated above, namely the provision of objective metrics of
diagnostic accuracy, privacy protection and demographic fairness towards
diverse patient subgroups.

##  2 Materials and Methods

###  2.1 Patient Cohort

We employed UKA-CXR [15, 16], a large cohort of chest radiographs. The dataset
consists of $N=193\,311$ frontal CXR images, all manually labeled by
radiologists. The available labels include: pleural effusion, pneumonic
infiltrates, and atelectasis, each separately for right and left lung,
congestion, and cardiomegaly. The labeling system for cardiomegaly included
five classes normal, uncertain, borderline, enlarged, and massively enlarged.
For the rest of the labels, five classes of negative, uncertain, mild,
moderate, and severe were used. Data were split into $N=153\,502$ training and
$N=39\,809$ test images using patient-wise stratification, but otherwise
completely random allocation [15, 16]. There was no overlap between the
training and test sets. Table 1 shows the statistics of the dataset.

| Training Set | Test Set | All
---|---|---|---
| N | percentage | N | percentage | N | percentage
Total | 153,502 |  | 39,809 |  | 193,311 |
Female | 52,843 | (34.42%) | 14,449 | (36.30%) | 67,292 | (34.81%)
Male | 100,659 | (65.58%) | 25,360 | (63.70%) | 126,019 | (65.19%)
Aged [0, 30) | 4,279 | (2.79%) | 1,165 | (2.93%) | 5,444 | (2.82%)
Aged [30, 60) | 42,340 | (27.58%) | 10,291 | (25.85%) | 52,631 | (27.23%)
Aged [60, 70) | 36,882 | (24.03%) | 10,025 | (25.18%) | 46,907 | (24.27%)
Aged [70, 80) | 48,864 | (31.83%) | 12,958 | (32.55%) | 61,822 | (31.98%)
Aged [80, 100) | 21,137 | (13.77%) | 5,370 | (13.49%) | 26,507 | (13.71%)
Cardiomegaly | 71,732 | (46.72%) | 18,616 | (46.75%) | 90,348 | (46.74%)
Congestion | 13,096 | (8.53%) | 3,275 | (8.22%) | 16,371 | (8.47%)
Pleural effusion right | 12,334 | (8.03%) | 3,275 | (8.22%) | 15,609 | (8.07%)
Pleural effusion left | 9,969 | (6.49%) | 2,602 | (6.53%) | 12,571 | (6.50%)
Pneumonic infiltration right | 17,666 | (11.51%) | 4,847 | (12.17%) | 22,513 | (11.64%)
Pneumonic infiltration left | 12,431 | (8.10%) | 3,562 | (8.94%) | 15,993 | (8.27%)
Atelectasis right | 14,841 | (9.67%) | 3,920 | (9.84%) | 18,761 | (9.71%)
Atelectasis left | 11,916 | (7.76%) | 3,166 | (7.95%) | 15,082 | (7.80%)
| Age Training Set | Age Test Set | Age All
| Mean | StD | Mean | StD | Mean | StD
Total | 66 | 15 | 66 | 15 | 66 | 15
Female | 66 | 15 | 66 | 16 | 66 | 15
Male | 65 | 14 | 66 | 14 | 65 | 14
Aged [0, 30) | 21 | 8 | 21 | 8 | 21 | 8
Aged [30, 60) | 50 | 8 | 51 | 8 | 51 | 8
Aged [60, 70) | 65 | 3 | 65 | 3 | 65 | 3
Aged [70, 80) | 75 | 3 | 75 | 3 | 75 | 3
Aged [80, 100) | 84 | 3 | 84 | 3 | 84 | 3
Table 1: Statistics over subgroups of the UKA-CXR dataset used in this study.
The upper part of the table shows the number of samples in each group and
their relative share in training and test set, as well as the complete
dataset. The lower part shows the mean and standard deviation of the age in
the subgroups again over training and test set as well as the complete
dataset.

###  2.2 Data Pre-processing

All the images were resized to ($512\times 512$) pixels. Afterward, a
normalization scheme as described previously by Johnson et al. [17] was
utilized by subtracting the lowest value in the image, dividing by the highest
value in the shifted image, truncating values, and converting the result to an
unsigned integer, i.e., the range of $[0,255]$. Finally, we performed
histogram equalization by shifting pixel values towards 0 or towards 255 such
that all pixel values $0$ through $255$ have approximately equal frequencies
[17].

We selected a binary classification paradigm for each label. The negative and
uncertain classes (normal and uncertain for cardiomegaly) were treated as
negative, while the mild, moderate, and severe classes (borderline, enlarged,
and massively enlarged for cardiomegaly) were treated as positive.

###  2.3 Deep Learning Process

####  2.3.1 Network Architecture

We employed the ResNet9 architecture introduced in [18] as our classification
architecture. The images were expanded to ($512\times 512\times 3$) for
compatibility with the neural network architecture. The final linear layer
reduces the ($512\times 1$) output feature vectors to the desired number of
diseases to be predicted, i.e., $8$. The sigmoid function was utilized to
convert the output predictions to individual class probabilities. The full
network contained a total of $4.9$ million trainable parameters. Our utilized
ResNet9 network employs the modifications proposed by Klause et al. [18] and
by He et al. [19] Instead of the batch normalization [20] layers, we used
group normalization [21] layers with groups of $32$ to be compatible with DP
processing. We pretrained the network on the MIMIC Chest X-ray JPG dataset
v2.0.0, [22] consisting of $N=210\,652$ frontal images. All training
hyperparameters were selected empirically based on their validation accuracy,
while no systematic/automated hyperparameter tuning was conducted.

####  2.3.2 Non-DP Training

The Rectified Linear Unit (ReLU) [23] was chosen as the activation function in
all layers. We performed data augmentation during training by applying random
rotation in the range of $[-10,10]$ degrees and medio-lateral flipping with a
probability of $0.50$. The model was optimized using the NAdam [24] optimizer
with a learning rate of $5\cdot 10^{-5}$. The binary weighted cross-entropy
with inverted class frequencies of the training data was selected as the loss
function. The training batch size was chosen to be $128$.

####  2.3.3 DP Training

Mish [25] was chosen as the activation function in all layers. No data
augmentation was performed during DP training as we found further data
augmentation during training to be harmful to accuracy. All models were
optimized using the NAdam [24] optimizer with a learning rate of $5\cdot
10^{-4}$. The binary weighted cross-entropy with inverted class frequencies of
the training data was selected as the loss function. The maximum allowed
gradient norm was chosen to be $1.5$ and the network was trained for $150$
epochs for each chosen privacy budget. Each point in the batch was sampled
with a probability of $8\cdot 10^{-4}$ ($128$ divided by $N=153,502$).

###  2.4 Quantitative Evaluation and Statistical Analysis

The area under the receiver-operator-characteristic curve (AUROC) was utilized
as the primary evaluation metric. We report the average AUROC over all the
labels for each experiment. The individual AUROC as well as all other
evaluation metrics of individual labels are reported in the supplemental
material (Tables 4–10). Bootstrapping was used with $1\,000$ redraws for each
measure to determine the statistical spread [26]. For calculating sensitivity,
specificity and accuracy, a threshold was chosen according to Youden’s
criterion [27], i.e., the threshold that maximized (true positive rate - false
positive rate).

To evaluate correlation between results of data subsets and their sample size
Pearson’s r coefficient was used. To analyze fairness between subgroups, the
statistical parity difference [28] was used which is defined as
$P(\hat{Y}=1|C=\mathrm{Minority})-P(\hat{Y}=1|C=\mathrm{Majority})$ where
$\hat{Y}=1$ represents correct model predictions and C is the group in
question. Intuitively, it is the difference in classification accuracy between
the minority and majority class and thus is optimally zero. Values larger than
zero mean that there is a benefit for the minority class, while values smaller
than zero mean that the minority class is discriminated against.

##  3 Results

###  3.1 High classification accuracy is attainable despite stringent privacy
guarantees

Table 2 shows the detailed evaluation results for non-private and private (at
$\varepsilon=7.89$) model training. In the case of non-private training, our
model achieves an AUROC of $0.90$ over all diagnoses. It performs best on
pneumonic infiltration on the right (AUROC=$0.94$) while struggling the most
to accurately classify cardiomegaly (AUROC=$0.84$). Training with DP decreases
all results slightly and achieves an overall AUROC of $0.87$. The per-
diagnosis performance ranges from $0.92$ (pleural effusion right) to $0.81$
AUROC (congestion). We next consider classification performance at a very
strong level of privacy protection (i.e. at $\varepsilon<1$). Here, at an
$\varepsilon$-budget of only $0.29$, our model achieves an average AUROC of
$0.83$ over all diagnoses. A visual overview is displayed in Figure 2, which
shows the average AUROC, accuracy, sensitivity, and specificity values over
all labels. Supplementary Tables 4–10 show the per-diagnosis evaluation
results for non-DP and DP training for different $\varepsilon$ values.

$\varepsilon$ | $0.29$ | $0.54$ | $1.06$ | $2.04$ | $4.71$ | $7.89$ | Non-private ($\infty$)
---|---|---|---|---|---|---|---
CDM | 0.79 ± 0.00 | 0.79 ± 0.00 | 0.80 ± 0.00 | 0.81 ± 0.00 | 0.81 ± 0.00 | 0.82 ± 0.00 | 0.84 ± 0.00
CNG | 0.78 ± 0.00 | 0.79 ± 0.00 | 0.80 ± 0.00 | 0.80 ± 0.00 | 0.81 ± 0.00 | 0.81 ± 0.00 | 0.85 ± 0.00
PER | 0.88 ± 0.00 | 0.89 ± 0.00 | 0.90 ± 0.00 | 0.90 ± 0.00 | 0.92 ± 0.00 | 0.92 ± 0.00 | 0.94 ± 0.00
PEL | 0.84 ± 0.00 | 0.84 ± 0.00 | 0.86 ± 0.00 | 0.87 ± 0.00 | 0.89 ± 0.00 | 0.89 ± 0.00 | 0.92 ± 0.00
PIR | 0.87 ± 0.00 | 0.88 ± 0.00 | 0.89 ± 0.00 | 0.90 ± 0.00 | 0.90 ± 0.00 | 0.91 ± 0.00 | 0.93 ± 0.00
PIL | 0.88 ± 0.00 | 0.88 ± 0.00 | 0.89 ± 0.00 | 0.90 ± 0.00 | 0.91 ± 0.00 | 0.91 ± 0.00 | 0.94 ± 0.00
ALR | 0.82 ± 0.00 | 0.83 ± 0.00 | 0.84 ± 0.00 | 0.85 ± 0.00 | 0.86 ± 0.00 | 0.87 ± 0.00 | 0.89 ± 0.00
ALL | 0.80 ± 0.00 | 0.81 ± 0.00 | 0.82 ± 0.00 | 0.83 ± 0.00 | 0.85 ± 0.00 | 0.85 ± 0.00 | 0.87 ± 0.00
Average | 0.83 ± 0.04 | 0.84 ± 0.04 | 0.85 ± 0.04 | 0.86 ± 0.04 | 0.87 ± 0.04 | 0.87 ± 0.04 | 0.90 ± 0.04
Table 2: Evaluation results of training with DP and without DP with different
$\varepsilon$ values for $\delta=6\cdot 10^{-6}$. The results show the
individual AUROC values for each label, including cardiomegaly (CDM),
congestion (CNG), pleural effusion right(PER), pleural effusion left (PEL),
pneumonic infiltration right (PIR), pneumonic infiltration left (PIL),
atelectasis right (ALR), and atelectasis left (ALL) tested on $N=39\,809$ test
images. The training dataset includes $N=153,502$ images. Figure 2: Average
results of training with DP with different $\varepsilon$ values for
$\delta=6\cdot 10^{-6}$. The curves show the average (A) area-under-the-
receiver-operator-curve (AUROC), (B) accuracy, (C) specificity, and (D)
sensitivity values over all labels, including cardiomegaly, congestion,
pleural effusion right, pleural effusion left, pneumonic infiltration right,
pneumonic infiltration left, atelectasis right, and atelectasis left tested on
$N=39\,809$ test images. The training dataset includes $N=153\,502$ images.
Note, that the AUROC is continuously increasing, while sensitivity,
specificity and accuracy exhibit more variation. This is due to the fact that
all training processes were optimized for the AUROC.

###  3.2 Diagnostic accuracy is correlated with patient age and sample size
for both private and non-private models

Table 2 shows the difference in classification performance for each diagnosis
between the non-private model evaluation and its private counterpart compared
to the sample size (that is, the number of available samples with a given
label) within our dataset. At an $\varepsilon=7.89$, the largest difference of
AUROC between the non-private and privacy-preserving model was observed for
congestion ($3.82\%$) and the smallest difference was observed for pleural
effusion right ($1.55\%$, see Table 2). Of note, there is a visible trend
(Pearson’s r: $0.44$) that classes where the model exhibits good diagnostic
performance in the non-private setting also suffer the smallest drop in the
private setting. On the other hand, classes that are already difficult to
predict in the non-private case deteriorate the most in terms of
classification performance with DP (see Figure 3). Both non-private (Pearson’s
r: $0.57$) and private (Pearson’s r: $0.52$) diagnostic AUROC exhibit a weak
correlation to the number of samples available for each class (see Figure 3).
However, the drop in AUROC between private and non-private is not correlated
with the sample size (Pearson’s r: $0.06$).

Figure 3: Relation of sample size to training performance for private and performance loss compared to non private training. Each dot marks the performance on the test set on one diagnosis of the private model at $\varepsilon=7.89$ (compare Table 2). Colors indicate the performance loss compared to the non private model.  Figure 4: Average results of training with DP with different $\varepsilon$ values for $\delta=6\cdot 10^{-6}$, separately for female and male samples. The curves show the average (A) AUROC, (B) accuracy, (C) specificity, and (D) sensitivity values over all labels, including cardiomegaly, congestion, pleural effusion right, pleural effusion left, pneumonic infiltration right, pneumonic infiltration left, atelectasis right, and atelectasis left tested on $N=39\,809$ test images. The training dataset includes $N=153\,502$ images. Note, that the AUROC is continuously increasing, while sensitivity, specificity and accuracy exhibit more variation. This is due to the fact that all training processes were optimized for the AUROC. $\varepsilon$ |  | Age | Patient Sex
---|---|---|---
| [0, 30) | [30, 60) | [60, 70) | [70, 80) | [80, 100) | Female | Male
$\infty$ | Mean | 0.92 | 0.91 | 0.90 | 0.89 | 0.88 | 0.90 | 0.89
StD | 0.04 | 0.03 | 0.04 | 0.04 | 0.04 | 0.04 | 0.04
PtD | 0.04 | 0.01 | 0.00 | 0.00 | -0.03 | 0.00
7.89 | Mean | 0.90 | 0.89 | 0.87 | 0.86 | 0.85 | 0.88 | 0.87
StD | 0.04 | 0.04 | 0.04 | 0.05 | 0.05 | 0.04 | 0.04
PtD | 0.04 | 0.01 | 0.01 | -0.01 | -0.03 | 0.01
4.71 | Mean | 0.89 | 0.89 | 0.87 | 0.86 | 0.85 | 0.87 | 0.87
StD | 0.03 | 0.04 | 0.04 | 0.05 | 0.05 | 0.04 | 0.04
PtD | 0.04 | 0.02 | 0.01 | -0.02 | -0.02 | 0.02
2.04 | Mean | 0.89 | 0.88 | 0.86 | 0.84 | 0.84 | 0.86 | 0.86
StD | 0.03 | 0.04 | 0.04 | 0.04 | 0.04 | 0.04 | 0.04
PtD | 0.06 | 0.02 | 0.01 | -0.03 | -0.02 | 0.00
1.06 | Mean | 0.88 | 0.87 | 0.85 | 0.84 | 0.83 | 0.85 | 0.85
StD | 0.03 | 0.04 | 0.04 | 0.05 | 0.04 | 0.04 | 0.04
PtD | 0.07 | 0.03 | 0.00 | -0.02 | -0.03 | 0.01
0.54 | Mean | 0.86 | 0.86 | 0.84 | 0.83 | 0.82 | 0.85 | 0.84
StD | 0.03 | 0.04 | 0.04 | 0.04 | 0.04 | 0.04 | 0.04
PtD | 0.07 | 0.01 | 0.02 | -0.03 | -0.01 | 0.02
0.29 | Mean | 0.86 | 0.85 | 0.83 | 0.82 | 0.81 | 0.84 | 0.83
StD | 0.04 | 0.04 | 0.04 | 0.04 | 0.04 | 0.04 | 0.04
PtD | 0.07 | 0.01 | 0.01 | -0.02 | -0.02 | 0.00
Table 3: Average evaluation results of training with DP and without DP with
different $\varepsilon$ values, for all age intervals, as well as the patient
sex. The results show the average AUROC values over all labels, tested on
$N=39\,809$ images. Mean is given over AUROC, StD is the standard deviation of
the AUROC, and PtD denotes the statistical parity difference between the
underrepresented class compared to all other patients. Positive values show a
benefit, negative values show discrimination.

Furthermore, we evaluate our models based on age range and patient sex (Table
3). Additionally, we calculate statistical parity difference for those groups
to obtain a measure of fairness (Table 3 and Figure 4). All models perform the
best on patients younger than 30 years of age. It appears that, the older
patients are, the greater the difficulty for the models to predict the labels
accurately. Statistical parity difference scores are slightly negative for the
age groups between 70 and 80 years and older than 80 years for all models,
indicating that the models discriminate slightly against these groups. In
addition, while for the aforementioned age groups the discrimination does not
change with privacy levels, younger patients become more privileged as privacy
increases. This finding indicates that –for models which are most protective
of data privacy– young patients benefit the most, despite the group of younger
patients being smaller overall. For patient sex, models show slightly better
performance for female patients and slightly discriminate against male
patients (Table 3). Statistical parity does not appear to correlate (Pearson’s
r: $0.13$) with privacy levels (Table 3).

##  4 Discussion

The main contribution of our paper is to demonstrate that the training of
highly accurate diagnostic AI models on a large-scale clinical chest
radiography database is possible while furnishing strong objective guarantees
of data privacy and without inducing patient discrimination.

Across all levels of privacy protection, training with DP only resulted in
mild AUROC reductions. The fact that the model maintained an AUROC of $0.83$
even at $\varepsilon=0.29$ is remarkable, and we are unaware of any prior work
to report such a strong level of privacy protection at this level of model
accuracy on clinical data. Our results thus exemplify that, through the use of
model pretraining on a related public dataset, specialized architecture
designs, and the availability of sufficient data samples, privately trained
models require only very small additional amounts of private information from
the training dataset to achieve high diagnostic accuracy on the tasks at hand.

Our analysis of the per-diagnosis performance of models that are trained with
and without privacy guarantees shows that models discriminate against
diagnoses that are underrepresented in the training set in both private and
non-private training. This finding is not unusual and several examples can be
found in [29]. However, the drop in performance between private and non-
private training is uncorrelated to the sample size. Instead, the difficulty
of the diagnosis seems to drive the difference in AUROC between the two
settings. Concretely, diagnostic performance under privacy constraints suffers
the most for those classes, which already have the lowest AUROC in the non-
private setting. Conversely, diagnoses that are predicted with the highest
AUROC suffer the least when DP is introduced.

Previous works investigating the effect of DP on fairness show that privacy
preservation amplifies discrimination [30]. This effect is very limited in our
study. Our models remain fair despite strong privacy guarantees, likely due to
our real-life dataset’s large size and high quality (whereas previous works
limited their scope to toy datasets). Moreover, our use of pre-training helps
to boost model performance and reduce the amount of additional information the
model needs to learn from scratch, which seems to benefit under-represented
groups in the dataset the most.

Our analysis of fairness related to patient age showed that older patients
(older than 70 years of age) are discriminated against both in the non-private
and the private setting, with discrimination against them remaining
approximately constant with stronger privacy guarantees. On the other hand,
patients below 30 years of age suffer overall lower model discrimination in
the non-private and the private setting. Interestingly, young patients seem to
profit more from stronger privacy guarantees, as they enjoy progressively more
fairness privilege with increasing privacy protection level. This holds
despite the fact that patients under 30 represent the smallest fraction of the
dataset. This effect is most likely due to a confounding variable, namely the
lower complexity of imaging findings in younger patients due to their improved
ability to cooperate during radiograph acquisition, resulting in a better
discrimination of the pathological finding on a more homogenous background
(i.e. cleaner) radiographs which are easier to diagnose overall [15, 31] (see
Figure 5). This hypothesis should be validated in cohorts with a larger
proportion of young patients, and we intend to expand on this finding in
future work. The analysis of model fairness related to patient sex shows that
female patients (which –similar to young patients– are an underrepresented
group) enjoy a slightly higher diagnostic accuracy than male patients for
almost all privacy levels. However, effect size differences were found to be
small, so that this finding can also be explained by variability between
models or by the randomness in the training process. Further investigation is
thus required to elucidate the aforementioned effects.

Figure 5: Exemplary radiographs from the utilized dataset. All examinations
share the diagnosis of pneumonic infiltrates on the right patient side (=left
image side). However, diagnosis in older patients is often more challenging
due to the more frequent presence of comorbidities and less cooperation during
image acquisition which results in lower image quality (A) 76-year-old male
patient, note the presence of a cardiac pacemaker that projects over part of
the left lung. (B) 74-year-old male patient with challenging image
acquisition: part of the lower right lung is not properly depicted. (C)
39-year-old male patient, the lungs are well inflated and pneumonic
infiltrates can be discerned even though they are less severe. (D) 33-year-old
male patient with challenging image acquisition, yet both lungs can be
assessed (almost) completely.

In conclusion, we analyzed the usage of privacy-preserving neural network
training and its implications on utility and fairness for a relevant
diagnostic task on a large real-world dataset. We showed that the utilization
of specialized architectures and targeted model pre-training allows for high
model accuracy despite stringent privacy guarantees. This enables us to train
expert-level diagnostic AI models even with privacy budgets as low as
$\varepsilon<1$, which – to our knowledge– has not been shown before, and
represents an important step towards the widespread utilization of
differentially private models in radiological diagnostic AI applications.
Moreover, our finding that the introduction of differential privacy mechanisms
to model training does not amplify unfair model bias regarding patient age,
sex or comorbidity signifies that –at least in our use case– the resulting
models abide by important non-discrimination principles of ethical AI. We are
hopeful that our findings will encourage practitioners and clinicians to
introduce advanced privacy-preserving techniques such as differential privacy
when training diagnostic AI models.

## References

  * [1] Dmitrii Usynin, Alexander Ziller, Marcus Makowski, Rickmer Braren, Daniel Rueckert, Ben Glocker, Georgios Kaissis, and Jonathan Passerat-Palmbach.  Adversarial interference and its mitigations in privacy-preserving collaborative machine learning.  Nature Machine Intelligence, 3(9):749–758, 2021.
  * [2] Jakub Konečnỳ, H Brendan McMahan, Daniel Ramage, and Peter Richtárik.  Federated optimization: Distributed machine learning for on-device intelligence.  arXiv preprint arXiv:1610.02527, 2016.
  * [3] Jakub Konečnỳ, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon.  Federated learning: Strategies for improving communication efficiency.  arXiv preprint arXiv:1610.05492, 2016.
  * [4] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
  * [5] Daniel Truhn, Soroosh Tayebi Arasteh, Oliver Lester Saldanha, Gustav Müller-Franzes, Firas Khader, Philip Quirke, Nicholas P West, Richard Gray, Gordon GA Hutchins, Jacqueline A James, et al.  Encrypted federated learning for secure decentralized collaboration in cancer image analysis.  medRxiv, pages 2022–07, 2022.
  * [6] Cynthia Dwork, Aaron Roth, et al.  The algorithmic foundations of differential privacy.  Found. Trends Theor. Comput. Sci., 9(3-4):211–407, 2014.
  * [7] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang.  Deep learning with differential privacy.  In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318, 2016.
  * [8] Borja Balle, Giovanni Cherubin, and Jamie Hayes.  Reconstructing training data with informed adversaries.  In 2022 IEEE Symposium on Security and Privacy (SP), pages 1138–1156. IEEE, 2022.
  * [9] Soham De, Leonard Berrada, Jamie Hayes, Samuel L Smith, and Borja Balle.  Unlocking high-accuracy differentially private image classification through scale.  arXiv preprint arXiv:2204.13650, 2022.
  * [10] Alexey Kurakin, Steve Chien, Shuang Song, Roxana Geambasu, Andreas Terzis, and Abhradeep Thakurta.  Toward training at imagenet scale with differential privacy.  arXiv preprint arXiv:2201.12328, 2022.
  * [11] Cuong Tran, Ferdinando Fioretto, Pascal Van Hentenryck, and Zhiyan Yao.  Decision making with differential privacy under a fairness lens.  In IJCAI, pages 560–566, 2021.
  * [12] Rachel Cummings, Varun Gupta, Dhamma Kimpara, and Jamie Morgenstern.  On the compatibility of privacy and fairness.  In Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization, pages 309–315, 2019.
  * [13] Sarthak Pati, Ujjwal Baid, Brandon Edwards, Micah Sheller, Shih-Han Wang, G Anthony Reina, Patrick Foley, Alexey Gruzdev, Deepthi Karkada, Christos Davatzikos, et al.  Federated learning enables big data for rare cancer boundary detection.  Nature communications, 13(1):7346, 2022.
  * [14] Georgios Kaissis, Alexander Ziller, Jonathan Passerat-Palmbach, Théo Ryffel, Dmitrii Usynin, Andrew Trask, Ionésio Lima Jr, Jason Mancuso, Friederike Jungmann, Marc-Matthias Steinborn, et al.  End-to-end privacy preserving deep learning on multi-institutional medical imaging.  Nature Machine Intelligence, 3(6):473–484, 2021.
  * [15] Firas Khader, Tianyu Han, Gustav Müller-Franzes, Luisa Huck, Philipp Schad, Sebastian Keil, Emona Barzakova, Maximilian Schulze-Hagen, Federico Pedersoli, Volkmar Schulz, et al.  Artificial intelligence for clinical interpretation of bedside chest radiographs.  Radiology, page 220510, 2022.
  * [16] Soroosh Tayebi Arasteh, Peter Isfort, Marwin Saehn, Gustav Mueller-Franzes, Firas Khader, Jakob Nikolas Kather, Christiane Kuhl, Sven Nebelung, and Daniel Truhn.  Collaborative training of medical artificial intelligence models with non-uniform labels.  arXiv preprint arXiv:2211.13606, 2022.
  * [17] Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng.  Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports.  Scientific data, 6(1):317, 2019.
  * [18] Helena Klause, Alexander Ziller, Daniel Rueckert, Kerstin Hammernik, and Georgios Kaissis.  Differentially private training of residual networks with scale normalisation.  Theory and Practice of Differential Privacy Workshop, ICML, 2022\.
  * [19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [20] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. pmlr, 2015.
  * [21] Yuxin Wu and Kaiming He.  Group normalization.  In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  * [22] Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng.  Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs.  arXiv preprint arXiv:1901.07042, 2019.
  * [23] Abien Fred Agarap.  Deep learning using rectified linear units (relu).  arXiv preprint arXiv:1803.08375, 2018.
  * [24] Timothy Dozat.  Incorporating nesterov momentum into adam.  2016\.
  * [25] Diganta Misra.  Mish: A self regularized non-monotonic activation function.  arXiv preprint arXiv:1908.08681, 2019.
  * [26] Frank Konietschke and Markus Pauly.  Bootstrapping and permuting paired t-test type statistics.  Statistics and Computing, 24:283–296, 2014.
  * [27] Ilker Unal.  Defining an optimal cut-point value in roc analysis: an alternative approach.  Computational and mathematical methods in medicine, 2017, 2017.
  * [28] Toon Calders and Sicco Verwer.  Three naive bayes approaches for discrimination-free classification.  Data mining and knowledge discovery, 21:277–292, 2010.
  * [29] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan.  A survey on bias and fairness in machine learning.  ACM Computing Surveys (CSUR), 54(6):1–35, 2021.
  * [30] Tom Farrand, Fatemehsadat Mireshghallah, Sahib Singh, and Andrew Trask.  Neither private nor fair: Impact of data imbalance on utility and fairness in differential privacy.  In Proceedings of the 2020 workshop on privacy-preserving machine learning in practice, pages 15–19, 2020.
  * [31] Joy T Wu, Ken CL Wong, Yaniv Gur, Nadeem Ansari, Alexandros Karargyris, Arjun Sharma, Michael Morris, Babak Saboury, Hassan Ahmad, Orest Boyko, et al.  Comparison of chest radiograph interpretations by artificial intelligence algorithm vs radiology residents.  JAMA network open, 3(10):e2022779–e2022779, 2020.
  * [32] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov.  Opacus: User-friendly differential privacy library in pytorch, 2021.

## Ethics Statement

The experiments were performed in accordance with relevant guidelines and
regulations. Approval by the local ethical committee has been granted for this
retrospective study (Reference No. EK 028/19). The institutional review board
did not require informed consent from subjects and/or their legal guardian(s).

## Acknowledgements

This work was (partially) funded / supported by the RACOON network under BMBF
grant number 01KX2021. The funders played no role in the design or execution
of the study.

## Author Contributions Statement

The formal analysis was conducted by STA, AZ, DT, and GK. The original draft
was written by STA and AZ and edited by DT and GK. The experiments were
performed by STA. The software was developed by STA. Statistical analyses were
performed by AZ. DT and GK provided clinical and technical expertise. All
authors read the manuscript and contributed to the interpretation of the
results and agreed to the submission of this paper.

## Competing Interests

The authors declare no competing interests.

## Code and Data Availability

All source codes for training and evaluation of the deep neural networks,
differential privacy, data augmentation, image analysis, and preprocessing are
publicly available at https://github.com/tayebiarasteh/DP_CXR. All code for
the experiments was developed in Python 3.10 using the PyTorch 1.13 framework.
The DP code was developed using Opacus 1.3.0 [32].

The UKA-CXR data is not publicly accessible as it is internal data of patients
of University Hospital RWTH Aachen in Aachen, Germany. Data access can be
granted upon reasonable request to the corresponding author.

## Hardware

The hardware used in our experiments were Intel CPUs with 18 cores and 32 GB
RAM and Nvidia RTX 6000 GPUs with 24 GB of VRAM.

##  Appendix A Tables

| AUROC | Accuracy | Specificity | Sensitivity
---|---|---|---|---
Cardiomegaly | 0.84 ± 0.00 | 0.75 ± 0.00 | 0.71 ± 0.02 | 0.79 ± 0.02
Congestion | 0.85 ± 0.00 | 0.75 ± 0.02 | 0.75 ± 0.02 | 0.79 ± 0.02
Pleural Effusion Right | 0.94 ± 0.00 | 0.83 ± 0.01 | 0.83 ± 0.02 | 0.91 ± 0.02
Pleural Effusion Left | 0.92 ± 0.00 | 0.83 ± 0.02 | 0.83 ± 0.02 | 0.86 ± 0.02
Pneumonic Infiltration Right | 0.93 ± 0.00 | 0.85 ± 0.02 | 0.85 ± 0.02 | 0.86 ± 0.02
Pneumonic Infiltration Left | 0.94 ± 0.00 | 0.86 ± 0.01 | 0.86 ± 0.02 | 0.87 ± 0.02
Atelectasis Right | 0.89 ± 0.00 | 0.78 ± 0.01 | 0.78 ± 0.01 | 0.84 ± 0.02
Atelectasis Left | 0.87 ± 0.00 | 0.78 ± 0.01 | 0.78 ± 0.02 | 0.81 ± 0.02
Average | 0.90 ± 0.04 | 0.81 ± 0.04 | 0.80 ± 0.05 | 0.84 ± 0.04

Table 4: Detailed evaluation results of training without DP. The results show
the average and individual area under the receiver-operator-characteristic
curve (AUROC), accuracy, specificity, and sensitivity values for each label
tested on $n=39,809$ test images. The training dataset includes $n=153,502$
images.

| AUROC | Accuracy | Specificity | Sensitivity
---|---|---|---|---
Cardiomegaly | 0.82 ± 0.00 | 0.73 ± 0.00 | 0.71 ± 0.02 | 0.76 ± 0.02
Congestion | 0.81 ± 0.00 | 0.72 ± 0.02 | 0.71 ± 0.03 | 0.76 ± 0.03
Pleural Effusion Right | 0.92 ± 0.00 | 0.82 ± 0.01 | 0.82 ± 0.01 | 0.88 ± 0.01
Pleural Effusion Left | 0.89 ± 0.00 | 0.79 ± 0.02 | 0.79 ± 0.02 | 0.84 ± 0.02
Pneumonic Infiltration Right | 0.91 ± 0.00 | 0.84 ± 0.01 | 0.83 ± 0.02 | 0.81 ± 0.02
Pneumonic Infiltration Left | 0.91 ± 0.00 | 0.84 ± 0.01 | 0.84 ± 0.01 | 0.83 ± 0.01
Atelectasis Right | 0.87 ± 0.00 | 0.78 ± 0.01 | 0.77 ± 0.01 | 0.81 ± 0.01
Atelectasis Left | 0.85 ± 0.00 | 0.76 ± 0.02 | 0.76 ± 0.02 | 0.79 ± 0.02
Average | 0.87 ± 0.04 | 0.79 ± 0.04 | 0.78 ± 0.05 | 0.81 ± 0.04

Table 5: Detailed evaluation results of DP training with $\varepsilon=7.89$,
$\delta=6\cdot 10^{-6}$. The results show the average and individual AUROC,
accuracy, specificity, and sensitivity values for each label tested on
$n=39,809$ test images. The training dataset includes $n=153,502$ images.

| AUROC | Accuracy | Specificity | Sensitivity
---|---|---|---|---
Cardiomegaly | 0.81 ± 0.00 | 0.73 ± 0.00 | 0.70 ± 0.01 | 0.77 ± 0.01
Congestion | 0.81 ± 0.00 | 0.71 ± 0.02 | 0.70 ± 0.02 | 0.77 ± 0.02
Pleural Effusion Right | 0.92 ± 0.00 | 0.82 ± 0.01 | 0.81 ± 0.01 | 0.87 ± 0.01
Pleural Effusion Left | 0.89 ± 0.00 | 0.80 ± 0.01 | 0.80 ± 0.02 | 0.81 ± 0.02
Pneumonic Infiltration Right | 0.90 ± 0.00 | 0.81 ± 0.01 | 0.81 ± 0.01 | 0.82 ± 0.01
Pneumonic Infiltration Left | 0.91 ± 0.00 | 0.82 ± 0.01 | 0.82 ± 0.01 | 0.85 ± 0.02
Atelectasis Right | 0.86 ± 0.00 | 0.76 ± 0.01 | 0.75 ± 0.02 | 0.83 ± 0.02
Atelectasis Left | 0.85 ± 0.00 | 0.78 ± 0.02 | 0.78 ± 0.03 | 0.76 ± 0.03
Average | 0.87 ± 0.04 | 0.78 ± 0.04 | 0.77 ± 0.05 | 0.81 ± 0.04

Table 6: Detailed evaluation results of DP training with $\varepsilon=4.71$,
$\delta=6\cdot 10^{-6}$. The results show the average and individual AUROC,
accuracy, specificity, and sensitivity values for each label tested on
$n=39,809$ test images. The training dataset includes $n=153,502$ images.

| AUROC | Accuracy | Specificity | Sensitivity
---|---|---|---|---
Cardiomegaly | 0.81 ± 0.00 | 0.73 ± 0.00 | 0.68 ± 0.02 | 0.78 ± 0.02
Congestion | 0.80 ± 0.00 | 0.70 ± 0.02 | 0.69 ± 0.03 | 0.76 ± 0.03
Pleural Effusion Right | 0.90 ± 0.00 | 0.80 ± 0.01 | 0.79 ± 0.01 | 0.86 ± 0.01
Pleural Effusion Left | 0.87 ± 0.00 | 0.75 ± 0.02 | 0.74 ± 0.02 | 0.84 ± 0.02
Pneumonic Infiltration Right | 0.90 ± 0.00 | 0.80 ± 0.01 | 0.80 ± 0.02 | 0.83 ± 0.02
Pneumonic Infiltration Left | 0.90 ± 0.00 | 0.83 ± 0.01 | 0.83 ± 0.02 | 0.81 ± 0.02
Atelectasis Right | 0.85 ± 0.00 | 0.74 ± 0.02 | 0.73 ± 0.02 | 0.82 ± 0.02
Atelectasis Left | 0.83 ± 0.00 | 0.73 ± 0.03 | 0.73 ± 0.03 | 0.77 ± 0.03
Average | 0.86 ± 0.04 | 0.76 ± 0.05 | 0.75 ± 0.05 | 0.81 ± 0.04

Table 7: Detailed evaluation results of DP training with $\varepsilon=2.04$,
$\delta=6\cdot 10^{-6}$. The results show the average and individual AUROC,
accuracy, specificity, and sensitivity values for each label tested on
$n=39,809$ test images. The training dataset includes $n=153,502$ images.

| AUROC | Accuracy | Specificity | Sensitivity
---|---|---|---|---
Cardiomegaly | 0.80 ± 0.00 | 0.72 ± 0.00 | 0.69 ± 0.02 | 0.76 ± 0.02
Congestion | 0.80 ± 0.00 | 0.70 ± 0.02 | 0.69 ± 0.02 | 0.75 ± 0.02
Pleural Effusion Right | 0.90 ± 0.00 | 0.80 ± 0.01 | 0.79 ± 0.02 | 0.86 ± 0.02
Pleural Effusion Left | 0.86 ± 0.00 | 0.73 ± 0.02 | 0.72 ± 0.02 | 0.83 ± 0.02
Pneumonic Infiltration Right | 0.89 ± 0.00 | 0.80 ± 0.02 | 0.80 ± 0.03 | 0.81 ± 0.03
Pneumonic Infiltration Left | 0.89 ± 0.00 | 0.79 ± 0.01 | 0.79 ± 0.02 | 0.83 ± 0.02
Atelectasis Right | 0.84 ± 0.00 | 0.74 ± 0.02 | 0.74 ± 0.02 | 0.80 ± 0.02
Atelectasis Left | 0.82 ± 0.00 | 0.70 ± 0.01 | 0.69 ± 0.02 | 0.79 ± 0.02
Average | 0.85 ± 0.04 | 0.75 ± 0.04 | 0.74 ± 0.05 | 0.80 ± 0.04

Table 8: Detailed evaluation results of DP training with $\varepsilon=1.06$,
$\delta=6\cdot 10^{-6}$. The results show the average and individual AUROC,
accuracy, specificity, and sensitivity values for each label tested on
$n=39,809$ test images. The training dataset includes $n=153,502$ images.

| AUROC | Accuracy | Specificity | Sensitivity
---|---|---|---|---
Cardiomegaly | 0.79 ± 0.00 | 0.72 ± 0.00 | 0.69 ± 0.01 | 0.74 ± 0.01
Congestion | 0.79 ± 0.00 | 0.67 ± 0.02 | 0.66 ± 0.02 | 0.78 ± 0.02
Pleural Effusion Right | 0.89 ± 0.00 | 0.77 ± 0.01 | 0.76 ± 0.02 | 0.86 ± 0.02
Pleural Effusion Left | 0.84 ± 0.00 | 0.71 ± 0.02 | 0.70 ± 0.03 | 0.84 ± 0.03
Pneumonic Infiltration Right | 0.88 ± 0.00 | 0.80 ± 0.01 | 0.80 ± 0.02 | 0.79 ± 0.02
Pneumonic Infiltration Left | 0.88 ± 0.00 | 0.77 ± 0.02 | 0.77 ± 0.03 | 0.83 ± 0.03
Atelectasis Right | 0.83 ± 0.00 | 0.74 ± 0.01 | 0.73 ± 0.01 | 0.79 ± 0.01
Atelectasis Left | 0.81 ± 0.00 | 0.70 ± 0.03 | 0.70 ± 0.03 | 0.77 ± 0.03
Average | 0.84 ± 0.04 | 0.73 ± 0.04 | 0.73 ± 0.05 | 0.80 ± 0.04

Table 9: Detailed evaluation results of DP training with $\varepsilon=0.54$,
$\delta=6\cdot 10^{-6}$. The results show the average and individual AUROC,
accuracy, specificity, and sensitivity values for each label tested on
$n=39,809$ test images. The training dataset includes $n=153,502$ images.

| AUROC | Accuracy | Specificity | Sensitivity
---|---|---|---|---
Cardiomegaly | 0.79 ± 0.00 | 0.71 ± 0.00 | 0.67 ± 0.01 | 0.75 ± 0.01
Congestion | 0.78 ± 0.00 | 0.68 ± 0.02 | 0.68 ± 0.02 | 0.74 ± 0.02
Pleural Effusion Right | 0.88 ± 0.00 | 0.77 ± 0.01 | 0.77 ± 0.02 | 0.83 ± 0.02
Pleural Effusion Left | 0.84 ± 0.00 | 0.73 ± 0.01 | 0.72 ± 0.02 | 0.80 ± 0.02
Pneumonic Infiltration Right | 0.87 ± 0.00 | 0.79 ± 0.01 | 0.79 ± 0.02 | 0.79 ± 0.02
Pneumonic Infiltration Left | 0.88 ± 0.00 | 0.79 ± 0.01 | 0.79 ± 0.01 | 0.81 ± 0.01
Atelectasis Right | 0.82 ± 0.00 | 0.73 ± 0.02 | 0.73 ± 0.02 | 0.77 ± 0.02
Atelectasis Left | 0.80 ± 0.00 | 0.71 ± 0.02 | 0.71 ± 0.02 | 0.75 ± 0.02
Average | 0.83 ± 0.04 | 0.74 ± 0.04 | 0.73 ± 0.05 | 0.78 ± 0.04

Table 10: Detailed evaluation results of DP training with $\varepsilon=0.29$,
$\delta=6\cdot 10^{-6}$. The results show the average and individual AUROC,
accuracy, specificity, and sensitivity values for each label tested on
$n=39,809$ test images. The training dataset includes $n=153,502$ images.

##  Appendix B Figures

Figure 6: Age histogram of the UKA-CXR dataset. (A) Training set. (B) Test
set. (C) Overall. Figure 7: Distribution of comorbidities over the UKA-CXR
dataset. Histograms of comorbidities are given for different subsets of the
dataset including subjects aging in the range of (A) $[0,30)$ years old with a
mean of $0.8\pm 1.2$ comorbidities, (B) $[30,60)$ years old with a mean of
$1.0\pm 1.3$ comorbidities, (C) $[60,70)$ years old with a mean of $1.1\pm
1.3$ comorbidities, (D) $[70,80)$ years old with a mean of $1.1\pm 1.2$
comorbidities, (E) $[80,100)$ years old with a mean of $1.1\pm 1.3$
comorbidities, as well as (F) females with a mean of $1.0\pm 1.2$
comorbidities, (G) males with a mean of $1.1\pm 1.3$ comorbidities, and (H)
overall with a mean of $1.1\pm 1.3$ comorbidities. Figure 8: Distribution of
comorbidities over the training set. Histograms of comorbidities are given for
different subsets of the training set including subjects aging in the range of
(A) $[0,30)$ years old with a mean of $0.8\pm 1.2$ comorbidities, (B)
$[30,60)$ years old with a mean of $1.0\pm 1.3$ comorbidities, (C) $[60,70)$
years old with a mean of $1.1\pm 1.3$ comorbidities, (D) $[70,80)$ years old
with a mean of $1.1\pm 1.2$ comorbidities, (E) $[80,100)$ years old with a
mean of $1.1\pm 1.3$ comorbidities, as well as (F) females with a mean of
$1.0\pm 1.2$ comorbidities, (G) males with a mean of $1.1\pm 1.3$
comorbidities, and (H) overall training set with a mean of $1.1\pm 1.3$
comorbidities. Figure 9: Distribution of comorbidities over the test set.
Histograms of comorbidities are given for different subsets of the test set
including subjects aging in the range of (A) $[0,30)$ years old with a mean of
$0.9\pm 1.4$ comorbidities, (B) $[30,60)$ years old with a mean of $1.0\pm
1.3$ comorbidities, (C) $[60,70)$ years old with a mean of $1.1\pm 1.3$
comorbidities, (D) $[70,80)$ years old with a mean of $1.1\pm 1.2$
comorbidities, (E) $[80,100)$ years old with a mean of $1.1\pm 1.3$
comorbidities, as well as (F) females with a mean of $1.0\pm 1.3$
comorbidities, (G) males with a mean of $1.1\pm 1.3$ comorbidities, and (H)
overall test set with a mean of $1.1\pm 1.3$ comorbidities.