# BayTTA: Uncertainty-aware medical image classification with optimized test-
time augmentation using Bayesian model averaging

Zeinab Sherkatghanad Moloud Abdar<EMAIL_ADDRESS>Mohammadreza Bakhtyari
Vladimir Makarenkov Departement d’Informatique, Universite du Quebec a
Montreal, H2X 3Y7, Montreal, QC, Canada  Institute for Intelligent Systems
Research and Innovation (IISRI), Deakin University, 3216, Geelong, VIC,
Australia  Mila - Quebec AI Institute, Montreal, QC, Canada

###### Abstract

Test-time augmentation (TTA) is a well-known technique employed during the
testing phase of computer vision tasks. It involves aggregating multiple
augmented versions of input data. Combining predictions using a simple average
formulation is a common and straightforward approach after performing TTA.
This paper introduces a novel framework for optimizing TTA, called BayTTA
(Bayesian-based TTA), which is based on Bayesian Model Averaging (BMA). First,
we generate a model list associated with different variations of the input
data created through TTA. Then, we use BMA to combine model predictions
weighted by their respective posterior probabilities. Such an approach allows
one to take into account model uncertainty, and thus to enhance the predictive
performance of the related machine learning or deep learning model. We
evaluate the performance of BayTTA on various public data, including three
medical image datasets comprising skin cancer, breast cancer, and chest X-ray
images and two well-known gene editing datasets, CRISPOR and GUIDE-seq. Our
experimental results indicate that BayTTA can be effectively integrated into
state-of-the-art deep learning models used in medical image analysis as well
as into some popular pre-trained CNN models such as VGG-16, MobileNetV2,
DenseNet201, ResNet152V2, and InceptionRes-NetV2, leading to the enhancement
in their accuracy and robustness performance. The source code of the proposed
BayTTA method is freely available at: https://github.com/Z-Sherkat/BayTTA.

###### keywords:

###### MSC:

41A05, 41A10, 65D05, 65D17 Deep Learning , Test-time Augmentation , Bayesian
Model Averaging, Uncertainty Quantification

††journal: Medical Image Analysis Fig. 1: Examples of original benign and
malignant skin cancer, breast cancer, and chest X-ray images and their
augmented versions considered in our study.

##  1 Introduction

Deep learning models are highly effective tools for identifying complex
patterns in data. However, they are also prone to overfitting and memorizing
the training data rather than generalizing real data patterns. Data
augmentation is a critical strategy to overcome this issue, which involves
synthesizing new training samples by applying to the original data a series of
transformations and perturbations, such as random erasing, flipping, rotating,
and scaling (see Fig. 1). Leveraging data augmentation techniques effectively
introduces variability in the training data, leading to a more robust and
general model. Machine learning (ML) and deep learning (DL) models equipped
with data augmentation can handle the complexities inherent in medical image
data, often leading to an enhanced diagnostic accuracy ([56, 8, 21, 9]).
Test-Time Augmentation (TTA) is a popular data augmentation technique which
can increase the prediction power of ML and DL models. TTA involves applying
data augmentation on test input rather than on training data. It provides a
more reliable estimate of the target variable by averaging the predictions
across augmented inputs to find the output result. While data augmentation is
mainly used for improving training data diversity to enhance the model
generalization on unseen data, TTA produces multiple versions of each test
image to obtain a more robust estimate of the target variable ([32, 40]).
Recently, Uncertainty Quantification (UQ) methods have been effectively
applied in a variety of research domains, contributing to the robustness and
reliability of the optimization and decision-making processes. There are two
primary categories of uncertainty ([1, 30]), referred to as aleatoric and
epistemic. While aleatoric uncertainty pertains to the inherent characteristic
of the data distribution, epistemic uncertainty relates to the lack of
knowledge about the optimal model. Uncertainty quantification is a vital tool
for addressing the limitations of data (aleatoric uncertainty) and models
(epistemic uncertainties) in various scientific and engineering applications,
and thus for improving the trustworthiness of predictions, especially in the
testing phase ([3]). Although deep learning models often outperform
traditional machine learning methods, overconfident predictions remain their
crucial issue, leading to excessive prediction errors ([51]). By combining
uncertainty quantification with TTA, we can gain a deeper insight into the
reliability and robustness of model predictions ([57, 6, 5]). This is
particularly beneficial for machine learning and deep learning applications in
healthcare, autonomous systems, and safety-critical domains, where
understanding and managing uncertainty may be critical.
Here, we describe a novel technique, called BayTTA, that employs Bayesian
Model Averaging (BMA) logistic regression to optimize TTA. Our new method
applies BMA on the predictions obtained through TTA. BMA can effectively
handle model uncertainty, combining multiple models generated by TTA and
offering precise and robust predictions. It aggregates predictions from all
candidate models, weighted by their posterior probabilities, and results in a
model-averaged prediction that accounts for model uncertainty.
This article showcases a number of contributions that are as follows:

  * 1.

We propose the BayTTA method for optimizing TTA using the BMA approach and
apply it to medical image classification.

  * 2.

We estimate the uncertainty associated with predictions obtained through TTA
and BMA, offering further insight into the reliability of our method.

  * 3.

We assess the performance of BayTTA on three public medical image datasets,
including skin cancer, breast cancer, and chest X-ray images available on the
Kaggle and Mendeley data repositories as well as on two popular gene editing
datasets, CRISPOR and GUIDE-seq, generated using the CRISPR-Cas9 technology
([23]). We demonstrate that our method can be applied successfully with
different deep learning network architectures. Our results indicates a
superior performance of the proposed method compared to standard averaging.

  * 4.

We evaluate the performance of BayTTA used in combination with some well-known
pre-trained deep learning networks, including VGG-16, MobileNetV2,
DenseNet201, ResNet152V2, and InceptionResNetV2.

  * 5.

We demonstrate how incorporating BayTTA enhances the predictive power and
robustness of state-of-the-art deep learning models used in medical image
analysis, compared to standard TTA.

##  2 Related work

###  2.1 Uncertainty quantification

Technical progress: Uncertainty quantification (UQ) is essential to enhance
the credibility of predictions during the testing phase. Since the excessive
confidence of deep neural networks may lead to prediction errors, it is
imperative to address the issue of overconfident predictions in order to
improve their reliability and trustworthiness ([24]). As deep learning models
are now constantly used in critical areas, the ability to quantify and manage
uncertainty becomes increasingly vital ([29, 41, 2]).

Nowadays, UQ has important applications in image processing, computer vision,
medical image analysis, diagnostic modeling, and healthcare decision-making
([1, 37, 38, 48]). There are several well-known methods for measuring
uncertainty such as Monte Carlo dropout ([20]), Variational Inference ([59,
47]), Deep Ensembles ([13, 46, 4]), and Bayesian Deep Ensembles ([26]).
Bayesian Model Averaging (BMA) is another effective technique used to take
into account prediction uncertainty ([60, 42, 31, 7]). Development of BMA in
the context of model uncertainty has been influenced by the seminal works of
[15, 16] and [28] as their original methods provide insight into the
quantification of both epistemic and aleatoric uncertainties.

UQ application in medical image analysis: Nowadays, UQ methods have become a
tool of choice for estimating the uncertainty associated with disease
detection, diagnosis, medical image segmentation, and identification of the
region of interest (ROI) in medical image analysis. In their recent work, [3]
have introduced an efficient hybrid dynamic model of uncertainty
quantification, called TWDBDL. The model is based on the Three-Way Decision
(TWD) theory and Bayesian Deep Learning (BDL) methods used together to improve
the trustworthiness of predictions in skin cancer detection. [17] used a
probabilistic variational autoencoders (VAEs), i.e. a Monte Carlo technique to
generate pixel uncertainty maps, and Stein’s Unbiased Risk Estimator (URE) to
provide accurate uncertainty estimations in knee magnetic resonance imaging.
[22] introduced the UA-ConvNet, i.e. an uncertainty-aware Convolutional Neural
Network (CNN) model, for COVID-19 detection in chest X-ray (CXR) images. The
model estimates the uncertainty based on the EfficientNet-B3 Bayesian network
supplemented with Monte Carlo dropout.

[41] designed a novel web server, called DUNEScan, for uncertainty estimation
in CNN models applied to skin cancer detection. The web server employs binary
dropout to compare the average model predictions, providing visualization for
uncertainty to diagnose precisely skin cancer cases. [25] proposed a novel
dynamic multi-scale convolutional neural network (DM-CNN) that leverages a
hierarchical dynamic uncertainty quantification attention (HDUQ-Attention)
submodel. HDUQ-Attention includes a tuning block for adjusting the attention
weights as well as Monte Carlo dropout for quantifying uncertainty. The
experiments conducted on skin disease images (HAM10000), colorectal cancer
images (NCT-CRC-HE-100K), and lung disease images (OCT2017 and Chest X-ray)
demonstrated that the DM-CNN model accurately quantifies uncertainty, showing
a stable performance.

UQ application in gene editing: Uncertainty quantification can be effectively
applied in the context of gene editing in order to improve the trustworthiness
of on- and off-target predictions in CRISPR-Cas9 experiments ([50]).

For example, [62] presented a deep learning model for off-target activity
prediction, employing data augmentation to mitigate the class imbalance issue.
The authors collected data from two source types, i.e. in vitro and cell-based
experiments, to increase the size of the positive class samples (off-targets).
They suggested synthetically expanding the number of positive samples by
rotating the sgRNA-DNA encoded images by 90, 180, and 270 degrees,
respectively, to enhance the model competency.

Moreover, [35] have recently introduced a pioneering method that incorporates
uncertainty into off-target predictions. Their approach offers an
interpretable evaluation of Cas9-gRNA and Cas12a-gRNA specificity through deep
kernel learning, estimating a gRNA’s cleavage efficiency with a corresponding
confidence interval.

Fig. 2: A schematic view of a conventional test-time augmentation (TTA)
process.

###  2.2 Test-time augmentation

Technical progress: Test-time augmentation (TTA) is a well-known technique
that applies data augmentation during the testing phase. TTA has multiple
benefits, including improving the model generalization and reliability
capacity ([53, 12]), estimating uncertainty in model predictions ([57, 6, 5]),
and boosting accuracy in classification and segmentation tasks ([36, 54, 27,
49]). Figure 2 presents a detailed flowchart of a typical TTA process.

Recently, some advanced TTA methods using diverse aggregation strategies have
been proposed. They include, among others, selective augmentation techniques
such as the instance-aware TTA based on a loss predictor ([34]), the instance-
level TTA with entropy weight method ([11]), and the selective-TTA method
([52]).

[39] presented a straightforward and efficient method based on a policy search
algorithm, known as greedy policy search (GPS), designed to optimize image
classification. The method aims to optimize and determine the most effective
data augmentation strategy to be applied during the testing phase of a machine
learning process, and thus to improve the prediction accuracy and robustness
of ML models. [34] developed an instance-aware test-time augmentation approach
that employs a loss predictor to dynamically select test-time transformations
based on the expected losses for individual instances. Furthermore, [11]
introduced an instance-level TTA with Entropy Weight Method (EWM) as an
innovative approach to improve the accuracy and robustness of classification
models.

The paper of [49] describes a new method for aggregating model predictions
obtained from TTA. Unlike the traditional approach of averaging model
predictions, this method focuses on learning different augmentation weights to
aggregate predictions obtained from transformations during TTA. The authors
noticed that existing aggregation methods, based on the mean or the maximum of
predictions obtained from augmented images, may not be optimal because they do
not consider the relationship between the original test image and its
augmented versions. The paper also offers insights into the point when TTA can
be helpful and provides guidance regarding the use of different TTA policies.
Moreover, Shanmugam and co-authors characterized the cases where TTA
transforms correct predictions into incorrect ones, and vice versa.

TTA application in medical image analysis: TTA has been extensively studied by
the medical imaging community due to its capability to contribute to model
robustness and improve the trustworthiness and generalization of predictions
during the testing phase. [57] conducted critical research in the context of
deep convolutional neural network (CNN)-based medical image segmentation.
Their work focuses on epistemic and aleatoric uncertainty analysis at pixel
and structure levels, providing valuable insights into the reliability of
segmentation results. [19] introduced a simple and powerful task-agnostic
semi-supervised active learning segmentation approach, called TAAL (TTA for
Active Learning). TAAL exploits unlabeled samples during training and sampling
phases by using a technique known as cross-augmentation consistency.

##  3 Methodology

In this section, we elaborate on the background, foundation, and comprehensive
explanation of the methodological procedures used within the proposed method.
Our new method, called BayTTA, aims to optimize TTA using BMA and uncertainty
estimation. The first component of our method involves formulating a
mathematical model that applies BMA to output predictions generated from
multiple transformed versions of the input data (e.g. input images). The
second component of the method estimates the uncertainty of the predictions
obtained from an augmented set of test images associated with image
transformations or noise. Together, these components offer a comprehensive
approach to enhance the robustness and reliability of ML or DL models during
the testing phase. An overview of the proposed BayTTA method is presented in
Figure 3.

Fig. 3: An overview of the proposed BayTTA method. During the testing phase:
(a) TTA generates a model list from a set of fixed augmented images, and (b)
BMA is then applied to combine and aggregate the predictions from the model
list.

Traditional TTA involves applying a number of data augmentation techniques,
including rotation, cropping, flipping, and brightness adjustments, to the
test data before making predictions. This widely-used approach leverages
ensemble methods, such as averaging or taking the majority vote of predictions
made on augmented samples, to reduce the impact of random noise and variation
in the test data. TTA usually produces stable and accurate predictions by
averaging multiple augmented versions of the test dataset ([57, 58]).

Our study also implements BMA to integrate predictions from the model list,
thus deviating from the conventional averaging approach used in TTA, to yield
more precise and robust outcomes. Let us focus on a single image $I_{y}$ from
the test dataset, where $y\in\\{0,1\\}$ denotes its classification label. In a
scenario generating $k$ augmented versions of the test image in TTA, the input
to the BMA comes from the vector ${\mathbf{x}=\\{x_{1},x_{2},...,x_{k+1}\\}}$,
where each element corresponds to a prediction from the TTA procedure. Here,
$x_{1}$ denotes the predictor variable derived from the original test image
$I_{y}$ and $\\{x_{2},x_{3},...,x_{k+1}\\}$ denote the predictor variables
from augmented versions of this image.

BMA involves generating various models by combining predictor variables and
selecting candidate models based on likelihood. To formalize this process, we
introduce $\mathcal{P(\mathbf{x})}$, a non-empty power set of $\mathbf{x}$,
representing all combinations of predictor variables. Subsequently, utilizing
$I\ \in\mathcal{P(\mathbf{x})}$, we select desired predictor variables from
$\mathbf{x}$ for model $M_{I}$, where $\mathbf{x}_{I}$ represents the input of
the respective model $M_{I}$. In the context of Bayesian model averaging
logistic regression, we further elucidate the mathematical formulations that
define our algorithm in the subsequent discussion.

According to the Bayes theorem, the posterior distribution for model $M_{I}$
is given by:

$p(M_{I}\mid\mathbf{x}_{I},y)=\frac{p(\mathbf{x}_{I},y\mid
M_{I})p(M_{I})}{\mathcal{L}_{total}},$ (1)

where $p(M_{I})$ denotes the prior probability of $M_{I}$,
$\mathcal{L}_{total}$ is the marginal likelihood, and $p(\mathbf{x}_{I},y\mid
M_{I})$ is the likelihood of $M_{I}$ estimated using the Bayesian Information
Criterion (BIC):

$\displaystyle BIC_{I}=p_{I}\ln(N)-2\ln{(\tilde{L}_{I})},$ (2) $\displaystyle
p({\mathbf{x}_{I}},y\mid M_{I})=e^{-BIC_{I}/2}.$

Here, $p_{I}$ is the number of parameters in model $M_{I}$, $N$ is the number
of data points in the input data, and $\tilde{L}_{I}$ is the maximized value
of the likelihood function for model $M_{I}$. Using the above formulas yields:

$p(M_{I}\mid\mathbf{x}_{I},y)=\frac{e^{-BIC_{I}/2}p(M_{I})}{\mathcal{L}_{total}},$
(3)

where:

$\mathcal{L}_{total}=\sum e^{-BIC/2}p(M),$ (4)

In BMA, the Bayes factor plays a crucial role in model selection. Thus,
recognizing the relationship between posterior model probabilities and Bayes
Factors (BF) application is crucial. While comparing two models, labeled $I$
and $J$, we can calculate the Bayes factor for model $M_{I}$ against model
$M_{J}$ using the following expression:

$BF_{IJ}=\frac{p(M_{I}\mid\mathbf{x}_{I},y)}{p(M_{J}\mid\mathbf{x}_{J},y)}.$
(5)

If the value of BF exceeds 1, the observed data strongly favors model $M_{I}$
over model $M_{J}$. In practical terms, this implies that the information
provided by BIC can guide us in selecting the best model that is the model
with the highest $log$ of marginal likelihood and, consequently, the smallest
BIC.

Computing output values for the Bayesian averaging logistic regression model
involves determining the probability of any predictor variable, $p(x_{i})$,
and the expected value of the coefficient associated with this predictor
variable, i.e. $E[\beta_{i}]$:

$\displaystyle p(x_{i})$ $\displaystyle=\sum_{M_{I}\ such\ that\
x_{i}\in\mathbf{x}_{I}}p(M_{I}\mid\mathbf{x}_{I},y),$ (6) $\displaystyle
E[\beta_{i}]$ $\displaystyle=\sum_{M_{I}\ such\ that\
x_{i}\in\mathbf{x}_{I}}p(M_{I}\mid\mathbf{x}_{I},y)\times\mathbf{\beta}^{I}_{i}.$

Here, $i\in\\{1,2,...,k+1\\}$, and $\beta^{I}_{i}$ is the coefficient of
$x_{i}$ in model $M_{I}$. The BayTTA model carries out a series of steps to
compute the probabilities: $p(\mathbf{x})=(p(x_{1}),p(x_{2}),...,p(x_{k+1}))$,
and the expected values:
$E[\boldsymbol{\beta}]=(E[\beta_{1}],E[\beta_{2}],...,E[\beta_{k+1}])$.
Algorithm 1 presents the pseudocode outlining the main steps of our method.

We define the set $I_{current}\subseteq\mathcal{P}(\mathbf{x})$ to identify
candidate models that we want to process at each iteration. Initially, we form
$I_{current}=\\{\\{1\\},\\{2\\},...,\\{k+1\\}\\}$, where each $I\in
I_{current}$ corresponds to a model $M_{I}$ with one (i.e. $m=1$) predictor
variable. Subsequently, logistic regression on each model is carried out to
calculate the BIC values and the coefficients of the predictor variables.
Following this step, we assess the model’s likelihood using the estimated
value of BIC, as outlined in Eq. 2. To finalize the model selection process in
BMA (see Eq. 5), we specify a uniform prior distribution for all candidate
models and set an initial threshold $\mathcal{L}_{max}$ to zero. For each
model $M_{I}$ whose likelihood exceeds $\mathcal{L}_{max}$, we perform the
following steps: (1) replace the threshold $\mathcal{L}_{max}$ with the
likelihood of the model $M_{I}$, (2) aggregate its likelihood to calculate
$\mathcal{L}_{total}$ (i.e. the denominator in Eq. 3), and (3) update the
probabilities and the coefficients of the predictor variables of the model
$M_{I}$ using Eqs. 6. At the subsequent iteration, we build a set of candidate
models with one additional predictor variable, $m=m+1$, based on models
meeting the threshold at the previous iteration. After repeating this
procedure $k+1$ times, and considering all possible combinations of predictor
variables for generating models, the BMA prediction can be calculated as
follows:

$y_{BMA}=\frac{1}{1+exp(-E[{\boldsymbol{\beta}}]^{T}\mathbf{x})}.$ (7)

In this procedure, the probability of each predictor variable, $p(x_{i})$,
corresponds to TTA augmentations, and is defined as the sum of the
probabilities of all models that incorporate this predictor variable.
Furthermore, the expected value for the coefficient of each predictor
variable, $E[\beta_{i}]$, is calculated as a weighted average of the
coefficients determined by the posterior probability across all models that
include this predictor variable. By implementing such a TTA optimization
technique that selects more confident augmentations during the testing phase,
we ensure that it improves predictive performance, uncertainty estimations,
and the overall robustness of deep learning models, compared to simple
averaging.

The uncertainty for the proposed method, $\sigma_{BayTTA}$ is defined based on
the mean accuracy values obtained through BMA:

$\sigma_{BayTTA}=\sqrt{\frac{1}{k+1}\sum_{i=1}^{k+1}\Bigl{(}p(x_{i})\times(acc_{i}-\mu_{BMA})\Bigr{)}^{2}},$
(8)

where $acc_{i}$ is the accuracy of the original image and its $k$ augmented
versions obtained using TTA, and $\mu_{BMA}$ is the accuracy obtained using
BayTTA.

Algorithm 1 Optimizing TTA using BMA (BayTTA)

1:Trained model

2:Original test image $X_{test}$

3:Set of $k$ transformations

4:Uncertainty estimation for test image $X_{test}$

5:$\mathbf{\hat{p}}\leftarrow\mathbf{0}_{k+1},\hat{E}[\boldsymbol{\beta}]\leftarrow\mathbf{0}_{k+1}$

6:$\mathcal{L}_{total}\leftarrow 0,\mathcal{L}_{max}\leftarrow 0$

7:for $i\leftarrow 1,...,k+1$ do

8: Calculate $I_{next}$ $\triangleright$
$\\{I_{next}\subseteq\mathcal{P(\mathbf{x})}\mid\text{length}(S\in
I_{next})=i\\}$

9: $I_{current}=\emptyset$

10: if $i==1$ then

11: $I_{current}=I_{next}$

12: $I_{previous}=\emptyset$

13: else

14: $A=\\{S\in I_{next}\mid S\ \text{contains}\ \text{an element of}\
I_{previous}\\}$

15: $I_{current}=I_{current}\cup A$

16: $I_{previous}=\emptyset$

17: end if

18: for $I\ \textbf{in}\ I_{current}$ do

19:
$BIC_{I},\boldsymbol{\beta}^{\prime}\leftarrow\text{logits}(\mathbf{x}_{I},y)$

20: $\mathcal{L}_{M_{I}}\leftarrow e^{-BIC_{I}/2}$ $\triangleright$ using Eq.
2

21: if $\mathcal{L}_{M_{I}}>\mathcal{L}_{max}$ then

22: $\mathcal{L}_{total}\mathrel{+}=\mathcal{L}_{M_{I}}$ $\triangleright$
using Eq. 4

23: $\mathcal{L}_{max}=\mathcal{L}_{M_{I}}$

24: for j in I do

25: $\mathbf{\hat{p}}_{j}\mathrel{+}=\mathcal{L}_{M_{I}}$ $\triangleright$ for
Eqs. 6

26:
$\hat{E}[\boldsymbol{\beta}_{j}]\mathrel{+}=\boldsymbol{\beta^{\prime}}_{j}\times\mathcal{L}_{M_{I}}$
$\triangleright$ for Eqs. 6

27: end for

28: $I_{previous}=I_{previous}\cup I$

29: end if

30: end for

31:end for

32:Calculate $p(\mathbf{x})$ and $E[\boldsymbol{\beta}]$ $\triangleright$
using Eqs. 6

33:Calculate $y_{BMA}$ $\triangleright$ using Eq. 7

34:Calculate the uncertainty

##  4 Experimental evaluations and discussion

In this section, we present the results of our comprehensive experimental
study conducted using different DL models in combination with TTA and BayTTA
methods. Additionally, we elucidate the experimental setup and the
optimization parameters employed in our study. Lastly, we present and discuss
the main findings derived from our experimental investigation.

###  4.1 Data used in evaluation

We conducted our experiments on three publicly available medical image
datasets: skin cancer, breast cancer, and chest X-ray data, which are freely
available on the Kaggle and Mendeley data repositories (see also Fig. 1).
Subsequently, we evaluated the efficacy of the proposed BayTTA method using
two well-known gene editing datasets, CRISPOR and GUIDE-seq, both generated
through the CRISPR-Cas-9 technology ([50]). We adopted an effective one-hot
encoding strategy, originally proposed by [10], to represent sgRNA and DNA
sequence pairs as binary images from which off-target effects can be predicted
by advanced DL models (see Fig. 4).

####  4.1.1 Medical image datasets

Skin cancer dataset is the first group of data taken from
Kaggle111https://www.kaggle.com/datasets/fanconic/skin-cancer-malignant-vs-
benign, encompassing images categorized into two distinct classes: Benign and
Malignant. This dataset comprises 2637 training images, among which 1440 fall
under the Benign category and 1197 under the Malignant category. Additionally,
it includes 660 test images, consisting of 360 Benign and 300 Malignant
samples.

Breast cancer dataset is the second group of data supplied by
Kaggle222https://www.kaggle.com/datasets/vuppalaadithyasairam/ultrasound-
breast-images-for-breast-cancer, comprising 8116 training images. This dataset
has 4074 samples categorized as Benign and 4042 samples categorized as
Malignant. Additionally, the dataset includes 900 test samples, 500 of which
are classified as Benign and 400 as Malignant.

Chest X-ray dataset is a large collection of X-ray images extracted from a
publicly available medical image
database333https://easy.dans.knaw.nl/ui/datasets/id/easy-dataset:102451
([33]). The dataset contains 5216 training images, among which 1341 are
categorized as Normal and 3875 as Pneumonia samples. Additionally, the test
dataset includes 624 images, among which 234 are classified as Normal and 390
as Pneumonia samples.

####  4.1.2 Gene editing datasets

CRISPOR database, organized and maintained by Maximilian Haeussler ([23])
aggregates widely-used public datasets aimed at quantifying on-target guide
efficiency and detecting off-target cleavage sites444http://crispor.tefor.net.
The dataset we selected from this database (see also [10]) comprises $18,211$
black and white training images (each with $8\times 23$ pixel dimension),
among which 18,112 are categorized as on-targets and 99 as off-targets.
Additionally, the dataset includes 7806 testing images of the same dimension,
7763 of which are classified as on-targets and 43 as off-targets.

GUIDE-seq is one of the pioneering off-target data repositories, derived from
the outcomes of the GUIDE-seq technique developed by [55]. It serves as an
accurate framework for genome-wide identification of off-target effects. The
sgRNAs used in GUIDE-seq target the following sites: VEGFA site 1, VEGFA site
2, VEGFA site 3, FANCF, HEK293 site 2, HEK293 site 3, and HEK293 site 4,
wherein 28 off-targets with a minimum modification frequency of $0.1$ were
identified (among 403 potential off-targets). This dataset consists of black
and white images with $8\times 23$ pixel dimension. It comprises 309 training
images, including 291 on-target and 18 off-target samples. Additionally, it
comprises 133 testing images, including 121 on-target and 12 off-target
samples.

Fig. 4: Examples of visualizing CRISPR-Cas9 sgRNA-DNA sequence pairs encoded onto 8×23 matrices, then transformed into black and white images from the (a) CRISPOR and (b) GUIDE-seq gene editing datasets, respectively ([10]). These images can be processed by neural networks to predict off-targets generated by CRISPR-Cas9 technology.  Table 1:  Hyperparameter configuration of the pre-trained deep learning models (VGG-16, MobileNetV2, DenseNet201, ResNet152V2, and InceptionResNetV2) for the Skin Cancer, Breast Cancer, and Chest X-ray medical image datasets considered in our study. | Skin Cancer | Breast Cancer | Chest X-ray
---|---|---|---
Models | Opt | LR | BS | Epoch | Opt | LR | BS | Epoch | Opt | LR | BS | Epoch
VGG-16 | Adam | 0.0005 | 256 | 124 | Adamax | 0.005 | 128 | 157 | Adamax | 0.0001 | 128 | 156
MobileNetV2 | Adam | 0.001 | 256 | 82 | SGD | 0.001 | 16 | 171 | SGD | 0.00001 | 256 | 100
DenseNet201 | Adamax | 0.0001 | 32 | 138 | SGD | 0.001 | 16 | 75 | SGD | 0.0001 | 256 | 145
ResNet152V2 | Adamax | 0.0005 | 128 | 127 | SGD | 0.001 | 16 | 251 | SGD | 0.00005 | 256 | 200
Inception | Adamax | 0.0001 | 16 | 256 | SGD | 0.001 | 16 | 249 | SGD | 0.00001 | 256 | 140
ResNetV2 |  |  |  |  |  |  |  |  |  |  |  |

###  4.2 Implementation details and model settings

To facilitate a comprehensive understanding of the proposed BayTTA method, we
provide additional insights into the selected experimental configuration.

Experimental Setup. For the training phase, we used a Compute Canada cluster
equipped with NVIDIA Tesla P100 and NVIDIA v100 GPUs (referred to as the Cedar
cluster). Additionally, we used a software environment with Python,
TensorFlow, and PyTorch stack.

Pre-trained CNN models. During the training phase of our first experiment, we
employed five well-established convolutional networks (CNNs), namely VGG-16,
MobileNetV2, DenseNet201, ResNet152V2, and InceptionResNetV2, as our feature
extraction backbone, initializing weights through pre-training on the ImageNet
dataset. The baseline versions of these models were initially used (without
applying TTA or BayTTA), followed by their application in combination with TTA
and BayTTA. All medical images were resized to the $224\times 224$ pixel size
to ensure uniformity in model inputs. The hyperparameters, including the
optimizer (Opt), the learning rate (LR), the batch size (BS), and the number
of epochs for each of the five pre-trained models and each medical image
dataset are summarized in Table 1. Additionally, we employed the early
stopping technique to handle overfitting ([44]). During our network training,
the data augmentation techniques were implemented using the following
parameters: rotation range = 20, width shift range = 0.2, height shift range =
0.2, shear range = 0.2, zoom range = 0.2, horizontal flip, and vertical flip.

Four pre-trained CNN models, VGG-16, MobileNetV2, DenseNet201, and
ResNet152V2, as well as a custom CNN with five layers, comprising two
convolutional and three fully connected dense layers, were used to predict
off-targets in highly imbalanced gene editing datasets. The models were
trained using the RMSprop optimizer, with a learning rate of $0.001$ and a
batch size of $64$. To address potential overfitting issue a set of callback
functions was used during training, specifically the ReduceLROnPlateau and
EarlyStopping functions.

State-of-the-art DL models intended for medical image analysis. To evaluate
the performance of BayTTA, we also compared its performance against TTA and
baseline, considering some state-of-the-art DL models recently used in medical
image analysis, namely Attention Residual Learning (ARL) ([61]), COVID-19
([43]), LoTeNet ([45]), IRv2+SA ([14]), and PCXRNet ([18]).

IRv2+SA ([14]) explores the efficacy of soft attention in deep neural
architectures. The core objective of this technique is to emphasize the
significance of essential features, while mitigating the influence of noise-
inducing ones. The ARL model, proposed by [61], is intended for classifying
skin lesions in dermoscopy images. The model architecture includes several ARL
blocks, a global average pooling layer, and a classification layer. Each ARL
block employs a combination of residual learning and an innovative attention-
learning mechanism to boost the capacity of discriminative representations.
[43] proposed a novel model based on the DarkNet method designed for automated
detection of COVID-19 cases using chest X-ray images to deliver precise
diagnostic results in the framework of binary and multi-class classifications.
PCXRNet ([18]) is an attention-driven convolutional neural network designed
for pneumonia diagnosis based on chest X-ray image analysis. PCXRNet
incorporates a novel Condensed Attention Module (CDSE) to harness the
information within the feature map channels. The LoTeNet (Locally order-less
Tensor Network) model ([45]) is based on the use of tensor networks, a crucial
tool for physicists to analyze complex quantum many-body systems.

###  4.3 Experimental results

Table 2:  Comparison of the baseline CNN model accuracy (%) $\pm$ STD
performance against the TTA and BayTTA versions on the skin cancer dataset.
The highest accuracy per column is in bold. The asterisk ($*$) denotes the
highest overall accuracy obtained by the models.

Models | VGG-16 | MobileNetV2 | DenseNet201 | ResNet152V2 | InceptionResNetV2
---|---|---|---|---|---
Baseline | 84.95 $\pm$ 0.40 | 85.75 $\pm$ 1.31 | 88.28 $\pm$ 0.76 | 83.33 $\pm$ 1.75 | 81.63 $\pm$ 1.70
TTA | 85.24 $\pm$ 0.39 | 87.39 $\pm$ 0.42 | 88.33 $\pm$ 0.58 | 83.82 $\pm$ 0.52 | 83.01 $\pm$ 0.77
BayTTA | 85.50 $\pm$ 0.14 | 87.52 $\pm$ 0.11 | 89.38 $\pm$ 0.17* | 84.04 $\pm$ 0.09 | 83.98 $\pm$ 0.46

Table 3:  Comparison of the baseline CNN model accuracy (%) $\pm$ STD
performance against the TTA and BayTTA versions on the breast cancer dataset.
The highest accuracy per column is in bold. The asterisk ($*$) denotes the
highest overall accuracy obtained by the models.

Models | VGG-16 | MobileNetV2 | DenseNet201 | ResNet152V2 | InceptionResNetV2
---|---|---|---|---|---
Baseline | 88.92 $\pm$ 1.70 | 86.70 $\pm$ 0.94 | 86.81 $\pm$ 1.06 | 91.52 $\pm$ 1.18 | 91.25 $\pm$ 0.98
TTA | 89.64 $\pm$ 0.52 | 87.84 $\pm$ 0.81 | 86.64 $\pm$ 0.74 | 93.64 $\pm$ 1.04 | 93.13 $\pm$ 0.71
BayTTA | 90.11 $\pm$ 0.64 | 88.36 $\pm$ 0.34 | 86.18 $\pm$ 0.59 | 93.81 $\pm$ 0.99* | 92.84 $\pm$ 0.69

Table 4:  Comparison of the baseline CNN model accuracy (%) $\pm$ STD
performance against the TTA and BayTTA versions on the chest X-ray dataset.
The highest accuracy per column is in bold. The asterisk ($*$) denotes the
highest overall accuracy obtained by the models.

Models | VGG-16 | MobileNetV2 | DenseNet201 | ResNet152V2 | InceptionResNetV2
---|---|---|---|---|---
Baseline | 71.02 $\pm$ 1.01 | 61.53 $\pm$ 0.73 | 66.77 $\pm$ 1.11 | 62.45 $\pm$ 0.17 | 63.31 $\pm$ 0.57
TTA | 71.11 $\pm$ 0.78 | 61.32 $\pm$ 0.62 | 68.20 $\pm$ 0.63 | 62.54 $\pm$ 0.19 | 63.45 $\pm$ 0.56
BayTTA | 72.49 $\pm$ 0.25* | 62.50 $\pm$ 0.27 | 69.98 $\pm$ 0.28 | 62.82 $\pm$ 0.06 | 64.30 $\pm$ 0.21

Table 5:  Comparison of state-of-the-art classification models against their
TTA and BayTTA counterparts, in terms of accuracy (%) and STD, after their
application on the skin cancer, breast cancer, and chest X-ray datasets
considered in our study. The highest overall accuracy per dataset is
highlighted in bold.

Models | Skin Cancer | Breast Cancer | Chest X-ray
---|---|---|---
ARL ([61]) | 86.21 $\pm$ 0.42 | 84.87 $\pm$ 0.15 | 64.58 $\pm$ 0.15
COVID-19 ([43]) | 85.65 $\pm$ 0.61 | 94.92 $\pm$ 0.88 | 84.07 $\pm$ 1.41
LoTeNet ([45]) | 74.89 $\pm$ 0.72 | 72.63 $\pm$ 1.24 | 79.48 $\pm$ 0.80
IRv2+SA ([14]) | 90.92 $\pm$ 0.32 | 95.84 $\pm$ 0.54 | 87.76 $\pm$ 1.20
PCXRNet ([18]) | 79.70 $\pm$ 0.21 | 93.05 $\pm$ 0.48 | 79.15 $\pm$ 1.46
COVID-19+TTA | 85.70 $\pm$ 0.74 | 94.88 $\pm$ 0.34 | 79.10 $\pm$ 1.96
COVID-19+BayTTA | 87.17 $\pm$ 0.31 | 95.55 $\pm$ 0.31 | 85.04 $\pm$ 1.43
IRv2+SA+TTA | 89.74 $\pm$ 0.71 | 94.48 $\pm$ 0.67 | 83.71 $\pm$ 1.36
IRv2+SA+BayTTA | 91.07 $\pm$ 0.04 | 96.66 $\pm$ 0.49 | 87.76 $\pm$ 1.20

Table 6:  Comparison of state-of-the-art classification models against their
TTA and BayTTA counterparts, in terms of precision (PR (%)), recall (RE (%)),
and F1-score (FS (%)), after their application on the skin cancer, breast
cancer, and chest X-ray datasets considered in our study. The highest overall
accuracy per dataset is highlighted in bold.

| Skin Cancer | Breast Cancer | Chest X-ray
---|---|---|---
Models | PR | RE | FS | PR | RE | FS | PR | RE | FS
ARL ([61]) | 86.66 | 85.80 | 86.09 | 89.60 | 83.15 | 84.33 | 81.46 | 51.46 | 41.57
COVID-19 ([43]) | 84.78 | 83.44 | 84.32 | 94.29 | 94.42 | 94.29 | 80.24 | 98.97 | 88.61
LoTeNet ([45]) | 75.38 | 86.50 | 66.80 | 75.22 | 62.25 | 68.13 | 77.53 | 96.41 | 85.94
IRv2+SA ([14]) | 89.60 | 90.55 | 90.04 | 97.68 | 95.16 | 95.52 | 84.24 | 98.97 | 91.03
PCXRNet ([18]) | 79.98 | 99.51 | 88.69 | 93.03 | 73.90 | 82.37 | 75.22 | 99.37 | 85.63
COVID-19+TTA | 83.81 | 84.61 | 84.17 | 94.55 | 94.05 | 94.26 | 99.17 | 75.67 | 85.61
COVID-19+BayTTA | 85.80 | 86.01 | 85.90 | 94.41 | 95.75 | 95.05 | 81.31 | 98.80 | 89.53
IRv2+SA+TTA | 87.22 | 84.63 | 90.37 | 98.59 | 91.58 | 95.19 | 79.92 | 99.57 | 88.66
IRv2+SA+BayTTA | 88.70 | 92.11 | 90.42 | 95.99 | 96.58 | 96.25 | 84.24 | 98.97 | 91.03

####  4.3.1 Results for medical image datasets

We present a thorough experimental assessment of the BayTTA model in two
stages.

First, we evaluate its performance in the case when it was used in combination
with five pre-trained CNN models, including VGG-16, MobileNetV2, DenseNet201,
ResNet152V2, and InceptionResNetV2, to determine the most suitable backbone
model for the skin cancer (Table 4), breast cancer (Table 4), and chest X-ray
(Table 4) datasets. In this study, we executed the baseline models three
times, computing the mean accuracy and the standard deviation (STD) of
accuracy. We applied TTA during the testing phase on the baseline models to
assess the model accuracy and conducted a comparative analysis with BayTTA, in
which the BMA method was used for model averaging. We considered each original
image and six of its random augmentations (obtained through rotations) during
the testing phase, evaluating the mean accuracy and STD obtained with these
random augmentations. The results reported in Tables 4 demonstrate that the
use of BayTTA allowed us to outperform the baseline and TTA-based models,
achieving the highest accuracy and significantly reducing the standard
deviation values for the skin cancer dataset. Regarding the breast cancer
dataset, BayTTA outperforms the baseline and TTA-based models in accuracy for
the three out of five pre-trained CNNs, i.e. VGG-16, MobileNetV2, and
ResNet152V2. Nonetheless, the BayTTA-based networks consistently exhibited the
lowest STD values, as outlined in Table 4. In the case of the imbalanced chest
X-ray dataset (see Table 4), the combination of TTA and BMA (i.e. BayTTA)
allowed us to improve the baseline and TTA results in all cases in terms of
both accuracy and STD.

Table 7:  Comparison of the baseline CNN model accuracy (%) $\pm$ STD
performance against their TTA and BayTTA versions on the CRISPOR dataset. The
highest accuracy per column is in bold. The asterisk ($*$) denotes the highest
overall accuracy obtained by the models.

Models | VGG-16 | MobileNetV2 | DenseNet201 | ResNet152V2 | CNN-5 layers
---|---|---|---|---|---
Baseline | 99.41 $\pm$ 0.005 | 99.55 $\pm$0.04 | 99.53 $\pm$ 0.05 | 99.62 $\pm$ 0.08 | 99.82 $\pm$ 0.057
TTA | 99.37 $\pm$ 0.018 | 99.46 $\pm$ 0.05 | 99.46 $\pm$ 0.03 | 99.53 $\pm$ 0.04 | 99.77 $\pm$ 0.016
BayTTA | 99.41 $\pm$ 0.011 | 99.55 $\pm$ 0.04 | 99.53 $\pm$ 0.03 | 99.62 $\pm$ 0.02 | 99.84 $\pm$ 0.008*

Table 8:  Comparison of the baseline CNN model accuracy (%) $\pm$ STD
performance against their TTA and BayTTA versions on the GUIDE-seq dataset.
The highest accuracy per column is in bold. The asterisk ($*$) denotes the
highest overall accuracy obtained by the models.

Models | VGG-16 | MobileNetV2 | DenseNet201 | ResNet152V2 | CNN-5 layers
---|---|---|---|---|---
Baseline | 93.51 $\pm$ 0.98 | 90.47 $\pm$ 0.70 | 90.97 $\pm$ 0.60 | 87.95 $\pm$ 0.02 | 94.22 $\pm$ 0.92
TTA | 91.61 $\pm$ 1.18 | 90.54 $\pm$ 0.55 | 90.83 $\pm$ 0.50 | 90.65 $\pm$ 0.79 | 94.45 $\pm$ 0.26
BayTTA | 93.51 $\pm$ 0.67 | 90.97$\pm$ 0.15 | 91.72 $\pm$ 0.30 | 90.98 $\pm$ 0.16 | 94.73 $\pm$ 0.11*

Table 9:  Comparison of state-of-the-art classification models against their
TTA and BayTTA counterparts, in terms of accuracy (%) and STD on the CRISPOR
and GUIDE-seq gene editing datasets. The highest accuracy per column is
highlighted in bold.

Models | CRISPOR | GUIDE-seq
---|---|---
ARL ([61]) | 94.20 $\pm$ 1.19 | 91.04 $\pm$ 0.31
COVID-19 ([43]) | 99.50 $\pm$ 0.20 | 93.87 $\pm$ 0.96
LoTeNet ([45]) | 98.85 $\pm$ 0.43 | 93.38 $\pm$ 0.99
IRv2+SA ([14]) | 99.38 $\pm$ 0.24 | 91.22 $\pm$ 1.03
PCXRNet ([18]) | 95.69 $\pm$ 2.51 | 91.42 $\pm$ 0.97
COVID-19+TTA | 99.39 $\pm$ 0.18 | 95.07 $\pm$ 0.90
COVID-19+BayTTA | 99.66 $\pm$ 0.02 | 96.56 $\pm$ 0.41
IRv2+SA+TTA | 95.24 $\pm$ 0.57 | 93.75 $\pm$ 0.71
IRv2+SA+BayTTA | 99.43 $\pm$ 0.38 | 94.29 $\pm$ 0.43

Table 10:  Comparison of state-of-the-art classification models against their
TTA and BayTTA counterparts, in terms of precision (PR (%)), recall (RE (%)),
and F1-score (FS (%)) on the CRISPOR and GUIDE-seq gene editing datasets. The
highest accuracy per column is highlighted in bold.

| CRISPOR | GUIDE-seq
---|---|---
Models | PR | RE | FS | PR | RE | FS
ARL ([61]) | 88.33 | 93.66 | 90.72 | 33.91 | 56.81 | 42.38
COVID-19 ([43]) | 98.03 | 98.07 | 98.04 | 20.73 | 33.30 | 37.93
LoTeNet ([45]) | 89.74 | 92.10 | 94.59 | 33.30 | 12.12 | 17.74
IRv2+SA ([14]) | 99.73 | 95.20 | 96.71 | 37.86 | 41.91 | 37.54
PCXRNet ([18]) | 74.13 | 98.01 | 84.43 | 32.90 | 69.67 | 43.47
COVID-19+TTA | 96.12 | 97.41 | 97.55 | 60.68 | 60.69 | 58.78
COVID-19+BayTTA | 99.36 | 98.08 | 98.70 | 80.79 | 70.03 | 72.64
IRv2+SA+TTA | 88.99 | 95.91 | 91.92 | 30.76 | 7.76 | 10.18
IRv2+SA+BayTTA | 99.73 | 95.61 | 97.49 | 50.01 | 8.61 | 14.54

Second, we conducted a comparative study of the TTA-based, BayTTA-based, and
baseline state-of-the-art classification models which have been recently used
in the literature to process medical image data (see Tables 6 and 6). To
ensure a thorough performance assessment, we reported the experimental results
for the four following key metrics: accuracy, precision, recall, and F1-score.
Our analysis revealed that among state-of-the-art models compared, COVID-19
and IRv2+SA provided the highest overall accuracy, surpassing the results of
ARL, LoTeNet, and PCXRNet models. Thus, we used the TTA and BayTTA frameworks
in combination with these two top-performing models. In this experiment, we
considered original images as well as ten random augmentation samples
generated for each of them using rotations during the test phase. Our results,
reported in Table 6, indicate that the use of TTA leads to a decrease in the
accuracy of IRv2+SA, but the performance of TTA is notably worse for the
imbalanced chest X-ray dataset. Nonetheless, BayTTA improves the models
accuracy even in cases of suboptimal performance of TTA, demonstrating its
high effectiveness. The model performances in terms of precision, recall, and
F1-score (see Table 6) also demonstrate a much higher robustness of the
proposed BayTTA computational framework, compared to the TTA-based and
baseline models.

####  4.3.2 Results for gene editing datasets

Similarly to medical image data, we first evaluated the performance of the
proposed BayTTA method on gene editing datasets using it in combination with
four pre-trained CNN models VGG-16, MobileNetV2, DenseNet201, ResNet152V2 as
well as a custom CNN model with 5 layers. A comparison against the TTA-based
and baseline models was carried out. The results obtained are reported in
Tables 10 and 10.

We executed each baseline model three times, computing the mean accuracy and
standard deviation (STD) of accuracy. During the testing phase, we used
original images along with six random augmented samples (generated for each
original image through rotations and random erasing). By observing the results
obtained for both CRISPOR and GUIDE-seq gene editing datasets, we can conclude
that BayTTA demonstrated an enhanced accuracy performance, while consistently
yielding lower standard deviation values.

Fig. 5: Comparison of the TTA and BayTTA method performance on the skin cancer
dataset in terms of accuracy and standard deviation, while considering pre-
trained baseline models with rotate, zoom, and shift augmentations.  Fig. 6:
Comparison of the TTA and BayTTA method performance on the breast cancer
dataset in terms of accuracy and standard deviation, while considering the
pre-trained baseline models with rotate, zoom, and shift augmentations. Fig.
7: Comparison of the TTA and BayTTA method performance on the chest X-ray
dataset in terms of accuracy and standard deviation, while considering the
pre-trained baseline models with rotate, zoom, and shift augmentations.

In addition to the comparison with pre-trained baseline models, we conducted a
comparative analysis between BayTTA and TTA used in combination with state-of-
the-art DL classification models whose input was adapted to gene editing
datasets. The results obtained in this experiment are presented in Tables 10
and 10. To conduct our assessment, we used original images with ten randomly
created samples (generated for each original image during the testing phase
using rotations and random erasing). The results reported in Table 10 suggest
that the proposed BayTTA computational framework used in combination with the
COVID-19 model provided the highest accuracy and the lowest STD values for
both CRISPOR and GUIDE-seq datasets. Similar model performances can be
observed in Table 10, where the obtained precision, recall, and F1-score
metric values are reported - the COVID-19+BayTTA model provided the best
results overall.

###  4.4 Assessing the impact of different data augmentations and of
increasing the number of samples

Furthermore, we conducted a number of supplementary experiments to determine
the key factors that may influence the performance of the TTA and BayTTA
methods considered in this study. Our analysis was performed on the above-
described skin cancer, breast cancer, and chest X-ray datasets to study the
influence of sample sizes and different data augmentation techniques and
sample sizes.

Fig. 8: Comparison of the TTA and BayTTA method performance in terms of
accuracy with different numbers of samples, while considering the best
baseline model from Tables 4, 4, and 4 for each medical image dataset.

###  4.5 Evaluation of different data augmentations

To analyze the impact of various data augmentation types and compare the
results with traditional TTA, we tested the performance of the proposed BayTTA
method on the three following augmentation types: rotation, zoom, and shift.
These augmentations were incorporated into the data augmentation process of
TTA and BayTTA at the testing phase. During testing, we used six augmented
samples in addition to the original image in our experiments including the
VGG-16, MobileNetV2, DenseNet201, ResNet152V2, and InceptionResNetV2 pre-
trained CNN models.

Our results are graphically represented in Figures 5, 6, and 7, corresponding
to the skin cancer, breast cancer, and chest X-ray datasets, respectively.
These graphs demonstrate how the accuracy and STD of the model alter as we
vary data augmentation types, highlighting which augmentations offer higher
accuracy and boost the robustness of CNN models.

We can conclude that BayTTA generally enhances classification performance by
increasing the models accuracy and reducing STD after integrating various data
augmentations. This outcome was expected, as our proposed method combines
predictions from each candidate model based on a constraint on model
likelihood obtained from logistic regression. This combination results in
model-averaged predictions that account for both the model’s uncertainty and
accuracy.

###  4.6 Evaluation of increasing the number of samples

We analyzed the impact of increasing the number of augmented samples on the
models accuracy during the testing phase. As indicated in Tables 4, 4 and 4,
the combination of BMA and TTA contributes to a higher accuracy of DenseNet201
on the skin cancer dataset, of ResNet152V2 on the breast cancer dataset, and
of VGG-16 on the chest X-ray dataset, compared to baseline models and TTA.
Thus, we examined the effect of increasing the number of augmented samples in
these three specific cases.

Figure 8 demonstrates that the accuracy of both TTA and BayTTA increases as
the number of samples in the test data with diverse transformations grows.
Hence, training the model to recognize patterns in their diverse forms
generally enhances its ability to make accurate predictions on unseen
examples. Obviously, the increase in the amount of augmented data results in a
higher accuracy at the expense of a higher computational cost.

##  5 Conclusion

In this study, we investigated how the combination of Test-Time Augmentation
(TTA) and Bayesian Model Averaging (BMA) techniques can improve the accuracy
of pre-trained and state-of-the-art deep learning models applied in the field
of medical image classification. In particular, we focused on the two
following modeling aspects: TTA optimization and uncertainty evaluation based
on posterior probabilities. Our empirical observations indicate that using BMA
as a method for combining model predictions is highly effective for enhancing
the performance of traditional TTA. BMA allows one to assign weights to
different models based on their posterior probabilities. Therefore, the
proposed BayTTA technique can be viewed as a novel and effective methodology
that harnesses the combined strengths of TTA and BMA. One of the key strengths
of our approach is its capacity to quantify model uncertainty, which means
that we not only obtain more accurate predictions, but also gain insight into
the level of confidence the model has in each of its predictions. This insight
is crucial, especially in applications where incorrect predictions can lead to
critical consequences, such as medical image and gene editing analyses.

##  6 Acknowledgments

This work was supported by le Fonds Québécois de la Recherche sur la Nature et
les Technologies [grant 173878] and the Natural Sciences and Engineering
Research Council of Canada [grant 249644].

## References

  * Abdar et al. [2021a] Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U.R., et al., 2021a.  A review of uncertainty quantification in deep learning: Techniques, applications and challenges.  Information fusion 76, 243–297.
  * Abdar et al. [2023] Abdar, M., Salari, S., Qahremani, S., Lam, H.K., Karray, F., Hussain, S., Khosravi, A., Acharya, U.R., Makarenkov, V., Nahavandi, S., 2023.  Uncertaintyfusenet: robust uncertainty-aware hierarchical feature fusion model with ensemble monte carlo dropout for covid-19 detection.  Information Fusion 90, 364–381.
  * Abdar et al. [2021b] Abdar, M., Samami, M., Mahmoodabad, S.D., Doan, T., Mazoure, B., Hashemifesharaki, R., Liu, L., Khosravi, A., Acharya, U.R., Makarenkov, V., et al., 2021b.  Uncertainty quantification in skin cancer classification using three-way decision-based bayesian deep learning.  Computers in biology and medicine 135, 104418.
  * Abe et al. [2022] Abe, T., Buchanan, E.K., Pleiss, G., Zemel, R., Cunningham, J.P., 2022.  Deep ensembles work, but are they necessary?  Advances in Neural Information Processing Systems 35, 33646–33660.
  * Ayhan and Berens [2022] Ayhan, M.S., Berens, P., 2022.  Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks, in: Medical Imaging with Deep Learning.
  * Bahat and Shakhnarovich [2020] Bahat, Y., Shakhnarovich, G., 2020.  Classification confidence estimation with test-time data-augmentation.  arXiv preprint arXiv:2006.16705 .
  * Bartoš et al. [2021] Bartoš, F., Gronau, Q.F., Timmers, B., Otte, W.M., Ly, A., Wagenmakers, E.J., 2021.  Bayesian model-averaged meta-analysis in medicine.  Statistics in Medicine 40, 6743–6761.
  * Biswas et al. [2023] Biswas, A., Bhattacharya, P., Maity, S.P., Banik, R., 2023.  Data augmentation for improved brain tumor segmentation.  IETE Journal of Research 69, 2772–2782.
  * Bozkurt [2023] Bozkurt, F., 2023.  Skin lesion classification on dermatoscopic images using effective data augmentation and pre-trained deep learning approach.  Multimedia Tools and Applications 82, 18985–19003.
  * Charlier et al. [2021] Charlier, J., Nadon, R., Makarenkov, V., 2021.  Accurate deep learning off-target prediction with novel sgrna-dna sequence encoding in crispr-cas9 gene editing.  Bioinformatics 37, 2299–2307.
  * Chun et al. [2022] Chun, S., Lee, J.Y., Kim, J., 2022.  Cyclic test time augmentation with entropy weight method, in: Uncertainty in Artificial Intelligence, PMLR. pp. 433–442.
  * Cohen et al. [2019] Cohen, J., Rosenfeld, E., Kolter, Z., 2019.  Certified adversarial robustness via randomized smoothing, in: international conference on machine learning, PMLR. pp. 1310–1320.
  * D’Angelo and Fortuin [2021] D’Angelo, F., Fortuin, V., 2021.  Repulsive deep ensembles are bayesian.  Advances in Neural Information Processing Systems 34, 3451–3465.
  * Datta et al. [2021] Datta, S.K., Shaikh, M.A., Srihari, S.N., Gao, M., 2021.  Soft attention improves skin cancer classification performance, in: Interpretability of Machine Intelligence in Medical Image Computing, and Topological Data Analysis and Its Applications for Medical Data: 4th International Workshop, iMIMIC 2021, and 1st International Workshop, TDA4MedicalData 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, September 27, 2021, Proceedings 4, Springer. pp. 13–23.
  * Draper [1995] Draper, D., 1995.  Assessment and propagation of model uncertainty.  Journal of the Royal Statistical Society Series B: Statistical Methodology 57, 45–70.
  * Draper [2013] Draper, D., 2013.  Bayesian model specification: heuristics and examples.  Bayesian theory and applications , 409–431.
  * Edupuganti et al. [2021] Edupuganti, V., Mardani, M., Vasanawala, S., Pauly, J., 2021.  Uncertainty quantification in deep mri reconstruction.  IEEE Transactions on Medical Imaging 40, 239–250.
  * Feng et al. [2022] Feng, Y., Yang, X., Qiu, D., Zhang, H., Wei, D., Liu, J., 2022.  Pcxrnet: Pneumonia diagnosis from chest x-ray images using condense attention block and multiconvolution attention block.  IEEE Journal of Biomedical and Health Informatics 26, 1484–1495.
  * Gaillochet et al. [2022] Gaillochet, M., Desrosiers, C., Lombaert, H., 2022.  Taal: Test-time augmentation for active learning in medical image segmentation, in: MICCAI Workshop on Data Augmentation, Labelling, and Imperfections, Springer. pp. 43–53.
  * Gal and Ghahramani [2016] Gal, Y., Ghahramani, Z., 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning, in: international conference on machine learning, PMLR. pp. 1050–1059.
  * Goceri [2023] Goceri, E., 2023.  Medical image data augmentation: techniques, comparisons and interpretations.  Artificial Intelligence Review , 1–45.
  * Gour and Jain [2022] Gour, M., Jain, S., 2022.  Uncertainty-aware convolutional neural network for covid-19 x-ray images classification.  Computers in biology and medicine 140, 105047.
  * Haeussler et al. [2016] Haeussler, M., Schönig, K., Eckert, H., Eschstruth, A., Mianné, J., Renaud, J.B., Schneider-Maunoury, S., Shkumatava, A., Teboul, L., Kent, J., et al., 2016.  Evaluation of off-target and on-target scoring algorithms and integration into the guide rna selection tool crispor.  Genome biology 17, 1–12.
  * Hamedani-KarAzmoudehFar et al. [2023] Hamedani-KarAzmoudehFar, F., Tavakkoli-Moghaddam, R., Tajally, A.R., Aria, S.S., 2023.  Breast cancer classification by a new approach to assessing deep neural network-based uncertainty quantification methods.  Biomedical Signal Processing and Control 79, 104057.
  * Han et al. [2024] Han, Q., Qian, X., Xu, H., Wu, K., Meng, L., Qiu, Z., Weng, T., Zhou, B., Gao, X., 2024.  Dm-cnn: Dynamic multi-scale convolutional neural network with uncertainty quantification for medical image classification.  Computers in Biology and Medicine 168, 107758.
  * He et al. [2020] He, B., Lakshminarayanan, B., Teh, Y.W., 2020.  Bayesian deep ensembles via the neural tangent kernel.  Advances in neural information processing systems 33, 1010–1022.
  * He et al. [2016] He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.
  * Hoeting et al. [1998] Hoeting, J.A., Madigan, D., Raftery, A.E., Volinsky, C.T., 1998.  Bayesian model averaging, in: Proceedings of the AAAI workshop on integrating multiple learned models, Citeseer. pp. 77–83.
  * Hoffmann et al. [2021] Hoffmann, L., Fortmeier, I., Elster, C., 2021.  Uncertainty quantification by ensemble learning for computational optical form measurements.  Machine Learning: Science and Technology 2, 035030.
  * Hüllermeier and Waegeman [2021] Hüllermeier, E., Waegeman, W., 2021.  Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.  Machine Learning 110, 457–506.
  * Izmailov et al. [2021] Izmailov, P., Nicholson, P., Lotfi, S., Wilson, A.G., 2021.  Dangers of bayesian model averaging under covariate shift.  Advances in Neural Information Processing Systems 34, 3309–3322.
  * Jin et al. [2018] Jin, H., Li, Z., Tong, R., Lin, L., 2018.  A deep 3d residual cnn for false-positive reduction in pulmonary nodule detection.  Medical physics 45, 2097–2107.
  * Kermany et al. [2018] Kermany, D., Zhang, K., Goldbaum, M., 2018.  Large dataset of labeled optical coherence tomography (oct) and chest x-ray images.  Mendeley Data 3.
  * Kim et al. [2020] Kim, I., Kim, Y., Kim, S., 2020.  Learning loss for test-time augmentation.  Advances in Neural Information Processing Systems 33, 4163–4174.
  * Kirillov et al. [2022] Kirillov, B., Savitskaya, E., Panov, M., Ogurtsov, A.Y., Shabalina, S.A., Koonin, E.V., Severinov, K.V., 2022.  Uncertainty-aware and interpretable evaluation of cas9–grna and cas12a–grna specificity for fully matched and partially mismatched targets with deep kernel learning.  Nucleic acids research 50, e11–e11.
  * Krizhevsky et al. [2012] Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems 25.
  * Lambert et al. [2022] Lambert, B., Forbes, F., Tucholka, A., Doyle, S., Dehaene, H., Dojat, M., 2022.  Trustworthy clinical ai solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis.  arXiv preprint arXiv:2210.03736 .
  * Loftus et al. [2022] Loftus, T.J., Shickel, B., Ruppert, M.M., Balch, J.A., Ozrazgat-Baslanti, T., Tighe, P.J., Efron, P.A., Hogan, W.R., Rashidi, P., Upchurch Jr, G.R., et al., 2022.  Uncertainty-aware deep learning in healthcare: a scoping review.  PLOS digital health 1, e0000085.
  * Lyzhov et al. [2020] Lyzhov, A., Molchanova, Y., Ashukha, A., Molchanov, D., Vetrov, D., 2020.  Greedy policy search: A simple baseline for learnable test-time augmentation, in: Conference on Uncertainty in Artificial Intelligence, PMLR. pp. 1308–1317.
  * Maron et al. [2021] Maron, R.C., Haggenmüller, S., von Kalle, C., Utikal, J.S., Meier, F., Gellrich, F.F., Hauschild, A., French, L.E., Schlaak, M., Ghoreschi, K., et al., 2021.  Robustness of convolutional neural networks in recognition of pigmented skin lesions.  European journal of cancer 145, 81–91.
  * Mazoure et al. [2022] Mazoure, B., Mazoure, A., Bédard, J., Makarenkov, V., 2022.  Dunescan: a web server for uncertainty estimation in skin cancer detection with deep neural networks.  Scientific Reports 12, 179.
  * Monteith et al. [2011] Monteith, K., Carroll, J.L., Seppi, K., Martinez, T., 2011.  Turning bayesian model averaging into bayesian model combination, in: The 2011 international joint conference on neural networks, IEEE. pp. 2657–2663.
  * Ozturk et al. [2020] Ozturk, T., Talo, M., Yildirim, E.A., Baloglu, U.B., Yildirim, O., Acharya, U.R., 2020.  Automated detection of covid-19 cases using deep neural networks with x-ray images.  Computers in biology and medicine 121, 103792.
  * Prechelt [2002] Prechelt, L., 2002.  Early stopping-but when?, in: Neural Networks: Tricks of the trade. Springer, pp. 55–69.
  * Raghavendra Selvan [2020] Raghavendra Selvan, E.B.D., 2020.  Tensor networks for medical image classification, in: International Conference on Medical Imaging with Deep Learning – Full Paper Track.  URL: https://openreview.net/forum?id=jjk6bxk07G.
  * Rahaman et al. [2021] Rahaman, R., et al., 2021.  Uncertainty quantification and deep ensembles.  Advances in Neural Information Processing Systems 34, 20063–20075.
  * Rudner et al. [2022] Rudner, T.G., Chen, Z., Teh, Y.W., Gal, Y., 2022.  Tractable function-space variational inference in bayesian neural networks.  Advances in Neural Information Processing Systems 35, 22686–22698.
  * Seoni et al. [2023] Seoni, S., Jahmunah, V., Salvi, M., Barua, P.D., Molinari, F., Acharya, U.R., 2023.  Application of uncertainty quantification to artificial intelligence in healthcare: A review of last decade (2013–2023).  Computers in Biology and Medicine , 107441.
  * Shanmugam et al. [2021] Shanmugam, D., Blalock, D., Balakrishnan, G., Guttag, J., 2021.  Better aggregation in test-time augmentation, in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 1214–1223.
  * Sherkatghanad et al. [2023] Sherkatghanad, Z., Abdar, M., Charlier, J., Makarenkov, V., 2023.  Using traditional machine learning and deep learning methods for on-and off-target prediction in crispr/cas9: a review.  Briefings in Bioinformatics 24, bbad131.
  * Sherkatghanad et al. [2020] Sherkatghanad, Z., Akhondzadeh, M., Salari, S., Zomorodi-Moghadam, M., Abdar, M., Acharya, U.R., Khosrowabadi, R., Salari, V., 2020.  Automated detection of autism spectrum disorder using a convolutional neural network.  Frontiers in neuroscience 13, 1325.
  * Son and Kang [2023] Son, J., Kang, S., 2023.  Efficient improvement of classification accuracy via selective test-time augmentation.  Information Sciences 642, 119148.
  * Song et al. [2017] Song, Y., Kim, T., Nowozin, S., Ermon, S., Kushman, N., 2017.  Pixeldefend: Leveraging generative models to understand and defend against adversarial examples.  arXiv preprint arXiv:1710.10766 .
  * Szegedy et al. [2015] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A., 2015.  Going deeper with convolutions, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9.
  * Tsai et al. [2015] Tsai, S.Q., Zheng, Z., Nguyen, N.T., Liebers, M., Topkar, V.V., Thapar, V., Wyvekens, N., Khayter, C., Iafrate, A.J., Le, L.P., et al., 2015.  Guide-seq enables genome-wide profiling of off-target cleavage by crispr-cas nucleases.  Nature biotechnology 33, 187–197.
  * Tsuneki [2022] Tsuneki, M., 2022.  Deep learning models in medical image analysis.  Journal of Oral Biosciences 64, 312–320.
  * Wang et al. [2019a] Wang, G., Li, W., Aertsen, M., Deprest, J., Ourselin, S., Vercauteren, T., 2019a.  Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks.  Neurocomputing 338, 34–45.
  * Wang et al. [2019b] Wang, G., Li, W., Ourselin, S., Vercauteren, T., 2019b.  Automatic brain tumor segmentation using convolutional neural networks with test-time augmentation, in: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th International Workshop, BrainLes 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Selected Papers, Part II 4, Springer. pp. 61–72.
  * Wang and Van Hoof [2020] Wang, Q., Van Hoof, H., 2020.  Doubly stochastic variational inference for neural processes with hierarchical latent variables, in: International Conference on Machine Learning, PMLR. pp. 10018–10028.
  * Wintle et al. [2003] Wintle, B.A., McCarthy, M.A., Volinsky, C.T., Kavanagh, R.P., 2003.  The use of bayesian model averaging to better represent uncertainty in ecological models.  Conservation biology 17, 1579–1590.
  * Zhang et al. [2019] Zhang, J., Xie, Y., Xia, Y., Shen, C., 2019.  Attention residual learning for skin lesion classification.  IEEE transactions on medical imaging 38, 2092–2103.
  * Zhang et al. [2020] Zhang, Y., Long, Y., Yin, R., Kwoh, C.K., 2020.  Dl-crispr: a deep learning method for off-target activity prediction in crispr/cas9 with data augmentation.  IEEE Access 8, 76610–76617.