11institutetext: Department of Computer Science Engineering, Shiv Nadar
Institute of Eminence, Delhi-NCR, India. 11email<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>
# Enhancing Eye Disease Diagnosis with Deep Learning and Synthetic Data
Augmentation

Saideep Kilaru 11 Kothamasu Jayachandra 11 Tanishka Yagneshwar 11 Suchi
Kumari∗ 11

###### Abstract

In recent years, the focus is on improving the diagnosis of diabetic
retinopathy (DR) using machine learning and deep learning technologies.
Researchers have explored various approaches, including the use of high-
definition medical imaging, AI-driven algorithms such as convolutional neural
networks (CNNs) and generative adversarial networks (GANs). Among all the
available tools, CNNs have emerged as a preferred tool due to their superior
classification accuracy and efficiency. Although the accuracy of CNNs is
comparatively better but it can be improved by introducing some hybrid models
by combining various machine learning and deep learning models. Therefore, in
this paper, an ensemble learning technique is proposed for early detection and
management of DR with higher accuracy. The proposed model is tested on the
APTOS dataset and it is showing supremacy on the validation accuracy ($99\%)$
in comparison to the previous models. Hence, the model can be helpful for
early detection and treatment of the DR, thereby enhancing the overall quality
of care for affected individuals.

keyword: Diatetic retinopathy, ensemble learning, convolution neural network,
eye disease diagnosis, APTOS

##  1 Introduction

Among all the sense organs, eyes are considered as a crucial tool for human
observation and perception to real worlds. The exposure of eyes with external
space leads to the increasing chance of contact with external pathogens. Some
viruses are entering to human body through the eyes, causing eye disease and
affecting the visual functions. Sometimes these diseases or some eye injury
cause visual impairment and blindness. It presents significant challenges in
the lives of the affected people to focus on their needs. Researchers from
various domains are providing various approaches to enhance the quality of
life for the visually impaired people. Timely and accurate diagnosis of eye
conditions can lead to prompt treatment and improved prognosis. Therefore, in
the current era, researchers analyzed artificial intelligence driven
technologies in the diagnosis of eye diseases, the development of visual aids,
assistive devices, etc.

Due to the advancement in the modern medical imaging technology, the high-
definition medical images play an important role in the disease diagnosis.
Fundus camera is used to capture such images and can also be used to capture
the complete structure of the retinal images such as vessels, optic cup, disc,
macula, etc. Analyzing structural changes aids in identifying various diseases
including Diabetic Retinopathy (DR) [6], Diabetic Macular Edema (DME),
Glaucoma, and others. Among all the diseases, DR is the most common disease,
caused due to the increasing rate of diabetic patients in the recent past.
According to WHO [26], approx. 422 million people worldwide have diabetes, the
majority living in low-and middle-income countries. Early screening for DR is
critical in preventing irreversible blindness, and this can be accomplished
through manual examination or the application of intelligent tools on fundus
images. Intelligent screening not only streamlines the process and reduces
manpower requirements but also facilitates large-scale DR screening [4]. Wang
et al. [25] provides an overview of the recent research works in artificial
intelligence-driven eye disease diagnosis and assistive technologies for the
visually impaired human beings. It outlines potential future directions where
artificial intelligence could further benefit the visually impaired.

In recent years, medical image processing based on deep learning has become an
important research direction, among which DR is an important research domain.
The accuracy of the diagnostic model heavily depends on the input images as
well as the labeling done by some professional doctors. The high-quality
images can be generated automatically using an unsupervised adversarial
generation model named as Generative Adversarial Network (GAN) [27]. It can
generate any number of high-quality images through the random distribution of
noise. However, GAN, is considered as inefficient and uncontrollable for
complex data. A controlled variant of GAN [16] is proposed by applying some
constraints to the generator and discriminator for pre-training of the data.
Liang et al. [14] proposed an end to end CGAN by applying constraint on the
loss function, namely class feature loss and retinal detail loss.

While GANs offer capabilities for image generation and data augmentation, deep
learning based model, particularly convolutional neural network (CNN) is
considered as a preferred tool for predicting diabetic retinopathy due to its
superior classification accuracy, interpretability, efficiency, and many more.
[18] proposed a CNN approach for diagnosing severity of DR from digital fundus
image with validation accuracy of $75\%$ on Kaggle dataset. Lam et al. [13]
used CNN and applied transfer learning on pretrained GoogLeNet and AlexNet
models to automatically detect DR, and multiple fundus image datasets, and
achieved an accuracy rate of $75.5\%$. A deep residual network (ResNet) model
[8] is also widely used for the diagnosis of DR. Wan et al. [24] achieved
accuracy rate of 95.68% by employing transfer learning architectures,
including AlexNet, VGGNet, GoogleNet, and ResNet. They utilized publicly
available data from the Kaggle platform for their study. Tufail et al. [23]
used 3D convolutional neural network (3D-CNN) architectures for binary and
multiclass (5 classes) classification of DR. Macsik et al. [15] proposed local
binary convolutional neural network (LBCNN) deterministic filter generation
which can do diagnosis less learnable parameters and also with less memory
usage in comparison with CNN.

Multiple hybrid models are also proposed considering different deep learning
(DL) models, and then the processed result is feed some classifiers for
classification. Ali et al. [1] used two deep learning models, Resnet50 and
Inceptionv3, for preprocessing the images and then feed them into the CNN for
classification. Nahiduzzaman et al. [17] utilized Ben Graham’s technique and
contrast limited adaptive histogram equalization (CLAHE) for preprocessing DR
images. They then introduced a novel hybrid CNN-singular value decomposition
(SVD) model to reduce input features for classifiers, followed by employing an
Extreme Learning Machine (ELM) algorithm for classification to minimize
training time. Bilal et al. [3] employed a multi-stage strategy encompassing
data preprocessing, feature extraction using a hybrid CNN-SVD model, and
classification utilizing Improved Support Vector Machine with Radial Basis
Function (ISVM-RBF), Decision Trees (DT), and K-Nearest Neighbors (KNN)
techniques.

Some of the researchers proposed various ensemble learning model to enhance
the accuracy of the DR image classification. Antal et al. [2] classifies
images based on characteristic features extracted by lesion detection and
anatomical part recognition algorithms and achieved $90\%$ 0f accuracy in
Messidor database. Somasundaram et al. [21] showed that the ensamble machine
learning model perform better than the machine learning model alone. Sikder et
al. [20] employed ensemble learning with feature extraction via histogram and
GLCM. Utilizing XGBoost, then Genetic Algorithm, we achieved $94.20\%$
classification accuracy by identifying relevant features. ensemble of five
deep CNN models trained on Kaggle dataset. Qummar et al. [19] demonstrate
improved detection across all DR stages, achieving better classification
accuracy at all the levels (0 to 4: 0—no DR, 1—mild, 2—moderate, 3—severe,
4—indicates proliferative). The authors have improved the prediction accuracy
but they have not considered the multi labeling of the dataset. In this paper,
the labels are converted into multi labels and a balanced class of the dataset
is created at the time of data preprocessing. Decisions from multiple base
models; Densenet121 and InceptionV3, are combined to improve the overall
performance of the model.

The key offerings of the manuscript are as follows:

  1. 1.

Propose an ensemble learning technique by considering two base models;
Densenet121 and InceptionV3, for early detection and management of diabetic
retinopathy.

  2. 2.

Generate a balanced dataset by oversampling of minority classes using the
multi label formatting is the data preprocessing phase.

  3. 3.

Evaluate the performance of the proposed ensemble model on the validation set
of the dataset.

  4. 4.

Provide a comparative analysis with the previously existing models.

The subsequent sections of the paper are structured as follows: Section 2
discusses the related deep learning models comprehensively. In Section 3, we
delve into the proposed strategies which are divided into multiple phases. The
results and in-depth analysis are presented in Section 4. Finally, Section 5
provides the conclusions drawn from the proposed research and outlines the
future direction.

##  2 Related Work

When it comes to diabetic retinopathy (DR) image classification, several
Convolutional Neural Network (CNN) models have been employed to achieve
accurate results. In this section, some of the notable CNN architectures are
discussed which are used in DR classification.

### Visual Geometry Group 16 (VGG16)

VGG16 [5] is a deep CNN architecture with 16 weight layers, comprising a
series of convolutional, pooling, and fully connected layers. It utilizes
small $3\times 3$ convolutional filters to capture fine details and complex
patterns in images. It is known for its uniform and simple structure and it
performs well on a variety of image classification tasks, including DR
classification. However, its deep structure can lead to higher computational
costs, making it less efficient for resource-limited environments.

### Inception V3 (GoogLeNet)

Inception V3[22], also known as GoogLeNet, employs inception modules that use
multiple filter sizes in parallel within a single layer to capture various
feature scales. The use of inception modules improves feature extraction while
keeping the model computationally efficient by reducing the number of
parameters. It provides high accuracy in DR classification, due to its ability
to capture a wide range of features.

### Xception

Xception [7] extends the Inception architecture by replacing standard
convolutions with depthwise separable convolutions. This modification reduces
the number of parameters and computational cost while maintaining or even
improving performance. It is well suited in DR classification as it can
efficiently extract complex features from images.

### DenseNet121

DenseNet121 [28] is a CNN with $121$ layers featuring dense connectivity,
where each layer is connected to every other layer in a feed-forward manner.
This dense connectivity allows for better feature reuse and efficient gradient
flow throughout the network. This helps the model to learn and leverage
complex features from images leading to strong performance on image
classification tasks especially DR classification.

### Ensemble model

Ensemble models have gained attention for their effectiveness in predicting
diabetic retinopathy due to their ability to combine multiple deep learning
architectures for improved performance. Chaurasia et al. [4] proposed a
transfer learning-driven ensemble model that integrates six CNN models,
including DenseNet-169, VGG-19, ResNet101-V2, MobileNet-V2, and Inception-V3,
achieving up to $98\%$ accuracy in detecting DR stages. Another research [19]
used an ensemble approach with publicly available retinal image datasets to
train five deep CNN models (ResNet50, InceptionV3, Xception, DenseNet121,
DenseNet169) to encode rich features and enhance classification across
different stages of DR. Moreover, Kale et al. [11] proposed a stacked ensemble
model utilized to classify five severity levels of DR (ranging from No-DR to
proliferative-DR), which improved accuracy in identifying the correct stage of
the disease.

In addition to these models, transfer learning [22, 11] has been a key
strategy in DR classification. By fine-tuning pre-trained models such as
VGG16, Inception V3, and DenseNet121 on specific DR datasets, researchers have
been able to achieve high accuracy and robust performance even with smaller
training datasets. These models leverage rich features learned from large
datasets, predictive capabilities of deep learning approaches and can be
adapted effectively to new tasks such as DR classification and detection.

##  3 Proposed Approach

In this section, the approaches to the design of the ensemble model is
discussed. The section contains architectural diagram, training details,
hyper-parameters, data preprocessing steps, transfer learning mechanism with
CNN, resampling and, multi label approach and evaluation and refinement
parameters.

###  3.1 Data Acquisition and Preprocessing

A public dataset, APTOS 2019 [12] of retinal images labeled with the
corresponding eye disease is chosen for the prediction of diabetic
retinopathy. Some of the images of the datasets before applying preprocessing
is shown in Fig. 1. The preprocessing of retinal images helps to improve model
performance and accuracy. Hence, some parameters such as random horizontal and
vertical flips, random zooming (zoom range: 0.15), constant fill mode for
points outside image boundaries, are considered for preprocessing the images.
Some of the preprocessed images are shown in Fig. 2. The steps to perform
preprocessing is as follows:

  * •

Step 1: Crop any extra dark pixels present at the sides of the images.

  * •

Step 2: Standardize the image size by resizing it.

  * •

Step 3: Draw a circle from the center of the image to provide a consistent
shape for all images.

  * •

Step 4: Apply a smoothing technique using Gaussian blur to remove noise caused
by varying resolutions and lighting conditions.

The formulation of the output image is provided in Eq. (1).

$\text{output}(x,y)=\frac{1}{{2\pi\sigma_{x}\sigma_{y}}}\cdot\sum_{i=-k}^{k}\sum_{j=-k}^{k}\text{input}(x+i,y+j)\cdot\exp\left(-\frac{i^{2}}{2\sigma_{x}^{2}}-\frac{j^{2}}{2\sigma_{y}^{2}}\right).$
(1)

Where $\text{output}(x,y)$ is the value of the pixel at coordinates $(x,y)$ in
the output image, $\text{input}(x+i,y+j)$ is the value of the pixel at
coordinates $(x+i,y+j)$ in the input image, $\sigma_{x}$ and $\sigma_{y}$ are
the standard deviations in the X and Y directions, which determine the amount
of blur, and $k$ is the half-size of the kernel in both the X and Y
directions.

(a) Image 1

(b) Image 2

(c) Image 3

Figure 1: Dataset Before Preprocessing

(a) Image 1

(b) Image 2

(c) Image 3

Figure 2: Dataset After Preprocessing

###  3.2 Convolutional Neural Network (CNN)

Due to multi-stage architecture, CNN is the most efficient deep learning model
for classifying images data. It consists of several layers, including
convolution (CONV), activation (ReLU), and pooling (POOL). Each of these
layers transforms the input data in specific ways. Here are the steps in CNN
[10]:

  1. 1.

Convolution (CONV) Layer: It accepts an input volume of size $[X_{1}\times
Y_{1}\times Z_{1}]$, where $X_{1}$ is the width, $Y_{1}$ is the height, and
$Z_{1}$ is the depth. The output volumes (i.e., convolution maps) are of size
$[X_{2}\times Y_{2}\times Z_{2}]$, is calculated using the following
equations:

$X_{2}=\left\lfloor\frac{X_{1}+2\cdot P-F}{S}\right\rfloor+1$
$Y_{2}=\left\lfloor\frac{Y_{1}+2\cdot P-F}{S}\right\rfloor+1$ $Z_{2}=K$

where, $F$ is the spatial extent of the filter, $K$ is the number of filters
or convolution kernels, $P$ is the amount of zero padding, and $S$ is the
stride (step size with which the filter slides).

  2. 2.

Activation (ReLU) Layer: This layer applies an activation function (such as
the max function) element-wise to introduce non-linearity to the output of the
CONV layer. The function commonly used is ReLU: $f(x)=\max(0,x)$. This
operation does not change the size of the volume.

  3. 3.

Pooling (POOL) Layer: It uses the MAX operation to optimize the spatial size
of the representation and reduce the number of parameters. The POOL layer
produces an output volume of size $[X_{2}\times Y_{2}\times Z_{2}]$ and is
calculated using:

$X_{2}=\left\lfloor\frac{X_{1}-F}{S}\right\rfloor+1$
$Y_{2}=\left\lfloor\frac{Y_{1}-F}{S}\right\rfloor+1$ $Z_{2}=Z_{1}$

Where $F$ is the size of the pooling region and $S$ is the stride.

  4. 4.

Flattening and Fully Connected (FC) Layers: After the feature extraction, the
output from the POOL layer is flattened into a feature vector. This vector is
then connected to a series of fully connected layers (multilayer perceptron)
to perform classification on the features.

These steps combine to form the feature extraction and classification process
in a CNN.

###  3.3 Transfer Learning with CNNs

Select a pre-trained CNN model such as InceptionV3, DenseNet121, or
EfficientNet, which are commonly trained on extensive image datasets like
ImageNet. Preserve the initial layers of the pre-trained model, as they are
responsible for extracting general image features and should remain unchanged.
Add new layers on top of the frozen pre-trained model. These new layers will
be specific to the eye disease classification task. The number and type of
layers will depend on the complexity of the problem.

###  3.4 Resampling

A resampling technique is proposed for addressing class imbalance in multi-
label classification tasks. By setting a desired number of samples per class,
we ensured balanced representation across all classes in the dataset. Our
approach involved replicating indices of samples for each class based on the
desired count which is 700, with additional random sampling to account for any
remainder. Through this process, we aimed to mitigate the biases introduced by
class imbalance, fostering the development of more equitable and accurate
multi-label classification models. Before resampling, our dataset exhibited
class imbalances: $1543$ samples in class $0$, $314$ samples in class $1$,
$849$ samples in class $2$, $164$ samples in class $3$, and $251$ samples in
class $4$. However, after resampling, all classes contained precisely 700
samples.

###  3.5 Multi Label

In the proposed work, the target encoding approach is modified for the
transition from a single-label to multilabel classification. Traditionally, a
class label after one hot encoding would be represented by a single $"1"$ in a
binary vector, with other classes indicated by $"0"$s. However, we redefined
this scheme to signify the inclusion of all preceding classes up to the target
class. For example, instead of encoding class $4$ retinopathy as $[0,0,0,1]$,
the proposed approach predicts $[1,1,1,1]$, encompassing all prior classes.
This adaptation acknowledges the progressive nature of retinopathy severity,
by enhancing the model’s capacity to differentiate variations in disease
progression. Detailed steps are provided in Flow diagram in Figure 3.

###  3.6 Architecture of the proposed ensemble model

The process for predicting DR disease is outlined in Fig. 3. Initially, data
preprocessing is performed to enhance accuracy. Subsequently, DenseNet121 and
InceptionV3 base models are trained on the APTOS dataset. In the third step,
the outcomes from both base models are aggregated and further trained.
Ultimately, the prediction score is assessed.

Data Preprocessing • Resize all images to $224\times 224$ pixels • Convert
labels to multi-label format (e.g., $[0,0,0,1]$ becomes $[1,1,1,1]$) •
Oversample the minority classes to create a balanced dataset Base Model
Training • Define and compile two base models: DenseNet121 and InceptionV3 •
Train both base models on the dataset • Monitor validation quadratic weighted
kappa during training Stacking Ensemble • Combine the predictions from both
base models by stacking them as input features • Define and compile the meta
model • Train the meta-model on the stacked features from base models •
Monitor validation quadratic weighted kappa during training Ensemble
Evaluation • Evaluate the performance of the stacking ensemble on the
validation set Figure 3: Flow diagram of the steps involved from data pre-
processing to Ensemble model evaluation of the diabetic retinopathy disease.

The architectural diagram of the proposed model is shown in Fig. 4. Transfer
learning was employed using pre-trained models, namely DenseNet121 [9] and
InceptionV3 [22]. The models were fine-tuned on the retinal image dataset
using a custom multi-label classification approach. The target variable was
transformed into a multi-label format, where each class encompasses all the
classes before it.

Figure 4: Model Architceture

The model consists of two parts. The first part uses DenseNet121 as the base
model, followed by a sequence of layers: pooling, dense (ReLU activation),
dropout, and sigmoid activation. The second part mirrors the first, but with
InceptionV3 as the base model. The outputs from both parts are combined and
passed through alternating dense (ReLU activation) and dropout layers, ending
with a sigmoid activation function for the final output. The model’s loss is
calculated using Binary Cross Entropy between the predicted output and the
true labels. The steps are provided here in 3.6.

$\displaystyle y_{11}$ $\displaystyle=\text{densenet121}-(1)$ $\displaystyle
y_{12}$ $\displaystyle=\text{pooling}(y_{11})-(2)$ $\displaystyle y_{13}$
$\displaystyle=\text{dense\\_relu}(y_{12})-(3)$ $\displaystyle y_{14}$
$\displaystyle=\text{dropout}(y_{13})-(4)$ $\displaystyle y_{15}$
$\displaystyle=\text{sigmoid}(y_{14})-(5)$ $\displaystyle y_{21}$
$\displaystyle=\text{inceptionV3}-(6)$ $\displaystyle y_{22}$
$\displaystyle=\text{pooling}(y_{21})-(7)$ $\displaystyle y_{23}$
$\displaystyle=\text{dense\\_relu}(y_{22})-(8)$ $\displaystyle y_{24}$
$\displaystyle=\text{dropout}(y_{23})-(9)$ $\displaystyle y_{25}$
$\displaystyle=\text{sigmoid}(y_{24})-(10)$ $\displaystyle y_{3}$
$\displaystyle=\text{stack}(y_{15},y_{25})-(11)$

$\displaystyle y_{31}$ $\displaystyle=\text{dense\\_relu}(y_{3})-(12)$
$\displaystyle y_{32}$ $\displaystyle=\text{dense\\_relu}(y_{31})-(13)$
$\displaystyle y_{33}$ $\displaystyle=\text{dropout}(y_{32})-(14)$
$\displaystyle y_{34}$ $\displaystyle=\text{dense\\_relu}(y_{33})-(15)$
$\displaystyle y_{35}$ $\displaystyle=\text{dense\\_relu}(y_{34})-(16)$
$\displaystyle y_{36}$ $\displaystyle=\text{dropout}(y_{35})-(17)$
$\displaystyle y_{37}$ $\displaystyle=\text{dense\\_relu}(y_{36})-(18)$
$\displaystyle y_{38}$ $\displaystyle=\text{dense\\_relu}(y_{37})-(19)$
$\displaystyle y_{39}$ $\displaystyle=\text{dense\\_relu}(y_{38})-(20)$
$\displaystyle\text{pred}_{i}$ $\displaystyle=\text{sigmoid}(y_{39})-(21)$
$\text{loss}=\text{BCrossEntropy}(\text{pred}_{i},y_{i})$

###  3.7 Training details and hyperparameters

The model training and evaluation process involved the multiple steps. In the
data Preprocessing, the images were resized to $224\times 224$ pixels and
preprocessed using the OpenCV and PIL libraries. A balanced dataset was
created by oversampling the minority classes to a desired number of samples
(500 per class) to handle class imbalance. The specification for the
hyperparameters are provided in Table 1.

Table 1: Hyperparameters Hyper parameters | Specification
---|---
Loss Function : | Binary Cross-Entropy
Optimizer : | Adam with a learning rate of 0.00005
Batch Size : | 32
Regularization: | L2 (0.001) on Dense layers
Dropout Rate: | 0.5
Data Augmentation: | Random zoom(0.15), horizontal flip, and vertical flip
Epochs: |  $15$ epochs for DenseNet121 and InceptionV3 models

After training the individual models, a stacking ensemble approach was used by
combining the predictions from DenseNet121 and InceptionV3 models. A meta-
model was trained on these stacked predictions to obtain the final
predictions. A neural network with multiple dense layers and dropout
regularization are used. Total 200 epochs, batch size of 64, binary cross-
entropy loss, and Adam optimizer. Finally the models were evaluated using
precision, recall, F1-score, quadratic weighted Kappa score, and accuracy.

###  3.8 Evaluation and Refinement

The performance of the fine-tuned CNN models are evaluated on a separate
validation dataset (not used for training). Metrics like accuracy, precision,
recall, F1 score, and confusion matrix can be used. The definition of all
these parameters are provided as follows:

  * •

Accuracy: Measures the proportion of correctly predicted outcomes over the
total number of predictions made

  * •

Recall:  It is defined as the number of positive samples the model correctly
recalled out of the total number of actual positive samples (TP + FN).

  * •

Precision: It is defined as the number of positive samples that the model was
able to precisely predict out of the total number of positive predicted
samples (TP + FP).

  * •

F1 score: It is a single metric that provides a way to combine both precision
and recall into one score by taking harmonic mean of precision and recall.

$F_{1}=\frac{2\times\text{precision}\times\text{recall}}{\text{precision}+\text{recall}}$

  * •

Confusion matrix: It compares model predictions against actual labels to
compute metrics like accuracy, precision, recall, and F1 score, providing
insights into classification performance for multi-class problems. The nearer
the value is to $1$, the more accurate and reliable the model’s predictions
become.

##  4 Results and Analysis

In this section, DR prediction accuracy is compared with some well-known
machine learning and deep learning models, and performance is evaluated
considering accuracy, precision, recall, and F1 score. A confusion matrix is
also plotted for the models.

###  4.1 Dataset description

This study utilizes the publicly available APTOS dataset for diabetic
retinopathy detection, which consists of fundus images. These images originate
from various camera models, potentially leading to differences in visual
appearance between left and right images. The dataset categorizes the
sensitivity of diabetic retinopathy presence on a scale of 0 to 4: 0—no DR,
1—mild, 2—moderate, 3—severe, 4—indicates proliferative DR. From the dataset,
a random sample of 5,590 images across five categories is selected.
Specifically, 3,662 images (3,112 for training and 1,928 for testing) are
employed. Prior to training, the dataset undergoes preprocessing steps to
ensure its quality and suitability for the task at hand.

###  4.2 Evaluation of the proposed ensemble model on the dataset

During both the training and validation phases, the performance of the
proposed ensemble model is assessed by plotting validation loss (in Fig. 5(a))
and accuracy curves (in Fig. 5(b)). Validation loss serves as an indicator of
the model’s error minimization on unseen data, while validation accuracy
reflects its predictive capability. From the figures, it can be analyzed that
the validation loss and accuracy is improving over epochs. Moreover, the gap
between training and validation metrics suggests that the model is not
overfitting to the training data.

(a) Loss Curve

(b) Accuracy Curve

Figure 5: Validation loss and validation Accuracy Curve

###  4.3 Discussion of results

Tabel 2 and Fig. 6 illustrate the superior performance of the proposed hybrid
model against other well-known deep learning architectures in detecting
diabetic retinopathy. Our model demonstrates higher accuracy and recall,
showcasing its effectiveness in this task. Specifically, our ensemble model,
leveraging InceptionV3 and Densenet121, achieves outstanding results across
all metrics. Notably, it achieves a specificity and precision of 0.99, an
accuracy rate of 0.99, and a Cohen’s Kappa Score of 0.98. These outcomes
underscore the capability of our model to capture both local and global
features in retinal images, thereby significantly enhancing diabetic
retinopathy detection. Consequently, our ensemble model emerges as a promising
tool to support clinicians in diagnosis.

Table 2: Comparative analysis among the proposed model and existing models
across evaluation metrics including Precision, Recall, Accuracy, Cohen’s Kappa
Score, and F1 Score.

Model | Precision | Recall | Accuracy | Cohen’s | F1
---|---|---|---|---|---
|  |  |  | Kappa Score |
Vgg16[5] | 0.95 | 0.99 | 0.96 | 0.92 | 0.97
Resnet50 | 0.98 | 0.97 | 0.97 | 0.94 | 0.97
Xception[7] | 0.97 | 0.99 | 0.98 | 0.95 | 0.97
InceptionV3[22] | 0.98 | 0.98 | 0.98 | 0.96 | 0.98
Densenet121[28] | 0.98 | 0.97 | 0.97 | 0.94 | 0.97
Proposed Model | 0.99 | 0.99 | 0.99 | 0.98 | 0.99

Figure 6: Model Perfomance

A confusion matrix Fig. 7 is a key tool for evaluating classification models,
detailing predictions versus actual labels. It reveals true positives, true
negatives, false positives, and false negatives, aiding in accuracy,
precision, recall, and F1-score calculations. Insights gained help identify
misclassification trends, guiding data gathering and model adjustments for
improved performance.

Furthermore, our proposed model outperforms other hybrid approaches across all
metrics, including accuracy, precision, recall, F1 score, and Cohen’s Kappa
score. This underscores the efficacy of combining InceptionV3 and Densenet121,
showcasing a robust strategy for improving performance in diabetic retinopathy
detection.

(a) Vgg16

(b) Resnet50

(c) Xception

(d) InceptionV3

(e) DenseNet121

(f) Proposed Ensemble Model

Figure 7: Confusion Matrices for Various Models

##  5 Conclusion and Future Scope

In this paper, an ensemble learning-based approach is provided by employing
two base models; Densenet121 and Inceptionv3, to enhance the accuracy and
early detection capabilities of diabetic retinopathy. During the data
preprocessing phase, images are resized to $224\times 224$ pixels, and a
multi-label formatting technique is adopted to establish a balanced dataset.
The proposed model undergoes training and testing using the APTOS dataset of
fundus images. Comparative analysis is conducted with five other models;
namely Vgg16, Resnet50, Xception, InceptionV3, and Densenet121. Results reveal
that the proposed model’s validation accuracy is $99\%$, Kohen’s Kappa score
is $98\%$ and F1 score is also $99\%$. Additionally, a confusion matrix is
generated to show misclassification trends and overall performance.

In the future, the proposed model could undergo testing on diverse datasets
and comparisons with a broader array of models. Furthermore, its applicability
can extend beyond diabetic retinopathy to encompass other classification
tasks.

## Declaration

The authors declare no competing interests and no finding is granted to this
work.

## References

  * [1] Ali, G., Dastgir, A., Iqbal, M.W., Anwar, M., Faheem, M.: A hybrid convolutional neural network model for automatic diabetic retinopathy classification from fundus images. IEEE Journal of Translational Engineering in Health and Medicine (2023)
  * [2] Antal, B., Hajdu, A.: An ensemble-based system for automatic screening of diabetic retinopathy. Knowledge-based systems 60, 20–27 (2014)
  * [3] Bilal, A., Imran, A., Baig, T.I., Liu, X., Long, H., Alzahrani, A., Shafiq, M.: Improved support vector machine based on cnn-svd for vision-threatening diabetic retinopathy detection and classification. Plos one 19(1), e0295951 (2024)
  * [4] Chaurasia, B.K., Raj, H., Rathour, S.S., Singh, P.B.: Transfer learning–driven ensemble model for detection of diabetic retinopathy disease. Medical & Biological Engineering & Computing 61(8), 2033–2049 (2023)
  * [5] Da Rocha, D.A., Ferreira, F.M.F., Peixoto, Z.M.A.: Diabetic retinopathy classification using vgg16 neural network. Research on Biomedical Engineering 38(2), 761–772 (2022)
  * [6] Fong, D.S., Aiello, L., Gardner, T.W., King, G.L., Blankenship, G., Cavallerano, J.D., Ferris III, F.L., Klein, R., Association, A.D.: Diabetic retinopathy. Diabetes care 26(suppl_1), s99–s102 (2003)
  * [7] Ganesh, M., Dulam, S., Venkatasubbu, P.: Diabetic retinopathy diagnosis with inceptionresnetv2, xception, and efficientnetb3. In: Artificial Intelligence and Technologies: Select Proceedings of ICRTAC-AIT 2020, pp. 405–413. Springer (2021)
  * [8] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [9] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4700–4708 (2017)
  * [10] Jmour, N., Zayen, S., Abdelkrim, A.: Convolutional neural networks for image classification. In: 2018 international conference on advanced systems and electric technologies (IC_ASET). pp. 397–402. IEEE (2018)
  * [11] Kale, Y., Sharma, S.: Detection of five severity levels of diabetic retinopathy using ensemble deep learning model. Multimedia Tools and Applications 82(12), 19005–19020 (2023)
  * [12] Karthik, Maggie, S.D.: Aptos 2019 blindness detection (2019), https://kaggle.com/competitions/aptos2019-blindness-detection
  * [13] Lam, C., Yi, D., Guo, M., Lindsey, T.: Automated detection of diabetic retinopathy using deep learning. AMIA summits on translational science proceedings 2018, 147 (2018)
  * [14] Liang, N., Yuan, L., Wen, X., Xu, H., Wang, J.: End-to-end retina image synthesis based on cgan using class feature loss and improved retinal detail loss. IEEE Access 10, 83125–83137 (2022)
  * [15] Macsik, P., Pavlovicova, J., Goga, J., Kajan, S.: Local binary cnn for diabetic retinopathy classification on fundus images. Acta Polytech. Hung 19(7), 27–45 (2022)
  * [16] Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
  * [17] Nahiduzzaman, M., Islam, M.R., Islam, S.R., Goni, M.O.F., Anower, M.S., Kwak, K.S.: Hybrid cnn-svd based prominent feature extraction and selection for grading diabetic retinopathy using extreme learning machine algorithm. IEEE Access 9, 152261–152274 (2021)
  * [18] Pratt, H., Coenen, F., Broadbent, D.M., Harding, S.P., Zheng, Y.: Convolutional neural networks for diabetic retinopathy. Procedia computer science 90, 200–205 (2016)
  * [19] Qummar, S., Khan, F.G., Shah, S., Khan, A., Shamshirband, S., Rehman, Z.U., Khan, I.A., Jadoon, W.: A deep learning ensemble approach for diabetic retinopathy detection. Ieee Access 7, 150530–150539 (2019)
  * [20] Sikder, N., Masud, M., Bairagi, A.K., Arif, A.S.M., Nahid, A.A., Alhumyani, H.A.: Severity classification of diabetic retinopathy using an ensemble learning algorithm through analyzing retinal images. Symmetry 13(4), 670 (2021)
  * [21] SK, S., P, A.: A machine learning ensemble classifier for early prediction of diabetic retinopathy. Journal of Medical Systems 41, 1–12 (2017)
  * [22] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2818–2826 (2016)
  * [23] Tufail, A.B., Ullah, I., Khan, W.U., Asif, M., Ahmad, I., Ma, Y.K., Khan, R., Kalimullah, Ali, M.S.: Diagnosis of diabetic retinopathy through retinal fundus images and 3d convolutional neural networks with limited number of samples. Wireless Communications and Mobile Computing 2021, 1–15 (2021)
  * [24] Wan, S., Liang, Y., Zhang, Y.: Deep convolutional neural networks for diabetic retinopathy detection by image classification. Computers & Electrical Engineering 72, 274–282 (2018)
  * [25] Wang, J., Wang, S., Zhang, Y.: Artificial intelligence for visually impaired. Displays 77, 102391 (2023)
  * [26] WHO: Health report on diabetic patient (2024), https://tinyurl.com/bdzz54nm
  * [27] Yi, X., Walia, E., Babyn, P.: Generative adversarial network in medical imaging: A review. Medical image analysis 58, 101552 (2019)
  * [28] Zhang, J., Xie, B., Wu, X., Ram, R., Liang, D.: Classification of diabetic retinopathy severity in fundus images with densenet121 and resnet50. arXiv preprint arXiv:2108.08473 (2021)


# Automated Security Assessment for the Internet of Things ††thanks:
5Corresponding author.

Xuanyu Duan1, Mengmeng Ge2, Triet Huynh Minh Le3, Faheem Ullah3,
Shang Gao1, Xuequan Lu1,5 and M. Ali Babar3,4  1School of Information
Technology, Deakin University, Geelong, Australia  2School of Computing
Technologies, RMIT University, Melbourne, Australia  3School of Computer
Science, The University of Adelaide, Adelaide, Australia  4Cyber Security
Cooperative Research Centre, Australia  Emails<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Internet of Things (IoT) based applications face an increasing number of
potential security risks, which need to be systematically assessed and
addressed. Expert-based manual assessment of IoT security is a predominant
approach, which is usually inefficient. To address this problem, we propose an
automated security assessment framework for IoT networks. Our framework first
leverages machine learning and natural language processing to analyze
vulnerability descriptions for predicting vulnerability metrics. The predicted
metrics are then input into a two-layered graphical security model, which
consists of an attack graph at the upper layer to present the network
connectivity and an attack tree for each node in the network at the bottom
layer to depict the vulnerability information. This security model
automatically assesses the security of the IoT network by capturing potential
attack paths. We evaluate the viability of our approach using a proof-of-
concept smart building system model which contains a variety of real-world IoT
devices and potential vulnerabilities. Our evaluation of the proposed
framework demonstrates its effectiveness in terms of automatically predicting
the vulnerability metrics of new vulnerabilities with more than 90% accuracy,
on average, and identifying the most vulnerable attack paths within an IoT
network. The produced assessment results can serve as a guideline for
cybersecurity professionals to take further actions and mitigate risks in a
timely manner.

###### Index Terms:

Internet of Things, Vulnerability Assessment, Machine Learning, Natural
Language Processing, Graphical Security Model

##  I Introduction

Internet of Things (IoT) based applications are being widely deployed for
providing intelligent services underpinning smart cyber-physical systems like
smart buildings and smart citizens [1, 2]. Nowadays, most of the efficiency
and optimization efforts in various personal, social, and business sectors,
such as healthcare, manufacturing, transportation, and smart home, are
incorporating IoT based applications [3]. At the same time, IoT systems are
also becoming a prime target for cybersecurity attacks [4], due to the ever-
growing amount of sensitive data that such systems support. Hence, there is an
increasing realization that it is critical to timely identify the risks within
the interconnected systems and deploy appropriate security countermeasures.

Researchers and practitioners have devised and applied a number of security
assessment models to identify the security issues in IoT systems. A security
assessment model analyzes an entity (e.g., IoT network) to make sure it
complies with certain security objectives [5]. One type of security assessment
models is graphical security models, that offer systematic ways for assessing
security vulnerabilities of systems [6]. Attack graphs (AGs) [7] and attack
trees (ATs) [8] are two of the most widely used ones. AGs analyze the security
of networked systems by identifying all conceivable attack sequences used by
attackers to reach the potential target areas of systems [9]. ATs describe the
security of systems formally and systematically by presenting various means by
which a system can be attacked [6]. However, these single-layered security
models (e.g., AGs and ATs) do not scale well to an increasing number of
devices in complex networks [10]. To address such scalability issue, a
Hierarchical Attack Representation Model (HARM) [6] was proposed. It combines
AGs and ATs into a two-layer model where they can be constructed in parallel.
HARM has been successfully applied to assess security of complex network,
including IoT systems [11].

Despite the performance improvement, the security assessment using HARM is not
fully automated [11]. In particular, the assessment metrics (i.e., the Common
Vulnerability Scoring System (CVSS) severity scores [12]) of vulnerabilities
are not always available in vulnerability databases (e.g., National
Vulnerability Database (NVD) [13]) as inputs for security assessment with
HARM.111Vulnerability assessment is different from security assessment as the
former is applied to individual devices and is a step in performing the
security assessment of an interconnected system containing multiple devices.
Some vulnerabilities only have vulnerability descriptions without the
assessment metrics when being added into NVD [14], and these metrics are
important for vulnerability assessment. Some approaches have been proposed to
automate software vulnerability assessment to address the issues in the
prediction of severity scores based on a vulnerability description [15, 16,
17, 18, 19]. However, to the best of our knowledge, there has been no work on
automated network-level security assessment with missing vulnerability-related
metrics.

To fill this gap, our paper aims to design and implement a novel end-to-end
security assessment framework for IoT networks. Specifically, the framework
consists of an automated vulnerability assessment model, a graphical security
model and a visualization model. The vulnerability assessment model uses
Natural Language Processing (NLP) and Machine Learning (ML) techniques to
process the vulnerability descriptions on NVD for predicting severity scores.
The reason to adopt ML techniques rather than rule-based techniques is the
emergence of new terms in the descriptions of new vulnerabilities which may
cause the frequent update of rules with the inclusion of new terms. Taking the
network connectivity information and vulnerability information of each node
(including severity scores from the vulnerability assessment model) as inputs,
the graphical security model produces potential attack paths and evaluates the
security of the network using the attack path information and security
metrics. Finally, the visualization model is used to visualize the attack
paths with the highest security risks.

The main contributions of this paper are summarized as follows:

  1. 1.

Propose an end-to-end security assessment framework combining automated
vulnerability assessment and graphical security modeling to determine the
attack probability, attack impact and risk in interconnected systems;

  2. 2.

Evaluate the framework using a smart building system;

  3. 3.

Develop a visualization interface to generate network diagrams for better
representation and understanding of security assessment results.

The rest of the paper is organized as follows. Section II introduces the
background knowledge and related work. Section III explains our proposed
framework. Section IV describes the implementation of the proposed framework
as well as presents the evaluation and visualization results. Section V
discusses the findings and limitations of the proposed framework, followed by
the conclusion and suggestions on future research directions in Section VI.

##  II Related Work

This section discusses current work on automated assessment of vulnerabilities
and security assessment for IoT networks.

###  II-A Automated Assessment of Vulnerabilities

Approaches have been proposed for automated assessment of vulnerabilities. Guo
et al. [20] offered a featherweight virtual machine (FVM) solution to address
the issue of vulnerability testing safety. The FVM technology enables a
vulnerability assessment tool to test the exact duplicate of a production-mode
network service while maintaining complete isolation of the production-mode
network service from the testing process. Besides of ensuring safety, the
vulnerability assessment support system presented in this work may also
automate the entire vulnerability testing process, making it possible to
conduct vulnerability testing automatically and frequently. Shah et al. [21]
developed NetNirikshak 1.0, an automated Vulnerability Assessment and
Penetration Testing (VAPT) tool that assists organizations in assessing their
applications/services and analyzing the security posture. This program
identifies vulnerabilities in a target system’s applications and services. The
automatic report generated by the tool is emailed to a given email address,
and all traces of the scan are deleted from the hard disk along with the
report, ensuring the report’s confidentiality. Le et al. [15] indicated that
the method of automatically assessing software vulnerabilities based on NLP
was affected by concept drift. Concept drift occurs as a result of lacking
proper handling of new (out of vocabulary) terms in the vulnerability
descriptions. Therefore, they proposed a vulnerability assessment method that
incorporates both character and word features. Furthermore, they designed a
time-based cross-validation method to identify the optimal ML model for
predicting each vulnerability metric. Blinowski et al. [22] used a
vulnerability classification scheme for IoT devices based on real-world data.
The authors first categorize the vulnerabilities into seven categories, then
further classify them using standard descriptors in the Common Platform
Enumeration (CPE). They leverage ML techniques to achieve automatic
classification to mitigate the threats posed by new vulnerabilities.

###  II-B Security Assessment for IoT Systems

The security of IoT systems has been explored by several studies. Radomirovic
et al. [23] proposed an asynchronous communication network and a security
model with fingerprint recognition capability based on fundamental assumptions
and observations about potential security and privacy threats. This security
model facilitates research on the security and privacy of protocols used in
IoT networks. Ge et al. [11] established a paradigm for modeling and assessing
IoT security. The framework is used to develop a graphical security model and
a security evaluator to automate the security analysis of IoT. The security
evaluator assesses security using a variety of security metrics and outputs
the analysis results via an analytic modeling and assessment tool called
Symbolic Hierarchical Automated Reliability and Performance Evaluator (SHARPE)
[24]. Park et al. [25] adopted an integrated fuzzy MCDM (FMCDM) technique to
develop a framework for assessing the security of IoT services. The integrated
method utilizes an analytic network process (ANP) combined with a decision-
making trial and evaluation laboratory (DEMATEL) technique based on fuzzy set
theory to raise the sensitivity of interrelationships amongst various security
metrics. Wang et al. [26] developed a security model based on blockchain and
InterPlanetary File System (IPFS). The study and experimental findings
demonstrate that many security issues associated with traditional IoT
architecture can be avoided. In addition, system performance has been
significantly improved on distributed large-capacity storage, concurrency and
query.

Bugeja et al. [27] designed an innovative IoT Security Model (IoTSM) based on
the Software Assurance Maturity Model (SAMM) framework. It has been extended
with new security practices and data collected from IoT practitioners. As a
result, IoTSM can be used to organize strategies and discourse IoT security
from an end-to-end perspective. Matheu-García et al. [28] presented an IoT-
specific security certification methodology that would enable various
stakeholders to evaluate security solutions for large-scale IoT deployments in
an automated manner. Additionally, it promotes customer openness at the IoT
security level, as the methodology includes a label as one of the primary
outcomes of the certification process. The certification methodology is an
implementation of the ETSI-presented Risk-based Security Assessment and
testing methodologies based on ISO 31000 and ISO 29119. It is built on top of
a variety of security testing and risk assessment technologies and approaches
targeting the IoT landscape. Martin et al. [29] proposed a novel security
model capable of analyzing the properties of IoT systems. The security model
can assist designers of IoT systems in creating more secure systems by
highlighting vulnerabilities and weakest links. Additionally, they intended to
cover the data access requirements in future work and support the
formalization and certification of access control mechanisms in IoT systems.
Waraga et al. [30] introduced an automated IoT security testbed for assessing
the security of IoT devices. The proposed testbed is based on open source
tools managed by an open source management system, where changes are provided
to users via an easy-to-use GUI. This approach allows other researchers to use
the modular structure of the testbed to create their own testing tools. This
IoT testbed keeps track of all exploits and CVEs discovered for the devices
under test, as well as the services it hosts through each port. This testbed
could also generate reports automatically that contain the results of all
devices.

###  II-C Summary

Many prominent studies have been conducted on automating vulnerability
assessment and utilizing security models to assess the security of IoT
devices. However, few studies focused on combining these assessments to
automate network assessment under missing vulnerability assessment metrics. To
the best of our knowledge, our paper is the first to propose a model for
automatically assessing the security of interconnected systems in terms of the
probability, risk and impact of network attacks based on node/device
vulnerabilities.

##  III Proposed Framework

The proposed framework divides the security assessment process into four
phases: system model generation, vulnerability assessment, graphical security
model modeling and assessment, and data visualization, as illustrated in Fig.
1. The implementation of the framework can be found at
GitHub.222https://github.com/mmge88/automated-security-assessment Each phase
is discussed in the following subsections.

Figure 1: The proposed framework for automated security assessment

###  III-A System Model Generation

In Phase 1, we first consider a real-world IoT system and generate a system
model based on the specifications of smart devices in the system and the
connectivity of all devices. We then extract the vulnerability records from
the NVD. Details of the vulnerability extraction can be found in Section
IV-A2. This system model will later be combined with the graphical security
model in Phase 3 for the security assessment. The framework can be applied to
any IoT system as long as the network connectivity and vulnerabilities of
devices can be obtained.

###  III-B Vulnerability Assessment

In Phase 2, we adopt the NLP techniques to preprocess the vulnerability
descriptions and apply ML techniques to predict CVSS version 2 metrics based
on the processed descriptions of vulnerabilities. We also utilize an over-
sampling approach to mitigate the impact of imbalanced data. We tune the ML
model to identify optimal hyperparameters and train the optimal model over the
dataset downloaded from NVD after the preprocessing and over-sampling steps to
predict the vulnerability scores of devices in the IoT system.

###  III-C Graphical Security Model and Assessment

In Phase 3, we adopt a graphical security model based on a previous IoT
network assessment research [11] in which a two-layer HARM was utilized, with
the upper layer (an attack graph for the network connectivity) and the lower
layer (an attack tree for each node) to capture the information of how
vulnerabilities can be exploited to gain a privilege. The approach has been
proven to be more efficient in identifying a network’s most severe attack
paths than single-layer attack graphs due to the parallel construction of the
two layered models. As a result, we choose to retain this structure and adjust
the program by integrating our automated vulnerability assessment module. The
predicted scores from the vulnerability assessment are used as inputs to the
graphical security model and compute the network-level security metrics to
evaluate the security of the IoT system. The network-level security metrics
are computed based on original scores of the vulnerabilities and the
expected/ground-truth results are compared with prediction results.

###  III-D Data Visualization

In Phase 4, we develop a web-based visualization interface to better present
the assessment results. A network graph is used to visualize the relationships
between all devices in the system and the attack path with highest risk based
on the security evaluation results. A flask [31] project is used to combine
all network graphs and present them on the website.

##  IV Evaluation

To evaluate the proposed framework, a smart building system is used as the
test system where IoT technologies are heavily adopted.

###  IV-A System Model Generation

The design of the test system is mainly derived from [32, 33]. Hachem et al.
[32] conducted a study on the smart building of the Adelaide University Health
and Medical School (AHMS). Jia et al. [33] undertook an assessment on the
adoption of IoT in smart building applications and categorized them according
to their critical functions and aims, which improved our test system based on
the AHMS. The test system includes major functionalities of a smart building
system that leverages IoT techniques.

Our test system incorporates a variety of smart sensors and consists of seven
subsystems, each of which serves a distinct purpose and is in charge of
implementing a part of functionalities in the smart building. The seven
subsystems are: lighting, audiovisual, security, fire detection, maintenance,
resource tracking and HVAC subsystems, as shown in Fig 2. Since several
subsystems have shared sensors, we obtain two combined subsystems by linking
them via the common devices. As shown in Fig 3, the security and the resource
tracking are combined via the burglar alarm device, named as a combined
security system. The audiovisual, lighting, fire detection and HVAC systems
are combined via the brightness sensor, occupancy sensor and thermometer,
referred to as a combined smart building automation system. The maintenance
system does not have a common device with other subsystems thus not being
included in any combined system.

(a) Lighting

(b) Audiovisual

(c) Security

(d) Fire detection

(e) Maintenance

(f) Resource tracking

(g) HVAC

Figure 2: Smart building subsystems

(a) Combined smart building automation system

(b) Combined security system

Figure 3: Combined systems in the smart building system

####  IV-A1 Design and generation of subsystems

The devices and functionality of each subsystem are described in Table IV-A1.
As devices of the same type are usually purchased from the same manufacturer
(e.g., light sensors in the lighting subsystem), we assume they have identical
vulnerabilities. We select one device from each type in the test system to
capture attack scenarios among different types of devices. More device types
can be added and modeled via the framework.

TABLE I: Devices and functionality of the subsystems

Subsystem |  Devices |  Functionality
---|---|---
Lighting |  2 light sensors, 1 light controller, 1 occupancy sensor, 1 brightness sensor, 1 motion sensor |  Illuminate the smart building by controlling the lights in response to occupancy and the brightness sensors in response to the motion sensor
Audiovisual |  1 media player, 1 screen projector, 1 smart TV, 1 brightness sensor, 1 speaker |  Regulate the operation of audiovisual devices in response to users
Security |  2 door and window alarm sensors, 1 burglar alarm, 1 electronic entrance guard, 1 surveillance camera |  Monitor the security of the smart building and respond to security incidents
Fire detection |  1 CO detection sensor, 1 thermometer, 1 smoke sensor, 1 fire alarm |  Detect and prevent potential fire incidents based on the carbon monoxide concentration, temperature and smoke concentration
Maintenance |  1 electrical current monitoring sensor, 1 water monitoring sensor, 1 repair alarm |  Monitor and record the usage of electricity and water and report problems in a timely manner
Resource tracking |  2 asset tags, 2 wearable devices, 1 burglar alarm |  Monitor and track the status of assets and wearable devices, and protect these resources from being stolen or damaged
HVAC |  1 heater, 1 air conditioner, 1 ventilator, 1 humidity sensor, 1 CO2 sensor, 1 thermometer, 2 window sensors, 1 occupancy sensor |  Increase the comfort level of the smart building by controlling the temperature and humidity

####  IV-A2 Vulnerability filtering

To filter vulnerabilities for the related devices, the vulnerability data
dated from 2002 to 2020 is first downloaded from NVD, which contains 144,345
vulnerability records. All of the records are merged and potential vendors of
IoT devices are identified. The records associated with the potential vendors
are extracted. Then, the records related to the subsystems are extracted after
we manually examine whether the vulnerability descriptions are associated with
the corresponding subsystems. Subsequently, an extraction based on the
potential subsystem devices is performed. This set of extracted vulnerability
data will be used for model training, validation and testing in the
vulnerability assessment process. The dataset is split into 80% for training,
10% for validation, and 10% for testing, excluding the evaluation set.

Finally, the vulnerability records related to the actual subsystem devices of
various vendors (e.g., Broadcom, Panasonic, and Schneider Electric) are
extracted. We utilize these device related vulnerabilities as evaluation data
in the vulnerability assessment process and take the predicted results as
inputs into the graphical security modeling and assessment. The number of
records used in the vulnerability assessment is shown in Table II. Note that
the vulnerabilities in each dataset are unique and do not overlap with the
other sets.

TABLE II: The number of unique vulnerabilities in each dataset used in our study Dataset |  No. of vulnerabilities/records
---|---
Training |  115,395
Validation |  14,424
Testing |  14,424
Evaluation |  102

###  IV-B Vulnerability Assessment

####  IV-B1 Data pre-processing

Each vulnerability record collected from NVD consists of a description, CVSS
v2 metrics (i.e., base score metric group captures the characteristics of a
vulnerability that do not change with time and across user environment), and
privileges. In particular, the base score is measured by impact and
exploitability. The impact is based on confidentiality, integrity, and
availability. The exploitability is identified by the access vector, access
complexity and authentication. There are three types of privileges, including
all (i.e., root), user, and other privileges [34].

To preprocess these records, the descriptions are first passed through NLP.
For each vulnerability record, the stop words are removed, and all words in
the description are converted to lowercase. Then term frequency-inverse
document frequency (TF-IDF) vectorizer with n-gram (1-3) is applied to
transform the text to vectors. TF-IDF is adopted because it has been proved to
be well performed for vulnerability analysis [15]. Other relevant algorithms
are to be investigated in our future work. The processed vulnerability
descriptions are taken as the input to the ML model. The CVSS metrics and
privilege values are predicted as the output.

(c) Number of label instances for each metric in the base score

(d) Number of label instances for privileges

Figure 4: Number of label instances before sampling

(a) Number of label instances for each metric in the base score.

(b) Number of label instances for privileges

Figure 5: Number of label instances after sampling

We evaluate the processed data before predicting the six metrics in CVSS base
score and types of privileges. As illustrated in Fig 3(c), each of the six
metrics has three distinct values. A multi-class classification is needed from
an ML perspective. In comparison, Fig 3(d) shows two values for each
privilege, addressing the binary classification in ML. Additionally, as
indicated in both figures, the distribution of each class is severely
imbalanced for access vector, access complexity, authentication, and the three
types of privileges. To mitigate the impact of imbalanced data and avoid the
ignorance of critical data during model training, we utilize an over-sampling
approach, RandomOverSampler [35], to duplicate the instances in the minority
class. By randomly reproducing instances in a minority class, this approach
raises the number of instances until the minority and majority classes are
balanced. The dataset after oversampling is shown in Fig 5.

####  IV-B2 Model tuning

The model tuning is done using the training and validation datasets, as shown
in Table II. Specifically, for tuning, a model is first trained on the
training set and then evaluated on the validation set. The Light Gradient
Boosting Machine (LGBM) [36] is used for the classification. LGBM has been
proven to outperform other ML models (such as Naive Bayes, Logistic
Regression, Support Vector Machine, Random Forest and XGBoost [37]) and thus
being the optimal model for classifying vulnerability assessment metrics [15].
Its primary advantage is scalability, as the sub-trees are formed leaf-wise
rather than level-wise compared to other gradient boosting techniques [36]. As
indicated previously, the prediction of metrics in the base score is a multi-
class classification problem, whereas the prediction of privileges is a binary
classification problem. As such, we set the objective for predicting metrics
of the base score to multi-class and privileges to binary. The complexity of
LGBM is mainly determined by the maximum depth and leaf count [36]. Therefore,
we use 100, 300 and 500 as the number of leaves for tuning. To limit the tree
depth and avoid overfitting [36], we set 100, 200 and 300 as the maximum depth
during tuning. These values follow the previous practices in this domain
(e.g., [15, 38, 39]).

Additionally, the grid search approach is adopted to determine the optimal
(with the highest value of evaluation metric on the validation set)
combination of hyperparameters by considering all possible hyperparameter
combinations. We set the evaluation metric as F1 score to tune the model for
the binary classification. F1-score is a measure of predictive effectiveness
and capable of balancing recall (i.e., the number of correct positive class
predictions among all actual positive instances) and precision (i.e., the
number of correct positive class predictions among all positive predictions)
[40].

Due to the fact that the prediction of metrics in the base score is a multi-
class classification problem, we quantify the result using the F1-weighted
method, i.e. calculate F1 score for each label and use the number of true
instances for each label as the weight for the average value. The model with
the highest F1-weighted score on the validation set is selected as the optimal
model. Table III illustrates the optimal combination of hyperparameters for
each privilege where “no. leaves” denotes number of leaves, “max depth”
denotes maximum depth, “std_dev” represents the standard deviation and the
time unit of “running time” is second. Table IV illustrates the optimal
combination of hyperparameters for each metric. We will perform sensitivity
analysis to examine the impact of different combinations of hyperparameters on
the model performance in the future work.

TABLE III: The validation results (F1 score) and the optimal combination of hyperparameters (i.e., max depth and number of leaves) for each privilege Privilege |  Max depth |  No. leaves |  Mean |  Std_dev |  F1 score |  Running time (s)
---|---|---|---|---|---|---
All privilege |  300 |  500 |  0.942 |  0.003 |  0.978 |  133.615
Other privilege |  300 |  500 |  0.914 |  0.006 |  0.965 |  130.612
User privilege |  300 |  500 |  0.964 |  0.003 |  0.988 |  132.743
TABLE IV: The validation results (F1-weighted score) and the optimal combination of hyperparameters (i.e., max depth and number of leaves) for each CVSS metric in base score Metric |  Max
depth |  No.
leaves |  Mean |  Std_dev |  F1-weighted |  Running
time (s)
---|---|---|---|---|---|---
Confidentiality |  200 |  500 |  0.825 |  0.007 |  0.895 |  394.004
Integrity |  100 |  500 |  0.848 |  0.004 |  0.914 |  407.055
Availability |  100 |  100 |  0.806 |  0.003 |  0.835 |  317.599
Access vector |  200 |  500 |  0.953 |  0.003 |  0.988 |  461.245
Access complexity |  300 |  500 |  0.825 |  0.005 |  0.913 |  401.231
Authentication |  100 |  300 |  0.944 |  0.002 |  0.974 |  389.540

####  IV-B3 Model testing

The optimal hyperparameters from the previous phase are used to build the
model for testing. The model is trained with the training and validation
datasets and is then tested on the testing set, as given in Table II. The F1
score and accuracy score are utilized to evaluate the model performance for
the binary classification problem. F1-weighted, F1-macro and accuracy are used
for the multi-class problem. The F1-macro calculates the F1 score of each
label and finds an unweighted mean.

As shown in Table V and Table VI, all scores for the prediction of privileges
are greater than 95%, and scores for the prediction of integrity and
exploitability metrics achieve 90%. Scores for predicting confidentiality are
approximately 90%, while scores for predicting availability are roughly 83%.
This indicates that the model performs well with the hyperparameters tuned in
the previous stage.

TABLE V: Test results for privileges Privilege | Accuracy Score | F1 Score
---|---|---
All privilege | 0.9786 | 0.9789
Other privilege | 0.9680 | 0.9693
User privilege | 0.9878 | 0.9879
TABLE VI: Test result for metrics in the base score Metric |  Accuracy Score |  F1-macro |  F1-weighted
---|---|---|---
Confidentiality |  0.8910 |  0.8908 |  0.8905
Integrity |  0.9145 |  0.9138 |  0.9140
Availability |  0.8358 |  0.8347 |  0.8344
Access vector |  0.9886 |  0.9886 |  0.9886
Access complexity |  0.9126 |  0.9124 |  0.9122
Authentication |  0.9741 |  0.9741 |  0.9741

####  IV-B4 Severity score prediction

With the trained model, we use the evaluation data (extracted vulnerability
data related to the subsystem devices, as described in Section IV-A2) to
predict metrics values in base score and privileges. The predicted
exploitability and impact metrics are then utilized to calculate the predicted
CVSS base/severity score using the CVSS equations [34], as given in equations
(1), (2), (3) and (4). The ConfImpact, IntegImpact and AvailImpact quantify
the impact of a successfully exploited vulnerability on the confidentiality,
integrity and availability of a system of interest, respectively. The
$f(\text{Impact})$ depends on the value of Impact. More details can be found
in [34].

$\displaystyle\text{Base
score}=(.6\times\text{Impact}+.4\times\text{Exploitability}-1.5)\times
f(\text{Impact})$ (1) $\displaystyle\text{Impact}={}$ $\displaystyle
10.41\times(1-(1-\text{ConfImpact})\times(1-\text{IntegImpact})~{}\times$ (2)
$\displaystyle(1-\text{AvailImpact}))$
$\displaystyle\text{Exploitability}=20\times\text{AccessComp.}\times\text{Authentication}\times\text{AccessVector}$
(3) $f(\text{Impact})=\Bigg{\\{}\begin{aligned} 0,\text{if Impact}=0\\\
1.176,\text{otherwise}\end{aligned}$ (4)

###  IV-C Graphical Security Model and Assessment

To better illustrate the assessment results in the network level, a graphical
security model is adopted. It is built on a previous Python project [11] with
modified modules to accommodate the requirements of this work.

Figure 6: Structure of the Graphical Security Model

####  IV-C1 Model description

As illustrated in Fig 6, the network generator contains the main function for
constructing a smart building network of devices and vulnerabilities in the
IoT context. It first initializes each device (i.e., node in the program) with
the prediction results from the vulnerability assessment by representing the
exploitation of vulnerabilities on each device based on privileges and
assigning impact and exploitability scores to each vulnerability for network-
level metric calculation. In particular, an integer value from 1 to 3 is
assigned to each Privilege (All/Other/User), representing the access level an
attacker obtains when a node is compromised. When “other privilege” is true,
the default privilege value of 1 is retained; the value rises to 2 if “user
privilege” is true, and to 3 if “all privilege” is true. We primarily focus on
the three security metrics for the security assessment of networks [12]: the
probability of a successful attack, the impact of an attack, and the risk of
an attack. The probability of a successful attack (ranging from 0 to 1) is
measured by the exploitability score (in equation (3) ranging from 0 to 10)
divided by 10, the impact of an attack refers to impact score (in equation
(2)), and the risk of attack is impact times probability. The network
generator then connects the devices according to the pre-defined topology of
each subsystem presented in Section IV-A.

After that, the generated network is taken as input to the graphical security
model generator, generating a Hierarchical Attack Representation Model (HARM)
[6]. This HARM consists of an upper layer attack graph and a lower layer
attack tree for each vulnerable node. To calculate the attack paths, we also
specify the potential entry points and attack targets in the network. Using
the lighting subsystem as an example, the sensors can be used as entry points
due to their lack of security protections; the light controller can be used as
the attack target as the attacker can use the compromised controller to
control all lights.

The most significant attack paths (e.g., with the maximum risk) can be
identified through the security metrics computed via the security evaluator.
We use a bottom-up approach to calculate the metrics via the HARM. Values in
an attack tree are calculated first and populated to the node via the security
evaluator. Multiple exploitable vulnerabilities may exist on a single device,
which can be analyzed by two types of gates in the attack tree: $AND$ gate and
$OR$ gate. An $AND$ gate means that all vulnerabilities must be exploited to
compromise a node. In contrast, an $OR$ gate implies that an attacker can gain
control of the node by exploiting only one of the vulnerabilities. These two
scenarios result in different evaluation methods in the security evaluator,
where $AND$ is accumulative and $OR$ is selective. In the attack path level,
we calculate the metric value by accumulating the values of all nodes along
the path (i.e., multiplication of probability values of nodes; sum of
impact/risk values of nodes). We then choose the attack path with the highest
risk value in the network-level assessment. Security evaluation results are
passed back to the main function for final output and visualization. When
comparing the ground-truth and predicted values at the network level, we use
the percentage error to determine the model’s prediction accuracy by Equation
5.

$\displaystyle\text{percentage error}=$ $\displaystyle(|\text{ground-truth
value}-\text{predicted value}|)$ (5) $\displaystyle/\text{ground-truth
value}*100$

####  IV-C2 Result analysis

The path with the highest risk in each subsystem and combined system is shown
in Table VII. The graphical security model is able to capture the attack path
with highest risk and present the sequences of nodes (in a subsystem or across
multiple subsystems) that can be potentially compromised by an attacker.

TABLE VII: The most risky attack paths in subsystems and combined systems Subsystem name |  Attack path with highest risk
---|---
Lighting |  attacker→motion-sensor→brightness-sensor →light1→occupancy-sensor→light2→light-controller
Audiovisual |  attacker→media-player→projector→speaker
Security |  attacker → door-window-alarm-sensor1 → door-window-alarm-sensor2 → burglar-alarm → surveillance → entrance-guard
Fire detection |  attacker→CO-sensor→fire-alarm
Maintenance |  attacker→electrical-current-monitoring-sensor→repair-alarm
Resource tracking |  attacker→wearable-device1→wearable-device2→burglar-alarm
HVAC |  attacker→window-sensor1→window-sensor2→heater
Combined smart building automation |  attacker→motion-sensor→brightness-sensor→light1→light2→light-controller→occupancy-sensor→air-conditioner
Combined security |  attacker→door-window-alarm-sensor1→door-window-alarm-sensor2→burglar-alarm→surveillance→entrance-guard

We show the percentage error of each subsystem and combined system in Table
VIII. According to Table VIII, the lowest percentage error of predicting
attack probability is 0, while the percentage errors of predicting impact,
risk, and base score range from 1.61% to 33.04%. Among the subsystems, HVAC
has the highest error for risk (i.e., 33.04%), which is reasonable as it is
the most complex subsystem with the most number of devices. The combined smart
building automation system contains HVAC subsystem, and thus also has higher
errors than the combined security system.

TABLE VIII: Percentage error in predicting probability, impact, risk, and base score for subsystems and combined systems Subsystem name |  Probability |  Impact |  Risk |  Base score
---|---|---|---|---
Lighting |  0.00% |  1.48% |  17.84% |  11.50%
Audiovisual |  0.00% |  2.94% |  18.62% |  1.61%
Security |  0.00% |  30.41% |  22.00% |  11.54%
Fire detection |  0.00% |  13.26% |  8.40% |  6.72%
Maintenance system |  0.00% |  22.83% |  31.50% |  15.73%
Resource tracking |  0.00% |  13.13% |  8.97% |  4.18%
HVAC |  0.00% |  2.21% |  33.04% |  10.05%
Combined smart building automation |  0.00% |  13.60% |  27.27% |  17.07%
Combined security |  0.00% |  17.73% |  5.34% |  11.54%

###  IV-D Data Visualization

We utilize a network graph by AnyChart [41] (i.e., library support for
javascript and HTML) to display the network connectivity and the attack path
with the highest risk, and merge all the network graphs into a flask project
for web presentation. All attack paths in Table VII can be visualized via the
web-based user interface. As an example, we demonstrate the attack path with
the highest risk in the lighting subsystem in Fig 7, where the sequence of
nodes compromised by the attacker along that path are also visualized.
Specifically, each node represents a device within the subsystem and has a
unique color in the connection diagram. The blue lines indicate the connection
between the nodes, while the red line indicates the riskiest attack path of
the subsystem. The bottom part of the figure gives the sequence of the nodes
on the most risky path from the attacker’s entry point (i.e., Motion Sensor)
to the targeted device/node (i.e., Light Controller).

Figure 7: Our web-based visualization of the attack path targeting the Light
Controller with the highest risk (highlighted in red color) in the lighting
subsystem

##  V Discussion

In this work, we integrated ML-driven vulnerability assessment and network
security assessment for the IoT, and identified a few interesting findings
from the analysis results.

Over 90% accurate predictions of security metrics using ML based on NLP: ML
models based on NLP can generate predictions with accuracy of approximately
90% for all security metrics of each vulnerability. We focus on the
professional vulnerability descriptions from the NVD source for both training
and testing. Future work can investigate the applicability of the framework to
predict user-reported descriptions and descriptions from various databases and
sources. Besides formal vulnerability sources like NVD, informal sources such
as technical forums like Stack Overflow [42] can also be explored for security
assessment of IoT systems. In addition, the usage of deep learning models [43]
is worth further exploration, as well as the examination of the degree to
which the NLP-based prediction model relies on the professionality and
consistency of vulnerability sources.

Higher prediction error at network level: The prediction error is more
significant at the network level compared to the errors for individual
vulnerability entries. This is expected as the inaccuracy of vulnerability
predictions accumulates along with the devices in the attack path. The
differences in errors among impact, risk and probability are significant. The
predicted probability is identical to the original probability, as the
majority of vulnerabilities in the network have a high possibility of being
exploited successfully. This demonstrates that the accuracy of the network-
level prediction is dependent upon the actual selection of vulnerabilities.

Limitations: A potential issue exists with the resource tracking subsystem.
Wearable devices are often movable in a realistic environment. In this work,
we only consider the situation that the wearable devices are directly
connected to the burglar alarm, and we assume they can maintain the connection
when they are moving. Therefore, the topology did not change when we used the
HARM as the graphical security model to compute attack paths. However, in a
complex network, the topology can change due to moving devices, and the
security model needs to be updated whenever the topology changes. To address
this issue, an extension to HARM, called temporal HARM can assess dynamically
changing networks [44]. Additionally, the graphical security model has
scalability issue when dealing with computationally complex networks,
necessitating further optimization of the algorithms for computing attack
paths. The impact of this issue can be potentially reduced by dividing the
network into multiple sub-networks.

##  VI Conclusion and Future Work

The growing use of the Internet of Things (IoT) exposes these systems to a
variety of vulnerabilities due to their hyper-connectivity. However, security
assessment based on vulnerability metrics is not fully automated due to manual
analysis of vulnerabilities, which is a time-consuming and tedious task.

This paper proposed a framework for automating the security assessment process
by employing ML techniques and graphical security modeling. To evaluate our
proposed framework, we developed a proof-of-concept smart building system that
simulates real-world IoT systems. Using data downloaded from the NVD, we
filtered IoT device-related vulnerabilities and utilized ML methods to predict
Common Vulnerability Scoring System (CVSS) metrics and privileges. The
proposed smart building system model and predicted data were used to evaluate
the security of IoT systems in terms of probability, risk, and impact using
the Graphical Security Model (GSM). The GSM computes all potential attack
paths, identifies the most vulnerable path, and visualizes them using a web-
based user interface, making it easier for system designers to identify
possible vulnerabilities.

In the future, we plan to study various IoT scenarios using the framework and
explore the usage of other ML and deep learning techniques to improve the
prediction accuracy. Additionally, we will investigate the impact of defences
on the change of vulnerability scores to compare the effectiveness of
defences.

## References

  * [1] A. Taivalsaari and T. Mikkonen, “On the development of iot systems,” in _2018 Third International Conference on Fog and Mobile Edge Computing (FMEC)_. IEEE, 2018, pp. 13–19.
  * [2] S. Madakam, V. Lake, V. Lake, V. Lake _et al._ , “Internet of things (iot): A literature review,” _Journal of Computer and Communications_ , vol. 3, no. 05, p. 164, 2015.
  * [3] J. Gubbi, R. Buyya, S. Marusic, and M. Palaniswami, “Internet of things (iot): A vision, architectural elements, and future directions,” _Future generation computer systems_ , vol. 29, no. 7, pp. 1645–1660, 2013.
  * [4] A. N. Duc, R. Jabangwe, P. Paul, and P. Abrahamsson, “Security challenges in iot development: a software engineering perspective,” in _Proceedings of the XP2017 Scientific Workshops_ , 2017, pp. 1–5.
  * [5] R. Leszczyna, “Standards on cyber security assessment of smart grid,” _International Journal of Critical Infrastructure Protection_ , vol. 22, pp. 70–89, 2018.
  * [6] J. Hong and D. S. Kim, “HARMs: Hierarchical attack representation models for network security analysis,” in _Proceedings of the 10th Australian Information Security Management Conference, AISM 2012_. SRI Security Research Institute, Edith Cowan University, Perth, Western Australia, dec 2012, pp. 74–81. [Online]. Available: https://ro.ecu.edu.au/ism/146
  * [7] O. Sheyner, J. Haines, S. Jha, R. Lippmann, and J. M. Wing, “Automated generation and analysis of attack graphs,” in _Proceedings 2002 IEEE Symposium on Security and Privacy_. IEEE, 2002, pp. 273–284.
  * [8] T. R. Ingoldsby, “Attack tree-based threat risk analysis,” _Amenaza Technologies Limited_ , pp. 3–9, 2010.
  * [9] S. Y. Enoch, J. B. Hong, and D. S. Kim, “Security modelling and assessment of modern networks using time independent graphical security models,” _Journal of Network and Computer Applications_ , vol. 148, p. 102448, 2019\.
  * [10] J. B. Hong and D. S. Kim, “Towards scalable security analysis using multi-layered security models,” _Journal of Network and Computer Applications_ , vol. 75, pp. 156–168, 2016.
  * [11] M. Ge, J. B. Hong, W. Guttmann, and D. S. Kim, “A framework for automating security analysis of the internet of things,” _Journal of Network and Computer Applications_ , vol. 83, pp. 12–27, 2017.
  * [12] Cve.mitre.org. (2021) Nvd - vulnerability metrics. [Online]. Available: https://nvd.nist.gov/vuln-metrics/cvss#
  * [13] Nvd.nist.gov. (2020) Nvd - general. [Online]. Available: https://nvd.nist.gov/general
  * [14] A. Feutrill, D. Ranathunga, Y. Yarom, and M. Roughan, “The effect of common vulnerability scoring system metrics on vulnerability exploit delay,” in _2018 Sixth International Symposium on Computing and Networking (CANDAR)_. IEEE, 2018, pp. 1–10.
  * [15] T. H. M. Le, B. Sabir, and M. A. Babar, “Automated software vulnerability assessment with concept drift,” in _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_. IEEE, 2019, pp. 371–382.
  * [16] G. Spanos and L. Angelis, “A multi-target approach to estimate software vulnerability characteristics and severity scores,” _Journal of Systems and Software_ , vol. 146, pp. 152–166, 2018.
  * [17] Y. Yamamoto, D. Miyamoto, and M. Nakayama, “Text-mining approach for estimating vulnerability score,” in _2015 4th International Workshop on Building Analysis Datasets and Gathering Experience Returns for Security (BADGERS)_. IEEE, 2015, pp. 67–73.
  * [18] G. Spanos, L. Angelis, and D. Toloudis, “Assessment of vulnerability severity using text mining,” in _Proceedings of the 21st Pan-Hellenic Conference on Informatics_ , 2017, pp. 1–6.
  * [19] T. H. M. Le, H. Chen, and M. A. Babar, “A survey on data-driven software vulnerability assessment and prioritization,” _arXiv preprint arXiv:2107.08364_ , 2021.
  * [20] F. Guo, Y. Yu, and T.-c. Chiueh, “Automated and safe vulnerability assessment,” in _21st Annual Computer Security Applications Conference (ACSAC’05)_. IEEE, 2005, pp. 10–pp.
  * [21] S. Shah and B. Mehtre, “An automated approach to vulnerability assessment and penetration testing using net-nirikshak 1.0,” in _2014 IEEE International Conference on Advanced Communications, Control and Computing Technologies_. IEEE, 2014, pp. 707–712.
  * [22] G. J. Blinowski and P. Piotrowski, “Cve based classification of vulnerable iot systems,” in _International Conference on Dependability and Complex Systems_. Springer, 2020, pp. 82–93.
  * [23] S. Radomirovic, “Towards a model for security and privacy in the internet of things,” in _Proc. First Int’l Workshop on Security of the Internet of Things_ , 2010, p. 6.
  * [24] R. A. Sahner, K. Trivedi, and A. Puliafito, _Performance and reliability analysis of computer systems: an example-based approach using the SHARPE software package_. Springer Science & Business Media, 2012.
  * [25] K. C. Park and D.-H. Shin, “Security assessment framework for iot service,” _Telecommunication Systems_ , vol. 64, no. 1, pp. 193–209, 2017.
  * [26] Z. Wang, X. Dong, Y. Li, L. Fang, and P. Chen, “Iot security model and performance evaluation: A blockchain approach,” in _2018 International Conference on Network Infrastructure and Digital Content (IC-NIDC)_. IEEE, 2018, pp. 260–264.
  * [27] J. Bugeja, B. Vogel, A. Jacobsson, and R. Varshney, “Iotsm: an end-to-end security model for iot ecosystems,” in _2019 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops)_. IEEE, 2019, pp. 267–272.
  * [28] S. N. Matheu-García, J. L. Hernández-Ramos, A. F. Skarmeta, and G. Baldini, “Risk-based automated assessment and testing for the cybersecurity certification and labelling of iot devices,” _Computer Standards & Interfaces_, vol. 62, pp. 64–83, 2019.
  * [29] T. Martin, D. Geneiatakis, I. Kounelis, S. Kerckhof, and I. N. Fovino, “Towards a formal iot security model,” _Symmetry_ , vol. 12, no. 8, p. 1305, 2020.
  * [30] O. A. Waraga, M. Bettayeb, Q. Nasir, and M. A. Talib, “Design and implementation of automated iot security testbed,” _Computers & Security_, vol. 88, p. 101648, 2020.
  * [31] M. Grinberg, _Flask web development: developing web applications with python_. ” O’Reilly Media, Inc.”, 2018\.
  * [32] J. E. Hachem, V. Chiprianov, M. A. Babar, T. A. Khalil, and P. Aniorte, “Modeling, analyzing and predicting security cascading attacks in smart buildings systems-of-systems,” _Journal of Systems and Software_ , vol. 162, p. 110484, 2020.
  * [33] M. Jia, A. Komeily, Y. Wang, and R. S. Srinivasan, “Adopting internet of things for the development of smart buildings: A review of enabling technologies and applications,” _Automation in Construction_ , vol. 101, pp. 111–126, 2019.
  * [34] S. R. Peter Mell, Karen Scarfone. (2021) A complete guide to the common vulnerability scoring system. [Online]. Available: https://www.first.org/cvss/v2/guide
  * [35] G. Lemaître, F. Nogueira, and C. K. Aridas, “Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning,” _Journal of Machine Learning Research_ , vol. 18, no. 17, pp. 1–5, 2017. [Online]. Available: http://jmlr.org/papers/v18/16-365.html
  * [36] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, “Lightgbm: A highly efficient gradient boosting decision tree,” _Advances in neural information processing systems_ , vol. 30, pp. 3146–3154, 2017.
  * [37] T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_ , 2016, pp. 785–794.
  * [38] T. H. M. Le, D. Hin, R. Croft, and M. A. Babar, “Puminer: Mining security posts from developer question and answer websites with pu learning,” in _Proceedings of the 17th International Conference on Mining Software Repositories_ , 2020, pp. 350–361.
  * [39] ——, “Deepcva: Automated commit-level vulnerability assessment with deep multi-task learning,” _arXiv preprint arXiv:2108.08041_ , 2021.
  * [40] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” _Journal of Machine Learning Research_ , vol. 12, pp. 2825–2830, 2011.
  * [41] AnyChart. (2021) Anychart. [Online]. Available: https://www.anychart.com
  * [42] T. H. M. Le, R. Croft, D. Hin, and M. A. Babar, “A large-scale study of security vulnerability support on developer q&a websites,” in _Evaluation and Assessment in Software Engineering_ , 2021, pp. 109–118.
  * [43] T. H. M. Le, H. Chen, and M. A. Babar, “Deep learning for source code modeling and generation: Models, applications, and challenges,” _ACM Computing Surveys (CSUR)_ , vol. 53, no. 3, pp. 1–38, 2020.
  * [44] S. Y. Enoch, M. Ge, J. B. Hong, H. Alzaid, and D. S. Kim, “Evaluating the Effectiveness of Security Metrics for Dynamic Networks,” in _Proceedings of the 16th IEEE International Conference On Trust, Security and Privacy in Computing and Communications (TrustCom 2017)_. IEEE, August 2017, pp. 277–284.