# Poisoning Attacks in Federated Edge Learning for Digital Twin 6G-enabled
IoTs: An Anticipatory Study

Mohamed Amine Ferrag  Technology Innovation Institute
9639 Masdar City, Abu Dhabi, UAE
<EMAIL_ADDRESS>Burak Kantarci  University of Ottawa
Ottawa, ON, Canada
<EMAIL_ADDRESS>Lucas C. Cordeiro  Technology Innovation Institute
9639 Masdar City, Abu Dhabi, UAE
<EMAIL_ADDRESS>Merouane Debbah  Technology Innovation Institute
9639 Masdar City, Abu Dhabi, UAE
<EMAIL_ADDRESS>Kim-Kwang Raymond Choo  University of Texas at San
Antonio
San Antonio, TX 78249-0631, USA
<EMAIL_ADDRESS>
###### Abstract

Federated edge learning can be essential in supporting privacy-preserving,
artificial intelligence (AI)-enabled activities in digital twin 6G-enabled
Internet of Things (IoT) environments. However, we need to also consider the
potential of attacks targeting the underlying AI systems (e.g., adversaries
seek to corrupt data on the IoT devices during local updates or corrupt the
model updates); hence, in this article, we propose an anticipatory study for
poisoning attacks in federated edge learning for digital twin 6G-enabled IoT
environments. Specifically, we study the influence of adversaries on the
training and development of federated learning models in digital twin
6G-enabled IoT environments. We demonstrate that attackers can carry out
poisoning attacks in two different learning settings, namely: centralized
learning and federated learning, and successful attacks can severely reduce
the model’s accuracy. We comprehensively evaluate the attacks on a new cyber
security dataset designed for IoT applications with three deep neural networks
under the non-independent and identically distributed (Non-IID) data and the
independent and identically distributed (IID) data. The poisoning attacks, on
an attack classification problem, can lead to a decrease in accuracy from
$94.93$% to $85.98$% with IID data and from $94.18$% to $30.04$% with Non-IID.

###### Index Terms:

Poisoning attack, Federated Learning, IoT, 6G, Security, Digital Twin.

##  I Introduction

The emergence of the digital twin (DT) paradigm with 6G wireless communication
networks is being viewed to transform applications and customer services
through the Internet of Things (IoT) to fully autonomous and smart systems
[1]. The main idea behind DT is to build a digital replication of wireless
networks’ physical devices and features to achieve low latency and highly
reliable connectivity while providing high performance and energy efficiency
in IoT networks. The architecture of digital twin 6G-enabled IoT can be
organized into four layers, namely, Physical, Networking, Services, and
Applications. The physical layer refers to IoT devices equipped with sensors
and actuators for collecting data from the environment. The networking layer
refers to DT services, telecommunication networks (6G), IEEE 802.15.4, long-
range WiFi, and communication protocols to connect IoT devices with other
operators and system components. The Ditto 111https://www.eclipse.org/ditto/
can be used as IoT middleware, supporting an IoT abstraction level for
interacting IoT solutions with physical devices via the digital twin model.
The services layer provides AI, storage, computation, and services to IoT
devices and Edge servers [2]. The applications layer refers to IoT
applications, including the Internet of Vehicles, the Internet of Sensing, and
the Internet of Energy…etc.

With distributed edge learning, intrusion poisoning attacks pose an even more
severe challenge than the conventional machine learning environment (i.e.,
centralized learning). However, the threat of intrusion poisoning attacks can
be challenging to overcome, as they can be difficult to identify.
Nevertheless, the availability of intrusion poisoning attacks has been proven
to be efficient in distributed edge learning in the following recent work: [3,
4, 5, 6, 7, 8]. Zhang et al. [4] proposed a defense mechanism at the model
level, which is mainly based on the detection of a poisoned model. More
precisely, the proposed mechanism uses the model’s important parameter
selection method based on the gradient to deliver the lowest dimensional
efficient performances of the local model parameters downloaded. Venkatesan et
al. [5] considered the poisoning availability attack framework, where an
attacker can introduce a set of poisonous samples during training to degrade
the deployed model’s accuracy. Aiken et al. [6] proposed an IDS system based
on an anomaly. The proposed system uses a tool for adversarial testing named
Hydra, which measures the influence of the threat of adversarial evasion
classifier attacks against the network intrusion detection system to reduce
the detection rate of malicious network traffic.

Motivated by the facts mentioned above, in this article, we propose an
anticipatory study for poisoning attacks in federated edge learning for
digital twin 6G-enabled IoTs. Specifically, we demonstrate that attackers who
conduct poisoning attacks in two different learning modes, namely, centralized
learning and federated learning, can severely reduce the model’s accuracy and
the detection rate of each intrusion. We comprehensively evaluate our attacks
on a new cyber security dataset designed for IoT applications, the Edge-IIoT
dataset. We use different deep-learning models for cyber security intrusion
detection. Furthermore, federated edge learning is evaluated under two data
distribution types: IID and Non-IID data. The study demonstrates that
attackers who conduct poisoning attacks in two different learning modes,
namely, centralized learning and federated learning, can lead to a decrease in
accuracy from 94.93% to 85.98% with IID data and from 94.18% to 30.04% with
Non-IID.

TABLE I: Notations used

Notation | Description
---|---
$\eta$ | The learning rate
$Epo$ | The number of local epochs
$Batch$ | The local minibatch size
$K_{H}$ | The honest clients
$k_{h}$ | The indexation of honest clients
$K_{M}$ | The malicious clients
$k_{m}$ | The indexation of malicious clients
$f_{t+1}^{k_{h}}$ |  The new local set of weights by honest client $k_{h}$
$f_{t+1}^{k_{m}}$ |  The new local set of weights by malicious client $k_{m}$
$C_{h}$ | The fraction of honest clients*
$C_{m}$ | The fraction of malicious clients*
$n_{k_{h}}$ |  The number of local examples for honest client $k_{h}$
$n_{k_{m}}$ |  The number of local examples for malicious client $k_{m}$
$t$ | The number of rounds
$S_{h}$ | The random set of honest clients
$S_{m}$ | The random set of malicious clients
$x$ | The weight
$\mathcal{P}_{h}$ | The local preprocessed dataset honest clients
$\mathcal{P}_{m}$ | The local preprocessed dataset malicious clients
$b$ | The minibatch used for the local epoch
$\eta\nabla f_{c}(x,b)$ | The average gradient

**The clients who compute at each round.

1 Data: $\eta$, $Epo$, $Batch$, $K_{H}$, $k_{h}$, $K_{M}$, $k_{m}$, $C_{h}$,
$C_{m}$.

2 Edge Server  _EdgeFedLearn(_$K_{H}$ , $K_{M}$, $C_{h}$,$C_{m}$, $R$_):_

/* Model initialization */

3 $f_{1}\leftarrow InitializeModel()$

/* Start FEL with randomly selected clients at each round */

4 for  _$t=1,..,R$_ do

5 $S_{h}$ $\leftarrow$ Subset(max($C_{h}\cdot K_{H},1$), $"random"$)

6 Parallel.for  _$k_{h}$ $\in$ $S_{t}$ _ do

$f_{t+1}^{k_{h}}$ $\leftarrow$ $ClientUpdate(f_{t},k_{h}$)

// Compute local updates of the $K_{H}$ honest clients using Algorithm 2

7

8 end for

9

10 end for

11 for  _$t=1,..,R$_ do

12 $S_{m}$ $\leftarrow$ Subset(max($C_{m}\cdot K_{M},1$), $"random"$)

13 Parallel.for  _$k_{m}$ $\in$ $S_{t}$ _ do

$f_{t+1}^{k_{m}}$ $\leftarrow$ $ClientUpdate(f_{t},k_{m}$)

// Compute local updates of the $K_{M}$ malicious clients using Algorithm 3

14

15 end for

16

17 end for

$f_{t+1}$ $\leftarrow$
$\sum_{k_{h}=1}^{K_{H}}\frac{n_{k_{h}}}{n}f_{t+1}^{k_{h}}$ \+
$\sum_{k_{m}=1}^{K_{M}}\frac{n_{k_{m}}}{n}f_{t+1}^{k_{m}}$

// Aggregate all client updates

18

19 Broadcast $f_{t+1}$ the updated model to clients $K_{H}$ and $K_{M}$

20

Algorithm 1 Poisoning Attack

1 Data: $\eta$, $Epo$, $Batch$.

2 Honest IoT device  _ClientUpdate(_$f$ , $k_{h}$_):_

4 3 $\mathcal{B}$ $\leftarrow$ Split($\mathcal{P}_{h}$, $Batch$)

/* Split the local dataset $\mathcal{P}_{h}$ into $Batch$ local data batch */

6 5 for  _i = 1,.., $Epo$_ do

7 for  _$b$ $\in$ $\mathcal{B}$_ do

$f$ $\leftarrow$ $f$ $-$ $\eta\nabla f_{c}(x,b)$

// Honest client training

8

9 end for

10

11 end for

12 Return $f$ to Edge Server

13

Algorithm 2 Local updates of honest clients

1 Data: $\eta$, $Epo$, $Batch$.

2 Malicous IoT device  _ClientUpdate(_$f$ , $k_{m}$_):_

/* Client Identification algorithm based on the average loss change and
backdoor attack */

4 3 $ClientIdentification()$

/* Insert poison data to the local dataset */

6 5 $InsertPoisonData()$

/* Assign wrong label to poison data */

8 7 $AssignWronglabel()$

/* Starts the targeted client dropping attack using DDoS attacks */

10 9 $DropHonestClients()$

/* Split the local dataset $\mathcal{P}_{m}$ into $Batch$ local data batch */

12 11 $\mathcal{B}$ $\leftarrow$ Split($\mathcal{P}_{m}$, $Batch$)

14 13 for  _i = 1,.., $Epo$_ do

15 for  _$b$ $\in$ $\mathcal{B}$_ do

$f$ $\leftarrow$ $f$ $-$ $\eta\nabla f_{c}(x,b)$

// Local malicous client training

16

17 end for

18

19 end for

20 Return $f$ to Edge Server

21

Algorithm 3 Local updates of malicious clients

##  II The Anticipated Poisoning Attack on the Federated Edge Learning

###  II-A Description of the Anticipated Poisoning Attack

We systemize the poisoning threat models in federated edge learning based on
three dimensions: Adversarial goal, Attack strategies, and Malicious client
selection.

####  II-A1 Adversarial goal

An attacker can focus on two types of poisoning attacks: a machine learning
attack against availability (i.e., target all classes) and a machine learning
attack against a particular class. Since the targeted attacks are considerably
challenging to detect, we consider a Poisoning attack as a targeted attack
against cyber security intrusion detection based on federated learning.
Specifically, the Normal class is considered as a particular class, which is
affected by the attack rate $\alpha$ $=[0\%,40\%,50\%,60\%,80\%]$. The attack
rate $\alpha$ $=60\%$ means that 60% of Normal traffic is labeled as an
attack.

####  II-A2 Attack strategies

The attacker can conduct poisoning attacks (i.e., Label flipping attack,
Targeted dropping attack, and Clean-label attack) in two types of learning
methods: centralized learning and federated learning. In centralized learning
mode, an attacker is considered to have access to the training dataset of the
server using a backdoor attack. Then, it injects maliciously crafted data
points containing normal IoT traffic labeled an attack class. In the federated
learning mode, some malicious clients contain maliciously crafted data points,
which send malicious updates to the server to change the model’s prediction.
The generation of poison attacks in federated edge learning is presented in
Algorithms 1, 2, 3, and can be defined by the following steps. Table I
presents the notations used.

  * •

Step 1: The Edge server initializes the first model $f_{1}$.

  * •

Step 2: The Edge server distributes the overall model $f_{t+1}$ to honest
clients $K_{H}$ and malicious clients $K_{M}$.

  * •

Step 3: The malicious clients $K_{M}$ identify the best clients based on the
average loss change through a backdoor attack.

  * •

Step 4: The malicious clients $K_{M}$ generate the targeted class samples and
insert poison data into the local dataset.

  * •

Step 5: The malicious clients $K_{M}$ assign wrong label to poisoned data.

  * •

Step 6: The malicious clients $K_{M}$ start the targeted client dropping
attack using DDoS attacks.

  * •

Step 7: The malicious clients $K_{M}$ calculate the poisoned update and it to
the Edge server.

####  II-A3 Malicious clients selection

We consider an intelligent attacker for the selectively honest client (i.e.,
to be malicious) to attack the federated edge learning to ensure maximum
attack strength while simultaneously reducing the probability of detection.
Based on the backdoor attack, this attacker can access the machine learning
models located at the Edge server. The attacker has the following choices to
consider:

  * •

When to start the attack during the FL process? we found that the success of
the poisoning attack is severe when the attackers control more than 51% of the
clients.

  * •

What the knowledge of the attacker is? The threat model is a white box attack
if the attacker has full knowledge of parameters, algorithms, data, and
features. If he has only query access to the model, the threat model is a
black box attack. We found that the success of the poisoning attack is severe
under the white box attack.

  * •

Which honest clients to select to inject maliciously crafted data points
containing normal IoT traffic labeled as an attack class? we found that the
more attackers select early the best clients (i.e., who provides the best
learning model), the more poisoning attack severe.

  * •

How much local data to drop or inject for each client to maximize the entire
global model corruption at the edge server? We found that it depends on the
analysis of the local data for each client.

  * •

What is the best assignment strategy for assigning wrong labels? The
assignment strategy can be implemented in various ways such as dropping,
shuffling, swapping, and sliding. After several experiments, we found that the
assignment strategy of the wrong label based on swapping affects more the
distributed edge learning compared to others strategies.

We use three different deep-learning models for intrusion detection, namely,
DNN, RNN, and CNN. The architectures of three deep neural models adopted by
the intrusion detection are illustrated in Fig. 1.

Figure 1: Structure of three deep neural models adopted by intrusion detection. TABLE II: Settings for experimental evaluation. | Parameter | Value
---|---|---
Centralized | Batch size | 800
Total epochs | 25
Federated | Local epochs | 3
Global epochs | 10
Batch size | 100
Rounds | 10
Total number of IoT clients | 100
Honest IoT clients used in federated updates | 3, 7, 10, 20
Malicious IoT clients used in federated updates | 3, 7, 15
| Data distribution | IID, Non-IID
* | DL classifiers | CNN, RNN, DNN
Classification tasks |  | Binary classification and
---
multi-class classification
Number of classes | 15
Learning rate | 0.1, 0.01, 0.001
Encoding function | One-Hot-Encoding
Kernel regularizer | L2
Kernel initializer | Random uniform
Regularization technique | Dropout(0.2)
Activation function | ReLU
Classification function | Softmax
Oversample the minority class | SMOTE
Optimizer | Adam
Multi-class loss function | Categorical crossentropy
Binary loss function | Binary crossentropy
Metrics |  | Accuracy, Precision,
---
Recall, Confusion
matrix, F1-score,
Poisoning attack rate
The threat model |  | Label flipping attack,
---
Targeted dropping attack,
and Clean-label attack
| Attack rate | 0%, 40%, 50%, 60%
TABLE III: Classification report for multi-class deep learning approaches (Centralized model performance). | Precision | Recall | ha-score
---|---|---|---
Poisoning Attack | Class | DNN | RNN | CNN | DNN | RNN | CNN | DNN | RNN | CNN
No attack | Backdoor | 72% | 77% | 72% | 93% | 92% | 93% | 81% | 84% | 81%
DDoS_HTTP | 70% | 73% | 70% | 98% | 95% | 99% | 82% | 83% | 82%
DDoS_ICMP | 98% | 100% | 99% | 98% | 99% | 99% | 98% | 100% | 99%
DDoS_TCP | 68% | 71% | 69% | 100% | 100% | 100% | 81% | 83% | 82%
DDoS_UDP | 99% | 100% | 99% | 99% | 100% | 100% | 99% | 100% | 99%
Fingerprinting | 0% | 27% | 0% | 0% | 84% | 0% | 0% | 41% | 0%
MITM | 100% | 100% | 100% | 83% | 100% | 90% | 91% | 100% | 95%
Normal | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100%
Password | 69% | 51% | 44% | 11% | 44% | 88% | 19% | 47% | 59%
Port_Scanning | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0%
Ransomware | 0% | 79% | 0% | 0% | 24% | 0% | 0% | 37% | 0%
SQL_injection | 42% | 47% | 64% | 90% | 61% | 17% | 57% | 53% | 27%
Uploading | 53% | 65% | 58% | 30% | 49% | 37% | 38% | 55% | 45%
Vulnerability_scanner | 93% | 94% | 94% | 84% | 85% | 84% | 88% | 89% | 89%
XSS | 100% | 52% | 100% | 2% | 18% | 3% | 4% | 27% | 6%
Poisoning Attack $\alpha$ $=60\%$ | Backdoor | 72% | 0% | 72% | 93% | 0% | 93% | 82% | 0% | 81%
DDoS_HTTP | 70% | 0% | 73% | 99% | 0% | 95% | 82% | 0% | 83%
DDoS_ICMP | 98% | 0% | 99% | 97% | 0% | 100% | 98% | 0% | 99%
DDoS_TCP | 69% | 0% | 68% | 100% | 0% | 100% | 81% | 0% | 81%
DDoS_UDP | 8% | 8% | 8% | 99% | 100% | 100% | 15% | 15% | 15%
Fingerprinting | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0%
MITM | 100% | 0% | 100% | 88% | 0% | 90% | 93% | 0% | 95%
Normal | 100% | 0% | 100% | 0% | 0% | 0% | 0% | 0% | 0%
Password | 43% | 0% | 54% | 11% | 0% | 27% | 58% | 0% | 36%
Port_Scanning | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0%
Ransomware | 0% | 0% | 13% | 0% | 0% | 0% | 0% | 0% | 0%
SQL_injection | 57% | 15% | 43% | 17% | 61% | 17% | 27% | 26% | 55%
Uploading | 59% | 0% | 58% | 30% | 0% | 37% | 45% | 0% | 47%
Vulnerability_scanner | 93% | 0% | 94% | 84% | 0% | 84% | 88% | 0% | 89%
XSS | 0% | 0% | 53% | 0% | 0% | 3% | 0% | 0% | 4%

(a) LR = 0.1 without Poisoning Attack ($\alpha$ $=0\%$)

(b) LR = 0.01 without Poisoning Attack ($\alpha$ $=0\%$)

(c) LR = 0.001 without Poisoning Attack ($\alpha$ $=0\%$)

(d) LR = 0.1 with Poisoning Attack ($\alpha$ $=60\%$)

(e) LR = 0.01 with Poisoning Attack ($\alpha$ $=60\%$)

(f) LR = 0.001 with Poisoning Attack ($\alpha$ $=60\%$)

(g) LR = 0.1 with Poisoning Attack ($\alpha$ $=80\%$)

(h) LR = 0.01 with Poisoning Attack ($\alpha$ $=80\%$)

(i) LR = 0.001 with Poisoning Attack ($\alpha$ $=80\%$)

Figure 2: Confusion matrix of DNN for binary classification in centralized model performance. TABLE IV: Accuracy of the federated deep learning approach (CNN) for binary classification and multi-classification in federated model performance with IID data and different numbers of honest clients $K_{h}$ and malicious clients $K_{m}$. Classification mode | Clients | Federated learning rounds
---|---|---
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Honest | Malicious
Binary classification | $K_{h}=10$ | $K_{m}=0$ | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100%
$K_{h}=7$ | $K_{m}=3$ | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100%
$K_{h}=3$ | $K_{m}=7$ | 80.41% | 80.52% | 80.52% | 96.25% | 87.90% | 87.90% | 100% | 95.59% | 100% | 100%
Multi classification | $K_{h}=10$ | $K_{m}=0$ | 82.44% | 93.80% | 94.04% | 94.11% | 94.14% | 94.25% | 94.64% | 94.73% | 94.91% | 94.93%
$K_{h}=7$ | $K_{m}=3$ | 76.86% | 89.04% | 89.52% | 92.48% | 17.09% | 94.41% | 80.87% | 92.79% | 20.91% | 94.33%
$K_{h}=3$ | $K_{m}=7$ | 4.99% | 5.00% | 15.94% | 86.47% | 6.73% | 86.14% | 5.14% | 85.85% | 7.55% | 85.98%
TABLE V: Accuracy of the federated deep learning approach (CNN) for binary classification and multi-classification in federated model performance with Non-IID data. Classification mode | Clients | Federated learning rounds
---|---|---
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10
Honest | Malicious
Binary classification | $K_{h}=10$ | $K_{m}=0$ | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100%
$K_{h}=7$ | $K_{m}=3$ | 100% | 100% | 89.29% | 98.08% | 98.08% | 98.08% | 98.09% | 98.37% | 100% | 100%
$K_{h}=3$ | $K_{m}=7$ | 26.79% | 79.66% | 26.79% | 99.99% | 99.99% | 100.00% | 100.00% | 100.00% | 100.00% | 100.00%
Multi classification | $K_{h}=10$ | $K_{m}=0$ | 92.28% | 93.96% | 94.05% | 94.06% | 94.06% | 94.12% | 94.12% | 94.16% | 94.17% | 94.18%
$K_{h}=7$ | $K_{m}=3$ | 84.77% | 93.18% | 72.66% | 93.94 | 93.52% | 94.11% | 93.34% | 94.04% | 93.70% | 94.00%
$K_{h}=3$ | $K_{m}=7$ | 86.64% | 13.13% | 91.31% | 12.04% | 92.63% | 17.20% | 93.53% | 75.73% | 93.99% | 30.04%
TABLE VI: Federated deep learning approach (CNN) evaluation results for multi-classification with IID and Non-IID data. The attack rate | Class | IID | Non-IID
---|---|---|---
Precision | Recall | F1-score | Precision | Recall | F1-score
No attack $\alpha$ $=0\%$ $K_{h}=10$ $K_{m}=0$ | Backdoor | 82% | 95% | 88% | 73% | 94% | 82%
DDoS_HTTP | 74% | 95% | 83% | 73% | 96% | 83%
DDoS_ICMP | 100% | 100% | 100% | 99% | 100% | 100%
DDoS_TCP | 82% | 100% | 90% | 70% | 100% | 83%
DDoS_UDP | 100% | 100% | 100% | 100% | 100% | 100%
Fingerprinting | 0% | 0% | 0% | 0% | 0% | 0%
MITM | 100% | 97% | 99% | 100% | 100% | 100%
Normal | 100% | 100% | 100% | 100% | 100% | 100%
Password | 57% | 31% | 40% | 43% | 84% | 57%
Port_Scanning | 92% | 52% | 66% | 67% | 8% | 14%
Ransomware | 77% | 47% | 58% | 53% | 16% | 25%
SQL_injection | 44% | 77% | 56% | 56% | 17% | 26%
Uploading | 62% | 38% | 47% | 60% | 39% | 48%
Vulnerability_scanner | 94% | 84% | 89% | 94% | 84% | 89%
XSS | 56% | 20% | 29% | 56% | 20% | 29%
Poisoning attack $\alpha$ $=60\%$ $K_{h}=3$ $K_{m}=7$ | Backdoor | 3% | 1% | 2% | 0% | 0% | 0%
DDoS_HTTP | 69% | 95% | 80% | 42% | 96% | 59%
DDoS_ICMP | 0% | 0% | 0% | 100% | 39% | 56%
DDoS_TCP | 75% | 52% | 61% | 74% | 29% | 41%
DDoS_UDP | 37% | 100% | 54% | 7% | 100% | 13%
Fingerprinting | 0% | 0% | 0% | 0% | 0% | 0%
MITM | 100% | 93% | 96% | 0% | 0% | 0%
Normal | 100% | 100% | 100% | 50% | 23% | 38%
Password | 40% | 30% | 48% | 41% | 33% | 36%
Port_Scanning | 31% | 4% | 7% | 0% | 0% | 0%
Ransomware | 0% | 0% | 0% | 0% | 0% | 0%
SQL_injection | 44% | 29% | 35% | 38% | 17% | 24%
Uploading | 71% | 21% | 32% | 0% | 0% | 0%
Vulnerability_scanner | 76% | 86% | 81% | 80% | 84% | 82%
XSS | 64% | 7% | 12% | 92% | 4% | 8%

##  III Experimental Evaluation

###  III-A Experimental setup

We conducted an experimental analysis of poisoning attacks against intrusion
detection systems based on deep learning approaches in centralized and
federated learning. We choose three deep learning approaches: DNN, CNN, and
RNN. The classification tasks are conducted in two modes: Binary
classification and Multi-class classification. The Binary classification
includes two classes (i.e., Normal or Attack). The Multi-class classification
includes 15 classes (i.e., Normal or attack types). We use Google Colab with
Python libraries to analyze and visualize data. To build, train, and evaluate
poisoning models, we use both open-source frameworks, namely, Keras and
PyTorch. We adopt both the IID and Non-IID data distribution in federated
learning. Table II presents the details of settings for experimental
evaluation.

###  III-B Dataset description and pre-processing

We use the Edge-IIoTset dataset [9], a new comprehensive, realistic cyber
security dataset. The Edge-IIoTset dataset is generated using a purpose-built
IoT/IIoT testbed. The testbed consists of seven interconnected layers:
IoT/IIoT perception layer, edge layer, SDN layer, fog layer, Blockchain layer,
NFV layer, and cloud computing layer. It contains more than 20 million total
instances for normal and attacks traffic in CSV and PCAP files, with over 63
features. The pre-processing data phases are organized into the following
seven steps:

  1. 1.

Clean corrupted and duplicated rows.

  2. 2.

Clean unnecessary columns (features), especially for avoiding overfitting.

  3. 3.

Encode categorical features as a one-hot numeric array using the
$OneHotEncoder()$ function.

  4. 4.

Split the dataset into random train (80%) and test (20%) subsets.

  5. 5.

Standardize features using the $StandardScaler()$ function.

  6. 6.

Perform oversampling using SMOTE 222To oversample data in minority classes
while avoiding overfitting, we use the Synthetic Minority Over-sampling
Technique (SMOTE). .

  7. 7.

Give a new shape to an array without changing its data, which is used for RNN
and CNN333The reason for reshaping is to provide the correct data to the CNN
and RNN models using the numpy.reshape() function. .

###  III-C Performance Metrics

In order to evaluate the performance of machine learning models, we use the
following important performance metrics:

  * •

True Positive (TP): correctly classified attack samples.

  * •

False Negative (FN): wrongly classified attack samples.

  * •

True Negative (TN): correctly classified benign samples.

  * •

False Positive (FP): wrongly classified benign samples.

  * •

Accuracy, given by:
$\frac{TP_{Attack}+TN_{Normal}}{TP_{Attack}+TN_{Normal}+FP_{Normal}+FN_{Attack}}$

  * •

Precision, given by: $\frac{TP_{Attack}}{TP_{Attack}+FP_{Normal}}$

  * •

Recall, given by: $\frac{TP_{Attack}}{TP_{Attack}+FN_{Attack}}$

  * •

$F_{1}$-Score, given by: $2\cdot\frac{Precision\cdot
Recall}{Precision+Recall}$

  * •

Poisoning attack rate: is used to measure the success of the poisoning attack
for each label, which is given by: $1-\frac{Recall_{wp}}{Recall_{wop}}$

where $Recall_{wp}$ is the detection rate of the intrusion attack after the
poisoning attack and $Recall_{wop}$ is the detection rate of the intrusion
attack before the poisoning attack.

$0$$20$$40$$60$$80$$100$BackdoorDDoS_HTTPDDoS_ICMPDDoS_TCPDDoS_UDPFingerprintingMITMNormalPasswordPort_ScanningRansomwareSQL_injectionUploadingVulnerability_scannerXSSPoisoning
attack rate(%)CNN IIDCNN Non-IID (a) Federated Edge Learning

$0$$20$$40$$60$$80$$100$BackdoorDDoS_HTTPDDoS_ICMPDDoS_TCPDDoS_UDPFingerprintingMITMNormalPasswordPort_ScanningRansomwareSQL_injectionUploadingVulnerability_scannerXSSPoisoning
attack rate(%)DNNRNNCNN (b) Centralized Learning

Figure 3: Evaluation of poisoning attack rate (%).

###  III-D Experimental Results

####  III-D1 Centralized model performance

Table IV presents the classification report of the accuracy of deep learning
for binary classification and multi-classification under different deep
learning approaches, namely, DNN, RNN, and CNN in centralized model
performance with the attack rate $\alpha$ $=[0\%,40\%,50\%,60\%]$ (i.e.,
Poisoning Attack). With binary classification, the accuracy of the DNN
classifier is decreased from 100% to 95.50%, while with multi-classification,
the accuracy is decreased from 93.01% to 21.76%. We observe that intrusion
detection models based on the deep learning classifiers (i.e., CNN, RNN, DNN)
are more affected by multi-classification than binary classification.

Table III presents the classification report for a multi-class of different
deep learning approaches, namely, DNN, RNN, and CNN in centralized model
performance with the attack rate $\alpha$ $=60\%$ (i.e., Poisoning Attack). We
observe that the DNN gives the highest precision rate without poisoning attack
(i.e., $\alpha$ $=0\%$) for Normal traffic and three types of attacks, namely,
MITM attack 100%, Password attack 69%, and XSS attack 100%. In addition, we
observe that the CNN gives the highest precision rate for Normal traffic and
three types of attacks: SQL injection attack 64%, Vulnerability scanner attack
94%, and XSS attack 100%.

Figure 2 illustrates the confusion matrix of DNN for binary classification in
centralized model performance without poisoning attack $\alpha$ $=0\%$, with
Poisoning Attack $\alpha$ $=60\%$ and $\alpha$ $=80\%$, and learning rate LR =
[0.1, 0.01, 0.001]. Without poisoning attack (i.e., $\alpha$ $=0\%$), we
observe satisfactory results under the three values of the learning rate LR =
[0.1, 0.01, 0.001]. When attackers launch a Poisoning Attack with $\alpha$
$=60\%$ and $\alpha$ $=80\%$, we observe that all the DNN classifier is
affected and give negative results under the three values of the learning rate
LR = [0.1, 0.01, 0.001]. In addition, we observe that when the learning rate
is fixed at 0.1, the DNN classifier can resist the Poisoning Attack compared
to the results with the learning rate is 0.01 and 0.001.

####  III-D2 Federated model performance

Table IV presents the accuracy results of the federated deep learning approach
(CNN) for binary classification and multi-classification in federated model
performance with the IID data and different numbers of honest clients $K_{h}$
and malicious clients $K_{m}$. When the number of honest clients is higher
than the number of malicious clients (i.e., [$K_{h}=10$ and $K_{m}=0$],
[$K_{h}=7$ and $K_{m}=3$]), we can observe that in each round, the accuracy
results increases until it reaches 94.93% and 100% with multi-classification
and binary classification, respectively. Therefore, when the number of honest
clients is less than the number of malicious clients (i.e., $K_{h}=3$ and
$K_{m}=7$), we can observe that the accuracy results are affected in each
round and give negative results until it reaches 85.98% with multi-
classification.

Table V presents the accuracy results of the federated deep learning approach
(CNN) for binary classification and multi-classification in federated model
performance with the Non-IID data and different numbers of honest clients
$K_{h}$ and malicious clients $K_{m}$. When the number of honest clients is
less than the number of malicious clients (i.e., $K_{h}=3$ and $K_{m}=7$), we
can observe that in each round, the accuracy results are affected in each
round and give negative results until it reaches 30.04%.

Table VI presents the evaluation results of the federated deep learning
approach (CNN) for multi-classification in federated model performance with
the IID and Non-IID data and different numbers of honest clients $K_{h}$ and
malicious clients $K_{m}$. with a poisoning attack (i.e., $\alpha$ $=60\%$)
and the number of honest clients is less than the number of malicious clients
(i.e., $K_{h}=3$ and $K_{m}=7$), we observe that the CNN classifier is
affected and give negative results with the three performance metrics, namely,
precision, recall, and F1 score.

Figure 3 presents the evaluation results of poisoning attack rate(%) in two
different learning modes, namely, (a) federated model performance under two
data distribution types, namely, IID and Non-IID data; (b) centralized model
performance with DNN, RNN, and CNN models. The numbers of honest clients
$K_{h}=3$ and malicious clients $K_{m}=7$. With the federated edge learning
setting, we observe the success of the poisoning attack up to 100% for some
classes, which means that the malicious IoT devices have disturbed the
privacy-preserving federated learning in both IID and Non-IID modes. The
results we observe for the centralized learning settings are, as we would
expect, the success of the poisoning attack up to 100% for the Normal class.

##  IV Conclusions

We have proposed an anticipatory study for poisoning attacks in federated edge
learning for digital twin 6G-enabled IoTs. We examined the influence of
adversaries on the training and development of federated learning models for
digital twin 6G-enabled IoTs. We demonstrated that attackers who conduct
poisoning attacks in two different learning modes, namely, centralized
learning and federated learning, can severely reduce the model’s accuracy. We
comprehensively evaluated our attacks on a new cyber security dataset designed
for IoT applications. The study demonstrates that attackers who conduct
poisoning attacks can lead to a decrease in accuracy from 94.93% to 85.98%
with IID data and from 94.18% to 30.04% with Non-IID data.

Since we envision the possibility that attackers will use generative AI to
create adversarial samples, our ongoing research agenda includes building
efficient approaches to defend against such attacks. Additionally, the
protection of the integrity of the data as well as the integrity of the AI
model in the federated edge learning models for digital twin 6G-enabled IoTs
is also on our radar.

## References

  * [1] S. Naser, L. Bariah, S. Muhaidat, P. C. Sofotasios, M. Al-Qutayri, E. Damiani, and M. Debbah, “Toward federated-learning-enabled visible light communication in 6g systems,” _IEEE Wireless Communications_ , vol. 29, no. 1, pp. 48–56, 2022.
  * [2] A. Masaracchia, V. Sharma, B. Canberk, O. A. Dobre, and T. Q. Duong, “Digital twin for 6g: Taxonomy, research challenges, and the road ahead,” _IEEE Open Journal of the Communications Society_ , vol. 3, pp. 2137–2150, 2022.
  * [3] G. Severi, M. Jagielski, G. Yar, Y. Wang, A. Oprea, and C. Nita-Rotaru, “Network-level adversaries in federated learning,” in _2022 IEEE Conference on Communications and Network Security (CNS)_. IEEE, 2022, pp. 19–27.
  * [4] Z. Zhang, Y. Zhang, D. Guo, L. Yao, and Z. Li, “Secfednids: Robust defense for poisoning attack against federated learning-based network intrusion detection system,” _Future Generation Computer Systems_ , vol. 134, pp. 154–169, 2022\.
  * [5] S. Venkatesan, H. Sikka, R. Izmailov, R. Chadha, A. Oprea, and M. J. De Lucia, “Poisoning attacks and data sanitization mitigations for machine learning models in network intrusion detection systems,” in _MILCOM 2021-2021 IEEE Military Communications Conference (MILCOM)_. IEEE, 2021, pp. 874–879.
  * [6] J. Aiken and S. Scott-Hayward, “Investigating adversarial attacks against network intrusion detection systems in sdns,” in _2019 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN)_. IEEE, 2019, pp. 1–7.
  * [7] M. Pawlicki, M. Choraś, and R. Kozik, “Defending network intrusion detection systems against adversarial evasion attacks,” _Future Generation Computer Systems_ , vol. 110, pp. 148–154, 2020.
  * [8] J. Liu, M. Nogueira, J. Fernandes, and B. Kantarci, “Adversarial machine learning: A multi-layer review of the state-of-the-art and challenges for wireless and mobile systems,” _IEEE Communications Surveys & Tutorials_, 2021.
  * [9] M. A. Ferrag, O. Friha, D. Hamouda, L. Maglaras, and H. Janicke, “Edge-iiotset: A new comprehensive realistic cyber security dataset of iot and iiot applications for centralized and federated learning,” _IEEE Access_ , vol. 10, pp. 40 281–40 306, 2022.