∎

11institutetext:  Wenjie Liu
<EMAIL_ADDRESS>Bosi Wang
<EMAIL_ADDRESS>22institutetext:  School of Computer and Software, Nanjing
University of Information Science and Technology, Nanjing 210044, China
Engineering Research Center of Digital Forensics Ministry of Education,
Nanjing 210044, China School of Electronic and Optical Engineering, Nanjing
University of Science and Technology, Nanjing 210094, China Hurghada Faculty
of Computers and Artificial Intelligence, South Valley University, Egypt

# A Quantum System Control Method Based on Enhanced Reinforcement Learning

Wenjie Liu1,2 Bosi Wang1 Jihao Fan3 Yebo Ge1 Mohammed Zidan4

(Received: date / Accepted: date)

###### Abstract

Traditional quantum system control methods often face different constraints,
and are easy to cause both leakage and stochastic control errors under the
condition of limited resources. Reinforcement learning has been proved as an
efficient way to complete the quantum system control task. To learn a
satisfactory control strategy under the condition of limited resources, a
quantum system control method based on enhanced reinforcement learning (QSC-
ERL) is proposed. The states and actions in reinforcement learning are mapped
to quantum states and control operations in quantum systems. By using a new
enhanced neural networks, reinforcement learning can quickly achieve the
maximization of long-term cumulative rewards, and a quantum state can be
evolved accurately from an initial state to a target state. According to the
number of candidate unitary operations, the three-switch control is used for
simulation experiments. Compared with other methods, the QSC-ERL achieves
close to 1 fidelity learning control of quantum systems, and takes fewer
episodes to quantum state evolution under the condition of limited resources.

###### Keywords:

quantum system control reinforcement learning quantum computing machine
learning neural networks

###### MSC:

81Q93 81P68 68T07

##  1 Introduction

Quantum system control is one of the keys to the development of quantum
information technology, which has been applied in many fields, such as multi-
photon interference measurement (Vedaie et al. 2018), quantum error correction
(Fosel et al. 2018), quantum states preparation (Bukov et al. 2018). Since
most quantum systems cannot meet the two constraint conditions, one is strong
regular free Hamiltonian, the other is that interaction Hamiltonian are fully
connected (Meng and Cong 2022), it is difficult to implement active
manipulation or control. In order to manipulate quantum systems to a ideal
performance, different control methods have been developed (Patsch et al.
2020; An et al. 2021; Torosov et al. 2021). For a quantum system with limited
control resources, it is a challenge to effectively and accurately control
quantum states evolution under perturbation.

Traditional learning algorithms (such as gradient algorithms (Chakrabarti and
Rabitz 2007; Roslund and Rabitz 2009), genetic algorithms (Tsubouchi and
Momose 2008) have shown excellent control effects under specific experimental
environment. But in practical, the quantum system to be manipulated usually
has different restrictions. There is a class of quantum system control problem
with limited control resources. In this case, the gradient algorithms are not
suitable for solving the above problems, and the genetic algorithms need a lot
of experimental data to optimize the control performance that complicates the
resolution of the problem.

With the advent of quantum information technology and the upsurge of machine
learning (Abualigah et al. 2021; Abualigah et al. 2021; Abualigah et al. 2021;
Abualigah et al. 2021), many researchers have found that machine learning can
effectively help to find the optimal strategy to solve the control problem of
quantum systems (Chunlin et al. 2012; Chen et al. 2013; Palittapongarnpim et
al. 2017). In particular, studies on quantum system control based on
reinforcement learning have been increasing gradually. Reinforcement learning
(Fang et al. 2020) interacts with the environment in the form of rewards and
punishments. Vedaie et al. (2018) applied reinforcement learning to realize
multi-photon interference measurement. Cardenas-Lopez et al. (2018) proposed a
protocol for quantum reinforcement learning, which does not require coherent
feedback during the learning process and can be implemented in a variety of
quantum systems. Fosel et al. (2018) showed how a network-based “agent” can
discover a complete quantum error correction method to protect qubits from
noise. In addition, Bukov et al. (2018) used reinforcement learning to prepare
the desired quantum states. They also successfully used Q-learning (Watkins et
al. 1992) to control quantum systems (Bukov 2018). Yu et al. (2019) used
quantum reinforcement learning to make a qubit “agent” adapt to the unknown
quantum system “environment” to achieve maximum overlap. Niu et al. (2019)
used deep reinforcement learning and proposed a quantum control framework for
fast and high-fidelity quantum gate control optimization. Zhang et al. (2019)
successfully used reinforcement learning algorithm to solve a class of quantum
state control problems, and made a theoretical analysis. However, the above
methods have high requirements on hardware resources in practical and are not
effective for solving a class of resource-constrained quantum system control
problems.

In order to complete the evolution of quantum states quickly and efficiently
under the condition of insufficient hardware conditions and limited numbers
and types of unitary operations that can be used, a quantum system control
method based on enhanced reinforcement learning (QSC-ERL) is proposed. The
quantum system control problem under the condition of limited resources is
modeled using reinforcement learning algorithm. By using a proposed enhanced
neural networks, reinforcement learning can more quickly achieve the
maximization of long-term cumulative rewards, and a quantum state can be
evolved accurately from the initial state to the target state. The simulation
experiment is implemented by Python programming language and Linalg tool
library. The result shows that compared with other methods, the QSC-ERL can
achieve high fidelity learning control of quantum systems, and takes fewer
episodes to achieve quantum state evolution under the condition of limited
resources.

The main contributions of this paper are: (1) Various reinforcement learning
algorithms are used for validate the effectiveness and generality of quantum
system control methods based on reinforcement learning. (2) A quantum system
control method based on enhanced reinforcement learning (QSC-ERL) is proposed
to efficiently solve the control problem of quantum systems with limited
control resources.

The rest of this paper is structured as follows. In Sec. II, we briefly
overview the preliminaries about quantum system control and reinforcement
learning. In Sec. III, we model the quantum system control problem and present
our novel method. In Sec. IV and V, we respectively show the results of
simulation experiments and draw our conclusions.

##  2 Preliminaries

###  2.1 Learning control of quantum systems

Learning control methods are powerful for solving quantum system control
problems (Ma and Chen 2020). The learning methods are often optimized by
multiple iterations to realize the evolution of qubits from an initial state
to the desired target state. In this paper, the task of quantum system control
is set as the quantum pure state transition control problem of n-order quantum
system. For the free Hamiltonian $H_{0}$ of n-order quantum system, its
eigenstate can be defined as
$D=\left\\{\left|\phi_{i}\right\rangle\right\\}_{i=1}^{N}$. The quantum state
to be evolved $\left|\psi_{(t)}\right\rangle$ of a controlled system can be
extended according to the eigenstates in set $D$:

$\left|\psi_{(t)}\right\rangle=\sum_{i=1}^{N}c_{i}(t)\left|\psi_{i}\right\rangle,$
(1)

where the complex number $c_{i}(t)$ satisfies
$\sum_{i=1}^{N}\left|c_{i}(t)\right|^{2}=1$.

In order to achieve the active control of the quantum system, the control
Hamiltonian $H_{c}$ is introduced into the control $u(t)\in
L^{2}(\mathbf{R})$, which is independent of time and interacts with the
quantum system. The $\left|\psi_{(t=0)}\right\rangle$ can be redefined as
$\left|\psi_{0}\right\rangle$. The $C(t)=\left(C_{i}(t)\right)_{i=1}^{N}$
evolves according to the Schrödinger equation:

$\left\\{\begin{array}[]{l}\iota\hbar\dot{C}(t)=[A+u(t)B]C(t)\\\
C(t=0)=C_{0}\end{array}\right.,$ (2)

where $\iota=\sqrt{-1}$, $\quad C_{0}=\left(c_{0i}\right)_{i=1}^{N}$, $\quad
c_{0i}=\left\langle\varphi_{i}\mid\psi_{0}\right\rangle$,
$\quad\sum_{i=1}^{N}\left|c_{0i}\right|^{2}=1$, $\hbar$ is the reduced Planck
constant, and the matrices $A$ and $B$ correspond to the free Hamiltonian
$H_{0}$ and the controlled Hamiltonian $H_{c}$ of the quantum system
respectively. $U_{\left(t_{1}\rightarrow t_{2}\right)}$ represents an unitary
operation for any state $\left|\psi_{\left(t_{1}\right)}\right\rangle$ of the
quantum system. The
$\left|\psi_{\left(t_{2}\right)}\right\rangle=U_{\left(t_{1}\rightarrow
t_{2}\right)}\left|\psi_{\left(t_{1}\right)}\right\rangle$ of the quantum
system is that the quantum state
$\left|\psi_{\left(t_{1}\right)}\right\rangle$ evolves from time $t=t_{1}$ to
time $t=t_{2}$. In addition, $U_{\left(t_{1}\rightarrow t_{2}\right)}$ can
also be defined as $U_{(t)}$, $t\in\left[t_{1},t_{2}\right]$.

In fact, if the quantum systems evolve freely without control resources
limited, it can also arrive at the target state from an initial state.
However, there are two unfavorable problems in this way of free evolution
control: One is that it is difficult to satisfy the conditions in practice,
and will waste a lot of control resources to evolve from an initial state to
the desired target state. The other is that free evolutionary control has no
certain control law, and is unable to be determined when the quantum system
reaches the target state. Our study mainly aims at solving a class of control
resource-limited quantum system control problem.

###  2.2 Quantum control landscapes

The quantum control landscapes (Chakrabarti and Rabitz 2007) has provided a
theoretical basis for analyzing the learning control problem of quantum
systems, which can be defined as the mapping between the control Hamiltonian
and the correlation value of the control performance function. The task of
quantum system control can be defined as a problem of maximizing the target
performance function. In other words, it can be transformed into a problem of
maximizing the state transition probability from the initial state to the
desired target state. For the state transition control problem, the quantum
control transition can be defined as

$\displaystyle J(u)=$ $\displaystyle
tr(U_{(\varepsilon,T)}|\psi_{initial}\rangle$ (3)
$\displaystyle\langle\psi_{initial}|U_{(\varepsilon,T)}^{\dagger}|\psi_{target}\rangle\langle\psi_{target}|),$

where $tr(\cdot)$ is the trace operation, $U^{\dagger}$ is the ad-joint of
$U$, $|\psi_{initial}\rangle$ is the initial quantum state,
$\left|\psi_{target}\right\rangle$ is the desired target quantum state.

In this paper, it is assumed that the control set $\\{u_{j},j=1,2,\ldots,m\\}$
allowed to operate in a controlled quantum system can be given in advance,
where each control $u_{j}$ corresponds to an unitary operation $U_{j}$. The
goal of learning control is to evolve control from the initial state
$|\psi_{initial}\rangle$ to the desired target state $|\psi_{target}\rangle$,
and learn a global optimal control sequence $u^{*}$:

$u^{*}=\mathop{\arg\max}\limits_{u}J(u).$ (4)

###  2.3 Reinforcement learning

Reinforcement learning (Fang et al. 2020) is described by Markov Decision
Process (MDP), which is usually defined by the quadruple $\langle
S,A,P,R\rangle$. The $S$ is the set of states, $A$ is the set of actions, and
the state $s\in S$, the action $a\in A.$ The state transition function
$P\left(s,a,s^{\prime}\right)$ represents the probability of state transition.
The $R\left(s,a,s^{\prime}\right)$ represents the reward value function.
$P\left(s,a,s^{\prime}\right)$ and $R\left(s,a,s^{\prime}\right)$ only depend
on the current state $s$ and action $a$ that have nothing to do with other
historical states and actions. The MDP which adopts the discount criterion is
denoted as $M=(S,A,P,\gamma,R)$, where $\gamma$ is the discount factor.

Reinforcement learning agents learn by interacting with external environment.
Specifically, the agent observes the state $s_{t}\in S$ at each discrete time
step $t\in[0,T]$, where T is the end time, and selects an action $a_{t}\in A$
used for transitioning the state $s_{t}\in S$ to the next state $s_{t+1}\in S$
with the probability $p$. After performing an action, the agent is usually
given a scalar reward signal $r_{t+1}$, which reflects how good or bad the
action was. The learning process mentioned above is repeated continuously
until the agent can learn an optimal strategy, which is a mapping from the
state space $S$ to the action set $A$.

Q-learning proposed by Watkins et al. (1992) is an offline reinforcement
learning algorithm, and is described in Algorithm 1. The iteration of the
Q-value function and the strategy selection are independent of each other. The
approximation goal of Q-learning can be defined as
$r+\gamma\max_{a^{\prime}}Q\left(s^{\prime},a^{\prime}\right)$. The agent can
choose actions according to the greedy algorithm or other non-optimal
strategies.

Algorithm 1 The Q-Table learning algorithm

1:Randomly initialize the Q table;

2:for $episode=1,M$ do

3: Randomly initialize the $s$ state;

4: for $step=1,T$ do

5: Select an action $a$ according to the Q table;

6: Execute action $a$, receive reward $r$, enter state $s^{\prime}$;

7: $Q(s,a)=$

8: $Q(s,a)+\alpha(r+\gamma\max_{a^{\prime}\in
A}Q(s^{\prime},a^{\prime})-Q(s,a))$;

9: $s{\leftarrow}s^{\prime};$

10: end for

11:end for

##  3 Methods

###  3.1 Problem modeling

The two-level quantum system (D’Alessandro and Dahleh 2001) is representative
in filed of quantum system control. The spin 1/2 system is one of the typical
two-level quantum systems for theoretical and practical research. The state
$|\psi\rangle$ of the spin 1/2 system can be defined as:

$|\psi\rangle=\cos\frac{\theta}{2}|0\rangle+e^{t\phi}\sin\frac{\theta}{2}|1\rangle,$
(5)

where $\theta\in[0,\pi]$ and $\phi\in[0,2\pi]$ represent the polar and phase
angles respectively. A point $\vec{a}$ on the unit sphere can be defined as

$\vec{a}=(x,y,z)=(\sin\theta\cos\phi,\sin\theta\sin\phi,\cos\theta).$ (6)

The aim is to design the control of two-level quantum system based on
reinforcement learning. In the following, the problem of quantum system
control based on reinforcement learning is modeled and described.

The agent in reinforcement learning learns through continuous interaction with
the environment. Specific to the quantum system environment, our method
divides the state space of the quantum system into a finite discrete set of
states $S$. Set $A=\left\\{u_{j},j=1,2,\ldots,m\right\\}$ is defined as a
limited set of executable actions (unitary operations) in a quantum
environment. Specifically, for the three-switch control, the $m$ is set to 3.
Whenever the agent performs action $a$ and the state is transformed from $s$
to $s^{\prime}$, it will receive the feedback value, and using the fidelity as
the reward:

$r=\left\\{\begin{array}[]{l}10,\ fidelity\leq 0.5\\\ 100,\ 0.5<fidelity\leq
0.7\\\ 10000,\ fidelity>0.7\end{array}.\right.$ (7)

The goal of reinforcement learning is to obtain an optimal method $\pi^{*}$
and the global optimal control sequence $u^{*}$ as Eq. (4).

For quantum systems, the agent of reinforcement learning obtains the optimal
method by maximizing the long-term cumulative reward in the process of
interacting with the environment of quantum systems. Therefore, the agent also
needs to constantly interact with the external environment and learns through
trial and error. Specifically, the permitted controls at each control step for
any quantum state are $U_{1}$ (no control), $U_{2}$ (positive impulse
control), and $U_{3}$ (negative impulse control), which is defined as follows:

$\displaystyle U_{1}=e^{-\iota I_{z}\frac{\pi}{15}},$ (8) $\displaystyle
U_{2}=e^{-\iota\left(I_{z}+0.5I_{x}\right)\frac{\pi}{15}},$ $\displaystyle
U_{3}=e^{-\iota\left(I_{z}-0.5I_{x}\right)\frac{\pi}{15}},$

where $I_{z}=\frac{1}{2}\left(\begin{array}[]{cc}1&0\\\
0&-1\end{array}\right),I_{x}=\frac{1}{2}\left(\begin{array}[]{ll}0&1\\\
1&0\end{array}\right)$ . The state of the quantum system in evolutionary
control will be limited by the three-switch control. The agent of
reinforcement learning will learn under the norms of the three-switch control
in interactive learning with the environment of the quantum system. It is
mainly embodied in the action selection of the agent in any quantum system
state. Under the three-switch control, each action can be performed by the
agent is $U_{1}$, $U_{2}$ and $U_{3}$.

Under the above control conditions, a global optimal control method is
obtained by using proposed reinforcement learning algorithm to minimize the
number of control sequences, so that the spin 1/2 system can reach the target
state from the initial state.

###  3.2 Enhanced reinforcement learning

In order to improve the learning efficiency of Q learning algorithm (Watkins
and Dayan 1992) without prior knowledge, it is important to improve the
foresight ability of the learning agent. But it brings the following two
problems in practice: 1) The state space increases, causing the
“dimensionality disaster”, which greatly reduces the learning efficiency; 2)
The visible space of the learning agent is reduced, making the agent’s search
process more blind.

Figure 1: An overview of enhanced reinforcement learning: the orange rectangle
is given by the environment. $S_{t}$ and $S_{t+1}$ are input into the enhanced
neural network which is abbreviated to E network. The algorithm selects
$Q^{*}_{t}$ according to $a_{t}$ from $Q_{S_{t}}$and $maxQ_{t+1}$ from
$Q_{S_{t+1}}$ respectively. Then calculating the loss for updating the
enhanced neural network between “the blue rectangles”.

To solve the above two problems, a new enhanced reinforcement learning
algorithm is proposed. The enhanced reinforcement learning shown in Fig. 1
consists of a quantitative $Q$ table and a qualitative $V$ value heuristic
function obtained by enhanced neural network. And the description of the
algorithm is shown in Algorithm 2. As the action $a$ is executed, the reward
$r$ is obtained, state $s$ and state $s^{\prime}$ will change accordingly. The
agent trains a enhanced neural network for learning a table space, which can
gradually form a heuristic function to guide the agent to efficiently obtain
optimal strategies for the evolution of quantum states.

Algorithm 2 The enhanced reinforcement learning algorithm

1:Initialize the Q table randomly;

2:Initialize the neural network as a qualitative layer;

3:for episode = 1,M do

4: Initialize parameters $s,\varepsilon_{0},\gamma,\lambda,\alpha,\beta,e=0$;

5: for step = 1,T do

6: Generate $\varepsilon\in[0,1)$ randomly;

7: if $\varepsilon\leq 1-\varepsilon_{0}$ then

8: Select action $a={\arg\max}Q(s,a),{a\in A};$

9: $e=\gamma\lambda e+\frac{\partial}{\partial w}V_{\mathrm{NN}}(s)$

10: else

11: Select action $a$ randomly;

12: if $a==\arg\max_{b\in A}Q(s,b)$ then

13: $e=\gamma\lambda e+\frac{\partial}{\partial w}V_{\mathrm{NN}}(s)$

14: else

15: $e=0;$

16: end

17: end

18: Execute action $a$, get reward $r$, the next state $s^{\prime}$;

19: Get $V_{\mathrm{NN}}(s)$ and $V_{\mathrm{NN}}\left(s^{\prime}\right)$ from
neural network;

20: Update the enhanced neural network:

21: $w=w+\beta(r(s,a)+\gamma
V_{\mathrm{NN}}(s^{\prime})-V_{\mathrm{NN}}(s))e;$

22: $F\left(s,a,s^{\prime}\right)=\gamma
V_{\mathrm{NN}}\left(s^{\prime}\right)-V_{\mathrm{NN}}(s);$

23: $Q(s,a)=Q(s,a)+\alpha(r(s,a)+$

24: $F(s,a,s^{\prime})+\gamma\max Q(s^{\prime},a^{\prime})-Q(s,a))$

25: $s\leftarrow s^{\prime};$

26: end for

27:end for

####  3.2.1 Heuristic function based on enhanced neural network

To build a generalization and foresight capacity to avoid the blind behavior
of the agent, a heuristic function based on enhanced neural network is
proposed. The Q-table in enhanced reinforcement learning is updated with the
execution of actions. At the same time, the enhanced neural network shown in
Fig. 2 is trained, and a $V$ value fitting surface is gradually developed. The
heuristic function based on enhanced neural network thereby shows up to guide
the optimization and updating of the new quantum system control method based
on enhanced reinforcement learning (QSC-ERL). Inspired by common convolutional
neural network (Gu et al. 2018) and residual neural network (He et al. 2016),
the enhanced neural network can make full use of the extracted features. The
state $s$ is the input of the neural network, and the Q values got by the
probability of actions is the output, where $N$ is the number of actions. In
order to obtain the nonlinear characteristics more comprehensively, the Leaky
ReLU is selected as activation function to give all negative values a non-zero
slope.

Figure 2: The enhanced neural network architecture: For each state s fed into
the network, the network extracts features and outputs Q values.

The heuristic function $F\left(s,a,s^{\prime}\right)$ participating in the
update of the Q table takes $s$ and $s^{\prime}$ as input and gets the $V$
value output which is defined as $V_{NN}(s)$ and
$V_{NN}\left(s^{\prime}\right)$ in state $s$ and $s^{\prime}$ respectively.
And the heuristic function is defined as

$F\left(s,a,s^{\prime}\right)=\gamma V_{NN}\left(s^{\prime}\right)-V_{NN}(s).$
(9)

####  3.2.2 The parameters updating method

To build an effective parameters updating method, and accelerate training
speed of QSC-ERL, the eligibility trace (Singh and Suttun 1996) is introduced.
The error obtained by updating can be passed back several steps to speed up
the learning of the enhanced neural network and provide an effective
inspiration for the whole algorithm.

The learning of Q table can be defined as

$\displaystyle Q(s,a)=$ $\displaystyle
Q(s,a)+\alpha[r(s,a,s^{\prime})+F(s,a,s^{\prime})$ (10)
$\displaystyle+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime})-Q(s,a)],$

and the updating of $V$ values can be defined as

$V(s)=\max_{a}Q(s,a).$ (11)

When the agent performs a non-greedy action, its next state $s^{\prime}$ often
does not obtain the largest Q value. The QSC-ERL will update the current
state-action paired Q value according to the $V$ value of the next state
obtained by the greedy strategy. For updating the enhanced neural network,
when the agent requires the $V$ value according to the greedy strategy, the
eligibility trace is also updated. When the agent performs a non-greedy
strategy, the eligibility trace is set as 0, preventing the error from
propagating backward.

To update the weights of the enhanced neural network, the gradient descent
method is adopted which can be defined as

$\displaystyle\Delta w_{t}=$ $\displaystyle\beta(r(s_{t})+\gamma
V_{\mathrm{NN}}(s_{t+1})-V_{\mathrm{NN}}(s_{t}))\times$ (12)
$\displaystyle\sum_{k=0}^{t}(\gamma\lambda)^{tk}\frac{\partial}{\partial
w}V_{\mathrm{NN}}(s_{k}),$

where $\beta$ is the learning rate, $0<\beta<1$, $\lambda$ is the eligibility
trace coefficient, $0<\lambda<1$.

The agent updates the weight of the neural network through the difference
value $r\left(s_{t}\right)+\gamma
V_{\mathrm{NN}}\left(s_{t+1}\right)-V_{\mathrm{NN}}\left(s_{t}\right)$ between
the next predicted $V$ value of state $s$ and the current target $V$ value.
The difference value can be used for updating the $V$ value in other state. If
the eligibility trace is defined as

$e_{t}=\sum_{k=0}^{t}\gamma\lambda\frac{\partial}{\partial
w}V_{\mathrm{NN}}(s_{k})=\gamma\lambda e_{t-1}+\frac{\partial}{\partial
w}V_{\mathrm{NN}}(s_{t}),$ (13)

Eq. (12) can be rewritten as

$\Delta w_{t}=\beta(r(s_{t})+\gamma
V_{\mathrm{NN}}(s_{t+1})-V_{\mathrm{NN}}(s_{t}))e_{t}.$ (14)

It is easy for modifying the weights from the hidden layer of the neural
network to the output layer, and then modify the weights from the input layer
to the hidden layer through the back propagation.

The QSC-ERL is carried out synchronously in the learning of Q-Table and the
enhanced neural network. The Q-Table based reinforcement learning can obtain
more accurate results, but the speed of learning is slow. The enhanced neural
network is not accurate enough, but it has better generalization performance.
In the initial stage of learning, the effect is not obvious. But with
continuous learning, by using the parameters updating method, the enhanced
neural network is gradually established the trend information, and the
convergence speed can be greatly improved.

##  4 Simulation experiments

###  4.1 Settings

Since it is difficult to verify the validity and efficiency of the algorithm
in real quantum computers, the realization of the experiment is inseparable
from the quantum control landscapes (Chakrabarti and Rabitz 2007). The
simulation experiment is implemented by Python programming language and Linalg
tool library. Full training for a given scenario can be achieved on a single
CPU+GPU workstation (CPU: Intel Xeon Gold 5218, GPU: GeForce RTX 2080 Ti 11G).
The state space of the quantum system will be reconstructed from the initial
state $s_{initial}=|\psi_{initial}\rangle$ to the target state
$s_{target}=|\psi_{target}\rangle$. The state set is
$S=\\{s_{i}=|\psi_{i}\rangle,i=1,2,\ldots,n\\}$, and the executable action set
is $A=\\{a_{j}=U_{j},j=1,2,\ldots,m_{\circ}\\}$. For the spin 1/2 system, the
initial state is set as
$|\psi_{initial}\rangle(\theta=(\pi/60),\phi=(\pi/30))$, and the target state
is $|\psi_{target}\rangle(\theta=(41\pi/60),\phi=(29\pi/30))$. The Eq. (3) and
Eq. (5) can be utilized to construct the whole quantum simulation environment.
The setting of the reward in QSC-ERL is according to the Eq. (7). Here is the
parameter settings shown as table 1.

Table 1: The parameter settings of the QSC-ERL Name | Value
---|---
maximum episode | 500
learning rate | 0.01
reward decay | 0.9
e greedy | 0.99
memory size | 2000

###  4.2 Evaluation index

Fidelity is a evaluation index to measure the distance between density
operators. It allows us to compare how the state of the system at any given
moment is different from the initial state, or how the state of a system is
different from a reference state. It allows us to measure quantitatively how
different two states really are. For two density matrices $\rho$, $\sigma$ it
is generalized as the largest fidelity between any two purifications of the
given states. And the fidelity function can be defined as

$F(\rho,\sigma)=(tr\sqrt{\sqrt{\rho}\sigma\sqrt{\rho}})^{2},$ (15)

where $\rho$ and $\sigma$ are the density matrix of source information and
target information respectively.

###  4.3 Results and analysis

The simulation experiments is carried out under the three switch control
paradigm. The goal of the experiment is to control the spin 1/2 system from
the initial state $|\psi_{initial}\rangle$ to the target state
$|\psi_{target}\rangle$. The main purpose is to explore the effectiveness of
reinforcement learning algorithm for solving quantum control problem.

Therefore, the simulation experiments is divided for two parts: one is that
the tabular Q-learning (TQL) (Sutton and Barto 2018), deep Q-learning (DQL)
(Mnih et al. 2015) and policy gradient (PG) (Sutton et al. 2000) are applied
to explore the effectiveness of reinforcement learning algorithm for solving
quantum control problem. The other is that the NN-QSC (Fosel et al. 2018) and
the DRL-QSC (An and Zhou 2019) are compared for verifying that the proposed
QSC-ERL performed better than its peers. The parameters of the reinforcement
learning algorithms involved in the experiment are set as follows: For all
state-action paired, the Q value is initialized to 0, the discount factor is
$\gamma=0.99$, the learning rate is $\alpha=0.1$, and the action selection
probability is initialized to $1/3$.

Figure 3: The comparison of fidelity between algorithms. (a)Fidelity of the
TQL algorithm. (b) Fidelity of the PG algorithm. (c) Fidelity of the DQL
algorithm. (d) Fidelity of the NN-QSC algorithm. (e) Fidelity of the DRL-QSC
algorithm. (f) Fidelity of the QSC-ERL algorithm.

Fig. 3 shows the comparison of fidelity between algorithms, where the X-axis
is the number of episode, and the Y-axis is fidelity. It can be seen from Fig.
3 that reinforcement learning has certain effects on solving the quantum
system control problems. Since the TQL algorithm can not converge rapidly
during training, the fidelity is the lowest. The PG algorithm has better
convergence, but only a little bit at one episode. The other four methods have
good results. The DQL algorithm adopted convolutional neural network to guide
the learning of the Q learning algorithm. Although it solves the problem of
not being able to update Q table well when there are many actions, it is
difficult for simple neural networks to learn the useful features of quantum
systems. If want to get a high fidelity between the final state and the target
state, it needs to train with more episodes and data set. Due to the high
correlation between states in the training process, the NN-QSC and the DRL-QSC
may fall into local optimum or be difficult to converge. Our QSC-ERL use the
enhanced neural network to effectively make use of the differential features
before and after the evolution of the quantum state. By introducing the
eligibility trace to update parameters, The QSC-ERL can quickly find the
optimal control strategy of the quantum system. Table 2 shows the number of
episodes when the fidelity can get the maximum, and total number of episodes
is set to 500. The data is taken from the average value of 100 experiments. It
represents that the ability of algorithms can make the quantum system from the
initial state to the desired target state.

Table 2: The comparison of the number of episodes between algorithms Name | Episodes | Fidelity
---|---|---
TQL | 452 | 0.73
PG | 311 | 0.99
DQL | 135 | 0.99
NN-QSC | 171 | 0.99
DRL-QSC | 60 | 0.99
QSC-ERL | 42 | 0.99

The experimental results show that most methods converge after training and
make the quantum system from the initial state to the desired target state.
Specifically, for the TQL, the maximum of the fidelity is 0.73, and others can
reach 0.99 after total training. The PG requires about 311 episodes and the
DQL requires about 135 episodes. It means that the reinforcement learning
based on neural network has the better performance in some degree than the
common RL algorithm. The NN-QSC requires about 171 episodes to control the
evolution of the quantum system from the initial state to the target state
while the DRL-QSC requires about 60 episodes, and the QSC-ERL requires the 42
episodes. So our proposed QSC-ERL algorithm is faster than the NN-QSC and the
DRL-QSC for controlling the evolution of the quantum system from the initial
state to the target state.

##  5 Conclusion

In this paper, a quantum system control method based on enhanced reinforcement
learning (QSC-ERL) is proposed to achieve the learning control of the spin 1/2
system. A satisfactory control strategy is obtained through enhanced
reinforcement learning so that the quantum system can be evolved accurately
from the initial state to the target state. Compared with other methods, our
method can achieve the quantum system control with high fidelity, and improve
the control efficiency of quantum systems.

It should be noted that our method is sufficient for the evolution of quantum
state in spin 1/2 system. Other difficult quantum control problems include
quantum error correction based on bosonic codes (Michael et al. 2016) and
quantum state preparation in the single-photon manifold (Vrajitoarea et al.
2020). And it is a valuable work to conduct a study on providing solutions by
using learning theories (Li et al. 2018; Zhang and Wang 2020) and neural
network (Xu et al. 2019; Hu et al. 2020), which is also one of our next
research.

###### Acknowledgements.

The authors would like to thank the anonymous reviewers and editors for their
comments that improved the quality of this paper. This work is supported by
the National Natural Science Foundation of China (62071240, 61802175), the
Natural Science Foundation of Jiangsu Province (BK20171458), and the Priority
Academic Program Development of Jiangsu Higher Education Institutions (PAPD).

## Declarations

Conflict of interest The authors declare that they have no conflict of
interest.
Ethical statement Articles do not rely on clinical trials.
Human and animal participants All submitted m-
anuscripts containing research which does not involve human participants
and/or animal experimentation.

## References

  * Laith Abualigah and Ali Diabat and Seyedali Mirjalili and Mohamed Abd Elaziz and Amir H. Gandomi. [2021] Abualigah L, Diabat A, Mirjalili S, Abd Elaziz M, Gandomi AH (2021a) The arithmetic optimization algorithm. Computer Methods in Applied Mechanics and Engineering 376:113609,DOI https://doi.org/10.1016/j.cma.2020.113609
  * Abualigah, Laith and Diabat, Ali and Sumari, Putra and Gandomi, Amir. [2021] Abualigah L, Diabat A, Sumari P, Gandomi A (2021b) Applications, deployments, and integration of internet of drones (iod): A review. IEEE Sensors Journal PP:1–1, DOI 10.1109/JSEN.2021.3114266
  * Abualigah, Laith and Elsayed Abd Elaziz, Mohamed and Sumari, Putra and Geem, Zong Woo and Gandomi, Amir. [2021] Abualigah L, Elsayed Abd Elaziz M, Sumari P, GeemZW, Gandomi A (2021c) Reptile search algorithm (rsa): A nature-inspired meta-heuristic optimizer. Expert Systems with Applications 191:116158, DOI 10.1016/j.eswa.2021.116158
  * Laith Abualigah and Dalia Yousri and Mohamed Abd Elaziz and Ahmed A. Ewees and Mohammed A.A. Al-qaness and Amir H. Gandomi. [2021] Abualigah L, Yousri D, Abd Elaziz M, Ewees AA, Alqaness MA, Gandomi AH (2021d) Aquila optimizer: A novel meta-heuristic optimization algorithm. Computers & Industrial Engineering 157:107250, DOI https://doi.org/10.1016/j.cie.2021.107250
  * An, Zheng and Zhou, DL. [2019] An Z, Zhou D (2019) Deep reinforcement learning for quantum gate control. EPL (Europhysics Letters) 126(6):60002
  * An, Zheng and Song, Hai-Jing and He, Qi-Kai and Zhou, DL. [2021] An Z, Song HJ, He QK, Zhou D (2021) Quantum optimal control of multilevel dissipative quantum systems with reinforcement learning. Physical Review A 103(1):012404
  * Bukov, Marin. [2018] Bukov M (2018) Reinforcement learning for autonomous preparation of floquet-engineered states: Inverting the quantum kapitza oscillator. Physical Review B 98(22):224305
  * Bukov, Marin and Day, Alexandre GR and Sels, Dries and Weinberg, Phillip and Polkovnikov, Anatoli and Mehta, Pankaj. [2018] Bukov M, Day AG, Sels D, Weinberg P, Polkovnikov A, Mehta P (2018) Reinforcement learning in different phases of quantum control. Physical Review X 8(3):031086
  * Cárdenas-López FA, Lamata L, Retamal JC, Solano E. [2018] Cárdenas-López FA, Lamata L, Retamal JC, Solano E (2018) Multiqubit and multilevel quantum reinforcement learning with quantum technologies. PloS one 13(7):e0200455
  * Chakrabarti R, Rabitz H . [2007] Chakrabarti R, Rabitz H (2007) Quantum control landscapes. International Reviews in Physical Chemistry 26(4):671–735
  * Chen, Chunlin and Dong, Daoyi and Li, Han-Xiong and Chu, Jian and Tarn, TzyhJong. [2013] Chen C, Dong D, Li HX, Chu J, Tarn TJ (2013) Fidelity-based probabilistic q-learning for control of quantum systems. IEEE transactions on neural networks and learning systems 25(5):920–933
  * Chunlin C, Frank J, Daoyi D. [2012] Chunlin C, Frank J, Daoyi D (2012) Hybrid control of uncertain quantum systems via fuzzy estimation and quantum reinforcement learning. In: Proceedings of the 31st Chinese Control Conference, IEEE, pp 7177-7182
  * D’Alessandro D, Dahleh M. [2001] D’Alessandro D, Dahleh M (2001) Optimal control of two-level quantum systems. IEEE Transactions on Automatic Control 46(6):866–876
  * Fang W, Pang L, Yi W . [2020] Fang W, Pang L, Yi W (2020) Survey on the application of deep reinforcement learning in image processing. Journal on Artificial Intelligence 2(1):39–58
  * Fösel T, Tighineanu P, Weiss T, Marquardt F . [2018] Fösel T, Tighineanu P, Weiss T, Marquardt F (2018) Reinforcement learning with neural networks for quantum feedback. Physical Review X 8(3):031084
  * Gu J, Wang Z, Kuen J, Ma L, Shahroudy A, Shuai B, Liu T, Wang X, Wang G, Cai J, et al. [2018] Gu J, Wang Z, Kuen J, Ma L, Shahroudy A, Shuai B, Liu T, Wang X, Wang G, Cai J, et al. (2018) Recent advances in convolutional neural networks. Pattern Recognition 77:354–377
  * He K, Zhang X, Ren S, Sun J. [2016] He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
  * Hu B, Zhao H, Yang Y, Zhou B, Raj ANJ . [2020] Hu B, Zhao H, Yang Y, Zhou B, Raj ANJ (2020) Multiple faces tracking using feature fusion and neural network in video. INTELLIGENT AUTOMATION AND SOFT COMPUTING 26(6):1549–1560
  * Li Z, Zhang J, Zhang K, Li Z . [2018] Li Z, Zhang J, Zhang K, Li Z (2018) Visual tracking with weighted adaptive local sparse appearance model via spatio-temporal context learning. IEEE Transactions on Image Processing 27(9):4478–4489
  * Ma H, Chen C . [2020] Ma H, Chen C (2020) Several developments in learning control of quantum systems. In: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE, pp 4165–4172
  * Meng F, Cong S . [2022] Meng F, Cong S (2022) Control design for state transition of open quantum system. In: Journal of Physics: Conference Series, IOP Publishing, vol 2183, p 012005
  * Michael MH, Silveri M, Brierley R, Albert VV, Salmilehto J, Jiang L, Girvin SM . [2016] Michael MH, Silveri M, Brierley R, Albert VV, Salmilehto J, Jiang L, Girvin SM (2016) New class of quantum error-correcting codes for a bosonic mode. Physical Review X 6(3):031006
  * Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, et al. [2015] Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, et al. (2015) Humanlevel control through deep reinforcement learning. nature 518(7540):529–533
  * Niu MY, Boixo S, Smelyanskiy VN, Neven H . [2019] Niu MY, Boixo S, Smelyanskiy VN, Neven H (2019) Universal quantum control through deep reinforcement learning. npj Quantum Information 5(1):1–8
  * Palittapongarnpim P, Wittek P, Sanders BC . [2017] Palittapongarnpim P, Wittek P, Sanders BC (2017) Robustness of learning-assisted adaptive quantumenhanced metrology in the presence of noise. In: 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE, pp 294–299
  * Patsch S, Maniscalco S, Koch CP . [2020] Patsch S, Maniscalco S, Koch CP (2020) Simulation of open-quantum-system dynamics using the quantum zeno effect. Physical Review Research 2(2):023133
  * Roslund J, Rabitz H . [2009] Roslund J, Rabitz H (2009) Gradient algorithm applied to laboratory quantum control. Physical Review A 79(5):053417
  * Singh SP, Sutton RS . [1996] Singh SP, Sutton RS (1996) Reinforcement learning with replacing eligibility traces. Machine learning 22(1):123–158
  * Sutton RS, Barto AG . [2018] Sutton RS, Barto AG (2018) Reinforcement learning: An introduction. MIT press
  * Sutton RS, McAllester DA, Singh SP, Mansour Y. [2000] Sutton RS, McAllester DA, Singh SP, Mansour Y (2000) Policy gradient methods for reinforcement learning with function approximation. In: Advances in neural information processing systems, pp 1057– 1063
  * Torosov BT, Shore BW, Vitanov NV . [2021] Torosov BT, Shore BW, Vitanov NV (2021) Coherent control techniques for two-state quantum systems: A comparative study. Physical Review A 103(3):033110
  * Tsubouchi M, Momose T . [2008] Tsubouchi M, Momose T (2008) Rovibrational wavepacket manipulation using shaped midinfrared femtosecond pulses toward quantum computation: Optimization of pulse shape by a genetic algorithm. Physical Review A 77(5):052326
  * Vedaie SS, Palittapongarnpim P, Sanders BC . [2018] Vedaie SS, Palittapongarnpim P, Sanders BC (2018) Reinforcement learning for quantum metrology via quantum control. In: 2018 IEEE Photonics Society Summer Topical Meeting Series (SUM), IEEE, pp 163–164
  * Vrajitoarea A, Huang Z, Groszkowski P, Koch J, Houck AA . [2020] Vrajitoarea A, Huang Z, Groszkowski P, Koch J, Houck AA (2020) Quantum control of an oscillator using a stimulated josephson nonlinearity. Nature Physics 16(2):211–217
  * Watkins CJ, Dayan P . [1992] Watkins CJ, Dayan P (1992) Q-learning. Machine learning 8(3-4):279–292
  * Xu F, Zhang X, Xin Z, Yang A . [2019] Xu F, Zhang X, Xin Z, Yang A (2019) Investigation on the chinese text sentiment analysis based on convolutional neural networks in deep learning. Comput Mater Contin 58(3):697–709
  * Yu S, Albarrán-Arriagada F, Retamal JC, Wang YT, Liu W, Ke ZJ, Meng Y, Li ZP, Tang JS, Solano E, et al. [2019] Yu S, Albarrán-Arriagada F, Retamal JC, Wang YT, Liu W, Ke ZJ, Meng Y, Li ZP, Tang JS, Solano E, et al. (2019) Reconstruction of a photonic qubit state with reinforcement learning. Advanced Quantum Technologies 2(7-8):1800074
  * Zhang XM, Wei Z, Asad R, Yang XC, Wang X . [2019] Zhang XM, Wei Z, Asad R, Yang XC, Wang X (2019) When does reinforcement learning stand out in quantum control? a comparative study on state preparation. npj Quantum Information 5(1):1–7
  * Zhang Y, Wang Z . [2020] Zhang Y, Wang Z (2020) Hybrid malware detection approach with feedback-directed machine learning. Information Sciences 63(139103):1–139103


# Don’t be a Fool: Pooling Strategies in Offensive Language Detection
from User-Intended Adversarial Attacks

Seunguk Yu, Juhwan Choi  Youngbin Kim
Chung-Ang University, Republic of Korea, Seoul
<EMAIL_ADDRESS>{gold5230<EMAIL_ADDRESS>

###### Abstract

Warning: this paper contains expressions that may offend the readers.

Offensive language detection is an important task for filtering out abusive
expressions and improving online user experiences. However, malicious users
often attempt to avoid filtering systems through the involvement of textual
noises. In this paper, we propose these evasions as user-intended adversarial
attacks that insert special symbols or leverage the distinctive features of
the Korean language. Furthermore, we introduce simple yet effective pooling
strategies in a layer-wise manner to defend against the proposed attacks,
focusing on the preceding layers not just the last layer to capture both
offensiveness and token embeddings. We demonstrate that these pooling
strategies are more robust to performance degradation even when the attack
rate is increased, without directly training of such patterns. Notably, we
found that models pre-trained on clean texts could achieve a comparable
performance in detecting attacked offensive language, to models pre-trained on
noisy texts by employing these pooling strategies.

##  1 Introduction

As the internet becomes an important part of our lives, the prevalence of
offensive language on online platforms, particularly social media, has become
a serious concern Zampieri et al. (2019). Deep learning models for filtering
offensive languages have been proposed to address this problem. However,
malicious users have consistently found ways to avoid them. One such way is
the deliberate insertion of additional typographical errors or substitution of
certain characters with visually similar alternatives Kurita et al. (2019); Wu
et al. (2018).

Despite numerous studies on this phenomenon in English, there has been a
comparatively limited exploration in Korean, which is a low-resource language
characterized by distinct linguistic features Sahoo et al. (2023); Kim et al.
(2021a). As the Korean communities also suffer from the use of abusive
language and cyberbullying McCurry (2022); Saengprang and Gadavanij (2021); Yi
and Cha (2020); Jun (2020), it is desirable to investigate the evasion tactics
utilized by malicious users and to formulate them. While recent studies have
discussed how to avoid offensive language detection Cho and Kim (2021); Kim et
al. (2021c); Ahn and Egorova (2021), their definitions are ambiguous, and no
clear solutions have been proposed to defend against the evasions.

In this paper, we propose the evasion methods as user-intended adversarial
attacks and incorporate them into offensive language from the perspective of
malicious users. Our proposed attacks are grounded in prevalent forms that can
be found in offensive language online, and reflect the distinct features of
Korean language, wherein a single character can be further subdivided Song
(2006). We test the proposed attacks on existing models for offensive language
detection, and the results reveal that the performance declines as the rate of
the proposed attacks increases.

Furthermore, we introduce simple yet effective pooling strategies in a layer-
wise manner to defend against the proposed attacks. Motivated by the
exploration of the impact of each layer in a pre-trained language model Oh et
al. (2022); Jawahar et al. (2019), we selectively integrate useful features
for the attacked offensive language across all layers111In our paper, we
denote offensive language as ‘attacked’ when an evasion method is applied to
it, viewed from the perspective of malicious users.. The attacked texts have
some changes in the tokens used, differing from the original texts. Therefore,
we implement pooling strategies to ensure that the model captures not only
high-level features but also low-level features, which are related to
offensiveness and token embeddings, respectively. This simple modification
enriches the understanding of the attacked offensive language, enhancing the
robustness of the model against user-intended adversarial attacks without
directly training of such patterns.

The contributions of our study are as follows:

  * •

We propose user-intended adversarial attacks that are often associated with
offensive language online from the perspective of malicious users. These
attacks are performed by inserting special symbols or leveraging the
distinctive features of Korean.

  * •

We introduce pooling strategies in a layer-wise manner to selectively utilize
all layers rather just than the last layer. This approach achieves a notable
performance when employed to a model pre-trained on clean texts, without
directly training of the attacks.

  * •

We demonstrate the effectiveness of layer-wise pooling strategies by assigning
distinct weights to each layer and employing them to the model depending on
the nature of the pre-trained texts. We especially note the efficacy of first-
last pooling and max pooling when the attacks are involved in offensive
language.

##  2 Related Work

###  2.1 Adversarial Attacks

Adversarial attacks involve perturbed input data that confuses the trained
model, whereas a situation in which the model consistently makes predictions
regardless of the nature of input data is referred to as defending Goyal et
al. (2023). Previous studies have explored this based on word-level
substitutions Jin et al. (2020); Ren et al. (2019), and others also have
explored them on character-level. We propose adversarial attacks that utilize
not only character-level but also smaller-scale alternations tailored to the
features of Korean language. Such attacks are commonly observed in the context
of offensive languages in various online communities Cho and Kim (2021); Ahn
and Egorova (2021).

TextBugger Li et al. (2019) was an early study that focused on character-level
alternations, such as replacing characters with visually similar ones (e.g.
replacing the alphabet ‘o’ with the number ‘0’). Other studies have suggested
simple leetspeaks that utilize symbols that resemble the alphabet Aggarwal and
Zesch (2022), or adversarial attacks that are not easily detected visually,
such as transforming Latin characters into similar-looking Cyrillic characters
Wolff and Wolff (2020). Although several studies also have explored visually
undetectable attacks Bajaj and Vishwakarma (2023); Boucher et al. (2022); Kim
et al. (2021b), we consider a more realistic attack scenario that can occur
online from the perspective of malicious users.

###  2.2 Korean Offensive Language Datasets

Owing to the increasing demand for online content in Korean language and the
growing threat of cyberbullying, previous studies have introduced offensive
language datasets collecting comments from diverse resources such as online
news, communities, and YouTube.

BEEP! Moon et al. (2020) was a pioneering study that utilizes hate speech
prevalent in news comments. KoLD Jeong et al. (2022) and K-MHaS Lee et al.
(2022) specified the target group of the offensive language. Subsequently,
KODOLI Park et al. (2023b) provided labels that refine the degree of
offensiveness, and built upon these efforts, K-HATERS Park et al. (2023a) was
built to incorporate the strengths of the preceding datasets.

Although numerous datasets have been proposed, there is still a lack of
definition for adversarial attacks that are frequently involved in offensive
language, and how to defend against them. In this study, we focus on
introducing pooling strategies for defending against these attacks, without
directly training attacked offensive langauge.

##  3 Method

###  3.1 User-Intended Adversarial Attacks

We present adversarial attacks designed to target offensive languages from the
perspective of malicious users. By referring to existing offensive language
datasets, we define frequently occurring attack types. These attacks are
categorized into three groups: Insert, Copy, and Decompose. Examples of each
attack type are listed in Table 1.

First, Insert involves adding incomplete Korean character forms, which are
often used online without significant meaning. For example, ‘ㅋㅋ’ (equivalent
‘lol’ or ‘lmao’ in English) is a commonly used and somewhat meaningless string
frequently used in online communications. In this case, Insert_zz is performed
by inserting the string at a specific location within the word, as in real
situations. Other types of Insert also include unnecessary spaces or special
symbols.

| User-Intended
---
Adversarial Attacks
Text Examples
original text |  쓰레기 같은 (piece of shit)
Insert_zz | 씈ㅋㅋㅋ레기 같은
Insert_space | 쓰 레기 같은
Insert_special | 쓰@레기 같은
Copy_initial | 쓸레기 같은
Copy_middle | 쓰레에기 같은
Copy_final | 쓰레기 가튼
Decompose_final | 쓰레기 가ㅌ은
Decompose_all | ㅆㅡ레기 같은
Table 1: Text examples of user-intended adversarial attacks with three categories: Insert, Copy, and Decompose. There are various attacks that involve special symbols or exploit the distinctive features of Korean. | User-Intended
---
Adversarial Attacks
Tokenized Examples
original text |  쓰레기, 같, ##은 (piece of shit)
Insert_zz | [UNK], 같, ##은
Insert_space | 쓰, 레, ##기, 같, ##은
Insert_special | 쓰, @, 레, ##기, 같, ##은
Copy_initial | 쓸, ##레기, 같, ##은
Copy_middle | 쓰, ##레, ##에, ##기, 같, ##은
Copy_final | 쓰레기, 가, ##튼
Decompose_final | 쓰레기, 가, ##ㅌ, ##은
Decompose_all | [UNK], 같, ##은
Table 2: Tokenized examples of user-intended adversarial attacks. Although the
texts have the same meaning, the tokens are represented differently.

The following two types of user-intended adversarial attacks take advantage of
the distinct features of the Korean language; a single character must have an
initial sound, a middle sound, and an optional final sound Song (2006). For
example, in the expression ‘쓰레기 같은’ in Table 1, the character ‘쓰’ has only the
initial and middle sounds, whereas the character ‘같’ has all three sounds.

Second, Copy utilizes the distinctive features, copying one of the three
sounds from the selected character to the other character. For example, the
character ‘레’ from the expression ‘쓰레기 같은’ has the initial and middle sounds
of ‘ㄹ’ and ‘ㅔ’. In this case, Copy_initial is performed by copying the initial
sound of that character ‘ㄹ’ to the final sound of the preceding character ‘쓰’.
Consequently, ‘쓰’ is transformed into ‘쓸’, leading to the attacked expression
‘쓸레기 같은’.

Finally, Decompose also utilizes the unique characteristics, isolating the
final sound of the selected character or breaking down the character itself.
For example, the single character ‘쓰’ from the expression ‘쓰레기 같은’ has the
initial and middle sounds of ‘ㅆ’ and ‘ㅡ’. In this case, Decompose_all is
performed by breaking down the character, resulting in the sounds being
independent of that character. Consequently, ‘쓰’ is transformed into ‘ㅆㅡ’,
leading to the attacked expression ‘ㅆㅡ레기 같은’. Further details and examples of
all user-intended adversarial attacks are provided in Appendix A.

###  3.2 Layer-Wise Pooling Strategies

In standard text classification tasks, pre-trained models such as BERT are
fined-tuned to the target domain. This is based on the assumption that the
$\left[CLS\right]$ token from the last layer effectively captures the sentence
representation Devlin et al. (2019). However, we notice inconsistencies in the
predictions of existing models regarding the proposed attacks222The results
for this experiment are included in Table 3.. Consequently, we conclude that
this information alone is insufficient for detecting the attacked offensive
language.

When using perturbed text to a trained model, the tokenization results differ
from those of the original text, as shown in Table 2. By involving special
symbols or exploiting the distinctive features of Korean, we observe that even
if the text had the same meaning to human readers, the tokenized outputs
differ significantly333The model used for tokenization in here is
$\text{BERT}_{\textit{clean}}$.. Therefore, we do not rely on the information
only from the last layer but utilize the preceding layers, which focus more on
token embeddings Ma et al. (2019). This also reflects the previous finding
that meaningful information for a certain task can be captured in the
preceding layers Oh et al. (2022).

We extend pooling strategies in a layer-wise manner, allowing us the
flexibility to utilize text representations from all layers. Denoting the
$\left[CLS\right]$ token of the $N$th layer as $h_{N}^{\textit{cls}}$, we
introduce four pooling strategies that optionally consider the
$\left[CLS\right]$ tokens from all the layers
$h_{1}^{\textit{cls}},...,h_{N}^{\textit{cls}}$.

Mean, Max Pooling: We apply mean pooling utilizing the $L^{1}$ norm, which
averages all $\left[CLS\right]$ tokens from all the layers, and max pooling
utilizing the $L^{\infty}$ norm, which takes a max-over-time operation on the
values corresponding to each dimension from all $\left[CLS\right]$ tokens.

When the dimension of $\left[CLS\right]$ token is $M$, and all the values of
$m$th dimension of $\left[CLS\right]$ tokens from all the layers are
concatenated and denoted by $h_{\textit{all}}^{m}$, these two poolings are
defined as follows:

$\displaystyle pool_{\textit{mean}}$
$\displaystyle=mean(h_{1}^{\textit{cls}},...,h_{N}^{\textit{cls}}),$ (1)
$\displaystyle pool_{\textit{max}}$
$\displaystyle=max(h_{\textit{all}}^{1}),...,max(h_{\textit{all}}^{M}),$ (2)

Weighted Pooling: We apply weighted pooling utilizing a learnable parameter
that determines the importance of each layer. Through adaptive incorporation
of the layers, we train weights that selectively capture both offensiveness
and token embeddings, initializing all weights to zero.

When the $w_{i}$ represents the weight of each layer and $\alpha_{i}$
represents its softmax distribution, the weighted pooling is defined as
follows:

$\displaystyle pool_{\textit{weighted}}$
$\displaystyle=\sum_{i=1}^{N}{\alpha_{i}h_{i}^{\textit{cls}}},$ (3)

First-last Pooling: We apply first-last pooling utilizing $\left[CLS\right]$
token from the first layer. Rather than considering all the layers, we focus
on leveraging information from layers directly associated with offensiveness
and token embeddings.

$\displaystyle pool_{\textit{first\text{-}last}}$
$\displaystyle=h_{N}^{\textit{cls}}+h_{1}^{\textit{cls}},$ (4)

Figure 1: Layer-wise pooling strategies that selectively use
$\left[CLS\right]$ tokens from all layers. From the upper left, there are
mean, max, weighted, and first-last pooling.

The layer-wise pooling strategies described above are illustrated in Figure 1.
We conducted experiments to verify the robustness of these strategies for
detecting offensive languages that reflect user-intended adversarial attacks.

##  4 Experiment

Model Original 30% Attacked 60% Attacked 90% Attacked P R F1 P R F1 $\Delta
atk$ P R F1 $\Delta atk$ P R F1 $\Delta atk$ BiLSTM 71.83 68.80 69.81 70.84
66.23 67.38 -3.48% 68.97 62.82 63.67 -8.79% 69.32 61.64 62.25 -10.82% BiGRU
71.32 65.71 66.91 70.40 63.32 64.26 -3.96% 68.84 60.31 60.52 -9.55% 68.05
58.83 58.48 -12.59% $\text{BERT}_{\textit{clean}}$ 79.81 77.79 78.64 79.51
73.35 75.19 -4.38% 77.74 66.38 67.96 -13.58% 76.14 62.01 62.44 -20.60%
$\text{BERT}_{\textit{multi}}$ 76.57 70.75 72.38 76.44 66.33 67.87 -6.23%
76.30 60.90 60.87 -15.90% 76.07 57.61 55.78 -22.93%
$\text{BERT}_{\textit{clean}}+mean$ 78.57 79.06 79.01 79.41 73.97 75.70 -4.18%
77.15 66.97 68.62 -13.15% 74.90 62.45 63.09 -20.14%
$\text{BERT}_{\textit{clean}}+max$ 78.51 78.81 78.65 78.47 74.03 75.54 -3.95%
77.80 66.66 68.29 -13.17% 76.79 61.51 61.72 -21.52%
$\text{BERT}_{\textit{clean}}+weighted$ 79.93 78.50 79.14 79.19 73.70 75.43
-4.68% 77.14 67.62 69.33 -12.39% 75.44 63.66 64.85 -18.05%
$\text{BERT}_{\textit{clean}}+first\text{-}last$ 79.05 79.37 79.21 78.89 75.85
77.02 -2.76% 77.58 69.38 71.21 -10.09% 76.08 64.33 65.49 -17.32%
$\text{BERT}_{\textit{noise}}$ 80.64 78.88 79.64 80.67 75.42 77.17 -3.10%
78.44 69.42 71.33 -10.43% 76.46 65.55 66.96 -15.92%
$\text{Ensemble}_{\textit{hard}}$ ($\text{BERT}_{\textit{clean}}$ \+
$\text{BERT}_{\textit{noise}}$) 81.63 79.42 80.36 81.57 75.03 77.04 -4.13%
80.47 68.60 70.60 -12.14% 78.68 64.24 65.35 -18.67%
$\text{Ensemble}_{\textit{soft}}$ ($\text{BERT}_{\textit{clean}}$ \+
$\text{BERT}_{\textit{noise}}$) 81.52 79.53 80.38 81.54 75.29 77.25 -3.89%
80.27 68.86 70.87 -11.83% 78.47 64.42 65.58 -18.41% DeBERTaV3 82.55 78.70
80.17 81.85 74.33 76.48 -4.60% 80.14 68.47 70.44 -12.13% 78.41 63.96 64.99
-18.93%

Table 3: Experimental results of offensive language detection when a certain
ratio of user-intended adversarial attacks are involved. P, R, and F1
represent macro precision, recall, and f1-score, respectively. $\Delta atk$
represents the performance drop in the f1-score as the attacks are involved in
the same model.

###  4.1 Datasets

We collected both the KoLD Jeong et al. (2022) and K-HATERS Park et al.
(2023a) datasets and divided them into train, validation, and test sets by
stratifying their labels. We randomly shuffled and split them in the ratio of
8:1:1. We set the attack rates to 30%, 60%, and 90%, corrupting a portion of
the words in a sentence. The details of how the attacks were carried out are
in Appendix A.

###  4.2 Baselines

We validate the effectiveness of the layer-wise pooling strategies with the
baselines, which are presented below. The experimental details including
hyperparameters and metrics are reported in Appendix B.

  * •

BiLSTM: This model addresses the long-term dependency problem by remembering
only the information in need Schuster and Paliwal (1997). It was built by
stacking two LSTMs, and the forward and backward $\left[CLS\right]$ tokens
from the last layer were combined and passed through the classification layer.

  * •

BiGRU: This model is derived from the BiLSTM and further evolved by reducing
the training parameters through the selective utilization of gates Cho et al.
(2014). Its configurations are the same as BiLSTM.

  * •

$\text{BERT}_{\textit{clean}}$: This model follows the BERT Devlin et al.
(2019) structure, which is built on a self-attention mechanism with masked
language modeling. It was pre-trained on preprocessed texts in Korean Park et
al. (2021). The $\left[CLS\right]$ token from the last layer is passed through
the classification layer.

  * •

$\text{BERT}_{\textit{multi}}$: This model is pre-trained on multilingual
data, including Korean Conneau et al. (2020). The model structure is based on
RoBERTa Liu et al. (2019), while the remaining configurations are the same as
$\text{BERT}_{\textit{clean}}$.

  * •

$\text{BERT}_{\textit{noise}}$: This model is pre-trained on noisy texts in
Korean, such as online comments Lee (2020). Its configurations are the same as
$\text{BERT}_{\textit{clean}}$.

  * •

$\text{Ensemble}_{\textit{hard}}$, $\text{Ensemble}_{\textit{soft}}$: These
models utilize both the $\text{BERT}_{\textit{clean}}$ and
$\text{BERT}_{\textit{noise}}$, employing voting methods from ensemble
techniques. Hard voting is conducted through a majority vote, but soft voting
occurs in the cases of tied votes. Soft voting averages the prediction
probabilities of each model.

  * •

DeBERTaV3: This model employs gradient-disentangled embeddings to enhance the
efficiency of pre-trianing when incorporating replaced token detection along
with DeBERTa He et al. (2023, 2021). We used the model fine-tuned in
Korean444https://huggingface.co/team-lucid/deberta-v3-base-korean.

##  5 Discussion

###  5.1 Experimental Results

The performances of the models when exposed to user-intended adversarial
attacks are presented in Table 3. Each of the best performances from the
layer-wise pooling strategies and the baselines in the original, and 30%, 60%,
and 90% attacked are highlighted in bold.

All baselines, including ensemble models and even a recent model like
DeBERTaV3, were susceptible to the proposed attacks. We observed that the
BERT-based models consistently outperformed RNN-based models in terms of the
f1-score across all attack rates. Under original and 30% attacked, employing
ensemble models with soft voting yielded the best scores, achieving f1-scores
of 80.38 and 77.25. As the attack rates increased to 60% and 90%, using a
single model pre-trained on noisy texts proved to be the most effective,
achieving f1 scores of 71.33 and 66.96, respectively.

However, ensemble models require twice computational resources for both
training and inference stages compared to a single model. In the case of
$\text{BERT}_{\textit{noise}}$, a large amount of noisy texts is required,
raising concerns regarding its adaptability when inference with attacked input
types is not encountered during the pre-training stage.

Model Average F1 $\Delta atk$ BiLSTM 64.43 -7.69% BiGRU 61.08 -8.70%
$\text{BERT}_{\textit{clean}}$ 68.52 -12.85% $\text{BERT}_{\textit{multi}}$
61.50 -15.02% $\text{BERT}_{\textit{clean}}+mean$ 69.13 -12.49%
$\text{BERT}_{\textit{clean}}+max$ 68.51 -12.88%
$\text{BERT}_{\textit{clean}}+weighted$ 69.86 -11.70%
$\text{BERT}_{\textit{clean}}+first\text{-}last$ 71.24 -10.05%
$\text{BERT}_{\textit{noise}}$ 71.82 -9.81% $\text{Ensemble}_{\textit{hard}}$
($\text{BERT}_{\textit{clean}}$ \+ $\text{BERT}_{\textit{noise}}$) 70.99
-11.64% $\text{Ensemble}_{\textit{soft}}$ ($\text{BERT}_{\textit{clean}}$ \+
$\text{BERT}_{\textit{noise}}$) 71.23 -11.37% DeBERTaV3 70.63 -11.88%

Table 4: Average from the experimental results of offensive language detection
when a certain ratio of user-intended adversarial attacks are involved.

When applying layer-wise pooling strategies to $\text{BERT}_{\textit{clean}}$,
we found that the performances were improved in almost all attack rates. They
only need to train an additional parameter equal to the size of all layers
(e.g. 12 for BERT-based models), or no parameters are required. Furthermore,
they are robust as the attack rate increases compared to models with no
pooling strategies.

The average performances when exposed to proposed attacks across all attack
rates are presented in Table 4. All layer-wise pooling strategies exhibited
robustness against attacks compared to their absence, except for
$\text{BERT}_{\textit{clean}}+max$, which exhibited a slight performance drop.
Moreover, even $\text{BERT}_{\textit{clean}}+first\text{-}last$, which only
incorporates information from the first layer without any parameters or
training noisy texts, showed comparable performance to
$\text{BERT}_{\textit{noise}}$ across all attack rates that were pre-trained
on noisy texts.

###  5.2 Focus on Performance Drop

The degree to which the f1-scores of the models decreased with the attack
rates is shown in Figure 2. We found that BiSLTM exhibited relatively modest
performance degradation across the models, which is depicted by the light
green triangles. Despite the modest decrease, the offensive language detection
scores of RNN-based models were not as good as that of the BERT-based models
because of the limitations of themselves.

Figure 2: Degree to which the f1-scores of the models decrease with the attack
rates 30%, 60%, and 90%. We selected several baseline models and
$\text{BERT}_{\textit{clean}}+first\text{-}last$ for the comparison.

Among the BERT-based models, $\text{BERT}_{\textit{clean}}$, which was pre-
trained on clean texts and is depicted by the light blue triangles, exhibited
the largest performance degradation. However,
$\text{BERT}_{\textit{clean}}+first\text{-}last$, which applied a simple
layer-wise pooling strategy to the model and is depicted by the orange
circles, successfully mitigated performance degradation by 1.62%, 3.49%, and
3.28% at each attack rate, respectively, achieving an average performance
degradation mitigation of 2.79%.

These results are similar or even better to those of
$\text{BERT}_{\textit{noise}}$ or $\text{Ensemble}_{\textit{soft}}$, which
used noisy texts in a pre-training stage and are depicted by the blue and deep
blue triangles, respectively. Therefore, we found that the model pre-trained
on clean texts with a simple pooling strategy can achieve a certain level of
performance, or even be more robust compared to the model pre-trained on noisy
texts in defending against user-intended adversarial attacks. It was even more
robust than the recent model DeBERTaV3, which is depicted by the purple
triangles.

###  5.3 Focus on Layer Weights

Figure 3: Initialized weights for each of the down-up and up-down poolings.
Each strategy shares the shape of a cosine function but varies in the range on
the x-axis depending on the layers to be focused.

Model Original 30% Attacked 60% Attacked 90% Attacked P R F1 P R F1 $\Delta
atk$ P R F1 $\Delta atk$ P R F1 $\Delta atk$ $\text{BERT}_{clean}$ 79.81 77.79
78.64 79.51 73.35 75.19 -4.38% 77.74 66.38 67.96 -13.58% 76.14 62.01 62.44
-20.60% $\text{BERT}_{clean}+down\text{-}up$ 79.80 77.64 78.55 80.02 73.05
75.02 -4.49% 77.20 65.70 67.15 -14.51% 76.61 61.84 62.19 -20.82%
$\text{BERT}_{clean}+up\text{-}down$ 80.35 77.10 78.36 79.95 71.67 73.73
-5.90% 78.59 65.28 66.67 -14.91% 76.81 60.87 60.81 -22.39%
$\text{BERT}_{clean}+first\text{-}last$ 79.05 79.37 79.21 78.89 75.85 77.02
-2.76% 77.58 69.38 71.21 -10.09% 76.08 64.33 65.49 -17.32%

Table 5: Experimental results of offensive language detection when using down-
up and up-down pooling strategies. They are initialized along with the cosine
function, assigning distinct weights to the layers depending on whether they
focus more on offensiveness and token embeddings.

Model Original 30% Attacked 60% Attacked 90% Attacked P R F1 P R F1 $\Delta
atk$ P R F1 $\Delta atk$ P R F1 $\Delta atk$ $\text{BERT}_{\textit{noise}}$
80.64 78.88 79.64 80.67 75.42 77.17 -3.10% 78.44 69.42 71.33 -10.43% 76.46
65.55 66.96 -15.92% $\text{BERT}_{\textit{noise}}+mean$ 81.59 77.73 79.18
80.81 73.79 75.82 -4.24% 78.35 68.32 70.17 -11.37% 75.89 65.54 66.94 -15.45%
$\text{BERT}_{\textit{noise}}+max$ 80.72 79.81 80.23 79.68 76.29 77.57 -3.31%
77.46 72.00 73.64 -8.21% 74.82 69.60 71.06 -11.42%
$\text{BERT}_{\textit{noise}}+weighted$ 81.31 78.06 79.34 80.71 75.10 76.91
-3.06% 77.73 69.73 71.57 -9.79% 75.13 66.77 68.29 -13.92%
$\text{BERT}_{\textit{noise}}+first\text{-}last$ 81.62 77.63 79.12 80.36 75.04
76.79 -2.94% 78.04 71.48 73.28 -7.38% 74.87 69.15 70.66 -10.69% DeBERTaV3
82.55 78.70 80.17 81.85 74.33 76.48 -4.60% 80.14 68.47 70.44 -12.13% 78.41
63.96 64.99 -18.93% DeBERTaV3$+mean$ 82.19 77.49 79.17 81.22 73.13 75.28
-4.91% 79.83 68.18 70.10 -11.45% 78.31 64.20 65.31 -17.50% DeBERTaV3$+max$
83.32 76.96 79.02 82.50 72.73 75.08 -4.98% 80.72 67.89 69.81 -11.65% 78.79
63.82 64.81 -17.98% DeBERTaV3$+weighted$ 81.74 77.71 79.21 80.88 73.02 75.13
-5.15% 79.72 68.24 70.16 -11.42% 78.14 64.18 65.28 -17.58%
DeBERTaV3$+first\text{-}last$ 82.90 78.32 79.99 81.65 73.75 75.92 -5.08% 79.88
68.71 70.69 -11.62% 78.21 64.54 65.74 -17.81%

Table 6: Experimental results of offensive language detection when using
layer-wise pooling strategies to the $\text{BERT}_{\textit{noise}}$ and
DeBERTaV3. The two models differ in whether the texts used for the pre-
training stage were preprocessed and their own model structures.

Additionally, we conducted experiments to determine whether useful information
could be captured not only in the first and last layer, but also in the layers
close to these two layers of the model. We hypothesized that layers close to
the last layer would capture the offensiveness that determine the text
representations, whereas layers close to the first layer would capture the
token embeddings that determine the degree to which a sentence contains
textual attacks. We set all weights to zero for the weighted pooling, however,
in this experiment, we assigned distinct weights to all layers.

The weights for the down-up and up-down poolings are shown in Figure 3. We
assigned relatively high weights to the layers close to the last and first
layers, while assigning low weights to the middle layers. This strategy is
referred to as down-up pooling, as its weight graph moves down then up. The
green line at the cosine function within the range $[0,2\pi]$ represents the
initial weights for down-up pooling. In contrast, we assigned relatively low
weights to the layers close to the last and first layers, while assigning high
weights to the middle layers. This strategy is referred to as up-down pooling,
as its weight graph moves up then down. The pink line at the cosine function
within the range $[\pi,3\pi]$ represents that of the up-down pooling.

The performances of the models for the distinct layer weights are presented in
Table 5. Down-up pooling outperformed for all attack rates and exhibited more
robustness against performance degradation compared to up-down pooling. This
demonstrates that it is more important to weigh the layers close to the last
and first layers than those of the middle for attacked offensive languages. In
summary, the strategy that focuses on layers capable of capturing
offensiveness and token embeddings yields a better performance.

Nevertheless, neither of these pooling strategies performed as well as the
$\text{BERT}_{\textit{clean}}$ or the model that employed the first-last
pooling. This indicates that while information close to the last and first
layers can be beneficial, there are some layers among them that hinder the
performance of the model, revealing that only the last and first layers are
the most helpful layers when detecting offensive language with user-intended
adversarial attacks.

###  5.4 Beyond the Limits: Broader Application
of Layer-Wise Pooling Strategies

We employed the layer-wise pooling strategies not only to
$\text{BERT}_{\textit{clean}}$, a model pre-trained on clean texts, but also
to models that utilize noisy texts or employ alternative methods in the pre-
training stage. The performances of the layer-wise pooling strategies for both
the $\text{BERT}_{\textit{noise}}$ and DeBERTaV3 are presented in Table 6. In
$\text{BERT}_{\textit{noise}}$, first-last pooling only performed well on the
performance degradation, but the strategy exhibiting more robustness to
detection metrics was max pooling.

In the case of $\text{BERT}_{\textit{clean}}$, because it did not directly
utilize noisy texts, first-last pooling performed well for attacked offensive
language, which focuses most on offensiveness and token embeddings from the
sentence. However, in the case of $\text{BERT}_{\textit{noise}}$, where noisy
texts were pre-trained into the model weights before the application of
pooling strategies, max pooling proved to be the most effective on detection
metrics by selecting prominent features among the information to predict the
labels invariant to small changes or disturbances.

The results for DeBERTaV3 showed that it performed better with only the model
itself when there were no attacks or 30% attacked, while performed better with
pooling strategies when the attack rate was increased. We found that the
distinct pre-training process of DeBERTaV3, different from the general BERT,
was capable of handling a certain level of textual attacks. However, the
introduced pooling strategy could prove beneficial in scenarios where the
attack rate increases.

The average performances using max pooling and first-last pooling, which had
the most impact on each model are presented in Table 7. For the model
$\text{BERT}_{\textit{clean}}$ and DeBERTaV3, first-last pooling improved the
f1-score by 2.72 and 0.15 and prevented a performance degradation of 2.8% and
0.38%, respectively. In particular, the naive BERT model with the pooling
strategies outperformed the ensemble models, and even surpassed the recent
model DeBERTaV3 with the pooling strategies. It suggests that the introduced
pooling strategies can be employed for a simple structure that solely relies
on naive BERT, enabling efficient detection of attacked offensive language
without the necessity for the training of noisy texts and strategic pre-
training.

In the case of $\text{BERT}_{\textit{noise}}$, max pooling improved its
f1-score by 2.26, reaching a relatively high score of 74.08. In terms of
performance degradation, first-last pooling exhibited a performance
degradation of 2.81%. Thus, we confirmed that implementing the pooling
strategies on a model pre-trained with noisy texts resulted in more robust and
improved detection of attacked offensive language exceeded others, such as
pooling strategies on a model pre-trained with clean texts. The performances
of these pooling strategies are remarkable considering that they do not
require additional parameters or direct training on noisy texts.

Model Average F1 $\Delta atk$ $\text{BERT}_{\textit{clean}}$ 68.52 -12.85%
$\text{BERT}_{\textit{clean}}+max$ 68.51 -12.88%
$\text{BERT}_{\textit{clean}}+first\text{-}last$ 71.24 -10.05%
$\text{BERT}_{\textit{noise}}$ 71.82 -9.81% $\text{BERT}_{\textit{noise}}+max$
74.08 -7.64% $\text{BERT}_{\textit{noise}}+first\text{-}last$ 73.57 -7.00%
$\text{Ensemble}_{\textit{hard}}$ ($\text{BERT}_{\textit{clean}}$ \+
$\text{BERT}_{\textit{noise}}$) 70.99 -11.64%
$\text{Ensemble}_{\textit{soft}}$ ($\text{BERT}_{\textit{clean}}$ \+
$\text{BERT}_{\textit{noise}}$) 71.23 -11.37% DeBERTaV3 70.63 -11.88%
DeBERTaV3$+max$ 69.89 -11.53% DeBERTaV3$+first\text{-}last$ 70.78 -11.50%

Table 7: Average from the experimental results of offensive language detection
when using max pooling and first-last pooling to the models which exhibited
the high scores among the baselines.

##  6 Conclusion

We proposed user-intended adversarial attacks that occur frequently in
offensive languages online from the perspective of malicious users. We
categorized them into three types: Insert, Copy, and Decompose, which add
special symbols or exploit the distinct features of the Korean language. The
involvement of attacks significantly affects the tokenization results from the
original text.

To address the proposed attacks, we introduced the pooling strategies in a
layer-wise manner. This extension utilizes not only the last layer, which
focuses on offensiveness, but also the preceding layers, which focus more on
token embeddings. The experimental results indicated that first-last pooling
was the most robust to the proposed attacks and could even achieve a
comparable performance to that of models pre-trained on noisy texts, when
applied to models pre-trained on clean texts. We especially demonstrated that
rather than the middle layers, the first and last layers can be effectively
employed to detect attacked offensive languages.

Furthermore, we experimented with the extent to which the introduced pooling
strategies could handle the proposed attacks. We observed that the first-last
pooling and max pooling are the most robust, depending on the nature of the
texts used for pre-training. It is noteworthy that these strategies, without
the explicit training of additional parameters or noisy texts, can effectively
defend against user-intended adversarial attacks.

## Limitations

Despite our efforts to define diverse types of adversarial attacks, there is
room for undefined attacks from real-world situations, which may have an
unexpected impact on the model performance. Additionally, although we used
some language-independent attacks such as inserting special symbols, most of
the proposed attacks were based on the characteristics of the Korean language.
Therefore, it is necessary to determine whether layer-wise pooling strategies
can effectively handle attacked offensive expressions written in other
languages for cross-lingual applications. The introduced pooling strategies
offer significant flexibility for models in other languages, requiring simple
modifications to the model structure.

## Ethics Statement

Given our use of offensive representations to describe the proposed attacks,
we have included a disclaimer at the beginning of this paper. From the
existing offensive language datasets, potential biases regarding race, gender,
political issues, and other factors might have been inherent in our
experiments. This should be considered when developing our research or
expanding it to other languages.

## Acknowledgements

This research was supported by Basic Science Research Program through the
National Research Foundation of Korea (NRF) funded by the Ministry of
Education (NRF-2022R1C1C1008534), and Institute for Information &
communications Technology Planning & Evaluation (IITP) through the Korea
government (MSIT) under Grant No. 2021-0-01341 (Artificial Intelligence
Graduate School Program, Chung-Ang University).

## References

  * Aggarwal and Zesch (2022) Piush Aggarwal and Torsten Zesch. 2022.  Analyzing the real vulnerability of hate speech detection systems against targeted intentional noise.  In _Proceedings of the Eighth Workshop on Noisy User-generated Text (W-NUT 2022)_ , pages 230–242.
  * Ahn and Egorova (2021) Sangcheol Ahn and Kyunney Egorova. 2021.  Morpho-phonological patterns of recent korean neologisms.  In _Conference on current problems of our time: the relationship of man and society (CPT 2020)_ , pages 62–72.
  * Bajaj and Vishwakarma (2023) Ashish Bajaj and Dinesh Kumar Vishwakarma. 2023.  Homochar: A novel adversarial attack framework for exposing the vulnerability of text based neural sentiment classifiers.  _Engineering Applications of Artificial Intelligence_ , 126:106815.
  * Boucher et al. (2022) Nicholas Boucher, Ilia Shumailov, Ross Anderson, and Nicolas Papernot. 2022.  Bad characters: Imperceptible nlp attacks.  In _2022 IEEE Symposium on Security and Privacy (SP)_ , pages 1987–2004. IEEE.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Çağlar Gu̇lçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using rnn encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1724–1734.
  * Cho and Kim (2021) Wonik Cho and Soomin Kim. 2021.  Google-trickers, yaminjeongeum, and leetspeak: An empirical taxonomy for intentionally noisy user-generated text.  In _Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021)_ , pages 56–61.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Goyal et al. (2023) Shreya Goyal, Sumanth Doddapaneni, Mitesh M Khapra, and Balaraman Ravindran. 2023.  A survey of adversarial defenses and robustness in nlp.  _ACM Computing Surveys_ , 55(14s):1–39.
  * He et al. (2023) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.  In _The Eleventh International Conference on Learning Representations_.
  * He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021.  Deberta: Decoding-enhanced bert with disentangled attention.  In _International Conference on Learning Representations_.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does BERT learn about the structure of language? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  * Jeong et al. (2022) Younghoon Jeong, Juhyun Oh, Jongwon Lee, Jaimeen Ahn, Jihyung Moon, Sungjoon Park, and Alice Oh. 2022.  KOLD: Korean offensive language dataset.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10818–10833, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Jin et al. (2020) Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020.  Is bert really robust? a strong baseline for natural language attack on text classification and entailment.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 8018–8025.
  * Jun (2020) Woochun Jun. 2020.  A study on the cause analysis of cyberbullying in korean adolescents.  _International journal of environmental research and public health_ , 17(13):4648.
  * Kim et al. (2021a) Bosung Kim, Juae Kim, Youngjoong Ko, and Jungyun Seo. 2021a.  Commonsense knowledge augmentation for low-resource languages via adversarial learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 6393–6401.
  * Kim et al. (2021b) Jinyong Kim, Jeonghyeon Kim, Mose Gu, Sanghak Ohh, Gilteun Choi, and Jaehoon Jeong. 2021b.  Hypocrite: Homoglyph adversarial examples for natural language web services in the physical world.
  * Kim et al. (2021c) Soomin Kim, Changhoon Oh, Wonik Cho, Donghoon Shin, Bongwon Suh, and Joonhwan Lee. 2021c.  Trkic g00gle: Why and how users game translation algorithms.  _Proceedings of the ACM on Human-Computer Interaction_ , 5(CSCW2):1–24.
  * Kurita et al. (2019) Keita Kurita, Anna Belova, and Antonios Anastasopoulos. 2019.  Towards robust toxic content classification.  _arXiv preprint arXiv:1912.06872_.
  * Lee et al. (2022) Jean Lee, Taejun Lim, Heejun Lee, Bogeun Jo, Yangsok Kim, Heegeun Yoon, and Soyeon Caren Han. 2022.  K-MHaS: A multi-label hate speech detection dataset in Korean online news comment.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 3530–3538, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Lee (2020) Junbum Lee. 2020.  Kcbert: Korean comments bert.  In _Proceedings of the 32nd Annual Conference on Human and Cognitive Language Technology_ , pages 437–440.
  * Li et al. (2019) J Li, S Ji, T Du, B Li, and T Wang. 2019.  Textbugger: Generating adversarial text against real-world applications.  In _26th Annual Network and Distributed System Security Symposium_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Ma et al. (2019) Xiaofei Ma, Zhiguo Wang, Patrick Ng, Ramesh Nallapati, and Bing Xiang. 2019.  Universal text representation from bert: An empirical study.  _arXiv preprint arXiv:1910.07973_.
  * McCurry (2022) Justin McCurry. 2022.  South korea under pressure to crack down on cyberbullying after high-profile deaths.  _The Guardian_.
  * Moon et al. (2020) Jihyung Moon, Won Ik Cho, and Junbum Lee. 2020.  BEEP! Korean corpus of online news comments for toxic speech detection.  In _Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media_ , pages 25–31, Online. Association for Computational Linguistics.
  * Oh et al. (2022) Dongsuk Oh, Yejin Kim, Hodong Lee, H. Howie Huang, and Heuiseok Lim. 2022.  Don’t judge a language model by its last layer: Contrastive learning with layer-wise attention pooling.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 4585–4592, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Park et al. (2023a) Chaewon Park, Soohwan Kim, Kyubyong Park, and Kunwoo Park. 2023a.  K-haters: A hate speech detection corpus in korean with target-specific ratings.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 14264–14278.
  * Park et al. (2023b) San-Hee Park, Kang-Min Kim, O-Joun Lee, Youjin Kang, Jaewon Lee, Su-Min Lee, and SangKeun Lee. 2023b.  “why do I feel offended?” - Korean dataset for offensive language identification.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 1142–1153, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Park et al. (2021) Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Ji Yoon Han, Jangwon Park, Chisung Song, Junseong Kim, Youngsook Song, Taehwan Oh, et al. 2021.  Klue: Korean language understanding evaluation.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Ren et al. (2019) Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019.  Generating natural language adversarial examples through probability weighted word saliency.  In _Proceedings of the 57th annual meeting of the association for computational linguistics_ , pages 1085–1097.
  * Saengprang and Gadavanij (2021) Suriya Saengprang and Savitri Gadavanij. 2021.  Cyberbullying: The case of public figures.  _LEARN Journal: Language Education and Acquisition Research Network_ , 14(1):344–369.
  * Sahoo et al. (2023) Nihar Sahoo, Niteesh Mallela, and Pushpak Bhattacharyya. 2023.  With prejudice to none: A few-shot, multilingual transfer learning approach to detect social bias in low resource languages.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 13316–13330, Toronto, Canada. Association for Computational Linguistics.
  * Schuster and Paliwal (1997) Mike Schuster and Kuldip K Paliwal. 1997.  Bidirectional recurrent neural networks.  _IEEE transactions on Signal Processing_ , 45(11):2673–2681.
  * Song (2006) Jaejung Song. 2006.  _The Korean language: Structure, use and context_.  Routledge.
  * Wolff and Wolff (2020) Max Wolff and Stuart Wolff. 2020.  Attacking neural text detectors.  _ICLR 2020 Workshop Towards Trustworthy ML_.
  * Wu et al. (2018) Zhelun Wu, Nishant Kambhatla, and Anoop Sarkar. 2018.  Decipherment for adversarial offensive language detection.  In _Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)_ , pages 149–159, Brussels, Belgium. Association for Computational Linguistics.
  * Yi and Cha (2020) Hyunyoung Yi and Sangmi Cha. 2020.  Cyber bullying, star suicides: The dark side of south korea’s k-pop world.  _Reuters_.
  * Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019.  Predicting the type and target of offensive posts in social media.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1415–1420, Minneapolis, Minnesota. Association for Computational Linguistics.

##  Appendix A User-Intended
Adversarial Attacks Details

###  A.1 Insert

We utilized three Insert types by adding special symbols that are not complete
characters. The detailed methods and examples are as follows:

  * •

Insert_zz: We randomly added between 2~5 sounds of ‘ㅋ’ to the word (on the
keyboard, the sound ‘ㅋ’ corresponds to the alphabet ‘z’). This is a commonly
used expression in online, conveying a meaningless and somewhat frivolous
tone. We placed it not only between characters but also in instances where the
final sound of a specific character was empty.

  * •

Insert_space: We randomly added a single space to the word. We expected the
same impact as the intentions of malicious users.

  * •

Insert_special: We randomly added a special character to the word: one of ‘~’,
‘!’, ‘@’, ‘1’, or ‘2’. We also expected the same impact as the intentions of
malicious users.

Insert Attacks Text Examples original text 틀딱이냐? (okay, boomer?) 기레기 여기 있었네
(presstitute right here) Insert_zz 틀ㅋㅋ딱이냐? 기렠ㅋㅋㅋ기 여기 있었네 Insert_space 틀 딱이냐?
기레 기 여기 있었네 Insert_special 틀@딱이냐? 기2레기 여기 있었네

Table 8: Text examples of user-intended adversarial attacks with the types of
Insert.

###  A.2 Copy

We utilized three Copy types based on the distinctive characteristics of the
Korean language. The detailed methods and examples are as follows:

  * •

Copy_initial: We copied the initial sound of the character to the final sound
of the preceding character. For example, if ‘기’ is chosen from ‘기레기’, the
initial sound ‘ㄱ’ would be copied as the final sound to the preceding ‘레’.
Consequently, ‘레’ is transformed into ‘렉’, leading to ‘기렉기’.

  * •

Copy_middle: We copied the middle sound of the character, onto the newly added
character. If the selected character had a final sound, it was also included.
For example, if ‘딱’ is chosen from ‘틀딱’, the middle sound ‘ㅏ’ would be copied
as the following character with the final sound ‘ㄱ’. Consequently, ‘악’ is
newly added, leading to ‘틀따악’.

  * •

Copy_final: We copied the final sound of the character to the initial sound of
the following character. For example, if ‘있’ is chosen from ‘있었네’, the final
sound ‘ㅆ’ would be copied as the initial sound to the following ‘었’.
Consequently, ‘었’ is transformed into ‘썼’, leading to ‘이썼네’.

Copy Attacks Text Examples original text 틀딱이냐? (okay, boomer?) 기레기 여기 있었네
(presstitute right here) Copy_initial 틀딱인냐? 기렉기 여기 있었네 Copy_middle 틀따악이냐? 기레기
여기이 있었네 Copy_final 틀딱기냐? 기레기 여기 이썼네

Table 9: Text examples of user-intended adversarial attacks with the types of
Copy.

Decompose Attacks Text Examples original text 틀딱이냐? (okay, boomer?) 기레기 여기 있었네
(presstitute right here) Decompose_final 틀따ㄱ이냐? 기레기 여기 이ㅆ었네 Decompose_all
ㅌㅡㄹ딱이냐? 기ㄹㅔ기 여기 있었네

Table 10: Text examples of user-intended adversarial attacks with the types of
Decompose.

###  A.3 Decompose

We utilized two Decompose types based on the distinctive characteristics of
the Korean language. The detailed methods and examples are as follows:

  * •

Decompose_final: We decomposed the final sound of the character into the newly
added sound. For example, if ‘딱’ is chosen from ‘틀딱’, the final sound ‘ㄱ’
would be decomposed as the following sound. Consequently, the expression will
be ‘틀따ㄱ’.

  * •

Decompose_all: We decomposed all the sounds of the character. For example, if
‘틀’ is chosen from ‘틀딱’, it would be decomposed as the initial, middle, and
final sounds. Consequently, the expression will be ‘ㅌㅡㄹ딱’.

###  A.4 How the Attack Works

In our experiments, the attacks were only applied to the test set to evaluate
the robustness of the model against user-intended adversarial attacks. We
randomly selected one of all the attacks according to the attack rates. For
instance, with a 30% attack rate and 9 words in a sentence, we attacked 3
words (30% of them), with each word randomly reflecting one of the proposed
attacks.

###  A.5 Experiments on Each Attack

We further investigated the impact for each attack type, selecting only one of
the attacks from only Insert, Copy, or Decompose. We chose
$\text{BERT}_{\textit{clean}}$ and the model that applied first-last pooling,
which exhibited the best performance among the layer-wise pooling strategies.

The performances according to the attack types are presented in Table 11. It
is noteworthy that a high performance on a specific attack type indicates that
it is easier to defend than others, and vice versa. Insert proved to be easier
than all attacks, with Copy only marginally harder than Insert. However,
Decompose was more difficult than these attacks, exhibiting a performance
degradation compared to all attacks. Therefore, we revealed that adversarial
attacks that reflect the characteristics of the Korean language are more
challenging than those that simply add special symbols that could be adapted
language independently.

Model All Attacks Only Insert P R F1 P R F1 $\text{BERT}_{\textit{clean}}$
77.74 66.38 67.96 78.12 68.80 70.66
$\text{BERT}_{\textit{clean}}+first\text{-}last$ 77.58 69.38 71.21 77.51 70.76
72.54 Model Only Copy Only Decompose P R F1 P R F1
$\text{BERT}_{\textit{clean}}$ 77.86 68.47 70.29 78.38 62.23 65.67
$\text{BERT}_{\textit{clean}}+first\text{-}last$ 77.33 70.50 72.26 77.00 66.47
68.04

Table 11: Experimental results of offensive language detection when the attack
ratio is 60% for each type of user-intended adversarial attacks.

##  Appendix B Experimental Details

###  B.1 Implementation Details

We used 6 layers with an embedding dimension of 768 and a dropout ratio of 0.1
for the RNN-based models containing BiLSTM and BiGRU. As for the BERT-based
models, containing $\text{BERT}_{\textit{clean}}$,
$\text{BERT}_{\textit{noise}}$, their ensembles, and DeBERTaV3, we fine-tuned
12 pre-trained layers with an embedding dimension of 768 and a dropout ratio
of 0.2.

We used the AdamW optimizer with a learning rate of 1e-5, trained the models
for 1~5 epochs, and considered the epoch with the lowest validation loss or
the last epoch. We set a batch size of 32 for all the models. The models were
implemented using PyTorch and NVIDIA GeForce RTX 3090 GPU. We also used the
HuggingFace library to leverage the weights of the pre-trained BERT models.

###  B.2 Metrics

We collected the datasets to train as many types of offensive languages as
possible. There is some label imbalance, therefore, we used macro precision,
recall, and f1-score to address this issue.

$\Delta atk$ represents the performance degradation in the f1-score. For
example, denoting the existing f1-score as $\text{F1}_{\textit{original}}$ and
the f1-score with the attacks as $\text{F1}_{\textit{attacked}}$, it is
computed as follows:

$\displaystyle\Delta atk$
$\displaystyle=(\text{F1}_{\textit{attacked}}-\text{F1}_{\textit{original}})/\text{F1}_{\textit{original}}*100.$
(5)

##  Appendix C Examples of the Attacks

We presented the following cases where offensive languages contained user-
intended adversarial attacks on popular online communities in South Korea,
such as Twitter555https://twitter.com and dcinside666https://dcinside.com.

  * •

Insert_zz:
지에지에 개 병ㅋㅋㅋ신
(ziezie so dumbzzzfuck)

  * •

Insert_space:
완전 또 라이 아니야…
(crazy ass we irdo)

  * •

Insert_special:
넥슨아 이거 개같으면 지워라 새@끼들아
(delete this if you dare Nexon you sons of bit@ches)

  * •

Copy_initial:
우리나라 남자샊끼들 진짜 재미없게 산다
(korean lossser men live in the most boring ass lives)

  * •

Copy_middle:
역쉬나 기레에기들..
(presstiiitutes of course)

  * •

Copy_final:
현생 어떻게 살아가누 너가튼게?
(how the fuck dooyou live in the real world?)

  * •

Decompose_final:
과몰입 중인 병신 트ㄹ딱들ㅋㅋ
(chronically online dumbass bo omers lmao)

  * •

Decompose_all:
너 같은 ㄱㅐ 새끼는 몽둥이로 맞아 죽어야 해
(p i ece of shit like you need to be beaten to death)