# Resonant jumps induced by stationary tidal perturbation: a two-for-one deal

Priti Gupta  Department of Physics, Kyoto University, Kyoto 606-8502, Japan
<EMAIL_ADDRESS>Takafumi Kakehi  Yukawa Institute for
Theoretical Physics,Kyoto University, Kyoto 606-8502, Japan  Center for
Gravitational Physics and Quantum Information, Yukawa Institute for
Theoretical Physics,Kyoto University, Kyoto 606-8502, Japan
<EMAIL_ADDRESS>Takahiro Tanaka  Department of Physics,
Kyoto University, Kyoto 606-8502, Japan  Center for Gravitational Physics and
Quantum Information, Yukawa Institute for Theoretical Physics,Kyoto
University, Kyoto 606-8502, Japan<EMAIL_ADDRESS>
###### Abstract

Extreme-mass-ratio inspirals (EMRIs) are promising target sources for space-
based interferometers such as LISA, Taiji, and Tianqin. Depending on the
astrophysical environment, such as close perturbers or an accretion disk, EMRI
orbital evolution may deviate from the predictions of general relativity in
vacuum. In particular, we focus on the resonance jumps, i.e., the changes of
the conserved quantities induced by a stationary perturbation to the
background Kerr geometry. Using Hamiltonian formulation, we provide a closed
relation between the jump in Carter constant and that in the axial component
of angular momentum. It is also shown that the obtained relation is consistent
with the fitting formulae computed for the tidal resonance in previous works.

##  1 Introduction

LIGO interferometers detected gravitational waves (GWs) for the first time in
2015, leading to the first direct observation of GWs [1]. With the space-based
interferometers (such as LISA [2], Taiji [3], and Tianqin [4]), we can detect
sources emitting GWs at mHz frequencies. There are a number of promising
sources for LISA, including extreme-mass-ratio inspirals (EMRIs), which
involve stellar-mass objects spiraling into supermassive black holes (SMBHs)
[5, 6, 7]. Given the extreme mass ratio, EMRIs will spend months to years in
the observable band before the plunge, allowing us to map strong gravity
spacetime and test the black hole geometry with great precision. High
precision theoretical prediction of the gravitational waveform is an urgent
task, and a lot of efforts have been made [8, 9].

Generic relativistic bound orbits around massive BHs have three frequencies,
i.e., the radial $\omega_{r}$, polar $\omega_{\theta}$, and azimuthal
$\omega_{\phi}$ frequencies. Due to the self-force [10, 11], these frequencies
evolve adiabatically as binary shrinks. The evolution track of an EMRI passes
through several self-force resonances, unless it takes a quasi-circular or
equatorial orbit. During such resonances, radial and polar frequencies become
commensurate, i.e., $n\,\omega_{r}+k\,\omega_{\theta}=0,$ where $n,k$ are
integers [12]. EMRIs can be significantly affected by a perturbation at
resonance points. At the resonance, a “jump” (depending on the orbital phase)
is induced in the “constants of motion” (orbital energy $E$, axial angular
momentum $L_{z}$, and Carter constant $Q$), which alters the post-resonance
orbital evolution [13, 14].

In a similar manner, when we have an external perturber, such as a neighboring
third body, a tidal gravitational perturbation is added to the background
spacetime of the EMRI, giving rise to tidal resonances111Generically, when the
EMRI’s three orbital frequencies are commensurate with those of the perturber,
a tidal resonance occurs. This paper treats the perturber as static, so its
orbital frequencies are irrelevant to resonance conditions. [15] when
$n\,\omega_{r}+k\,\omega_{\theta}+m\,\omega_{\phi}=0$. It has been shown that
EMRIs may encounter multiple tidal resonances and may dephase the waveform
significantly depending on the magnitude of the tidal perturbation as well as
on the orbital parameters and phase. A slowly changing tidal perturbation will
result in jumps in the azimuthal angular momentum and Carter constant, and
fitting formulae for different resonance combinations have already been
developed [16, 17]. It may, however, be necessary to evaluate a broader range
of resonances in the future. Recently, a simpler method of computing the
resonant jump for angular momentum was proposed [18]. However, this method
does not directly apply to the jump in the Carter constant. In this paper, we
show that there is a simple relation between the two jumps caused by a
stationary external perturbation, which is not restricted to the tidal
perturbation with $|m|\leq 2$. This relation allows us to deduce the jump in
the Carter constant from the jump in the angular momentum.

##  2 Brief review of the Hamiltonian approach

Self-force motion can be understood using a perturbed Hamiltonian expressed in
terms of action-angle variables [19]. Here, we argue that the same method as
for the conservative part of the self-force, whose existence was first pointed
out in [20] and very recently confirmed by [21], can also be used in the case
of external perturbation. In particular, we consider a tidal perturbation
added to the background Kerr spacetime. The full metric is given by

$\displaystyle g_{\mu\nu}=g_{\mu\nu}^{(0)}+h_{\mu\nu}\,,$ (1)

where

$\displaystyle g^{(0)}_{\mu\nu}dx^{\mu}dx^{\nu}=$
$\displaystyle-\left(1-\frac{2Mr}{\Sigma}\right)dt^{2}-\frac{4Mar\sin^{2}\\!\theta}{\Sigma}dt\,d\phi+\frac{\Sigma}{\Delta}dr^{2}$
(2)
$\displaystyle+\Sigma\,d\theta^{2}+\left(r^{2}+a^{2}+\frac{2Ma^{2}r}{\Sigma}\sin^{2}\\!\theta\right)\sin^{2}\\!\theta\,d\phi^{2}\,,$
(3)

with

$\displaystyle\Sigma\equiv r^{2}+a^{2}\cos^{2}\\!\theta,\qquad\Delta\equiv
r^{2}-2Mr+a^{2}\,,$ (4)

is the infinitesimal line element of the Kerr spacetime, and $h_{\mu\nu}$
represents a stationary external perturbation, which is given independently of
the particle motion. The perturbed metric admits a stationary Killing vector

$(\partial_{t})^{\mu}-\Omega(\partial_{\phi})^{\mu}\,,$

with $\Omega$ being the frequency of the perturber. We consider the geodesic
motion of a particle in this perturbed spacetime, which is governed by the
following Hamiltonian

$\displaystyle
H=H^{(0)}+H^{(1)}=\frac{1}{2}g_{\mu\nu}\left(x(\tau)\right)u^{\mu}(\tau)u^{\nu}(\tau)\,,$
(5)

where $H^{(0)}$ and $H^{(1)}$ describe the contributions coming from
$g_{\mu\nu}^{(0)}$ and $h_{\mu\nu}$, respectively. Our focus is on the change
rates of $E,L_{z}$ and $Q$, which are defined by

$\displaystyle E\equiv-\mu u_{t}\,,$ (6) $\displaystyle L_{z}\equiv\mu
u_{z}\,,$ (7) $\displaystyle
Q\equiv\mu^{2}(u_{\theta}^{2}+a^{2}\cos^{2}\\!\theta(1-u_{t}^{2})+\cot^{2}\\!\theta
u_{\phi}^{2})\,,$ (8)

with $u^{\mu}$ being the four-velocity of the particle [19]. Additionally, the
mass of the particle $\mu^{2}$ is also treated as a constant of motion such
that

$\displaystyle\mu^{2}=-g^{\mu\nu}p_{\mu}p_{\nu}\,.$ (9)

where $p_{\mu}$ is the conjugate momenta of the particle. The complete set of
quantities $\\{E,L_{z},Q,\mu^{2}\\}$ is denoted by $\\{I_{a}\\}$. For the
following formal discussion, we use a set of action-angle variables,
$(w^{a},J_{a})$, instead of the pair of the Boyer-Lindquist coordinates and
the specific four momentum $(x^{\mu},p_{\mu})$. The relation between these two
sets of variables is given by

$\displaystyle p_{\mu}=\frac{\partial W}{\partial x^{\mu}}(x^{\nu},J_{a})\,,$
(10) $\displaystyle w^{a}=\frac{\partial W}{\partial J_{a}}(x^{\nu},J_{a})\,,$
(11)

using the generating function $W(x^{\nu},J_{a})$ defined as

$\displaystyle
W(x^{\mu},J_{a})=-J_{t}t+J_{\phi}\phi\pm\int^{r}dr^{\prime}\frac{\sqrt{V_{r}(r^{\prime})}}{\Delta}\pm\int^{\theta}d\theta^{\prime}\sqrt{V_{\theta}(\theta^{\prime})}\,,$
(12)

where the functions $V_{r}$ and $V_{\theta}$ are given by

$\displaystyle
V_{r}(r,E,L_{z},Q,\mu^{2})=\left(E(r^{2}+a^{2})-aL_{z}\right)^{2}-\Delta\left[r^{2}\mu^{2}+(L_{z}-aE)^{2}+Q\right]\,,$
(13) $\displaystyle
V_{\theta}(\theta,E,L_{z},Q,\mu^{2})=Q-\cos^{2}\\!\theta\left[a^{2}(\mu^{2}-E^{2})+\frac{L_{z}^{2}}{\sin^{2}\\!\theta}\right]\,.$
(14)

The expression for $W(x^{\mu},J_{a})$ contains $E,L_{z},Q$ and $\mu^{2}$,
which are to be understood as functions of $J_{a}$ determined by solving

$\displaystyle J_{t}=-E\,,\quad J_{\phi}=L_{z}\,,\quad$ (15) $\displaystyle
J_{r}=\frac{1}{2\pi}\oint\frac{\sqrt{V_{r}(r,E,L_{z},Q,\mu^{2})}}{\Delta(r)}dr\,,$
(16) $\displaystyle
J_{\theta}=\frac{1}{2\pi}\oint\sqrt{V_{\theta}(\theta,E,L_{z},Q,\mu^{2})}d\theta\,.$
(17)

Here, $\oint$ means the integral over one period of oscillation of the
integration variable, and the signature of the square root is chosen to be
positive (negative) when the integration variable increases (decreases).

The change rate of $J_{a}$ is given by the corresponding Hamiltonian equation
of motion,

$\displaystyle\frac{dJ_{a}}{d\tau}=-\frac{\partial H}{\partial
w^{a}}=-\frac{\partial H^{(1)}}{\partial w^{a}_{\rm i}}\,,$ (18)

where $w^{a}_{\rm i}$ is the initial value of
$w^{a}=\\{w^{t},w^{r},w^{\theta},w^{\phi}\\}$. In the second equality we used
the facts that $H^{(0)}$ is written solely in terms of $J_{a}$, and $w^{a}$
for the background geodesic motion is expressed as
$w^{a}=\omega^{a}\tau+w^{a}_{\rm i}$ with $\omega^{a}$ being the angular
velocity of the variable $w^{a}$. For the non-resonant orbit, the average of
the right-hand side of Eq. (18) vanishes because the long time average of
$H^{(1)}$ (denoted by $\left\langle H^{(1)}\right\rangle$), should be
independent of the choice of the initial phase. Hence, $\left\langle
H^{(1)}\right\rangle$ is also a function of $J_{a}$ only, as in the case of
$H^{(0)}$. By contrast, when the resonant condition is satisfied, i.e., when

$\displaystyle
n\omega^{r}+k\omega^{\theta}+m(\omega^{\phi}+\Omega\omega^{t})=0\,,$ (19)

holds for some set of small non-trivial set of integers $\\{n,k,m\\}$, the
averaging must be treated differently. We refer to such a case as the
resonance combination $n:k:m$, although in general neither the ratio of
frequencies nor that of their inverses takes a simple integer ratio, it is the
ratio of the coefficients in Eq. (19) that takes the simple integer ratio
$n:k:m$. In such a case, the background geodesics taking different initial
values of

$\displaystyle\Delta w\equiv nw^{r}+kw^{\theta}+m(w^{\phi}+\Omega w^{t})\,,$
(20)

modulo $2\pi$, are distinguishable. Hence, the average value $\left\langle
H^{(1)}\right\rangle$ depends on $\Delta w$ as well as $J_{a}$. As a result,
Eq. (18) does not vanish. Furthermore, $\left\langle H^{(1)}\right\rangle$
depends on the initial phase only through the combination $\Delta w$, which
implies

$\displaystyle\left\langle\frac{dJ_{a}}{d\tau}\right\rangle=-\frac{\partial\left\langle
H^{(1)}\right\rangle}{\partial\Delta w}n_{a}\,,$ (21)

where we have defined $n_{a}\equiv\\{m\Omega,n,k,m\\}$. The above relation
immediately explains why the jumps of “constants of motion” vanish when $k+m=$
odd as follows (see also the previous discussions [17, 18]). Under the parity
transformation $\vec{x}\to-\vec{x}$, the tidal field remains invariant.
However, the angle variables transform like
$\\{w^{t},w^{r},w^{\theta},w^{\phi}\\}\to\\{w^{t},w^{r},w^{\theta}+\pi,w^{\phi}+\pi\\}$.
Therefore, under the parity transformation, $\Delta w$ transforms as $\Delta
w\to\Delta w+(k+m)\pi$ indicating that $\left\langle H^{(1)}(\Delta
w)\right\rangle=\left\langle H^{(1)}(\Delta w+\pi)\right\rangle$ when $k+m$ is
odd. Further, $H^{(1)}$ can be expanded as a Fourier series of $w^{a}$ as

$\displaystyle H^{(1)}=\sum_{n,k,m}e^{in_{a}w^{a}}H^{(1)}_{n,k,m}\,.$ (22)

From the expression (22), we find that the long-time average of $H^{(1)}$,

$\displaystyle\left\langle
H^{(1)}\right\rangle=\sum_{s\in\mathbb{Z}}e^{is\Delta w}H^{(1)}_{sn,sk,sm}$
(23)

is a periodic function with respect to $\Delta w$ with a period of $2\pi$. Let
us focus on the dominant contribution coming from $s=\pm 1$. Then, if
$\left\langle H^{(1)}\right\rangle$ takes a maximum value at $\Delta w$, it
should take a minimum value at $\Delta w+\pi$. Therefore, we can conclude that
the function $\left\langle H^{(1)}(\Delta w)\right\rangle$ restricted to the
contribution from $s=\pm 1$ becomes constant for $k+m=$ odd. Nonetheless, it
is possible to have the contribution coming from the terms with $s=$ even in
the resonance with $k+m=$ odd. Such higher-order contributions, which have not
been specifically discussed in literature [17, 16], do not necessarily vanish.
To provide more complete fitting formulae for resonance jumps, we will return
to this point in our future publication.

So far, we did not consider the lowest order (adiabatic) self-force, i.e., the
secular change of the “constants of motion” caused by the linear perturbation
induced by the particle itself. In the actual evolution of EMRIs, this
adiabatic self-force is expected to dominate the evolution of the “constants
of motion”, while the tidal resonance gives just minor corrections. In such
cases, we can compute the jumps, i.e., the extra changes induced in the
“constants of motion” due to a tidal resonance by integrating along the
evolution caused by the adiabatic self-force across the resonance point. At
least, in this approximation we obtain the relations

$\displaystyle\centering\Delta J_{t}(=-\Delta E):\Delta J_{r}:\Delta
J_{\theta}:\Delta J_{\phi}(=\Delta L_{z})\@add@centering$ $\displaystyle=$
$\displaystyle\left\langle\frac{dJ_{t}}{d\tau}\right\rangle:\left\langle\frac{dJ_{r}}{d\tau}\right\rangle:\left\langle\frac{dJ_{\theta}}{d\tau}\right\rangle:\left\langle\frac{dJ_{\phi}}{d\tau}\right\rangle$
(24) $\displaystyle=$ $\displaystyle m\Omega:n:k:m.$

This is the main message of this short paper. To understand the resonant jump,
it is sufficient to compute only one of $\Delta J_{a}$, e.g., $\Delta L_{z}$.
An exception are $m=0$ resonances, for which $\Delta L_{z}=0$ is obvious due
to axisymmetry of $m=0$ perturbation and therefore does not convey information
about other jumps.

##  3 Demonstration of the validity of the relation among resonance jumps

In the following, we demonstrate the validity of the relation shown in Eq.
(24). It appears that this equation gives three independent relations among
$\Delta J_{a}$. One is just the conservation law coming from the presence of
Killing vector, i.e., $dJ_{t}/d\tau-\Omega dJ_{\phi}/d\tau=0$, which reduces
to the familiar relation $dE/d\tau=0$ in the static limit. Another is the
conservation of $\mu^{2}\equiv-2H^{(0)}$, meaning that there is only one
independent relation. The full Hamiltonian $H$ is conserved under the
influence of any kind of perturbation to the background space-time. Hence,
$H^{(0)}$ can vary in time but only within the magnitude of the variation of
$H^{(1)}$. Therefore, $H^{(0)}=-2\mu^{2}$ is not allowed to violate
indefinitely in a secular manner. After a long time average for any kind of
perturbation, we should have

$\displaystyle\
0=\left\langle\frac{dH^{(0)}}{d\tau}\right\rangle=\left\langle\frac{\partial
H^{(0)}}{\partial
J_{a}}\frac{dJ_{a}}{d\tau}\right\rangle=\omega^{a}\left\langle\frac{dJ_{a}}{d\tau}\right\rangle\,.$
(25)

If we substitute the relation (24), we find that the vanishing of the right-
hand side is nothing but the resonance condition itself. Hence, only one of
three independent relations derived from Eq. (24) is the non-trivial one that
must be satisfied by generic perturbation.

Next, to obtain the formula for the ratio $\langle dQ/d\tau\rangle/\langle
dL_{z}/d\tau\rangle$, it is easier to consider that $Q$ is just the function
of $\\{E,L_{z},J_{r}\\}$ by setting $\mu^{2}=1$. In this treatment,
$J_{\theta}$ is a variable depending on $\\{E,L_{z},J_{r}\\}$. Then, we have

$\displaystyle 1=\left(\frac{\partial Q}{\partial
Q}\right)_{\\!\\!E,L_{z}}=\left(\frac{\partial Q}{\partial
J_{r}}\right)_{\\!\\!E,L_{z}}\left(\frac{\partial J_{r}}{\partial
Q}\right)_{\\!\\!E,L_{z}}\,,$ (26)

and

$\displaystyle 0=\left(\frac{\partial Q}{\partial
L_{z}}\right)_{\\!\\!E,Q}=\left(\frac{\partial Q}{\partial
J_{r}}\right)_{\\!\\!E,L_{z}}\left(\frac{\partial J_{r}}{\partial
L_{z}}\right)_{\\!\\!E,Q}+\left(\frac{\partial Q}{\partial
L_{z}}\right)_{\\!\\!E,J_{r}}\,.$ (27)

From these relations, we immediately obtain

$\displaystyle\frac{dQ}{d\tau}$ $\displaystyle=\left(\frac{\partial
Q}{\partial
J_{r}}\right)_{\\!\\!E,L_{z}}\frac{dJ_{r}}{d\tau}+\left(\frac{\partial
Q}{\partial L_{z}}\right)_{\\!\\!E,J_{r}}\frac{dL_{z}}{d\tau}$ (28)
$\displaystyle=\left[\frac{n}{m}-\left(\frac{\partial J_{r}}{\partial
L_{z}}\right)_{\\!\\!E,Q}\right]\left(\frac{\partial J_{r}}{\partial
Q}\right)_{\\!\\!E,L_{z}}^{-1}\frac{dL_{z}}{d\tau}\,,$ (29)

where in the second equality we used the relation (24). In the same manner we
can obtain an equivalent expression in terms of $J_{\theta}$, which is given
by

$\displaystyle\frac{dQ}{d\tau}=\left[\frac{k}{m}-\left(\frac{\partial
J_{\theta}}{\partial L_{z}}\right)_{\\!\\!E,Q}\right]\left(\frac{\partial
J_{\theta}}{\partial Q}\right)_{\\!\\!E,L_{z}}^{-1}\frac{dL_{z}}{d\tau}\,.$
(30)

The above relation can be more explicitly written as

$\displaystyle\frac{dQ}{d\tau}=\frac{\displaystyle\frac{\pi
ak}{m}\sqrt{1-E^{2}}\,y_{+}-{2L_{z}}\left(K\left[\left(\frac{y_{-}}{y_{+}}\right)^{2}\right]-\Pi\left[y_{-}^{2},\left(\frac{y_{-}}{y_{+}}\right)^{2}\right]\right)}{\displaystyle{K\left[\left(\frac{y_{-}}{y_{+}}\right)^{2}\right]}}\frac{dL_{z}}{d\tau}\,.$
(31)

where $y_{\pm}~{}(y_{+}>y_{-})$ are the values of $\cos^{2}\\!\theta$ at
zeroes of the potential $V_{\theta}$ and $K[m],\,\Pi[n,m]$ are complete
elliptic integrals of the first and third kind, respectively. Explicit
expressions for all the quantities are given in A.

Now, we confirm Eq. (31) using the fitting formulae derived in Ref. [17].
These formulae are derived under the assumption of a static perturber with
$\Omega=0$. Figure 1 shows an agreement for different resonance combinations
such as $n:k:m=3:0:-2$, $n:k:m=3:-4:2$, and $n:k:m=-3:1:1$. The horizontal
axis is the spin of the central black hole and orbital eccentricity ($e$) and
inclination ($I$) is fixed (see the figure caption for details). We find that
the fitting formulae satisfy the relation (29) or (31) within the error of
0.01%.

Figure 1: Dependence of the ratio of the average change rate of Carter
constant and axial angular momentum on the spin of the central BH. The figures
are plotted for three different resonance combinations: $3:0:-2$ (top-left),
$3:-4:2$ (top-right), $-3:1:1$ (bottom). The perturber is placed on the
equatorial plane in the case of $m=\pm 2$ resonance, whereas in the case of
$m=\pm 1$ resonance, it is positioned at an inclination of $45^{\circ}$. The
dots represent the values obtained from the relation given in Eq. (31) and
curves denotes the fitting given in Ref. [16, 17]

.

##  4 Discussion

Our work establishes a simple relation between the jump of the Carter constant
and that in the angular momentum caused by the tidal resonance effect, derived
from the fact that the interaction Hamiltonian due to the tidal perturbation
depends on the time independent combination of the orbital phases at the
resonance. The ratio between the change rate of the Carter constant and that
of the the angular momentum is algebraically given by the background values of
the “constants of motion”. Hence, the jump of the Carter constant across the
resonance point can be easily calculated from the jump of the angular
momentum.

Additionally, we validated this new relation by comparing them with the
fitting formula provided in Ref. [16, 17], which is also a consistency check
of our previously obtained fitting formulae. However, even if there were some
mistake in the expressions for the metric perturbation adopted to compute the
tidal force, this proportionality relation is still valid as long as
$dJ_{a}/d\tau$ is correctly calculated from the same metric perturbation. This
means that the applicability of this proportionality relation is not limited
to the case of tidal resonance. An interesting application would be the
evaluation of the resonance jumps in the context of non-GR modification to the
background Kerr geometry [22, 23]. We hope to return to this issue in our
future publication.

In contrast to the tidal resonance case, we cannot use this relation in
evaluating the conservative part of the self-force resonance, because the
self-force resonance appears only when the resonance condition with $m=0$ is
satisfied, and hence the jump of the angular momentum trivially vanishes. The
current discussion does not apply to the dissipative part of the self-force,
either. In this case, to compute $\dot{J}_{a}$, the variation of the
interaction Hamiltonian $H^{(1)}$ with respect to the phase variables $q^{a}$
should be taken first before setting the initial phases of the source orbit to
those of the argument of the perturbation field. Afterward, the long-time
average of the expression for $\dot{J}_{a}$ is taken. Therefore, $\dot{J}_{a}$
is not given by a simple variation of a function of the initial phases.

We thank Soichiro Isoyama for valuable comments as well as Béatrice Bonga for
feedback on our draft. This work was supported by Japan Society for the
Promotion of Science (JSPS) Grants-in-Aid for Scientific Research (KAKENHI)
Grant Numbers JP17H06357, JP17H06358, JP20K03928 (T.T.). PG is supported by
JSPS fellowship and KAKENHI Grant Number 21J15826.

##  Appendix A Explicit expression

Here, we give closed form expressions for ${\partial J_{\theta}}/{\partial
L_{z}}$ and ${\partial J_{\theta}}/{\partial Q}$ requisite for Eq. (31).

$\displaystyle\frac{\partial J_{\theta}}{\partial L_{z}}$
$\displaystyle=\frac{2L_{z}}{\pi
a\sqrt{1-E^{2}}}\int_{0}^{y_{-}}\frac{dy}{\sqrt{(y_{-}^{2}-y^{2})(y_{+}^{2}-y^{2})}}\left\\{1-\frac{1}{1-y^{2}}\right\\}$
(32) $\displaystyle=\frac{2L_{z}}{\pi
a\sqrt{1-E^{2}}\,y_{+}}\left(K\left[\left(\frac{y_{-}}{y_{+}}\right)^{\\!2}\right]-\Pi\left[y_{-}^{2},\left(\frac{y_{-}}{y_{+}}\right)^{\\!2}\right]\right)\,,$
(33) $\displaystyle\frac{\partial J_{\theta}}{\partial Q}$
$\displaystyle=\frac{1}{\pi
a\sqrt{1-E^{2}}}\int_{0}^{y_{-}}\frac{dy}{\sqrt{(y_{-}^{2}-y^{2})(y_{+}^{2}-y^{2})}}$
(34) $\displaystyle=\frac{1}{\pi
a\sqrt{1-E^{2}}\,y_{+}}K\left[\left(\frac{y_{-}}{y_{+}}\right)^{\\!2}\right]\,,$
(35)

where $K[m]$ and $\Pi[n,m]$ are complete elliptic integral of the first and of
the third kind, respectively, defined by

$\displaystyle
K[m]\equiv\int_{0}^{\frac{\pi}{2}}\frac{d\phi}{\sqrt{1-m\sin^{2}\\!\phi}}=\int_{0}^{1}\frac{dx}{\sqrt{(1-x^{2})(1-mx^{2})}}\,,$
(36)
$\displaystyle\Pi[n,m]\equiv\int_{0}^{\frac{\pi}{2}}\frac{d\phi}{(1-n\sin^{2}\\!\phi)\sqrt{1-m\sin^{2}\\!\phi}}=\int_{0}^{1}\frac{dx}{(1-nx^{2})\sqrt{(1-x^{2})(1-mx^{2})}}\,,$
(37)

and $y^{2}_{\pm}$ are given by

$\displaystyle
y_{\pm}^{2}=\frac{a^{2}\left(1-E^{2}\right)+L_{z}^{2}+Q\pm\sqrt{\left(a^{2}\left(1-E^{2}\right)+L_{z}^{2}+Q\right)^{2}+4a^{2}\left(E^{2}-1\right)Q}}{2a^{2}\left(1-E^{2}\right)}\,,$
(38)

which are the values of $\cos^{2}\\!\\!\theta$ at the zeros of the potential
$V_{\theta}$. Since $1-E^{2}>0$ and $Q>0$, we find $y_{+}>y_{-}>0$. In
deriving Eqs. (33) and (35), we have also used the fact that the potential is
written as

$\displaystyle
V_{\theta}(\theta)=a^{2}(1-E^{2})\frac{(y_{+}^{2}-\cos^{2}\\!\theta)(y_{-}^{2}-\cos^{2}\\!\theta)}{\sin^{2}\\!\theta}\,.$
(39)

## References

## References

  * [1] B. P. Abbott et al. [LIGO Scientific and Virgo], Phys. Rev. Lett. 116 (2016) no.6, 061102.
  * [2] P. Amaro. Seoane et al., arXiv:1702.00786 [astro-ph.IM].
  * [3] W. H. Ruan, Z. K. Guo, R. G. Cai and Y. Z. Zhang, Int. J. Mod. Phys. A 35 (2020) no.17
  * [4] J. Mei et al. [TianQin], PTEP 2021 (2021) no.5, 05A107
  * [5] S. Babak, J. Gair, A. Sesana, E. Barausse, C. F. Sopuerta, C. P. L. Berry, E. Berti, P. Amaro-Seoane, A. Petiteau and A. Klein, Phys. Rev. D 95 (2017) no.10, 103012
  * [6] C. P. L. Berry, S. A. Hughes, C. F. Sopuerta, A. J. K. Chua, A. Heffernan, K. Holley-Bockelmann, D. P. Mihaylov, M. C. Miller and A. Sesana, [arXiv:1903.03686 [astro-ph.HE]].
  * [7] E. Barausse, E. Berti, T. Hertog, S. A. Hughes, P. Jetzer, P. Pani, T. P. Sotiriou, N. Tamanini, H. Witek and K. Yagi, et al. Gen. Rel. Grav. 52 (2020) no.8, 81
  * [8] L. Barack and A. Pound, Rept. Prog. Phys. 82 (2019) no.1, 016904
  * [9] A. Pound and B. Wardell, [arXiv:2101.04592 [gr-qc]].
  * [10] Y. Mino, M. Sasaki and T. Tanaka, Phys. Rev. D 55 (1997), 3457-3476
  * [11] T. C. Quinn and R. M. Wald, Phys. Rev. D 56 (1997), 3381-3394
  * [12] E. E. Flanagan and T. Hinderer, Phys. Rev. Lett. 109 (2012), 071102
  * [13] É. E. Flanagan, S. A. Hughes and U. Ruangsri, Phys. Rev. D 89 (2014) no.8, 084028
  * [14] C. P. L. Berry, R. H. Cole, P. Cañizares and J. R. Gair, Phys. Rev. D 94 (2016) no.12, 124042
  * [15] B. Bonga, H. Yang and S. A. Hughes, Phys. Rev. Lett. 123 (2019) no.10, 101103
  * [16] P. Gupta, B. Bonga, A. J. K. Chua and T. Tanaka, Phys. Rev. D 104 (2021) no.4, 044056
  * [17] P. Gupta, L. Speri, B. Bonga, A. J. K. Chua and T. Tanaka, [arXiv:2205.04808 [gr-qc]].
  * [18] M. Silva and C. Hirata, [arXiv:2207.07733 [gr-qc]].
  * [19] T. Hinderer and E. E. Flanagan, Phys. Rev. D 78 (2008), 064028
  * [20] S. Isoyama, R. Fujita, H. Nakano, N. Sago and T. Tanaka, PTEP 2013 (2013) no.6, 063E01
  * [21] Z. Nasipak, [arXiv:2207.02224 [gr-qc]].
  * [22] G. Lukes-Gerakopoulos, T. A. Apostolatos and G. Contopoulos, Phys. Rev. D 81 (2010), 124005
  * [23] J. R. Gair, C. Li and I. Mandel, Phys. Rev. D 77 (2008), 024035


[1]Matouš Kozák

[1]Faculty of Information Technology, Czech Technical University in Prague,
Prague, Czechia

2]Department of Computer Science, San Jose State University, San Jose,
California, USA

# A Comparison of Adversarial Learning Techniques for Malware Detection

Pavla Louthánová<EMAIL_ADDRESS><EMAIL_ADDRESS>Martin Jureček
<EMAIL_ADDRESS>Mark Stamp<EMAIL_ADDRESS>*  [

###### Abstract

Machine learning has proven to be a useful tool for automated malware
detection, but machine learning models have also been shown to be vulnerable
to adversarial attacks. This article addresses the problem of generating
adversarial malware samples, specifically malicious Windows Portable
Executable files. We summarize and compare work that has focused on
adversarial machine learning for malware detection. We use gradient-based,
evolutionary algorithm-based, and reinforcement-based methods to generate
adversarial samples, and then test the generated samples against selected
antivirus products. We compare the selected methods in terms of accuracy and
practical applicability. The results show that applying optimized
modifications to previously detected malware can lead to incorrect
classification of the file as benign. It is also known that generated malware
samples can be successfully used against detection models other than those
used to generate them and that using combinations of generators can create new
samples that evade detection. Experiments show that the Gym-malware generator,
which uses a reinforcement learning approach, has the greatest practical
potential. This generator achieved an average sample generation time of 5.73
seconds and the highest average evasion rate of 44.11%. Using the Gym-malware
generator in combination with itself improved the evasion rate to 58.35%.

###### keywords:

Adversarial Examples, Malware Detection, Machine Learning, PE Files

##  1 Introduction

With the rapid development of information technology, computer systems have
become increasingly important in the daily lives of people. Unfortunately, the
rapid development of these technologies is accompanied by a similarly rapid
increase in cyberattacks.

Malicious software (malware) is one of the most significant security threats
today, comprising several different categories of malicious code, such as
viruses, trojans, worms, spyware, and ransomware. To protect computers and the
Internet from malware, early detection is necessary. However, this is
problematic, as a large amount of new malicious code is generated every day
[1]. Since it is not possible to analyze each sample individually, automatic
mechanisms are required to detect malware.

Antivirus companies often rely mainly on signature-based detection techniques
[2] for malware detection. Signatures are specific patterns that allow for the
recognition of malicious files. For example, they can be a byte sequence, a
file hash, or a string. When inspecting a file, the antivirus system compares
its content with the signatures of already-known malware stored in the
database. If a match is found, the file is reported as malware. Signature-
based detection methods are fast and effective in detecting known malware.
However, malware authors can modify their code to change the signature of the
program, thereby avoiding detection. Some malware can hide in the system using
various obfuscation techniques [3], such as encryption, oligomorphic,
polymorphic, metamorphic, stealth, and packing methods, to make the detection
process more difficult.

Machine learning (ML) models are commonly used today in various fields. Their
application can be found, for example, in technologies such as self-driving
cars, weather forecasting, face recognition, or language translation systems.
Machine learning has also proved to be a useful tool for automatic malware
detection [4]. Unlike the signature-based method, it is capable of detecting
previously unknown or obfuscated malware. However, it can be difficult to
explain why the model classifies a certain file as malicious or benign [5],
which can cause hidden vulnerabilities that attackers can exploit.

Machine learning models are vulnerable to adversarial attacks [6]. Attackers
purposely design adversarial examples, which are deliberately designed inputs
to a machine learning model, to cause the model to make a mistake in its
predictions. Adversarial machine learning is a field that deals with attacks
on machine learning algorithms and defenses against such attacks.

Malware detection is thus a battle between defenders and malware authors, in
which each side attempts to devise new and effective ways to outwit the other.
Each detection method has its own advantages and disadvantages. In various
scenarios, one method may be more successful than another. Thus, the creation
of an effective malware detection method is a very challenging task, and new
research and methods are necessary.

The main contributions of this paper are to compare works that focus on
adversarial machine learning in the area of malware detection. Specifically,

  * •

we applied some existing methods in the field of adversarial learning to
selected malware detection systems.

  * •

we combined these methods to create more sophisticated adversarial generators
capable of bypassing top-tier AV products.

  * •

we evaluated the single and combined generators in terms of accuracy and
usability in practice.

The rest of the paper is organized as follows: In Section 2, we describe
state-of-the-art techniques used to generate adversarial examples present.
Section 3 provides an overview of the publications focused on creating
adversarial portable executable malware samples. Section 4 describes the
experiments performed and the metrics used for evaluation. Section 5 presents
and discusses the experimental results, and Section 6 concludes this work.

##  2 Background

In this section, we describe the different methods used to create adversarial
examples. We also introduce and describe selected attacks for experimentation.

###  2.1 Methods for Creating Adversarial Examples

In this section, we describe various methods to create adversarial examples.

####  2.1.1 Gradient-based Approaches

Gradient-based methods are a popular approach to generate adversarial
examples. These methods work by computing the gradient of a loss function with
respect to the input data. This gradient is then used to iteratively modify
the input to minimize the loss. The Fast Gradient Sign Method [7] and the
Jacobian-based Saliency Map Approach [8] are two popular gradient-based
methods used for malware generation.

Given a trained model $f$ and an input example $x$, gradient-based methods
generate an adversarial example $x^{\prime}$ by adding a small perturbation
$\delta$ to the input that maximizes the loss function $L(f(x^{\prime}),y)$,
where $y$ is the true input label.

The perturbation $\delta$ is calculated as follows:

$\displaystyle\delta=\epsilon\cdot\textrm{sign}(\nabla_{x}L(f(x),y))$

where $\epsilon$ is a small constant that controls the size of the
perturbation, and the sign function is used to ensure that the perturbation
has the same sign as the gradient, allowing efficient computation and ensuring
that the perturbation always increases the loss.

The gradient of the loss function with respect to the input
$(\nabla_{x}L(f(x),y))$ is computed by backpropagation through the model $f$.
This gradient gives the direction in which the loss function increases the
most for a small change in the input and is used to determine the direction of
the perturbation.

Gradient-based attacks are performed using the addition or insertion method
for perturbations generated using the gradient of the cost function. When
using the append method, the data (payload) is appended at the end of the
file. When the insertion method is used, the payload is inserted into the
slack region where the physical size is greater than the virtual size.

####  2.1.2 Generative Adversarial Network-based Approaches

Generative adversarial networks (GANs) were developed and presented by
Goodfellow et al. [9] in 2014.

GAN is a system consisting of two neural networks, a generator and a
discriminator, which compete against each other. The goal of the generator is
to create examples that are indistinguishable from the real examples in the
training set, thus fooling the discriminator. In contrast, the objective of
the discriminator is to distinguish the false examples produced by the
generator from the real examples that come from the training data set, thus
preventing it from being fooled by the generator. The generator learns from
the feedback it receives from the discriminator’s classification [10].

These two neural networks are trained simultaneously. The generator is
constantly improving its ability to generate realistic samples, so the
discriminator must continually improve its ability to distinguish between real
and generated samples. This mutual competition forces both networks to
continuously improve through the training process. Once this training process
is completed, the generator can be used to generate new samples that are
indistinguishable from the real samples [11].

Denote the generator as $G$ and the discriminator as $D$. As described in [9],
networks $G$ and $D$ play the following two-player minimax game with value
function $V(G,D)$:

$\displaystyle\underset{G}{\text{min }}\underset{D}{\text{max
}}V(D,G)=\mathbb{E}_{x\sim p_{data}(x)}[logD(x)]$
$\displaystyle+\mathbb{E}_{z\sim p_{z}(z)}[log(1-D(G(z)))]$

that $G$ tries to minimize, while $D$ tries to maximize. $D(x)$ is the
discriminator’s estimate of the probability that the original data $x$ is
real, $G(z)$ is the generator’s output when it receives noise $z$ as input,
$D(G(z))$ is the discriminator’s estimate of the probability that a synthetic
sample $G(z)$ of data is real, $\mathbb{E}_{x}$ is the expected value over all
real data instances, and $\mathbb{E}_{z}$ is the expected value over all
generated fake instances.

####  2.1.3 Reinforcement Learning-based Approaches

Reinforcement learning (RL) is a type of machine learning technique along with
supervised and unsupervised machine learning. In supervised machine learning,
the model is trained using a training set of labeled examples. Based on the
given inputs and expected outputs, the model creates a mapping equation that
the model can use to predict the labels of the inputs in the future. In
unsupervised machine learning, the model is trained only on inputs without
labels. The model divides the input data into classes that have similar
properties, and during prediction, the inputs are labeled based on the
similarity of their properties to one of the classes. Unlike supervised and
unsupervised machine learning, reinforcement learning algorithms learn by
interacting with an environment and getting feedback in the form of rewards or
penalties rather than relying on pre-labeled instances.

A reinforcement learning model consists of two main parts: an agent and an
environment. The agent learns to perform a task through repeated interactions
with the environment through trial and error. In addition to the agent and the
environment, the reinforcement learning model has four main subelements: a
policy, a reward signal, a value function and, optionally, an environment
model. The policy defines the behavior of the agent at a particular time. In
other words, it is a strategy that the agent uses to determine the next action
based on the current state to achieve the highest reward. The reward signal is
the feedback from the environment to the agent, indicating the success or
failure of the agent’s action in a given state. At each time step, the agent
is in some state and sends the selected action as its output to the
environment, which then returns a new state to the agent along with a reward
signal. While the reward signal shows what is beneficial in the present, the
value function describes what is beneficial in the long term. The value
function provides an estimate of the expected cumulative reward from the
current state of the environment in the future. The agent’s objective is to
maximize the total reward. The environment model mimics the behavior of the
environment, making it possible to predict future states and rewards. This is
an optional part of the system [12].

Reinforcement learning is defined as repeated interactions between an agent
and an environment. Individual interactions (signals exchange) are performed
in time steps. At time step $t$, the environment is in state $s_{t}\in S$,
where $S$ is the set of all possible states in the environment. The agent
receives state $s_{t}$ and then chooses action $a_{t}\in A$ based on policy
$\pi$, where $A$ is the set of all possible actions defined by the
environment. After the environment receives information about the chosen
action $a_{t}$ from the agent, it calculates the reward
$r_{t}=R(s_{t},a_{t},s_{t+1})$ and sends it to the agent in the form of
feedback. At the same time, the environment transitions to a new state
$s_{t+1}$. When this cycle is complete, we say that one time step has passed.
This cycle can repeat forever or end when it reaches a terminal state or a
maximum time step $t=T$. We call the triplet of signals $(s_{t},a_{t},r_{t})$
an experience. The time elapsed between $t=0$ and the end of the environment
is called an episode. A trajectory is a sequence of experiences during an
episode, $\tau=(s_{0},a_{0},r_{0}),(s_{1},a_{1},r_{1}),\dots$ [12].

In the field of creating adversarial malware samples, interactions between the
agent and the environment occur in discrete time steps. The agent has a set of
operations available for modifying PE files while maintaining the
functionality of the malware. The goal of the agent is to perform a sequence
of operations on the malware to prevent its detection.

####  2.1.4 Evolutionary Algorithm-based Approaches

Evolutionary algorithms (EA) are a useful tool for solving optimization
problems. They are based on the Darwinian principle of evolution and attempt
to mimic these processes. The search for the best or at least satisfactory
solution to a problem takes the form of competition between gradually
developing solutions within the population. Variants of EA include, for
example, evolutionary strategies, evolutionary programming, genetic
algorithms, and genetic programming. All of these variants share the same
principle of operation but differ in their implementation.

When solving a problem, it is necessary first to define the representation of
candidate solutions. These candidate solutions are called individuals or
phenotypes. Since the phenotype can have a complex structure, an encoding is
used to represent the individuals in an appropriate way, which is called a
chromosome or genotype. Next, an initial population of individuals is created,
with each individual representing a coded solution. Then, each member of the
population is evaluated using a fitness function that numerically expresses
the quality of the solution. The individuals with the best score are then
selected and used to create a new generation. This is done using the crossover
operator, which usually takes pairs of chromosomes and exchanges information
between them to create new offspring. This is followed by the mutation
operator, which changes a small portion of the offspring so that it is no
longer just a mixture of parental genes. This introduces new genetic material
into the new generation. This entire cycle (fitness evaluation, selection,
crossover, mutation) is repeated until the termination condition is reached.

###  2.2 Selected Attacks

To generate samples of malicious software, we utilized three distinct
techniques: gradient-based techniques, evolutionary algorithm-based
techniques, and reinforcement learning-based techniques. Specifically, we
selected Partial DOS [13] and Full DOS [14] attacks from gradient-based
techniques. From the evolutionary algorithms-based techniques, we chose GAMMA
padding [15] and GAMMA section-injection [15] attacks. Finally, for
reinforcement learning techniques, we selected the Gym-malware [16] attack. In
this section, we describe each of these attacks in detail.

Partial DOS attack and Full DOS attack focus on modifying bytes in the DOS
header of a portable executable (PE) file. The DOS header contains only two
important pieces of information. The initial 2 bytes represent the magic
number, while the final 4 bytes at offset 0x3C show the location of the PE
signature in the NT headers. Partial DOS attacks modify only bytes within the
range of 0x02 to 0x3B, inclusive. Full DOS attacks expand this range to
include all bytes up to the PE signature. The position of this signature may
vary in individual files, but it can be found at offset 0x3C.

GAMMA padding attack and GAMMA section-injection attack are based on inserting
parts extracted from benign files into malware files. These are black-box
attacks. Gamma attacks are formalized as a constrained optimization problem.
The goal is to minimize the probability of detection but also to minimize the
size of the injected content. This optimization problem is solved using a
genetic optimizer. First, a random matrix is created that represents the
initial population of manipulation vectors. The algorithm then iterates in
three steps: selection, crossover, and mutation. During selection, the
objective function is evaluated and the $N$ best candidate manipulation
vectors from the current population and the population created in the previous
iteration are selected. This is followed by the crossover function, where the
candidates from the previous step are modified by mixing the values of pairs
of randomly selected candidate vectors and a new set of $N$ candidates is
returned. The last operation is a mutation, which randomly changes the
elements of each vector with a low probability. At each iteration, $N$ queries
are made to the target model to evaluate the objective function on new
candidates and keep the best candidate population. When the maximum number of
queries is reached or no further improvement in the objective function value
is observed, the best manipulation vector from the current population is
returned. The resulting adversarial malware sample is obtained by applying the
optimal manipulation vector to the input malware sample through the
manipulation operator.

The Gym-malware attack is based on reinforcement learning. The environment
consists of a sample of malware and the attack target, which is an anti-
malware engine. At each step, the agent receives feedback that is composed of
a reward value and a vector of features that summarize the state of the
environment. Based on the feedback, the agent selects mutations from a set of
actions, such as adding a function to the import address table that will never
be used; manipulating existing section names; creating new sections that will
not be used; adding bytes to extra space at the end of sections; creating a
new entry point that immediately jumps to the original entry point;
manipulating debug information; packing or unpacking the file; modifying the
header checksum; adding bytes to the end of the PE file. This process is
repeated in several rounds, and rounds can be prematurely terminated if the
agent bypasses the anti-malware engine before 10 mutations are completed.

##  3 Related Work

In this section, the publications on modern methods to create adversarial
examples are summarized. The section is divided into several parts, depending
on the area to which the method belongs, with all publications compared based
on the attacker’s knowledge and strategy in Table 1.

Table 1: Summary of State-of-the-Art Adversarial Attacks against PE Malware Detection. Paper | Year | Attack framework | Knowledge | Attack Strategy
---|---|---|---|---
[17] | 2019 | AIMED | black-box | evolutionary algorithm
[18] | 2020 | MDEA | black-box | evolutionary algorithm
[15] | 2021 | GAMMA | black-box | evolutionary algorithm
[16] | 2018 | Gym-malware | black-box | reinforcement learning
[19] | 2019 | DQEAF | black-box | reinforcement learning
[20] | 2020 | MAB-malware | black-box | reinforcement learning
[21] | 2021 | AIMED-RL | black-box | reinforcement learning
[22] | 2018 | AMB | white-box | gradient
[23] | 2018 | – | white-box | gradient
[13] | 2019 | – | white-box | gradient
[24] | 2018 | – | white-box | gradient
[14] | 2020 | RAMEN | white-box | gradient
[25] | 2017 | MalGAN | black-box | GAN
[26] | 2019 | Improved-MalGAN | black-box | GAN
[27] | 2020 | GAPGAN | black-box | GAN

###  3.1 Evolutionary Algorithm-based Attacks

In [17], an AIMED system was designed and implemented to generate adversarial
examples using genetic programming by Castro et al. The system enables the
automatic finding of optimized modifications that are applied to previously
detected malware and lead to its incorrect evaluation by the malware
classifier. It is ensured that all generated adversarial examples are valid PE
files. The system implements genetic operations such as selection, crossover,
and mutation. If modified PE malware cannot bypass the malware detector,
genetic operations are repeated until the generated adversary PE malware can
bypass the malware classifier. Experiments have shown that the time to
generate successful adversarial examples is reduced by up to 50% compared to
random approaches. Furthermore, adversarial examples generated using the given
malware classifier were shown to be successful against other malware detectors
in 82% of cases.

Wang and Miikkulainen proposed an adversarial malware detection model named
MDEA [18]. This model combines the convolutional neural network to classify
raw data from malicious binary and evolutionary optimization to modify
detected malware. The action space consists of 10 different methods to modify
binary programs. The genetic algorithm evolves different action sequences by
selecting actions from the action space until the generated adversarial
malware can bypass the target malware detectors. After the successful
discovery of action sequences, these sequences are applied to the
corresponding malware samples and create a new training set for the detection
model. Unlike AIMED, malware samples generated by MDEA are not tested for
functionality.

Demetrio et al. introduced a black-box attack framework called GAMMA [15]. The
black-box attack is the most challenging case since the attacker knows nothing
about the target classifier besides the final prediction label. GAMMA attacks
are based on the principle of injecting harmless content extracted from
goodware into malicious files. Harmless content is inserted into some newly
created section or at the end of the file, while the functionality of the file
is preserved. The attack is formalized as a constrained optimization problem
that minimizes the probability of escaping detection and also penalizes the
size of the injected content through a specific penalty.

###  3.2 Reinforcement Learning-based Attacks

Anderson et al. focused on automating the manipulation of malicious PE files
in [16]. The goal is to modify the original malicious PE file so that it is no
longer detected as malicious and, at the same time, its format and
functionality are not violated. They proposed an attack known as Gym-malware.
This is a black-box attack based on reinforcement learning. The authors
defined the RL agent’s action space as a set of binary manipulation actions.
Over time, the agent learns which combinations of actions make malware more
likely to bypass antivirus systems.

Song et al. proposed a MAB-Malware framework [20] based on reinforcement
learning to generate adversarial PE malware examples. The action selection
problem is modeled as a multi-armed bandit problem. The results showed that
the MAB-Malware framework achieves an evasion rate of 74% to 97% against
machine learning detectors (EMBER [28] and MalConv [29]) and an evasion rate
of 32% to 48% against commercial antivirus (AV). Furthermore, they also showed
that the transferability of adversarial attacks between ML-based classifiers
(i.e., adversarial examples generated against one classifier can be used
successfully against another) is greater than 80%, and the transferability of
attacks between pure ML and commercial AV is only up to 7%.

Fang et al. proposed a framework named DQEAF [19] that uses reinforcement
learning to evade antimalware engines. DQEAF is similar in methodology to Gym-
malware but has many benefits and a higher evasion success rate of adversarial
examples compared to it. DQEAF uses a subset of modifications used in Gym-
malware and ensures that all modifications will not cause corruption in the
modified malware files. DQEAF is able to reduce instability caused by higher
dimensions by representing executable files with only 513 features, which is
much lower than that in Gym-malware. DQEAF takes priority into account when
replaying past transitions. This helps to replay higher-value transitions more
frequently, and thus optimize RL networks.

Another reinforcement learning approach is presented in [21], in which Labaca-
Castro et al. presented the AIMED-RL adversarial attack framework. This attack
can generate adversarial examples that lead machine learning models to
misclassify malicious files without compromising their functionality. The
authors demonstrated the importance of a penalty technique and introduced a
new penalization for the reward function with the aim of increasing the
diversity of the generated sequences of modifications while minimizing the
number of modifications. The results showed that the agents with penalty
outperform the agents without penalty in terms of both the best and the
average evasion rates.

###  3.3 Gradient-based Attacks

Kolosnjaji et al. in [22] introduced a gradient-based white-box attack to
generate adversarial malware binaries against MalConv. A white-box scenario
occurs when the attacker gets access to the system and may examine its
internal configuration or training datasets. The basic idea of the attack is
to manipulate some bytes of each malicious software to maximize the likelihood
that the input samples are classified as benign. To ensure that malicious
binary functionality is maintained, only padding bytes at the end of the file
are considered. The results show that the evasion rate is linearly correlated
with the length of the injected sequence and, despite the fact that less than
1% of the bytes are modified, the modified binary evades the target network
with high probability and that the precision of MalConv can be reduced by more
than 50%.

In [23] Kreuk et al. presented an improved gradient-based white-box attack
method against MalConv. The authors proposed two methods for inserting a
sequence of bytes into a file; the payload is inserted either at the end of
the file or into slack regions. Unlike [22], the evasion rate of [23] is
invariant to the length of the injected sequence.

Demetrio et al. in [13] presented a gradient-based variant white-box attack
that is similar to [22]. The main difference is that [22] injects adversarial
bytes to the end of the PE file, while this attack is limited to changing
bytes within a specific disk operating system (DOS) header in the PE header.
The results show that a change of a few bytes is sufficient to evade MalConv
with high probability.

Suciu et al. in [24] describe the FGM append and the FGM slack attacks and
compare their effectiveness against MalConv. Experimental results show that
the FGM slack attack achieves better results than the FGM append attack with
fewer modified bytes.

Demetrio et al. in [14] propose RAMEN, a general adversarial attack framework
against PE malware detectors. This framework generalizes and includes previous
attacks against machine learning models, as well as three new attacks based on
manipulations of the PE file format that preserve its functionality. The first
attack is a full DOS attack, which edits all the available bytes in the DOS
header. The second attack is called Extend, which enlarges the DOS header,
thus enabling manipulation of the extra DOS bytes. The third is the Shift
attack, which shifts the content of the first section, creating additional
space for the adversarial payload.

###  3.4 Generative Adversarial Network-based Attacks

Hu and Tan proposed in [25] a model called MalGAN, which is based on a
generative adversarial network (GAN). This model enables the generation of
adversarial malware examples that are capable of bypassing black-box ML-based
detection models. MalGAN includes two neural networks: a generator and a
substitute detector. The generator is used to generate adversarial examples,
which are dynamically generated according to the feedback of the malware
detector and are able to fool the substitute detector. The results showed that
MalGAN can reduce the accuracy of detector to almost zero.

Later, Kawai et al. presented Improved MalGAN in [26]. The authors discuss the
problems of MalGAN and try to improve them. For example, the original MalGAN
uses multiple malware samples to train MalGAN, in contrast, the improved
MalGAN uses only one malware sample. Furthermore, the original MalGAN trains
the generator and the malware detector with the same application programming
interface (API) call list, while the Improved MalGAN trains with different API
call lists.

Yuan et al. in [27] introduced GAPGAN, a GAN-based black-box adversarial
attack framework. GAPGAN allows end-to-end black-box attacks at the byte level
against deep learning-based malware binaries detection. In this approach, a
generator and a discriminator are trained concurrently. The generator is used
to generate adversarial payloads that are appended to the end of the original
data to craft a malware adversarial sample while ensuring the preservation of
their functionality. The discriminator tries to imitate the black-box malware
detector to recognize both the original benign samples and the adversarial
samples generated. When the training process is completed, the trained
generator is able to generate an adversarial sample in less than 20
milliseconds. Experiments show that GAPGAN is capable of achieving a 100%
attack success rate against the MalConv malware detector by only inserting
adversarial payloads with the size of 2.5% of the total length of the input
malware samples.

##  4 Experiments

This section describes the setup and procedure for each experiment. First, the
hardware configuration is introduced, followed by a description of the
datasets used. Next, the setup of the different algorithms used to generate
adversarial samples is described. Finally, the experiments performed are
described.

###  4.1 Experimental Setup

The experiments are carried out on the NVIDIA DGX Station A100 server. This
server is equipped with an AMD EPYC 7742 processor with a base frequency of
2.25 GHz, 64 cores and 512 GiB of RAM. We also use a virtual machine with
Windows 11 operating system and another virtual machine with Kali Linux
operating system for testing and analysis.

We use two datasets for our experiments. The first dataset contains a total of
3,625 harmless executable files, which were collected from a newly installed
Windows 11 system. The second dataset contains 3,625 malicious executable
files obtained from the VirusShare repository [30]. All the files in both
datasets are PE files.

###  4.2 Attack settings

In total, we compare five adversarial attack strategies. Partial DOS and Full
DOS attacks are performed in a white-box setting against the MalConv detector
and the maximum number of iterations was set to 50. GAMMA padding and GAMMA
section-injection attacks are performed in a black-box setting against the
MalConv detector with the maximum number of queries set to 500, regularization
parameter set to $10^{-5}$, and a total of 100 .data sections extracted from
benign programs used as injection content. The last attack we use is the Gym-
malware attack with its default settings performed in a black-box setting
against the GDBT detector. The Gym-malware model was trained on a dataset of
3,000 malicious samples and a validation set of 1,000 files.

###  4.3 Evaluation Metrics

We use several metrics to evaluate the experiments. A key metric in the area
of adversarial machine learning is the evasion rate ($ER$), the proportion of
malware files misclassified by the target malware classifier, and can be
calculated as follows:

$ER=\frac{misclassified}{total}$ (1)

where $misclassified$ is the number of malware samples misclassified as benign
and $total$ is the total number of files submitted to the target classifier
after discarding files that were already incorrectly predicted before
modification.

The evasion rate mentioned above is a universal metric that can be used to
evaluate both single attacks and combinations of attacks. In both cases, we
are interested in the percentage of malware that escaped detection by the
antivirus program. Additionally, we use the following metrics to evaluate the
combination of attacks.

The first two metrics that we chose to evaluate the success of the combination
are the absolute improvement and the relative improvement in the evasion rate
when using the second attack in the combination compared to the first attack.
Absolute improvement ($AI$) can be described by the following formula:

$AI=ER_{C}-ER_{1}$ (2)

where $ER_{C}$ is the total evasion rate when using a combination of methods,
and $ER_{1}$ is the evasion rate after using the first attack in the
combination alone. The result is the percentage increase in evasion rate
between the first and second attack in the combination. For example, if the
evasion rate after the first attack in the combination is $0.01$ and after the
second attack is $0.1$, then the absolute improvement is $0.09$, which means
that the second attack improved the overall evasion rate by $9\%$.

Similarly, the relative improvement ($RI$) can be expressed using the formula:

$RI=\frac{ER_{C}-ER_{1}}{ER_{1}}$ (3)

where the meaning of the variables is the same as in the previous formula (2).
However, in the previous case, the result expressed a percentage increase over
all samples tested. In the case of relative improvement, we limit ourselves to
the set of samples that escaped the antivirus program after the first attack.
For example, if the evasion rate after the first attack of the combination is
$0.01$ and after the second attack it is $0.1$, then the relative improvement
is $9$, i.e. the second attack improved the evasion rate of the first attack
by $900\%$.

Next, we need to compare the combination of attacks with performing the
attacks separately to see if the combination of attacks adds any value. To do
this, we use a simple comparison of the evasion rate of the combination of
attacks with the evasion rate of the attacks performed separately. We call it
evasion rate comparison ($ERC$), and it has the following calculation:

$ERC=ER_{C}-\max(ER_{1},ER_{2})$ (4)

where $ER_{C}$ is the evasion rate of the combination attack, while $ER_{1}$
and $ER_{2}$ are the evasion rates of the first and second attacks,
respectively. If the result is positive, it means that the combination of
attacks performed better than the combination of the two attacks in the
combination that would have been performed alone. If the result is negative or
zero, it means that the execution of the attack combination was pointless
because one of the attacks that were part of the combination performed better
or was equal to the combination in terms of evasion rate.

###  4.4 Experiments Description

We present four experiments that explore different characteristics of the
adversarial attack methods mentioned above.

####  4.4.1 Sample Generation Time

In the first experiment, we measure the time it took to generate individual
samples using all the aforementioned selected algorithms. These results, along
with other data collected during the experiments, can help compare the
effectiveness of various generators and determine the most effective method to
generate adversarial malware samples.

####  4.4.2 Sample Size

In the second experiment, we investigate how the size of the original malware
samples changes by applying various adversarial malware generators. Generally,
the attacker’s goal is to minimize the increase in the size of the generated
adversarial files to make it harder to distinguish them from the original
malware samples.

####  4.4.3 Bypassing commercial AV products

In the third experiment, we analyze the effectiveness of created adversarial
malware samples against real-world AV detectors. Based on a comparative study
[31], 10 AV programs were selected for experimentation, and their names were
intentionally anonymized in the following results. Note that in the subsequent
results, only nine AVs are listed as two of the selected AVs reported the same
results.

The modified malware files from different adversarial algorithms are submitted
to the VirusTotal server [32] to obtain the evasion rate for each adversarial
malware generator. To avoid bias in the results, we only analyzed malware
samples that were correctly classified by all selected AV products before
modification. We also discarded samples from which we were unable to obtain
file analysis from VirusTotal, e.g., due to broken behavior of modified
examples. In total, we use a set of 530 genuine malware samples along with a
modified version for each malware generator.

####  4.4.4 Combination of Multiple Techniques

In the last experiment, we test the effectiveness of using a combination of
methods to generate malware samples [33]. The goal of this experiment is to
test whether using multiple adversarial example generators per malware sample
would significantly increase the malware evasion rate.

An overview of the experiment is shown in Figure 1. First, the original
malware samples are processed by the first generator. These modified samples
are then tested against a real AV detector that is not part of the generator.
The result is a set of samples divided into two sets. The first set consists
of evasive examples that successfully evaded the given malware detector, and
this set is no longer processed. In contrast to adversarial examples, which
are generated against the target classifier, evasive examples are samples that
have evaded detection by the AV program, although this AV program was not used
to generate these samples. The second set consists of failed examples that
failed to evade the detector and are used as input to the second generator.
The second generator processes the failed examples, and the resulting modified
samples are again tested against real the AV detector. The result is again a
set of samples divided into two sets: evasive and failed. The set of evasive
examples produced by the first generator and the set of evasive examples
produced by the second generator together form the set of resulting successful
adversarial examples produced by combining these two generators. The failed
examples obtained after using the second generator are the resulting samples
that did not evade detection.

Figure 1: Method for generating adversarial examples by combining two
generators.

##  5 Results

This section presents the recorded results of the individual experiments
described in Section 4.

###  5.1 Sample generation time

Firstly, we look at the results of the sample generation time. Average times
in seconds and standard deviation for each attack are listed in Table 2 and in
the box plot in Figure 2.

Figure 2: Time required to generate a sample for each sample generator. Table 2: Average sample generation time for each sample generator. Attack | Average Duration [s] | Standard Deviation [s]
---|---|---
Partial DOS | $99.27$ | $31.13$
Full DOS | $169.08$ | $104.53$
GAMMA Padding | $87.61$ | $39.28$
GAMMA Section-injection | $118.47$ | $69.44$
Gym-malware | $5.73$ | $7.52$

From Figure 2, we can see that Gym-malware took the least amount of time, on
average less than 6 seconds, to generate adversarial examples with some
outliers below 100 seconds. It should be noted that Gym-malware requires a
preceding training phase that is not taken into account in this experiment. On
the contrary, the Full DOS attack recorded the longest time to create
adversarial examples, with an average duration of more than 160 seconds. The
remaining three methods achieved similar sample generation times of around 100
seconds in spite of the fact that the results of the GAMMA section-injection
attack contain several outliers with extensive long durations of more than 300
seconds. However, the measured times may be affected by the settings of
individual algorithms.

###  5.2 Sample Size

Secondly, we present the results of the sample size experiment. Partial DOS
and Full DOS attacks are based on changing bytes in the DOS header. Thus,
modifying the malware samples by these methods does not alter the size of the
resulting adversarial examples. On the other hand, the remaining tested
methods can change the initial file size. The general results of this
experiment are shown in Table 3.

The GAMMA padding and section-injection attacks are based on inserting parts
extracted from benign files into the malware file. In the case of the GAMMA
padding attack, the file size increased on average by 223,605 bytes and in the
case of the GAMMA section-injection attack, by 1,940,352 bytes. As we can see
from Table 3, the adversarial examples generated by the GAMMA section-
injection attack exhibit significant differences in the final file sizes.

The Gym-malware attack uses various types of file manipulations. The file size
can be reduced, increased, or unchanged based on the chosen modification. On
average, the file size was reduced by 149,273 bytes. The observed file size
reduction was probably due to the authors’ implementation of file
manipulations, as they used the LIEF library [34], which significantly alters
the structure of the initial malware file.

Table 3: Changes to the sample size of generated adversarial samples for each generator. Attack | Average Size [B] | Standard Deviation [B]
---|---|---
Partial DOS | $0$ | $0$
Full DOS | $0$ | $0$
GAMMA Padding | $223,604.57$ | $48,403.25$
GAMMA Section-injection | $1,940,351.63$ | $78,088,981.28$
Gym-malware | $-149,272.75$ | $754,660.5$

###  5.3 Bypassing commercial AV products

Table 4: Evasion rate (in %) of generated adversarial samples achieved against
AV products on the VirusTotal server.

Attack | AV1 | AV2 | AV3 | AV4 | AV5 | AV6 | AV7 | AV8 | AV9 | Average
---|---|---|---|---|---|---|---|---|---|---
GAMMA padding | 0.00 | 1.79 | 0.45 | 0.22 | 0.45 | 0.90 | 1.34 | 0.56 | 0.45 | 0.68
Partial DOS | 0.78 | 2.57 | 0.78 | 1.01 | 0.78 | 0.78 | 1.90 | 1.45 | 0.78 | 1.21
Full DOS | 0.67 | 1.34 | 0.78 | 0.90 | 0.78 | 0.78 | 4.14 | 1.23 | 0.78 | 1.27
GAMMA section-injection | 18.46 | 5.37 | 6.38 | 4.36 | 4.47 | 9.06 | 43.62 | 1.23 | 5.37 | 10.92
Gym-malware | 45.53 | 19.02 | 44.86 | 67.23 | 41.61 | 53.58 | 53.80 | 26.51 | 44.86 | 44.11

Thirdly, we list the effectiveness of generated adversarial samples on
commercially available AV programs. The results are shown in Table 4. Each
column represents one of the selected AV programs and each row represents one
of the algorithms used to generate adversarial samples. The values in the
table represent the achieved evasion rates, expressed as a percentage, for the
corresponding algorithm and AV products.

The Gym-malware attack achieved the highest evasion rates among all selected
AV programs, successfully bypassing the top AVs 19.02% to 67.23% of the time.
The second-best results were recorded by the GAMMA section-injection attack,
which recorded evasion rates between 1.23% and 43.62%.

In contrast, the GAMMA padding attack achieved the worst results, failing to
mislead any detector tested in more than 1.5% of cases. Full and Partial DOS
attacks scored slightly better than the GAMMA padding attack, with the Full
DOS marginally outperforming the Partial DOS attack.

###  5.4 Combination of Multiple Techniques

Based on the results from the previous experiment, we chose the three most
successful adversarial example generators and tested all nine possible
combinations. Namely, we used Gym-malware, GAMMA section-injection, and Full
DOS adversarial malware generators.

This section contains two types of tables. The first type of table lists the
measured minimum, average, and maximum values for particular AVs across the
nine combinations of the three selected generators. The second type of table
lists the measured minimum, average, and maximum values for a particular
combination of generators across the nine AVs. The First Generator column
contains the first generator used, and the Second Generator column contains
the second generator used as described in Section 4.4.4.

#### Evasion Rate

Table 5: Evasion rate (in %) for each AV using all combinations of generators. AV | Minimum | Average | Maximum
---|---|---|---
AV1 | 0.78 | 32.39 | 55.26
AV2 | 1.45 | 17.79 | 29.53
AV3 | 0.90 | 31.15 | 63.09
AV4 | 1.45 | 38.59 | 78.19
AV5 | 0.78 | 26.76 | 57.61
AV6 | 0.90 | 35.91 | 73.60
AV7 | 5.26 | 49.32 | 74.50
AV8 | 1.57 | 17.80 | 41.39
AV9 | 0.78 | 30.79 | 62.75
Table 6: Evasion rate (in %) for each generator combination against all AVs. First Generator | Second Generator | Minimum | Average | Maximum
---|---|---|---|---
Full DOS | Full DOS | 0.78 | 1.54 | 5.26
Full DOS | GAMMA section-injection | 1.57 | 10.48 | 43.96
Full DOS | Gym-malware | 23.15 | 38.69 | 61.63
GAMMA section-injection | Full DOS | 6.26 | 15.05 | 45.30
GAMMA section-injection | GAMMA section-injection | 1.90 | 14.03 | 44.52
GAMMA section-injection | Gym-malware | 25.39 | 46.97 | 74.50
Gym-malware | Full DOS | 26.51 | 46.16 | 67.34
Gym-malware | GAMMA section-injection | 27.18 | 49.22 | 67.79
Gym-malware | Gym-malware | 29.53 | 58.34 | 78.19

First, we present the results of the evasion rate metric. For all AVs, we
examine the minimum, average, and maximum of the results of all combinations
of generators tested in the experiment. These results can be found in Table 5.
For the minimum values of the evasion rate, we can see that none of the
antivirus programs reached a detection rate of 100%. On the other hand, all
these values are relatively low compared to the average and maximum values,
which tells us that some of the nine generator combinations were not very
successful. The average evasion rates range from about 18% to 49%, and the
maximum values range from 30% to 78%. The best result was achieved by
combinations of generators against AV7, where the average evasion rate is
around 49%, while the least successful was against AV2, where the average
evasion rate is around 18%.

Table 6 shows the results of the evasion rate achieved by each combination of
generators. Here, we can see that the most successful combination was the one
in which the Gym-malware generator was used twice in a row. This achieved a
minimum evasion rate of around 30% for all AVs. The average value for this
combination is around 58%, and for at least one AV we achieved an evasion rate
of around 78% with this combination. On the other hand, the worst combination
in terms of evasion rate is the one in which the Full DOS generator was used
twice. In this case, we have an average evasion rate of about 2%, while for
all other combinations, this value exceeds 10%, and for some even more
significantly. The maximum value of the evasion rate for this combination is
about 5%, for the others we have at least about 44%.

To determine the effectiveness of using generator combinations instead of
individual generators alone, we can compare these results with the values from
the previous experiment. However, we have additional metrics for this, which
are described in Section 4.3. We analyze the results of these metrics in the
following parts of this section.

#### Absolute Improvement

Next, we evaluate the metric that we identified as an absolute improvement in
Section 4.3. In short, it is the percentage difference in the evasion rate
between the evasion rate achieved by combining both generators and the evasion
rate achieved by using only the first generator.

Table 7: Absolute improvement (in %) for each AV using all combinations of generators. AV | Minimum | Average | Maximum
---|---|---|---
AV1 | 0.11 | 10.84 | 29.98
AV2 | 0.11 | 9.21 | 22.48
AV3 | 0.11 | 13.81 | 41.16
AV4 | 0.00 | 14.43 | 60.74
AV5 | 0.00 | 11.14 | 35.01
AV6 | 0.11 | 14.77 | 50.22
AV7 | 0.90 | 15.46 | 40.72
AV8 | 0.34 | 8.14 | 26.51
AV9 | 0.00 | 13.78 | 41.39
Table 8: Absolute improvement (in %) for each generator combination against all AVs. First Generator | Second Generator | Minimum | Average | Maximum
---|---|---|---|---
Full DOS | Full DOS | 0.00 | 0.27 | 1.12
Full DOS | GAMMA section-injection | 0.67 | 9.21 | 39.82
Full DOS | Gym-malware | 21.92 | 37.42 | 60.74
GAMMA section-injection | Full DOS | 0.78 | 4.13 | 5.93
GAMMA section-injection | GAMMA section-injection | 0.00 | 3.11 | 10.29
GAMMA section-injection | Gym-malware | 20.02 | 36.04 | 51.23
Gym-malware | Full DOS | 0.11 | 2.05 | 7.49
Gym-malware | GAMMA section-injection | 0.56 | 5.11 | 10.63
Gym-malware | Gym-malware | 7.72 | 14.23 | 20.02

First, we focus on Table 7, which shows the absolute improvement values for
each AV in the nine combinations of generators. Here we can see that for some
AVs we were unable to improve the evasion rate by applying the second
generator. Specifically, this refers to the minimum value for AV4, AV5, and
AV9. Table 8 helps us to identify the relevant generators. We can see that the
only non-improving combinations are the ones in which we use the same
generator twice, namely, the Full DOS generator or the GAMMA section-injection
generator. This may indicate that the use of these combinations is not
entirely effective. For the average values in Table 7, we can see that they do
not differ significantly, in all cases between about 8% and 15%. For the
maximum values, we have a slightly higher range, approximately 22% to 61%.
This means that if we use a second generator, we will improve the evasion rate
by about 10% on average after using the first generator.

However, significantly more intriguing is Table 8, which shows the absolute
improvement for each generator combination against all AVs. Here, we can see
that the use of Full DOS as the second generator does not result in a
significant absolute improvement in the evasion rate for the minimum, average,
and maximum values. This means that using Full DOS as the second generator in
a combination is the least effective. On the other hand, we can see that we
get the best absolute improvement by using the Gym-malware method as the
second generator in the combination.

In Table 8, we can observe another interesting fact. As we have already
mentioned, the use of two identical generators in combination is not very
effective. However, this statement does not apply to the Gym-malware
generator. In contrast, if we use the Gym-malware generator as the first
generator in the combination, then the best choice to select the second
generator seems to be the Gym-malware generator again. Full DOS and GAMMA
section-injection generators have minimal absolute improvements in the role of
the second generator. These results show that the Gym-malware generator is
successful in both cases, used alone and in combination with itself.

The best absolute improvement values are achieved when we choose Full DOS as
the first generator and Gym-malware as the second. This results in an absolute
improvement in the evasion rate in all AVs of at least around 22%, on average
37% and up to a maximum of 61%. On the other hand, the worst results in terms
of absolute evasion rate improvement are obtained when we choose Full DOS as
both generators. In this case, we obtain a minimum absolute improvement of 0%,
an average of 0.3% and a maximum of 1.1% across all antivirus programs. We can
conclude that the Full DOS generator is likely to be the least successful
generator, both when used alone and in combination.

#### Relative Improvement

Table 9: Relative improvement (in %) for each AV using all combinations of generators. AV | Minimum | Average | Maximum
---|---|---|---
AV1 | 0.61 | 788.88 | 4466.67
AV2 | 8.33 | 307.66 | 1675.00
AV3 | 3.74 | 732.66 | 5014.29
AV4 | 0.00 | 914.67 | 6787.50
AV5 | 0.00 | 633.29 | 4257.14
AV6 | 1.46 | 846.29 | 6000.00
AV7 | 2.05 | 232.70 | 983.78
AV8 | 3.80 | 510.39 | 2154.55
AV9 | 0.00 | 758.01 | 5285.71
Table 10: Relative improvement (in %) for each generator combination against all AVs. First Generator | Second Generator | Minimum | Average | Maximum
---|---|---|---|---
Full DOS | Full DOS | 0.00 | 18.93 | 62.50
Full DOS | GAMMA section-injection | 75.00 | 715.51 | 2400.00
Full DOS | Gym-malware | 983.78 | 4027.99 | 6787.50
GAMMA section-injection | Full DOS | 3.85 | 104.90 | 409.09
GAMMA section-injection | GAMMA section-injection | 0.00 | 58.50 | 181.25
GAMMA section-injection | Gym-malware | 70.77 | 741.93 | 2154.55
Gym-malware | Full DOS | 0.17 | 7.31 | 39.41
Gym-malware | GAMMA section-injection | 0.83 | 13.60 | 42.94
Gym-malware | Gym-malware | 16.31 | 35.90 | 56.12

We follow the absolute improvement results with a relative improvement in the
evasion rate. Analogously to the improvement in the absolute evasion rate, we
can see the results of relative improvement in Table 9. We can see that AV7
performed the best on average in terms of relative improvement of the evasion
rate. Conversely, AV4 performed the worst, where in some cases the second
generator managed to increase the number of successfully modified samples
(which evaded detection) by more than 67 times.

Table 10 confirms that the Full DOS generator, which was used as the second
generator in combination, does not increase the evasion rate significantly. On
the other hand, if this generator is chosen as the first generator in the
combination, the GAMMA section-injection or Gym-malware generator can increase
the number of samples that evade detection by up to 67 times. This shows that
the Full DOS generator is not a very strong generator on its own. Regarding
the Gym-malware generator, we can see that when it is used as the first
generator in a combination, the second generator does not significantly
increase the result, even when Gym-malware is used again as the second
generator. However, the Gym-malware generator is very successful on its own as
it achieves a significantly higher evasion rate than other generators.
Therefore, a relative improvement in the evasion rate from 39% to 56% can
cause a drastic increase in the total evasion rate of the combination (up to
20%). We can also see that when the Gym-malware generator is used as a second
generator in other combinations, there are huge relative improvements over the
first generator. Again, we can see that Gym-malware is very effective when
used in any combination of generators.

#### Evasion Rate Comparison

The last metric examined, evasion rate comparison, helps us find the answer to
the question of whether it is better to use individual generators separately
or to use them in combination. A negative value of this metric indicates that
we would achieve a better evasion rate by using the better of the two
generators individually (better in terms of evasion rate). On the contrary, a
positive value indicates that we recorded a better result using the
combination of both generators.

Table 11: Evasion rate comparison (in %) for each AV using all combinations of generators. AV | Minimum | Average | Maximum
---|---|---|---
AV1 | -14.88 | 0.87 | 9.73
AV2 | 0.11 | 5.28 | 10.52
AV3 | -4.81 | 4.02 | 18.23
AV4 | -11.63 | -0.31 | 10.96
AV5 | -7.49 | 2.06 | 16.00
AV6 | -5.82 | 3.03 | 20.02
AV7 | -8.95 | 4.43 | 20.69
AV8 | -3.36 | 2.52 | 14.88
AV9 | -2.69 | 3.99 | 17.90
Table 12: Evasion rate comparison (in %) for each generator combination against all AVs. First Generator | Second Generator | Minimum | Average | Maximum
---|---|---|---|---
Full DOS | Full DOS | 0.00 | 0.27 | 1.12
Full DOS | GAMMA section-injection | -2.80 | -0.45 | 1.57
Full DOS | Gym-malware | -14.88 | -5.42 | 4.81
GAMMA section-injection | Full DOS | 0.78 | 4.13 | 5.93
GAMMA section-injection | GAMMA section-injection | 0.00 | 3.11 | 10.29
GAMMA section-injection | Gym-malware | -11.63 | 2.86 | 20.69
Gym-malware | Full DOS | 0.11 | 2.05 | 7.49
Gym-malware | GAMMA section-injection | 0.56 | 5.11 | 10.63
Gym-malware | Gym-malware | 7.72 | 14.23 | 20.02

Looking at the minimum values in table 11, we can see that for almost all AVs
there was a combination of generators that was not effective due to the
negative values in this column. The exception is AV2, where we achieved a
better evasion rate for all combinations of generators used. However, the
average and maximum values show that in most cases the combination is more
effective than the better of the two generators used separately. Only AV4 has
a negative average value, which means that a separate use of a single
generator is a better option against this AV. On the other hand, when
attacking AV2 it is better to use a combination of generators as it achieved
the highest average evasion rate comparison value.

Table 12 shows that combining generators using Full DOS as the first generator
and either GAMMA section-injection or Gym-malware as the second generator
yields negative results, as evidenced by both the minimum and average values
being negative. Based on the results from the previous parts of this section,
we can say that it is more advantageous to use GAMMA section-injection and
Gym-malware generators separately in such cases. We can also see other
negative values in the minimum value column for the combination of GAMMA
section-injection and Gym-malware generators used in this order. Nonetheless,
the average value of the combination is positive, indicating that it is
beneficial to use this combination on average. For the remaining combinations,
we can conclude that using them results in a better evasion rate than using
the better of the two generators separately.

##  6 Conclusion

In this paper, we explored the use of adversarial learning techniques in
malware detection. Our goal was to apply existing methods for generating
adversarial malware samples, test their effectiveness against selected malware
detectors, and compare the evasion rates achieved and the practical
applicability of these methods.

For our experiments, we chose five adversarial malware sample generators:
Partial DOS, Full DOS, GAMMA padding, GAMMA section-injection, and Gym-
malware. This selection represents a spectrum of adversarial techniques based
on gradient, evolutionary algorithms, and reinforcement learning. These
adversarial malware generators were evaluated on nine commercially available
antivirus products.

To validate and compare the different characteristics and properties of the
methods used, we performed four experiments. These included tracking the time
taken to generate samples, changes in sample size after using applying
adversarial modifications, testing effectiveness against antivirus programs,
and evaluating combinations of generators.

The results indicate that making optimized modifications to previously
detected malware can cause the classifier to misclassify the file and label it
as benign. Furthermore, the study confirmed that generated malware samples
could be used successfully against detection models other than those used to
generate them. Using combination attacks, a significant percentage of new
samples were created that could evade detection by antivirus programs.

Experiments showed that the Gym-malware generator, which uses a reinforcement
learning approach, has the greatest practical potential. This generator
produced malware samples in the shortest time, with an average sample
generation time of 5.73 seconds. The Gym-malware generator also achieved the
highest evasion rate among all selected antivirus products, with the highest
average evasion rate of 44.11% against nine AVs. Furthermore, the Gym-malware
generator was effective when combined with another generator, especially with
itself, where it achieved the highest average evasion rate of 58.35%.
Additionally, this generator could significantly improve the performance of
other generators with absolute and relative improvements ranging between
36.04%–37.42% and 741.93%–4027.99%, respectively.

For future experiments, we propose to study in more detail how the sample
generation time is affected by the input genuine malware size and investigate
the correlation between the time and the evasion rate of the resulting
adversarial examples. Further experiments could be done in the area of
combining generators where more than two generators could be combined to
achieve even higher evasion rates. Our work highlights the importance of
developing new techniques to detect malware and identify adversarial attacks.
More research is needed in this area to successfully combat these novel
threats and attacks.

## ACKNOWLEDGEMENTS

This work was supported by the Student Summer Research Program 2022 of FIT CTU
in Prague and by the OP VVV MEYS funded project CZ.02.1.01/0.0/0.0/16
019/0000765 “Research Center for Informatics” and by the Grant Agency of the
CTU in Prague, grant No. SGS23/211/OHK3/3T/18 funded by the MEYS of the Czech
Republic.

## Declarations

The authors have no relevant financial or non-financial interests to disclose.

## References

  *   * AV-TEST [2022] AV-TEST: Malware Statistics & Trends Report | AV-TEST. AV-TEST (2022). https://www.av-test.org/en/statistics/malware
  * Al-Asli and Ghaleb [2019] Al-Asli, M., Ghaleb, T.A.: Review of signature-based techniques in antivirus products. In: 2019 International Conference on Computer and Information Sciences (ICCIS), pp. 1–6 (2019). https://doi.org/10.1109/ICCISci.2019.8716381 . IEEE
  * Singh and Singh [2018] Singh, J., Singh, J.: Challenge of malware analysis: malware obfuscation techniques. Şeref SAĞIROĞLU (2018). https://dergipark.org.tr/en/download/article-file/2160186
  * Singh and Singh [2021] Singh, J., Singh, J.: A survey on machine learning-based malware detection in executable files. Journal of Systems Architecture 112, 101861 (2021) https://doi.org/10.1016/j.sysarc.2020.101861
  * Dolejš and Jureček [2022] Dolejš, J., Jureček, M.: Interpretability of Machine Learning-Based Results of Malware Detection Using a Set of Rules. Springer (2022). https://doi.org/10.1007/978-3-030-97087-1_5
  * Rosenberg et al. [2021] Rosenberg, I., Shabtai, A., Elovici, Y., Rokach, L.: Adversarial machine learning attacks and defense methods in the cyber security domain. ACM Computing Surveys (CSUR) 54(5), 1–36 (2021) https://doi.org/10.1145/3453158
  * Goodfellow et al. [2015] Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and Harnessing Adversarial Examples (2015). https://arxiv.org/abs/1412.6572
  * Papernot et al. [2016] Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z.B., Swami, A.: The limitations of deep learning in adversarial settings. In: 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 372–387 (2016). https://doi.org/10.1109/EuroSP.2016.36
  * Goodfellow et al. [2014] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative Adversarial Networks. arXiv (2014). https://doi.org/10.48550/ARXIV.1406.2661 . https://arxiv.org/abs/1406.2661
  * Dutta et al. [2020] Dutta, I.K., Ghosh, B., Carlson, A., Totaro, M., Bayoumi, M.: Generative adversarial networks in security: a survey. In: 2020 11th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), pp. 0399–0405 (2020). https://doi.org/10.1109/UEMCON51285.2020.9298135 . IEEE
  * Wang et al. [2017] Wang, K., Gou, C., Duan, Y., Lin, Y., Zheng, X., Wang, F.-Y.: Generative adversarial networks: introduction and outlook. IEEE/CAA Journal of Automatica Sinica 4(4), 588–598 (2017) https://doi.org/10.1109/JAS.2017.7510583
  * Sutton and Barto [2018] Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction. MIT press (2018). https://doi.org/10.1016/S1364-6613(99)01331-5
  * Demetrio et al. [2019] Demetrio, L., Biggio, B., Lagorio, G., Roli, F., Armando, A.: Explaining vulnerabilities of deep learning to adversarial malware binaries (2019) https://doi.org/10.48550/arXiv.1901.03583
  * Demetrio et al. [2020] Demetrio, L., Coull, S.E., Biggio, B., Lagorio, G., Armando, A., Roli, F.: Adversarial exemples: A survey and experimental evaluation of practical attacks on machine learning for windows malware detection. arXiv preprint arXiv:2008.07125 (2020) https://doi.org/10.1145/3473039
  * Demetrio et al. [2021] Demetrio, L., Biggio, B., Lagorio, G., Roli, F., Armando, A.: Functionality-preserving black-box optimization of adversarial windows malware. IEEE Transactions on Information Forensics and Security 16, 3469–3478 (2021) https://doi.org/10.1109/TIFS.2021.3082330
  * Anderson et al. [2018] Anderson, H.S., Kharkar, A., Filar, B., Evans, D., Roth, P.: Learning to evade static pe machine learning malware models via reinforcement learning. arXiv preprint arXiv:1801.08917 (2018) https://doi.org/10.48550/arXiv.1801.08917
  * Castro et al. [2019] Castro, R.L., Schmitt, C., Dreo, G.: Aimed: Evolving malware with genetic programming to evade detection. In: 2019 18th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/13th IEEE International Conference On Big Data Science And Engineering (TrustCom/BigDataSE), pp. 240–247 (2019). https://doi.org/10.1109/TrustCom/BigDataSE.2019.00040 . IEEE
  * Wang and Miikkulainen [2020] Wang, X., Miikkulainen, R.: Mdea: Malware detection with evolutionary adversarial learning. In: 2020 IEEE Congress on Evolutionary Computation (CEC), pp. 1–8 (2020). https://doi.org/10.1109/CEC48606.2020.9185810 . IEEE
  * Fang et al. [2019] Fang, Z., Wang, J., Li, B., Wu, S., Zhou, Y., Huang, H.: Evading anti-malware engines with deep reinforcement learning. IEEE Access 7, 48867–48879 (2019) https://doi.org/10.1109/ACCESS.2019.2908033
  * Song et al. [2020] Song, W., Li, X., Afroz, S., Garg, D., Kuznetsov, D., Yin, H.: Mab-malware: A reinforcement learning framework for attacking static malware classifiers. arXiv preprint arXiv:2003.03100 (2020) https://doi.org/10.48550/arXiv.2003.03100
  * Labaca-Castro et al. [2021] Labaca-Castro, R., Franz, S., Rodosek, G.D.: Aimed-rl: Exploring adversarial malware examples with reinforcement learning. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 37–52 (2021). https://doi.org/10.1007/978-3-030-86514-6_3 . Springer
  * Kolosnjaji et al. [2018] Kolosnjaji, B., Demontis, A., Biggio, B., Maiorca, D., Giacinto, G., Eckert, C., Roli, F.: Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executables. arXiv (2018). https://arxiv.org/abs/1803.04173
  * Kreuk et al. [2018] Kreuk, F., Barak, A., Aviv-Reuven, S., Baruch, M., Pinkas, B., Keshet, J.: Deceiving End-to-End Deep Learning Malware Detectors using Adversarial Examples. arXiv (2018). https://arxiv.org/abs/1802.04528
  * Suciu et al. [2018] Suciu, O., Coull, S.E., Johns, J.: Exploring adversarial examples in malware detection (2018)
  * Hu and Tan [2017] Hu, W., Tan, Y.: Generating adversarial malware examples for black-box attacks based on gan. arXiv preprint arXiv:1702.05983 (2017) https://doi.org/10.48550/arXiv.1702.05983
  * Kawai et al. [2019] Kawai, M., Ota, K., Dong, M.: Improved malgan: Avoiding malware detector by leaning cleanware features. In: 2019 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), pp. 040–045 (2019). https://doi.org/10.1109/ICAIIC.2019.8669079 . IEEE
  * Yuan et al. [2020] Yuan, J., Zhou, S., Lin, L., Wang, F., Cui, J.: Black-box adversarial attacks against deep learning based malware binaries detection with GAN. IOS Press (2020). https://doi.org/10.3233/FAIA200388
  * Anderson and Roth [2018] Anderson, H.S., Roth, P.: Ember: An open dataset for training static pe malware machine learning models. CoRR abs/1804.04637 (2018) https://doi.org/10.48550/ARXIV.1804.04637 . Accessed 2023-13-01
  * Raff et al. [2017] Raff, E., Barker, J., Sylvester, J., Brandon, R., Catanzaro, B., Nicholas, C.: Malware detection by eating a whole exe (2017) https://doi.org/10.48550/ARXIV.1710.09435 . Accessed 2023-13-01
  * VirusShare [2023] VirusShare: VirusShare. https://www.virusshare.com/. Accessed at 13/01/2023 (2023)
  * AV-Comparatives [2022] AV-Comparatives: Malware Protection Test September 2022. https://www.av-comparatives.org/tests/malware-protection-test-september-2022/. Accessed at 13/01/2023 (2022)
  * VirusTotal [2023] VirusTotal: VirusTotal. https://www.virustotal.com/. Accessed at 13/01/2023 (2023)
  * Kozák and Jureček [2023] Kozák, M., Jureček, M.: Combining generators of adversarial malware examples to increase evasion rate. In: Proceedings of the 20th International Conference on Security and Cryptography - SECRYPT, pp. 778–786 (2023). https://doi.org/10.5220/0012127700003555
  * Thomas [2017] Thomas, R.: LIEF - Library to Instrument Executable Formats (2017). https://lief.quarkslab.com/ Accessed 2023-13-01