Landau theory of $180^{o}$ domain walls in $BaTiO_{3}$ type ferroelectric
particles: microcomposite materials

O. Hudak

Department of Theoretical Physics, Faculty of Mathematics, Physics and
Informatics, Comenius University, Mlynska dolina F2, 845 01 Bratislava

## Abstract

The Landau theory of $180^{o}$ domain walls in $BaTiO_{3}$ type ferroelectric
particles is presented . Results of exact description of domain walls in bulk
enabled us to formulate variational approach to theory of domain walls in
corresponding small particles. The depolarization field effects and the space-
charge layers are taken into account in the samples of the cube form. It was
found that at low temperatures well known hyperbolic tangent wall profile is a
good approximation for description of domain walls in particles. Near the
transition temperature it is more and more appropriate to speak about two
walls separating ferroelectric, paraelectric and ferroelectric domains
correspondingly as a result of splitting of a single ferroelectric wall into
two subwalls in small $BaTiO_{3}$ particles. Domain wall energy density,
average interwall distance and change of the dielectric response of thick
walls in small ferroelectric particles in microcomposites is qualitatively
found. Our results qualitatively describe observed dependencies better than
those theories which exist up to date. In temperature region near transition
from the ferroelectric to paraelectric phase in micrcomposites our
quantitative results for response, for Curie-like transition temperature and
other parameters may be verified.

##  1 Introduction

The transition temperature of the ferroelectric phase - paraelectric phase
phase transition decreases with decreasing particle diameter in $\rm
BaTiO_{3}$ [1], $\rm PbTiO_{3}$ [2] and in KDP [3] microcomposites. It was
found experimentaly that there is no phase transition bellow a critical size
of a particle in the microcomposite. It was observed also that a higher phase
transition temperature in smaller particles sometimes occurs. In thin films of
$\rm KNbO_{3}$ [4] and TGS [5, 6] both increase and decrease of the transition
temperature were observed with decreasing particle size. In $\rm PbTiO_{3}$
microcomposites [7] the dielectric constant decreases with decreasing particle
size, probably due to the gradual creation of a multidomain state in larger
particles. An unusual peak of dielectric constant versus particle size was
observed in $\rm BaTiO_{3}$ [8].

There are two main theoretical approaches to explanation of the size effects
in the ferroelectric - dielectric microcomposites of the above type. The
significance of depolarization field that tries to break up the ferroelectric
particle into domains with different polarizations. The multidomain state then
exhibits higher permittivity, but its creation becomes difficult for small
enough particles [9]. Another possibility is that a surface layer with
different transition temperature than that in the bulk is phenomenologically
considered without the depolarization field explicitly taken into account.
Then a shift of the phase transition temperature in the particle is present as
well as the inhomogeneous distribution of the polarization [10]. In [10] the
static properties of such particles is studied by numerical calculations only.
The ferroelectric phase transition in small spherical particles was studied in
[11]. Size effects were studied analyticaly as concerning temperature effects
and polarization profile. Our model in [11] did not take into account
depolarization fields, assuming full compensation of surface charges. The
dynamic susceptibility deviates from Debye-like behaviour then and exhibits a
broadening at higher frequencies. The Curie-Weiss law is exhibited by the
static susceptibility, there exists a divergency at the point of the size-
driven transition. Dielectric response of microcomposite ferroelectrics was
studied in [12]. A two-phase composite of ferroelectric and dielectric
materials was described, and studied by different effective medium theories
(isolated particles, Maxwell-Garnet theory, the effective medium approximation
and the Bergman representation of the effective dielectric function). There
are present modes due to geometric resonances which lead to new low-frequency
absorption peaks near the percolation threshold of the ferroelectric material.
Those peaks are always located higher than the peak of the transverse polar
mode of the bulk component. The soft mode with renormalized mode strength but
unchanged frequency exists only in the composite with percolated ferroelectric
clusters. Below the percolation threshold the finite clusters of the
ferroelectric material soften only to some degree, there is no soft mode.
Results of exact description of domain walls in bulk have been found and were
preliminary published in [13]. They enabled us to formulate variational
approach to the theory of domain walls in corresponding small particles.

In this paper we study a Landau theory of $180^{o}$ domain walls in
$BaTiO_{3}$ type ferroelectric particles. The depolarization field effects and
the space-charge layers are taken into account in the samples of the cube
form. We have found that at low temperatures well known hyperbolic tangent
wall profile is a good approximation for description of domain walls in
particles. Nearby the first-order transition temperature a free energy
contribution due to paraelectric domain state within a wall center becomes
important leading to unusually large wall thickness. Approaching the
transition temperature it is more and more appropriate to speak about two
walls separating ferroelectric domains as a result of splitting a single
ferroelectric wall into two walls with a paraelectric phase between them.
Consequently the primary ferroelectric domain wall width becomes very large
which influences also size dependence and other characteristics of the
ferroelectric transition to the paraelectric phase in small $BaTiO_{3}$
particles. Domain wall energy density, average interwall distance and change
of the dielectric response of thick walls in small ferroelectric particles is
qualitatively described and quantitatively approximately found in this
temperature region.

##  2 Domain walls configurations in a $BaTiO_{3}$ particle

The free energy density F expansion into polar mode amplitude P following [14]
has the form

$F=\frac{\alpha}{2}{\bf P}^{2}+\frac{\beta}{4}{\bf P}^{4}+\frac{\sigma}{6}{\bf
P}^{6}+C\mid{\bf\nabla P}\mid^{2}.$ (1)

where the coefficient is given by $\alpha=a_{0}(T-T_{0\infty}),$ here $a_{0}$
is a constant, $T_{0\infty}$ is the Curie-Weiss temperature for the bulk phase
transition.

Numerical values of the coefficients of the Landau free energy expansion (1)
of $BaTiO_{3}$ are given in [14].

Assuming the domain wall localized at x=0, the polarization P oriented in the
z-direction and that this polarisation changes in the x direction, P(x), the
domain-wall energy $\gamma$ is defined by

$\gamma=\int^{+\infty}_{-\infty}\triangle F(P(x))dx,$ (2)

where $\triangle F(P(x))=F(P(x))-F(P)$ is the difference of free energy
densities between the state with a domain-wall configuration P(x) and the
equilibrium state with a homogeneous polarization P.

The single domain-wall configuration P(x) in $BaTiO_{3}$ was in [14] described
approximately by the following well-known profile

$P(x)=P.tanh(\frac{x}{\xi}),$ (3)

with P and $\xi$ as variational parameters. The domain-wall configuration P(x)
in a $BaTiO_{3}$ particle has a wall energy density which may be approximately
calculated as in [14] by

$\gamma=\int^{+D}_{-D}\triangle F(P(x))dx,$ (4)

with 2D a distance between two neighbouring walls. It is a variational
parameter. After calculating the wall energy density $\gamma$ one can then
easily incorporate space-charge layer effects, characterised by a parameter t,
in the same way is in [14], too.

##  3 Exact solution for wall configurations

Let us transform the free energy part (1) describing local potential of the
sixth order in the polarization P to a more transparent form

$V(P)=b[\frac{P^{6}}{6}-(a^{2}+c^{2})\frac{P^{4}}{4}+a^{2}c^{2}\frac{P^{2}}{2}].$
(5)

The coefficients a, b and c above and the free energy density expansion
coefficients $\alpha,\beta,\gamma$ are connected by relations:

$a^{2}=\frac{1}{2}[\frac{\beta}{\sigma}+\sqrt{\frac{\beta}{\sigma}-\frac{4\alpha}{\sigma}}],$
(6)
$c^{2}=\frac{1}{2}[\frac{\beta}{\sigma}-\sqrt{\frac{\beta}{\sigma}-\frac{4\alpha}{\sigma}}],$
$b=\sigma.$

To discuss properties of the local effective potential (5) it is convenient to
introduce a dimensionless parameter $\rho$

$\rho=(\frac{c}{a})^{2},$ (7)

which we will assume to be a nonegative quantity. The case of negative values
leads to a two well potential of the sixth order, which is similar to the
well-known double-well potentials.

The local potential V(P) has two absolute minima at $\pm a$ and there is a
local maximum at P=0 for $\rho=0.$ The potential V(P) has two absolute minima
$\pm a$ and there is a local minimum at P=0, and at $\pm c$ there are two
local maxima for $0<\rho<\frac{1}{3}.$ The potential V(P) has three absolute
minima at $\pm a$ and at P=0, in $\pm c$ there are two local maxima for
$\rho=\frac{1}{3}.$ The potential V(P) has only single absolute minimum at
P=0, at $\pm a$ there two local minima and in $\pm c$ there are two local
maxima for $\frac{1}{3}<\rho<1.$ For $\rho>1$ the situation just described
repeats with the reversed role of the a and c constants.

The Lagrange-Euler equation for configurations of polarisation P which
extremize the free energy (1) has the form

$2C\frac{dP}{dx}=bP.(P^{2}-a^{2}).(P^{2}-c^{2}).$ (8)

We assume that $a>c,$ the equilibrium state configurations then realizes at
points $\pm a$ (ferroelectric phase) or at P=0 (paraelectric phase).

Severals years ago [18] we succeeded to find exact solutions of the equation
(8). Similar solutions in different forms were obtained also elsewhere, [19]
and [20]. For $0\leq\rho<\frac{1}{3}$ the walls are described in the form

$P(x)=\pm
a\frac{\sqrt{q}tanh(\frac{x}{\xi}+R)}{\sqrt{1+q-tanh^{2}(\frac{x}{\xi}+R))}}$
(9)

where R is an arbitrary constant, describing the position of the center of the
wall, and where we define $q=\frac{1-3\rho}{3}.$ These walls interconnect two
ferroelectric domains with opposite polarization.

The parameter $\xi$ describes the rate of change of the polarization at the
wall configuration and is given by

$\xi=\frac{1}{\sqrt{\frac{1+q}{3}}}\frac{1}{\sqrt{\frac{ba^{4}}{2C}}}.$ (10)

##  4 $180^{o}$ domain wall configurations

The exact solution (9) describes a profile of a domain wall. It is convenient
to rewrite the form (9) into a more transparent form

$P(x)=P_{0}(x).tanh(\frac{x}{\xi}+R),$ (11)

where the first factor in (11) we call a ”modulated amplitude” $P_{0}(x)$ of
the wall configuration tanh. It is given by

$P_{0}(x)=\pm a\frac{\sqrt{q}}{\sqrt{1+q-tanh^{2}(\frac{x}{\xi}+R))}}.$ (12)

While at larger distances from the center of the wall, localized at $x=0$ for
R=0, the ”modulated amplitude” takes the equilibrium configuration values

$P_{0}(\pm\infty)=\pm a.$ (13)

at the center of the wall this amplitude decreases its value to

$P_{0}(x=0)=\pm
a\frac{\sqrt{q}}{\sqrt{1+q}}=P_{0}(\pm\infty).\sqrt{\frac{1-3\rho}{3-3\rho}}.$
(14)

It is clear from (14) that the nearer we are to the phase transition the
nearer the constant $\rho$ to the value $\frac{1}{3},$ and consequently the
smaller ”modulated amplitude” of the wall at the center. This decrease
reflects the increasing role of the paraelectric phase region in the center of
the wall.

The wall thus consists of two subwalls connecting domains with nonzero
polarization with a region with almost zero polarization. At temperatures near
the phase transition temperature the distance $d_{w}$ between the two subwalls
(walls separating the ferroelectric domains and the paraelectric phase domain
in the center) becomes larger. One can find that at this temperature region we
have the distance $d_{w}$ given by

$d_{w}=-\frac{\xi}{2}ln(\frac{1-\sqrt{3\rho}}{1+\sqrt{3\rho}}),$ (15)

which describes logarithmic divergence of the distance $d_{w}$, because the
nearer we are to the phase transition the nearer the constant $\rho$ to the
value $\frac{1}{3}.$

On the other hand the lower temperature the larger the ”modulated amplitude”
$P_{0}(x=0),$ approaching the same value as the amplitude at domains
$P_{0}(\pm\infty).$ One can find that in this temperature region

$d_{w}=\xi\sqrt{3\rho},$ (16)

which is a finite quantity.

One can easily estimate temperature region in which the condition
$0<\rho<\frac{1}{3}$ is satisfied and thus in which very thin walls may exist
the distance $d_{w}$ of which logarithmicaly divergences. Using parameters for
$BaTiO_{3}$ from [14] we found that the mentioned inequalities are equivalent
with temperature region $112.24^{o}C<T<121.48^{o}C.$ In this region thus the
ferroelectric domain wall has in the center the large paraelectric region and
consists of two walls /subwalls/ connecting this region with the corresponding
ferroelectric region.

From this result it follows that at $25^{o}C$ the ”tanh” wall profile, used in
[14] for approximate calculations, is an adequate approximation. At this lower
temperatures this material is characterized by an effective potential with
only two absolute minima. Near phase transition from the ferroelectric phase
to the paraelectric phase in the temperature region
$112.24^{o}C<T<121.48^{o}C$ the ”tanh” wall profile used for approximate
calculations is an inadequate approximation. Our exact description of the
domain wall is more appropriate in this region.

##  5 The dielectric response of a ferroelectric particle

The dielectric response of a ferroelectric particle at temperature region near
the transition temperature, where the wall width behaves according to (16),
may be estimated in the following way. The effect of the depolarization field
and of the space-charge layer may be taken into account using the same
variational approach as in [14]. However, as the wall thickness parameter the
domain subwall distance as a domain wall width $d_{w}$ instead of $\xi$ from
[14] is taken.

This approach leads to the following wall energy density $\gamma$

$\gamma\approx\frac{8cP_{\infty}^{2}}{3d_{w}}.$ (17)

The equilibrium interwall distance 2D between two ferroelectric walls in a
particle of a cube form with the side width L is found to be given by D

$D=\sqrt{\frac{8c}{5.1d_{w}t}}L.$ (18)

where t characterizes changes of the polarisation near the surface. We see
that increasing temperature to the transition temperature the domain wall
width measured by the subwall distance $d_{w}$ increases while the interwall
distance D decreases. Remaining structure consists of strips of ferroelectric
phase domains separated by wide regions (domains) of the paraelectric phase.

If both characteristic lengths, the interwall distance characteristic quantity
D and the subwall distance characteristic quantity $d_{w}$ become comparable,
$D\approx d_{w},$ then the ferroelectric domains become very narrow and almost
vanish with respect to usual situation in which ferroelectric domain wall
width is very small with respect the ferroelectric domain width. The
corresponding temperature is slightly below the transition temperature at
which $\rho=\frac{1}{3}.$

In the regime in which $\xi<<D,$ one finds the Curie-Weiss law with
renormalised transition temperature

$\alpha^{{}^{\prime}}=\alpha+3.4\frac{Dt}{L^{2}}+[-\frac{2d_{w}\alpha}{D}+\frac{8c}{3d_{w}}].$
(19)

From this expression one finds that the temperature coefficient a is
multiplied by $1-\frac{2d_{w}}{D}$ giving $a^{{}^{\prime}}.$

The critical Curie-Weiss temperature becomes

$T_{0}=T_{0\infty}-\frac{1}{a^{\prime}}[3.4\frac{Dt}{L^{2}}+\frac{8c}{3d_{w}}].$
(20)

If the wall thickness (15) increases increasing temperature there will be much
stronger renormalization of the temperature coefficient to zero approximately
for those wall configurations at which the the interwall distance 2D becomes
comparable with the width parameter $d_{w}.$ From (20) it follows that
simultaneously the Curie-Weiss temperature reduces to zero quickly, the
ferroelectric phase vanishes even before the wall thickness becomes comparable
with the interwall distance characteristic length D. It is also clear that the
existence of the ferroelectric phase becomes more sensitive to the length L of
the crystal due to decreased value of the temperature coefficient
$a^{{}^{\prime}}$ and due to the domain wall width $d_{w}$ dependence of the
interwall distance 2D (18).

For thin walls $(d_{w}<\frac{D}{8})$ the transition temperature dependence
coefficient on the particle size L is larger for the domain walls as described
in our paper than for those described approximately by simple $tanh$
dependence used in [14] to qualitatively describe transition temperature
dependence on L. Experimentally found transition temperature dependence on L
is steeper than that predicted by the simple tanh wall model [14]. Our exact
description od domain walls for the bulk free energy potential in $BaTiO_{3}$
describing the first order phase transition which is near the second order
phase transition and used for the description of the particle domain structure
as in [14] is more appropriate for real crystals $BaTiO_{3}$. Note that for
thick walls $(d_{w}>\frac{D}{8})$ the transition temperature dependence
coefficient on the particle size L is smaller for the domain walls as
described in our paper than for those described approximately by simple $tanh$
dependence. Thus mechanism of the increasing domain wall width due to
splitting of the domain wall into two walls (sequence ferroelectric -
paraelectric - ferroelectric domains) may explain much more abrupt change of
the Curie-Weiss temperature al low L values unexplained by the spontaneous
strain mechanism in [14].

## Acknowledgement

The author wishes to express his sincere thanks to V.Dvorak, J.Petzelt,
I.Rychetsky, J.Holakovsky and M.Glogarova for their discussions about
microcomposite materials. This paper was supported by the grant VEGA
1/0250/03, Faculty of Mathematics, Physics and Informatics, Comenius
University, Bratislava.

## References

  * [1] K. Uchino, E. Sadanaga and T. Hirose: J. Am. Ceram. Soc. 72 (1989) 1555
  * [2] K. Ishikawa, K. Yoshikawa and N. Okada: Phys. Rev. B 37 (1988) 5852
  * [3] F. Jona and G. Shirane: Ferroelectric Crystals (Dover, New York, 1962)
  * [4] J. F. Scott, H. M. Duiker, P. D. Beale, B. Pouligny, K. Dimmler, M. Parris, D. Butler and S. Eaton: Physica B 150 (1988) 160
  * [5] A. Hadi and R. Thomas: Ferroelectrics 59 (1984) 221
  * [6] I. P. Batra and B. D. Silverman: Solid State Commun. 11 (1972) 291
  * [7] M.-H. Lee, A. Halliyal and R. E. Newnham: Ferroelectrics 87 (1988) 71
  * [8] G. Arlt, D. Hennings and G. de With: J. Appl. Phys. 58 (1985) 1619
  * [9] Wan Y. Shih, Wei-Heng Shih and Ilhan A. Aksay: Phys. Rev. B 50 (1994) 15575
  * [10] W. L. Zhong, Y. G. Wang, P. L. Zhang and B. D. Qu: Phys. Rev. B 50 (1994) 698
  * [11] I. Rychetsky and O. Hudak, J.Phys.: Condens. Matter 9 (1997) 4955-4965
  * [12] O. Hudak, I. Rychetsky and J. Petzelt: Ferroelectrics 208-209 (1998) 429-447
  * [13] O. Hudak: CO-MAT-TECH’99 7th International Scientific Conference Proceedings, 19-20 October 1999, Trnava.
  * [14] W.Y. Shih, W.-H. Shih and I.A. Aksay: Phys.Rev. B50 (1994) 15 575 - 15 585
  * [15]
    * •

W.J.Merz: Phys.Rev. 118 (1954) 690

    * •

W.Kinase and H.Takahashi: J.Phys.Soc.Jpn. 12 (1957) 464

    * •

W.Kinase: Prog.Theor.Phys. (Kyoto) 13 (1955) 529

    * •

V.A.Zhirnov: Zh.Eksp.Teor.Fiz. 35 (1959) 1175

    * •

A.F.Devonshire: Philos.Mag. 42 (1951) 1065

    * •

W.Cao and E.Cross: Phys.Rev. B 44 (1991) 5

  * [16] M.E.Lines and A.M.Glass: Principles and Application of Ferroelectrics and Related Materials, Clarendon Press, Oxford, 1977
  * [17] W.L.Zhong, Y.G.Wang, P.L.Zhang and B.D.Qu: Phys.Rev. B50 (1994) 698-703
  * [18] O.Hudak: published in 1981 as a part of papers for thesis exams, Institute of Physics, Czechoslovak Academy of Sciences, Prague, 1981
  * [19] F.Falk: Z.Phys. B 51 (1983) 177-185
  * [20] W.Cao and E.Cross: Phys.Rev. B 44 (1991) 5


# On Generating and Labeling Network Traffic with Realistic, Self-Propagating
Malware ††thanks: The research reported in this document/presentation was
performed in connection with contract number W911NF-18-C- 0019 with the U.S.
Army Contracting Command - Aberdeen Proving Ground (ACC-APG) and the Defense
Advanced Research Projects Agency (DARPA).

Molly Buchanan, Jeffrey W. Collyer, Jack W.
Davidson, Jason D. Hiser, Alastair Nottingham  University of Virginia,
Charlottesville, VA, U.S.A.
<EMAIL_ADDRESS>Saikat Dey, Mark Gardner, Jeffry
Lang  Virginia Tech, Blacksburg, VA, U.S.A.
<EMAIL_ADDRESS>Alina Oprea  Northeastern University,
Boston, MA, U.S.A.
<EMAIL_ADDRESS>
###### Abstract

Research and development of techniques which detect or remediate malicious
network activity require access to diverse, realistic, contemporary data sets
containing labeled malicious connections. In the absence of such data, said
techniques cannot be meaningfully trained, tested, and evaluated.
Synthetically produced data containing fabricated or merged network traffic is
of limited value as it is easily distinguishable from real traffic by even
simple machine learning (ML) algorithms. Real network data is preferable, but
while ubiquitous is broadly both sensitive and lacking in ground truth labels,
limiting its utility for ML research.

This paper presents a multi-faceted approach to generating a data set of
labeled malicious connections embedded within anonymized network traffic
collected from large production networks. Real-world malware is defanged and
introduced to simulated, secured nodes within those networks to generate
realistic traffic while maintaining sufficient isolation to protect real data
and infrastructure. Network sensor data, including this embedded malware
traffic, is collected at a network edge and anonymized for research use.

Network traffic was collected and produced in accordance with the
aforementioned methods at two major educational institutions. The result is a
highly realistic, long term, multi-institution data set with embedded data
labels spanning over 1.5 trillion connections and over a petabyte of sensor
log data. The usability of this data set is demonstrated by its utility to our
artificial intelligence and machine learning (AI/ML) research program.

###### Index Terms:

Artificial Intelligence, Machine Learning, Cyber security, Network Data
Collection, Attack Recreation, Mirai

##  I Introduction

Computing networks face a variety of attacks, but large network traffic
volumes and the need to keep connected devices operational make it infeasible
for limited personnel to manually detect and respond to every attack. Though
desirable, automatic or semi-automatic attack detection and remediation
techniques are difficult to develop due to the dearth of publicly available,
representative data sets [1]. Considering their ability to model network
traffic and pick out unusual patterns, AI/ML techniques are particularly
suited to automatic network attack detection. AI methods can be _supervised_
(learning to classify malicious and benign traffic) [2, 3], or _unsupervised_
(learning the normal traffic distribution and detecting anomalies) [4, 5].
Supervised AI techniques rely on the availability of ground truth samples of
malicious traffic for training; both methods need malicious samples for
testing and evaluation. However, infected machines usually generate copious
legitimate traffic, thus necessitating fine-grained labeling of the
exclusively malicious activity. One of the main challenges in developing AI/ML
for network security is _obtaining realistic network traffic including benign
and malicious connections with fine-grained labeling_ [6].

Of course, one cannot simply permit unmodified malware to infect in-use
machines and record the ensuing traffic to generate the necessary data. A
real-world network cannot reasonably and responsibly risk the operational
interruption and data loss that malware can cause. But continuously recording
traffic and waiting for an inevitable infection is also insufficient due to
the uncertainty and difficulty of correctly identifying malicious traffic.
Modern tools like VirusTotal are not adequately accurate to provide sufficient
information to determine which network nodes were infected and when, or which
traffic from said nodes was malicious [7].

To address the AI/ML requirements and concerns of using real malware on a real
network, these data sets can be generated by introducing defanged malware into
the real network and capturing and anonymizing the resulting traffic, which
will include a realistic mix of benign and malicious traffic. The remainder of
the paper describes how this goal was achieved to facilitate AI/ML research
and experimentation.

The major contributions of this work are:

  * •

a workable system to capture and anonymize network traffic,

  * •

a model to safely deploy malware in a real network, including a realization of
the Mirai virus, and

  * •

a model for fine-grained labeling of the generated malicious traffic in the
network capture.

##  II Anonymous Data Collection and Use

This section briefly explores the collection and anonymization of enterprise
scale network sensor data to produce a diverse, realistic, long-term data set
for developing and training machine learning algorithms and AI approaches.

The data set is composed of anonymized network sensor data from two large,
disjoint, and geographically separated enterprise networks at the University
of Virginia (UVA) and Virginia Tech (VT); these networks differ in topology,
service and application ecosystems, and baseline behavior. Figure 1 provides
an overview of the data collection and anonymization process.

Figure 1: Overview of the collection, anonymization and processing of network
sensor data.

###  II-A Data Collection

The generated data set includes a variety of network sensor and appliance logs
that collectively provide a detailed trace of the network state and
user/device behavior. While packet traces may provide a more complete view of
network behavior and additional metrics, sensor logs are preferred because
they are (a) pre-aggregated and normalized, which simplifies anonymization
without sacrificing application level detail, (b) appreciably smaller and more
compressible, making long term storage less expensive, and (c) significantly
less computationally expensive and complicated to process at enterprise scale.

Zeek [8] provides the primary source of network activity data; it generates a
wide selection of inter-connected logs across a broad range of protocols and
field types. Zeek logs are supplemented by a range of other sensor and
appliance logs that provide context regarding the network’s state. These
include logs generated by FireEye [9] — which enumerate detected threats such
as known malware callbacks — and STINGAR honeypot logs [10] — which provide
community-sourced threat intelligence from a distributed honeypot network.
Together, these provide a valuable source of ground truth to seed discovery of
natural malicious behavior in Zeek logs. UVA additionally collects NAT [11]
(Network Address Translation), DHCP (Dynamic Host Configuration Protocol), and
asset registration logs to map between public and private addresses.

The creation of this data set is achieved through partnership with and
oversight from Information Security departments and professionals at UVA and
VT. As network traffic transitions across the UVA and VT network edges,
traffic is mirrored through a packet broker to sensor appliances within the
respective networks for log creation. All logs are archived, allowing future
retrieval and reprocessing when necessary.

Though collection is broadly similar at both institutions, it differs in
traffic composition, the types and placement of sensor appliances, and how
data is transitioned through anonymization to be made accessible to
researchers. Some of the primary differences between UVA and VT data
collection are shown in Table I.

| UVA | VT
---|---|---
Network Border Traffic
IPv6 | No | Yes
NAT Type[11] | Carrier Grade NAT | Traditional NAT
Sensor Logs
Zeek | Yes [Corelight [12]] | Yes [Custom]
FireEye | Yes | No
STINGAR | Yes | Yes
NAT, DHCP, Asset | Yes | No
Current Zeek Dataset (Combined)
Connections Records |  ~1.5 Trillion
Total Volume |  ~1 Petabyte
TABLE I: Summary of UVA and VT data collection.

###  II-B Log Anonymization

Collected network sensor data contains significant PII (Personally
Identifiable Information) and other sensitive information. To protect network
users and organizational security, sensitive fields must be obfuscated or
removed from logs before the data set can be made accessible to researchers.
Anonymization through overwriting or otherwise destroying field data [13, 14],
while effective, significantly reduces the fidelity of anonymized records
[15]. In particular, fields anonymized by destructive methods cannot be
leveraged effectively in inter-record and cross-network comparisons.

To retain a measure of fidelity and comparability in field data, the removal
or destruction of field values is avoided wherever possible. Instead, a
combination of careful field parsing and one-way cryptographic functions are
used to eliminate identifying data while preserving many of the features and
patterns inherent in the raw logs. Such functions include prefix preserving IP
address anonymization [16, 17] and custom hash-based functions for other field
types. Hashes use a 256-bit secret key as salt to (a) add entropy to fields
and protect against collision-based attacks on anonymization and (b) ensure
consistency across log types and institutions over time. Anonymization is
subject to extensive testing to identify emergent issues, and is regularly
updated and improved. When necessary, archived logs are re-anonymized to take
advantage of new features and correct identified anonymization errors.

While every effort is made to ensure all data elements are correctly
anonymized, the complex nature of the produced data set presents an
unavoidable known risk of anonymization errors in edge cases. This risk is
addressed through mitigating policy and infrastructure-based controls to
protect edge cases when anonymization is insufficient or otherwise fails.

To this end, sensor data is only made available on Sentinel: a dedicated,
access-controlled High Performance Computing (HPC) cluster provisioned to
support AI/ML research (see Figure 1). Data access is limited to approved
researchers and subject to restrictions that prohibit data exfiltration and
govern how records may be used in publications. Anonymized sensor data is
published daily (i) as compressed files, (ii) in Apache Kafka [18] as a
message streams, and (iii) in Apache Druid [19] which supports fast SQL-like
(Structured Query Language-like) queries. Figure 2 provides a high-level view
of the data set available on Sentinel.

Figure 2: Data volume and number of distinct connections per day for the
combined Zeek data set. It currently takes between 30 and 45 minutes to
anonymize a days worth of traffic at a given institution using a single
server.

As the data collected at the network edge is composed of real traffic from
institutions that actively defend their networks, actual instances of known
malware campaigns and malicious traffic are uncommon. The following sections
present a method for safely and realistically introducing attack traffic into
sensor logs through attack recreation exercises. These provide additional and
verifiable ground truth for malicious activity in the data set.

##  III Attack Recreation

A key aspect of the proposed data set production approach is the introduction
of realistic malware traffic to the network. Said generated traffic should
combine with the network’s natural traffic using standard network protocols,
yet machines and appliances should be secure from compromise. To achieve this
balance of realism and safety, real-world malware is defanged and carefully
deployed on real networked systems.

This approach has two key advantages. First, as malware traffic is naturally
mixed in the network, no artifacts from mixing are evident in the data set.
This feature is important because otherwise machine learners can easily
determine which traffic is which by examining artifacts unintentionally added
during the traffic merging process. Second, the malware never resides on
operational systems with sensitive data. This feature is essential to obtain
permission for deployment from network operators, as normal network operation
can not be interrupted, and intentional data exposure will not be tolerated.

As a proof of concept, the Mirai virus was defanged and deployed on the
University of Virginia (UVA) and Virginia Tech (VT) networks. Mirai, as
depicted in Figure 3, operates by leveraging already infected nodes to scan
the network to locate new nodes that respond to telnet or ssh. It then brute-
forces the passwords using a built-in dictionary of common/default credentials
[20, 21]. When a new vulnerable node is detected, an infected node reports
this to the report server, which instructs the malware loader to infect the
vulnerable node. After infection, the vulnerable node reports to the command
and control server as a bot; at this point, the bot master can leverage the
new addition to its botnet for other purposes, such as a DDoS attack.

Figure 3: Mirai malware, as deployed for collecting datasets

Mirai is an excellent candidate for experimentation because it was
particularly successful real-world malware, variants of which continue to be
seen in the wild. Mirai infected over 65,000 on its first day of deployment
and totaled over 600,000 nodes at its peak. Additionally, since Mirai and its
control components were open-sourced, it was easy to safely defang. Mirai was
defanged by: 1) Removing portions of the infection that might cause damage. 2)
Replacing the default credential dictionary with 20 and 25 character randomly-
selected alpha-numeric usernames and passwords, respectively, which were
statistically unlikely to match any real-world accounts. 3) Adjusting the
malware loader to safeguard user nodes by only permitting infection of nodes
in the pre-selected vulnerable nodes pool. 4) Limiting scanning traffic to
permissible and non-critical nodes - for example, UVA’s hospital system is not
scanned.

To provide further evidence to the UVA and VT network and security groups,
unit and integration tests were developed for defanged components. Test
deployments in a sandbox verified that only acceptable IPs would be contacted.
While these techniques are specific to Mirai, safely handling other malware
should be possible, especially via careful application of network firewall
rules.

The precautions installed in Mirai were sufficient to convince UVA and VT’s
network operation centers to allow Mirai to scan a large subset of public IP
space. The defanged Mirai has scanned a large swath of over 500,000 IPs at UVA
and VT.

Nodes sending only malicious traffic might be conspicuous to machine learners,
so the Caldera _human_ plugin was run to add mitigating benign traffic. The
plugin randomly browses websites, performs google searches of random
sentences, and provides realistic delays between operations [22, 23, 24].

To mimic an outside attacker’s command and control (C2) server, C2 components
were positioned on the third-party NSF-funded Chameleon Cloud system [25].
Traffic generated as infected nodes in each network contact external C2
components or scan vulnerable nodes in the adjacent network is collected as it
transits the network edge, resulting in realistic merging of the attack
traffic in sensor logs.

##  IV Generated Labeled Traffic

A key requirement for both training and evaluating AI/ML algorithms is labeled
data. Each connection captured in the anonymized logs includes associated
metadata which determines whether it was natural traffic on the network or
generated during the attack recreation. Ideally, separate labels would be
generated for each phase of the attack; for example, scanning traffic might be
labeled differently than infection traffic that downloads malware to a new
system.

In this experimental setup, such labels can typically be applied in a post-hoc
manner by inspecting particular fields in the network traffic and applying a
set of simple heuristics. If Mirai uses port 80 of a particular web server to
download the malware from a particular set of URLs, simply filter traffic that
matches those criteria and label them as malware download traffic.

A library was developed to label traffic captured during a Mirai attack
recreation event according to the aforementioned filtering technique. For the
defanged Mirai, scanning traffic occurs on ports 23 and 2323, virus download
happens from a web sever set up on port 80, and C2 communication happens on
ports 23 and 48101. Furthermore, a record is kept of the (anonymized) IPs of
all the vulnerable nodes in the experiment. Any traffic not to or from these
IPs is labeled as _natural_. Traffic to/from these IPs which matches one of
the above rules is labeled as indicated; other traffic from the experimental
nodes is labeled as generated, benign traffic.

These simple rules were able to label 99.9% of all traffic during a 6 hour
experiment. Only a handful of unlabeled connections remained, all of which
seemed to be oddities of the capture process (e.g., when traffic capture was
started during an already active connection).

One might wonder whether the network’s natural traffic is benign or could
contain traffic from an infected node. In a network of sufficient size and
complexity, it is likely that some traffic is malicious (hence the application
of the natural rather than benign label.) However, the traffic labeled
malicious during attack recreations is verifiable so, and can be accurately
used for training and evaluation of AI/ML techniques. Further, during
evaluation of a (good) algorithm, little traffic should be detected as
malicious; such traffic can (and should) be analyzed by an expert to determine
whether it is truly malicious or a false positive detection. In fact, this
type of event happened during the associated AI/ML research, where previously
unknown malware was detected in the network.

The labeled Mirai attack was used to evaluate an ML-based system for self-
propagating malware detection [5]. The PORTFILER system creates aggregated
network traffic features for a set of monitored ports, learns the legitimate
network profiles on those ports during training in an unsupervised manner, and
detects and ranks anomalies at testing time. Labeling the different attack
stages (e.g., scanning, C2) was particularly useful for evaluating this
detector. In use, PORTFILER detected scanning traffic with high precision and
recall and low false positive rates; detection results were more accurate at
higher scanning rates. The detailed analysis of PORTFILER’s performance was
made possible by fine-grained labeling of individual connections and attack
stages. Moreover, by analyzing the traffic of all infected machines, it was
possible to identify and confirm the C2 server IP based on the number of
connections on port 23 and the large duration of those connections compared to
legitimate connections. In future experiments, likewise defanged evasive
variants of Mirai and other malware will be generated, and novel ML techniques
will be developed to detect them more robustly.

##  V Related Work

While IP address anonymization is well understood [16, 17, 26] and significant
related work exists in the area of network traffic anonymization, most
approaches focus on packet traces. A wide variety of approaches to trace
anonymization have been developed, both in software [27, 28, 14, 29, 30, 13,
31, 32, 33], and in hardware [26, 34, 35], although many of these approaches
are now defunct. While some implementations focus on appliance and service
logs [36, 37, 38, 39, 40], these approaches are typically quite destructive as
they are intended for broad sharing. Published available data sets [41, 42,
43] tend to have a limited duration, are more specialized and/or synthetic,
and are not representative of normal enterprise traffic [1].

Previous work on generating and labeling data synthetically also exists [44,
45, 46, 47, 48]. Synthetic techniques suffer to various degrees with
artificial artifacts, but issues with unrealistic data have been generally
accepted in light of the working assumption that real data on real networks is
too cumbersome to generate and too sensitive to provide to researchers. The
approach outlined in this paper contrasts said assumption by demonstrating the
viability of a more natural means of data production and illustrating its
successful use in ML research.

##  VI Future Work and Summary

A good way to create data sets for AI/ML training and evaluation is to outfit
existing networks for collection and anonymization. This approach can be
achieved with commodity hardware and software solutions. Malicious traffic can
be carefully added to the existing network by defanging real malware and
controlling where in the network it is deployed. Labeling the traffic can be
achieved through simple heuristics based on knowledge of the malware’s
behavior and the known IPs of the infected machines.

Utilizing this approach and working with UVA and VT’s network operators,
extant networks were instrumented, traffic collection and anonymization were
measured, defanged Mirai was deployed, and secure access infrastructure for
researchers was provided. Simple labeling heuristics were able to generate
useful labels for internal research programs; their success in using the
produced data set to train models that detected malware demonstrates the
proposed approach’s feasibility.

In the future, the range of collected sensor data will be expanded, and field
coverage for anonymization will continue to be refined and improved. In
particular, incorporation of host-based sensor data is planned to supplement
network logs. Additionally, there are plans to leverage Mitre’s Caldera
infrastructure to deploy more sophisticated, subtle attacks [24].

## Acknowledgments

The authors would like to thank the late Dr. Malathi Veeraraghavan for her
significant contributions to this work.

## References

  * [1] W. Haider, J. Hu, J. Slay, B. Turnbull, and Y. Xie, “Generating realistic intrusion detection system dataset based on fuzzy qualitative modeling,” _J. Netw. Comput. Appl._ , vol. 87, pp. 185–192, 2017.
  * [2] L. Bilge, S. Sen, D. Balzarotti, E. Kirda, and C. Kruegel, “Exposure: A passive DNS analysis service to detect and report malicious domains,” _ACM Trans. Inf. Syst. Secur._ , vol. 16, no. 4, Apr. 2014. [Online]. Available: https://doi.org/10.1145/2584679
  * [3] A. Oprea, Z. Li, R. Norris, and K. Bowers, “MADE: Security analytics for enterprise threat detection,” in _In 34th Annual Computer Security Applications Conference_ , 2018, pp. 124–136.
  * [4] T.-F. Yen, A. Oprea, K. Onarlioglu, T. Leetham, W. Robertson, A. Juels, and E. Kirda, “Beehive: Large-scale log analysis for detecting suspicious activity in enterprise networks,” in _In 29th Annual Computer Security Applications Conference_ , 2013, pp. 199–208.
  * [5] T. Ongun, O. Spohngellert, B. Miller, S. Boboila, A. Oprea, T. Eliassi-Rad, J. Hiser, A. Nottingham, J. Davidson, and M. Veeraraghavan, “PORTFILER: Port-level network profiling for self-propagating malware detection,” https://www.ccs.neu.edu/home/tongun/files/TrafficProfiler.pdf, 2021.
  * [6] R. Sommer and V. Paxson, “Outside the closed world: On using machine learning for network intrusion detection,” in _Proceedings of the 2010 IEEE Symposium on Security and Privacy_ , ser. SP ’10. USA: IEEE Computer Society, 2010, p. 305–316. [Online]. Available: https://doi.org/10.1109/SP.2010.25
  * [7] (2021) The VirusTotal website. [Online]. Available: https://www.virustotal.com/
  * [8] The Bro Project, “The Zeek Network Security Monitor,” Online, 2021. [Online]. Available: https://www.zeek.org/
  * [9] FireEye, Inc., “FireEye,” Online, 2021. [Online]. Available: https://www.fireeye.com/
  * [10] Duke STINGAR, “Shared Threat Intelligence for Network Gatekeeping and Automated Response,” Online, last Updated: 10/2020. [Online]. Available: https://stingar.security.duke.edu/
  * [11] K. B. Egevang and P. Srisuresh, “RFC3022: Traditional IP Network Address Translator (Traditional NAT),” USA, 2001.
  * [12] Corelight, Inc., “Corelight,” Online, 2021. [Online]. Available: https://www.corelight.com/
  * [13] ISI, “dag scrubber,” Online, Jan. 2016. [Online]. Available: https://ant.isi.edu/software/dag_scrubber/index.html
  * [14] W. Yurcik, C. Woolam, G. Hellings, L. Khan, and B. Thuraisingham, “SCRUB-tcpdump: A multi-level packet anonymizer demonstrating privacy/analysis tradeoffs,” in _2007 Third International Conference on Security and Privacy in Communications Networks and the Workshops - SecureComm 2007_ , Sep. 2007, pp. 49–56.
  * [15] W. Brockelsby and R. Dutta, “A Graded Approach to Network Forensics with Privacy Concerns,” in _2019 International Conference on Computing, Networking and Communications (ICNC)_ , Feb 2019, pp. 292–297.
  * [16] J. Fan, J. Xu, M. H. Ammar, and S. B. Moon, “Prefix-preserving ip address anonymization: measurement-based security evaluation and a new cryptography-based scheme,” _Computer Networks_ , vol. 46, no. 2, pp. 253 – 272, 2004. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S1389128604001197
  * [17] Y. Pradkin, “cryptopANT IP Address Anonymization Library,” Online, 2018. [Online]. Available: https://ant.isi.edu/software/cryptopANT/index.html
  * [18] Apache Software Foundation, “Apache kafka,” Online, 2021. [Online]. Available: https://kafka.apache.org/
  * [19] ——, “Apache druid,” Online, 2021. [Online]. Available: https://druid.apache.org/
  * [20] C. Kolias, G. Kambourakis, A. Stavrou, and J. Voas, “DDoS in the IoT: Mirai and other botnets,” _Computer_ , vol. 50, no. 7, pp. 80–84, 2017\.
  * [21] M. Antonakakis, T. April, M. Bailey, M. Bernhard, E. Bursztein, J. Cochran, Z. Durumeric, J. A. Halderman, L. Invernizzi, M. Kallitsis _et al._ , “Understanding the mirai botnet,” in _26th $\\{$USENIX$\\}$ security symposium ($\\{$USENIX$\\}$ Security 17)_, 2017, pp. 1093–1110.
  * [22] (2021) The Caldera Human Plugin on GitHub. [Online]. Available: https://github.com/mitre/human
  * [23] J. A. Booz, “Towards scalable automated vulnerability scanning & exploitation,” Ph.D. dissertation, Carnegie Mellon University, 2020.
  * [24] (2021) Mitre’s Caldera on GitHub. [Online]. Available: https://github.com/mitre/caldera
  * [25] K. Keahey, J. Anderson, Z. Zhen, P. Riteau, P. Ruth, D. Stanzione, M. Cevik, J. Colleran, H. S. Gunawi, C. Hammock, J. Mambretti, A. Barnes, F. Halbach, A. Rocha, and J. Stubbs, “Lessons learned from the chameleon testbed,” in _Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC ’20)_. USENIX Association, July 2020.
  * [26] H. Kim and A. Gupta, “ONTAS: Flexible and Scalable Online Network Traffic Anonymization System,” in _Proceedings of the 2019 Workshop on Network Meets AI & ML_, ser. NetAI’19. New York, NY, USA: ACM, 2019, pp. 15–21. [Online]. Available: http://doi.acm.org/10.1145/3341216.3342208
  * [27] Y. Lin, P. Lin, S. Wang, I. Chen, and Y. Lai, “PCAPLib: A System of Extracting, Classifying, and Anonymizing Real Packet Traces,” _IEEE Systems Journal_ , vol. 10, no. 2, pp. 520–531, June 2016.
  * [28] F. Gringoli, “tcpanon,” Online, Jan. 2009. [Online]. Available: http://netweb.ing.unibs.it/~ntw/tools/tcpanon/
  * [29] Lawrence Brkley National Laboratory and ICSI, “tcpmkpub,” Online, 2005. [Online]. Available: http://www.icir.org/enterprise-tracing/tcpmkpub.html
  * [30] CAIDA, “Anonymization Tools Taxonomy,” Online, 2004. [Online]. Available: https://www.caida.org/tools/taxonomy/anontaxonomy.xml
  * [31] ISI, “LANDER Trace Software,” Online, May 2017. [Online]. Available: https://ant.isi.edu/software/lander/index.html
  * [32] D. Koukis, S. Antonatos, D. Antoniades, E. P. Markatos, and P. Trimintzios, “A Generic Anonymization Framework for Network Traffic,” in _2006 IEEE International Conference on Communications_ , vol. 5, June 2006, pp. 2302–2309.
  * [33] R. Pang and V. Paxson, “A High-level Programming Environment for Packet Trace Anonymization and Transformation,” in _Proceedings of the 2003 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications_ , ser. SIGCOMM ’03. New York, NY, USA: ACM, 2003, pp. 339–351. [Online]. Available: http://doi.acm.org/10.1145/863955.863994
  * [34] S. Shohata, Y. Nakamura, and H. Nishi, “Hardware for Accelerating Anonymization Transparent to Network,” in _2018 Sixth International Symposium on Computing and Networking (CANDAR)_ , Nov 2018, pp. 181–187.
  * [35] A. Fukuhara, T. Iwai, Y. Sakuma, and H. Nishi, “Implementation of Content-Based Anonymization Edge Router on NetFPGA,” in _2019 IEEE 13th International Symposium on Embedded Multicore/Many-core Systems-on-Chip (MCSoC)_ , Oct 2019, pp. 123–128.
  * [36] A. Slagell, J. Wang, and W. Yurcik, “Network log anonymization: Application of crypto-pan to cisco netflows,” 2004.
  * [37] K. Luo, Y. Li, A. Slagell, and W. Yurcik, “Canine: A netflows converter/anonymizer tool for format interoperability and secure sharing,” 01 2005.
  * [38] Y. Li, A. Slagell, K. Luo, and W. Yurcik, “Canine: A combined conversion and anonymization tool for processing netflows for security,” 01 2005.
  * [39] A. Slagell, K. Lakkaraju, and K. Luo, “FLAIM: A multi-level anonymization framework for computer and network logs,” in _20th Large Installation System Administration Conference (LISA 06)_. Washington, D.C.: USENIX Association, Dec. 2006. [Online]. Available: https://www.usenix.org/conference/lisa-06/flaim-multi-level-anonymization-framework-computer-and-network-logs
  * [40] ISI, “Dnsanon: extract DNS traffic from pcap to text with optionally anonymization,” Online, May 2019. [Online]. Available: https://ant.isi.edu/software/dnsanon/index.html
  * [41] M. J. M. Turcotte, A. D. Kent, and C. Hash, _Unified Host and Network Data Set_. World Scientific, nov 2018, ch. Chapter 1, pp. 1–22. [Online]. Available: https://www.worldscientific.com/doi/abs/10.1142/9781786345646_001
  * [42] A. D. Kent, “Comprehensive, Multi-Source Cyber-Security Events,” Los Alamos National Laboratory, 2015.
  * [43] N. Moustafa and J. Slay, “Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set),” in _2015 Military Communications and Information Systems Conference (MilCIS)_ , Nov 2015, pp. 1–6.
  * [44] M. Landauer, F. Skopik, M. Wurzenberger, W. Hotwagner, and A. Rauber, “Have it your way: Generating customized log datasets with a model-driven simulation testbed,” _IEEE Transactions on Reliability_ , pp. 1–14, 2020.
  * [45] I. Sharafaldin, A. H. Lashkari, and A. Ghorbani, “Toward generating a new intrusion detection dataset and intrusion traffic characterization,” in _ICISSP_ , 2018.
  * [46] S. Choi, J. Choi, J.-H. Yun, B.-G. Min, and H. Kim, “Expansion of ics testbed for security validation based on mitre att&ck techniques,” in _CSET @ USENIX Security Symposium_ , 2020.
  * [47] W. Haider, J. Hu, J. Slay, B. Turnbull, and Y. Xie, “Generating realistic intrusion detection system dataset based on fuzzy qualitative modeling,” _J. Netw. Comput. Appl._ , vol. 87, pp. 185–192, 2017.
  * [48] F. Skopik, G. Settanni, R. Fiedler, and I. Friedberg, “Semi-synthetic data set generation for security software evaluation,” _2014 Twelfth Annual International Conference on Privacy, Security and Trust_ , pp. 156–163, 2014.