# NNG-Mix: Improving Semi-supervised Anomaly Detection with Pseudo-anomaly
Generation

Hao Dong1 Gaëtan Frusque2 Yue Zhao3 Eleni Chatzi1 Olga Fink2
1ETH Zürich 2EPFL 3University of Southern California

###### Abstract

Anomaly detection (AD) is essential in identifying rare and often critical
events in complex systems, finding applications in fields such as network
intrusion detection, financial fraud detection, and fault detection in
infrastructure and industrial systems. While AD is typically treated as an
unsupervised learning task due to the high cost of label annotation, it is
more practical to assume access to a small set of labeled anomaly samples from
domain experts, as is the case for semi-supervised anomaly detection. Semi-
supervised and supervised approaches can leverage such labeled data, resulting
in improved performance. In this paper, rather than proposing a new semi-
supervised or supervised approach for AD, we introduce a novel algorithm for
generating additional pseudo-anomalies on the basis of the limited labeled
anomalies and a large volume of unlabeled data. This serves as an augmentation
to facilitate the detection of new anomalies. Our proposed algorithm, named
Nearest Neighbor Gaussian Mixup (NNG-Mix), efficiently integrates information
from both labeled and unlabeled data to generate pseudo-anomalies. We compare
the performance of this novel algorithm with commonly applied augmentation
techniques, such as Mixup and Cutout. We evaluate NNG-Mix by training various
existing semi-supervised and supervised anomaly detection algorithms on the
original training data along with the generated pseudo-anomalies. Through
extensive experiments on $57$ benchmark datasets in ADBench, reflecting
different data types, we demonstrate that NNG-Mix outperforms other data
augmentation methods. It yields significant performance improvements compared
to the baselines trained exclusively on the original training data. Notably,
NNG-Mix yields up to $16.4\%$, $8.8\%$, and $8.0\%$ improvements on Classical,
CV, and NLP datasets in ADBench. Our source code will be available at
https://github.com/donghao51/NNG-Mix.

##  1 Introduction

Anomaly detection (AD) aims to identify irregular patterns or observations
that deviate significantly from the expected behavior within a dataset. It
finds applications across various domains, including social media analysis
yu2017ring , rare disease detection zhao2021suod , complex systems
frusque2023non ; zhao2023dynamic ; michau2021unsupervised , and autonomous
driving bogdoll2022anomaly ; SuperFusion ; dong2023jras . Due to the rarity of
anomaly events and high annotation cost han2022adbench , AD is typically
treated as an unsupervised learning problem ruff2018deep with the assumption
that the majority of the unlabeled data in the training dataset represents
normal instances. Unsupervised AD algorithms typically learn a model or
distribution that accurately describes normal behavior and detect deviations
from this description as anomalies. Various unsupervised methods have been
proposed, including shallow machine learning approaches like One-Class SVM
scholkopf2001estimating , Kernel Density Estimation
vandermeulen2013consistency , Isolation Forest liu2008isolation , and
Empirical Cumulative Distribution Functions li2022ecod along with deep-
learning-based approaches such as Deep Support Vector Data Description
(DeepSVDD) ruff2018deep and Deep Autoencoding Gaussian Mixture Model (DAGMM)
zong2018deep . Recent literature indicates that unsupervised methods are
effective only when their assumptions align with the characteristics of the
considered dataset. For instance, the $k$ nearest neighbor
ramaswamy2000efficient functions optimally as a global detector when the
dataset indeed contains global anomalies, and local outlier factor
breunig2000lof performs well with local outliers only. However, this scenario
is not consistently mirrored in real-world applications. The real outliers can
be a mix of multiple types of outliers and are difficult to identify
beforehand.

Beyond the option of adopting fully unsupervised methods, accessing some
labeled anomalies in addition to unlabeled data is feasible in many real-world
applications, such as the inspection of complex systems frusque2023non ;
zhao2023dynamic ; michau2021unsupervised . A recent study han2022adbench
demonstrated that the introduction of a small number of anomalies, whether
through a semi-supervised or supervised approach, consistently resulted in
improvements in anomaly detection with respect to the unsupervised case.
Anomalies are often associated with safety-critical events, leading to their
rare occurrences. Therefore, the majority of samples in a dataset usually
represent normal data. While some anomalous samples may have been identified
and labeled by domain experts, it is important to note that not all anomalous
samples may have been recognized or captured. The number of labeled anomalies
is often scarce compared to the unlabeled data. With only a small number of
such labeled samples available, several semi-supervised AD algorithms have
been proposed, including Deep Semi-supervised Anomaly Detection (DeepSAD)
DeepSAD , Pairwise Relation prediction Network (PReNet) pang2023deep , and
Extreme Gradient Boosting Outlier Detection (XGBOD) zhao2018xgbod . For
instance, DeepSAD DeepSAD introduces a novel loss function to bring normal
data closer to a fixed center and position anomalies farther away. In
contrast, supervised methods such as MLP rosenblatt1958perceptron , Feature
Tokenizer + Transformer (FTTransformer) gorishniy2021revisiting , and
Categorical Boosting (CatB) prokhorenkova2018catboost treat anomaly detection
as binary classification. However, they are limited to detecting known types
of anomalies aggarwal2017introduction , as the training dataset may not cover
all possible types of anomalies. Moreover, for both semi-supervised and
supervised approaches, deep learning models might struggle to learn effective
representations if labeled anomalies are extremely limited.

Unlike existing solutions that primarily focus on creating novel semi-
supervised DeepSAD ; pang2023deep ; zong2018deep or supervised
rosenblatt1958perceptron ; gorishniy2021revisiting ; prokhorenkova2018catboost
algorithms to better leverage the use of limited labeled data for anomaly
detection, our approach focuses on harnessing the potential of generating
pseudo-anomalies to improve AD performance. More specifically, we generate
pseudo-anomalies by leveraging the limited set of labeled anomalies and a
large amount of unlabeled data. Then we train various semi-supervised or
supervised AD algorithms on the original training data in conjunction with the
generated pseudo-anomalies. With the availability of more anomaly data for
training, we demonstrate performance improvements compared with the original
AD algorithms.

Figure 1: Pseudo-anomaly generation using Mixup zhang2018mixup and our
proposed NNG-Mix. Mixup using all training data introduces noise samples
within the distribution of unlabeled data, while using only labeled anomalies
underestimates the information from unlabeled data and also injects some noise
samples that are within unlabeled data. In contrast, our NNG-Mix fully
exploits the information from both labeled anomalies and unlabeled data
samples to generate pseudo-anomalies effectively.

Pseudo-anomaly generation is similar to data augmentation, a technique
employed to mitigate overfitting by training models on slightly modified
copies of existing data. Well-established data augmentation methods like Mixup
zhang2018mixup , Cutout devries2017cutout , CutMix yun2019cutmix , and the
addition of random Gaussian noises can be readily adapted for pseudo-anomaly
generation. For example, Mixup creates new samples by the convex combinations
of sample pairs. However, traditional data augmentation methods commonly
adopted in computer vision may not be optimal for the anomaly detection task.
If we mix the data randomly sampled from all training data using Mixup, there
may exist scenarios where generated pseudo-anomalies fall within the
distribution of unlabeled data. Fig. 1 (a) shows normal samples surrounded by
two different anomaly types. Mixup generates new anomaly samples that overlap
with the distribution of the unlabeled samples, as demonstrated in Fig. 1 (b).
Given the general assumption that the majority of unlabeled data is normal,
the introduction of such generated pseudo-anomalies could significantly hinder
the performance of anomaly detection. Conversely, if we only mix the data
randomly sampled from labeled anomalies, as illustrated in Fig. 1 (c), the
output space for generated pseudo-anomalies becomes excessively constrained,
because of the limited utilization of information from unlabeled data. Mixing
only the labeled anomalies can also introduce noise samples that are within
the distribution of unlabeled data, potentially adversely impacting the
performance of anomaly detection.

In this paper, we propose a Nearest Neighbor Gaussian Mixup (NNG-Mix)
algorithm for generating pseudo-anomalies that optimally exploit information
from both labeled anomalies and unlabeled data. To prevent the generation of
anomaly samples that are within the distribution of the normal data, we
exclusively mix the anomalies with their top $k$ nearest neighbors from the
unlabeled data. Additionally, we introduce random Gaussian noise to the
samples before applying Mixup, thereby expanding the potential space for
generating pseudo-anomalies. As shown in Fig. 1 (d), NNG-Mix effectively
generates pseudo-anomalies without introducing undesired noise samples. Our
main contributions are the following:

  * •

Novel Anomaly Detection Framework. We investigate improving semi-supervised
anomaly detection performance from a novel viewpoint, by generating additional
pseudo-anomalies based on the limited labeled anomalies and a large amount of
unlabeled data.

  * •

In-depth Analysis. We systematically evaluate four different data augmentation
methods for pseudo-anomaly generation.

  * •

Novel algorithm. We introduce NNG-Mix, a simple and effective pseudo-anomaly
generation algorithm, that optimally utilizes information from both labeled
anomalies and unlabeled data.

  * •

Evaluation and Effectiveness. We conduct extensive experiments on $57$
datasets in ADBench, reflecting diversified domains, including applications in
healthcare, audio and language processing, image processing, finance, etc.,
and on five semi-supervised and supervised anomaly detection algorithms. Our
NNG-Mix yields substantial performance improvements compared to different
baseline data augmentation methods.

##  2 Related Work

Anomaly Detection aims to identify data instances that deviate significantly
from the majority of data objects. Different types of AD methods have been
proposed depending on the availability of labels, including unsupervised,
semi-supervised, and supervised methods. Unsupervised AD methods usually
assume no access to labeled data and often suggest that anomalies are situated
in low-density regions bergman2019classification ; zhao2019pyod . For example,
Local Outlier Factor (LOF) breunig2000lof measures the local deviation of the
density of a given sample with respect to its neighbors. DeepSVDD ruff2018deep
trains a neural network that minimizes the volume of a hypersphere to
encapsulate the network representations of the data, driving the network to
extract the common factors of variation. In a recent study by han2022adbench ,
various unsupervised anomaly detection approaches were compared across
multiple datasets. The findings revealed that no single algorithm consistently
outperformed others across all datasets. Furthermore, the study demonstrates
that the introduction of a small number of anomalies, whether through a
supervised or semi-supervised approach, resulted in consistent improvements in
the AD tasks.

Supervised AD methods consider anomaly detection as binary classification, and
prevailing classifiers such as MLP rosenblatt1958perceptron , FTTransformer
gorishniy2021revisiting , and CatB prokhorenkova2018catboost are frequently
used for this purpose. MLP rosenblatt1958perceptron uses the binary cross
entropy loss to update network parameters, while FTTransformer
gorishniy2021revisiting presents an effective adaptation of the Transformer
architecture vaswani2017attention specifically designed for tabular data. CatB
prokhorenkova2018catboost operates as a fast, scalable, high-performance
gradient boosting technique based on decision trees. One of the main
challenges encountered when using supervised algorithms for anomaly detection
is the limited availability of anomalous data, often failing to encompass the
entire range of potential anomalies.

Hence, semi-supervised methods DeepSAD ; zhao2018xgbod stand out as an
effective strategy, utilizing partial supervision from labeled data to enhance
anomaly detection performance. There are different types of directions for
semi-supervised approaches. The Extreme Gradient Boosting Outlier Detection
(XGBOD) zhao2018xgbod , for example, augments the feature space by extracting
useful representations using multiple unsupervised AD algorithms and
concatenating them together as features to enhance predictive capabilities.
DeepSAD DeepSAD integrates a term for the labeled anomalies in the loss
function to leverage the supervision from labeled data. FEAWAD zhou2021feature
enhances AD performance by utilizing hidden representations, reconstruction
residual vectors, and reconstruction errors from an autoencoder as the new
representations. They use labeled anomalies to force the anomaly scores of
normal samples to be close to $0$ while the anomaly scores for labeled
anomalies away from $0$ by a predefined margin.

In this study, our objective is to enhance the anomaly detection performance
by generating additional pseudo-anomalies as training data to improve the
generalization ability of different AD methods. We combine our augmentation
algorithm with current State-of-the-Art (SOTA) semi-supervised and supervised
AD methods for experiments to demonstrate its flexibility and versatility.

Data Augmentation is a technique in deep learning to mitigate overfitting by
generating modified versions of existing data samples to increase diversity
and improve generalization and has been widely used in many areas like
computer vision (CV) yang2022image and natural language processing (NLP)
feng2021survey . In CV, basic data augmentation methods chen2020simple involve
image manipulations, including cropping, resizing, color distortion, rotation,
addition of Gaussian noise and blur, and techniques like Cutout
devries2017cutout . Another method involves different mixing strategies, which
combine two or more images or their sub-regions. For example, Mixup
zhang2018mixup and its variants yao2022cmix ; li2022who perform convex
combinations between image pairs and their labels and encourage the model to
behave linearly between training samples. This behavior can lead to a smoother
and more robust model that generalizes better to unseen data. CutMix
yun2019cutmix replaces removed regions with a patch from another image.
Advanced approaches such as AutoAugment cubuk2019autoaugment automatically
search for improved data augmentation policies. In NLP, basic augmentation
methods involve random insertion, deletion, and swap wei2019eda , while mixed
sample data augmentation explores interpolating inner components
guo2020nonlinear . While many data augmentation methods can be adapted for
pseudo-anomaly generation, their optimality might be limited without
considering the specific characteristics of anomaly detection tasks. In this
work, we propose a novel pseudo-anomaly generation algorithm that fully
leverages information from both labeled anomalies and unlabeled data.

Self-supervised AD is motivated by the recent success of self-supervised
learning jing2020self , where a model learns a generalizable representation
from unlabelled data by solving one or several pretext tasks. These tasks use
data as labels and are not directly related to the final downstream task. In
self-supervised AD, the pretext task is designed to guide the model in
learning a representation tailored for anomaly detection. For example,
CutPaste li2021cutpaste generates local irregular patterns in the training
phase and detects similar irregularities, representative of unseen real
defects, during the testing phase. In a different line of research, SSL-OE
hendrycks2019using uses an auxiliary rotation loss to improve the robustness
of AD models. SSD sehwag2021ssd uses contrastive learning
winkens2020contrastive for unsupervised representation learning adopting by a
Mahalanobis distance in the feature space as the probabilistic metric that
enables anomaly detection. While these approaches have shown promising results
in AD, they are entirely unsupervised and only effective when their
assumptions align with the characteristics of the considered dataset. Contrary
to these approaches, our setting has access to a small set of labeled
anomalies, making it more realistic.

##  3 Methodology

###  3.1 Preliminaries and Problem Definition

In this work, we aim to generate additional pseudo-anomalies based on the
limited labeled anomalies and a large volume of unlabeled data as an
augmentation to facilitate the detection of new anomalies. Let us consider a
scenario where we have a limited number of labeled anomaly data $\mathcal{A}$
and a large amount of unlabeled data $\mathcal{H}$, where the majority of data
samples in $\mathcal{H}$ represents normal data and we treat them as normal
data during training. The label of sample is denoted by $y$, where $y=1$
represents anomalies and $y=0$ represents normal data. The primary objective
of semi-supervised and supervised AD methods is to train a model $\mathbf{F}$
on the combined dataset $\\{\mathcal{A},\mathcal{H}\\}$ to generate an anomaly
score for each sample. This work delves into generating additional pseudo-
anomaly data $\mathcal{D}$ using available data $\mathcal{A}$ and
$\mathcal{H}$. Subsequently, $\mathbf{F}$ is trained on the extended dataset
$\\{\mathcal{A},\mathcal{H},\mathcal{D}\\}$ with the aim of enhancing anomaly
detection performance and improving generalization ability.

###  3.2 Baseline Pseudo-anomaly Generation Methods

We evaluate the performance improvement and generalization ability of other
commonly applied data augmentation approaches that are not specifically
tailored for anomaly generation, including Mixup zhang2018mixup , Cutout
devries2017cutout , CutMix yun2019cutmix , and random Gaussian noises. Then,
we introduce the proposed novel Nearest Neighbor Gaussian Mixup method to
enhance the effectiveness of pseudo-anomaly generation.

Mixup for Pseudo-anomaly Generation.  Mixup zhang2018mixup generates a new
sample $(\mathbf{d},y)$ by creating convex combinations of a pair of samples
and their respective labels:

$\begin{split}\mathbf{d}&=\lambda\mathbf{a_{1}}+(1-\lambda)\mathbf{a_{2}},\\\
y&=\lambda y_{1}+(1-\lambda)y_{2},\\\ \end{split}$ (1)

where $(\mathbf{a_{1}},y_{1})$ and $(\mathbf{a_{2}},y_{2})$ are two samples
randomly drawn from the training data, and
$\lambda\sim\text{Beta}(\alpha,\alpha)$, for $\alpha\in(0,\infty)$. In our
experiments, $(\mathbf{a_{1}},y_{1})$ and $(\mathbf{a_{2}},y_{2})$ are
exclusively drawn from the labeled anomaly data $\mathcal{A}$. We show in our
ablation study that applying Mixup between anomaly and unlabeled data without
any constraints yields inferior performance. We derive pseudo-anomaly
$\mathbf{d}$ using
$\mathbf{d}=\lambda\mathbf{a_{1}}+(1-\lambda)\mathbf{a_{2}}$. Concerning the
label $y$, instead of utilizing the convex combinations of $1$ and $0$, we
assume $y$ to be $1$ for all samples, which implies that all generated data
are assumed to be anomalies.

Cutout for Pseudo-anomaly Generation.  Cutout devries2017cutout randomly masks
out some square or continuous regions in a sample. Given $\mathbf{a_{1}}$
drawn from $\mathcal{A}$, we generate the pseudo-anomaly $\mathbf{d}$ as
follows:

$\begin{split}\mathbf{d}&=\mathbf{m}\odot\mathbf{a_{1}},\\\ \end{split}$ (2)

where $\mathbf{m}\in\\{0,1\\}^{m}$ denotes a binary mask indicating where to
drop out from $\mathbf{a_{1}}$, and $\odot$ represents element-wise
multiplication, $m$ is the length of $\mathbf{a_{1}}$.

CutMix for Pseudo-anomaly Generation.  CutMix yun2019cutmix involves cutting
and pasting patches among training samples, where the ground truth labels are
also mixed proportionally based on the area of the patches. This process is
mathematically represented as:

$\begin{split}\mathbf{d}&=\mathbf{m}\odot\mathbf{a_{1}}+(\mathbf{1}-\mathbf{m})\odot\mathbf{a_{2}}\\\
y&=\lambda y_{1}+(1-\lambda)y_{2},\\\ \end{split}$ (3)

where $\mathbf{m}\in\\{0,1\\}^{m}$ denotes a binary mask indicating areas to
be replaced and filled from two samples, and $\mathbf{1}$ represents a binary
mask filled with ones. Similar to Mixup, in our experiment, the pairs
$(\mathbf{a_{1}},y_{1})$ and $(\mathbf{a_{2}},y_{2})$ are drawn exclusively
from $\mathcal{A}$. We generate pseudo-anomaly samples using
$\mathbf{d}=\mathbf{m}\odot\mathbf{a_{1}}+(\mathbf{1}-\mathbf{m})\odot\mathbf{a_{2}}$
while fixing $y$ to be 1.

Gaussian Noise for Pseudo-anomaly Generation.  In this baseline method, we
generate pseudo-anomaly $\mathbf{d}$ by adding random Gaussian noises (GN) to
the labeled anomaly data $\mathbf{a_{1}}$ from $\mathcal{A}$ as represented by
the equation:

$\begin{split}\mathbf{d}=\mathbf{a_{1}}+\boldsymbol{\epsilon},\\\ \end{split}$
(4)

where $\boldsymbol{\epsilon}$ is drawn from a centered Gaussian distribution
with a standard deviation of $\boldsymbol{\sigma}$, denoted as
$\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\boldsymbol{\sigma})$.

###  3.3 Nearest Neighbor Gaussian Mixup

Motivation. While Mixup offers the potential to generate new samples by
blending unlabeled and anomaly samples, this approach poses the risk of
producing mislabeled samples. For instance, C-Mixup yao2022cmix demonstrates
that directly applying vanilla Mixup on regression tasks can lead to
arbitrarily incorrect labels and proposes an adjustment in the sampling
probability based on label similarity. In our scenario, as illustrated in Fig.
1 (b) and (c), vanilla Mixup introduces a considerable number of noisy samples
within the distribution of the unlabeled data. This influx of noise could
significantly hinder anomaly detection performance. Moreover, the exclusive
mixing of labeled anomaly samples presents the drawback of failing to leverage
information from the unlabeled data, which could be valuable in producing more
relevant anomalies. To effectively utilize the unlabeled data and avoid
generating unwanted noise samples, we propose Nearest Neighbor Gaussian Mixup,
a simple yet effective pseudo-anomaly generation algorithm.

Proposed Approach. Instead of randomly selecting an anomaly sample from
$\mathcal{A}$ and an unlabeled sample from $\mathcal{H}$ to mix, our approach
involves obtaining a labeled anomaly sample $\mathbf{a_{1}}$ from
$\mathcal{A}$ and identifying its top $k$ nearest neighbors, denoted as
$\mathcal{M}$ from the unlabeled set $\mathcal{H}$. We then randomly select
one unlabeled sample $\mathbf{a_{2}}$ from $\mathcal{M}$ for mixup with
$\mathbf{a_{1}}$. This process aims to prevent the generation of anomalies
that reside within the distribution of unlabeled data, as shown in Fig. 1 (d).
To further refine the selection process, we propose a $50\%$ probability of
choosing the top $k$ nearest neighbors from $\mathcal{A}$ instead. This
approach ensures the combination of two anomalies belonging to the same
anomaly category, thereby alleviating the generation of unrealistic artificial
anomalies that lie within the distribution of unlabeled data and could
complicate the training process. Before applying the convex combination of the
two samples using the parameter $\lambda$ drawn from a beta distribution, we
augment both $\mathbf{a_{1}}$ and $\mathbf{a_{2}}$ by adding random Gaussian
noise with a zero mean and a standard deviation of $\boldsymbol{\sigma}$. This
augmentation broadens the potential space for generating pseudo-anomalies.
Then, NNG-Mix can be defined by the following equations:

$\displaystyle\mathbf{d}=\lambda(\mathbf{a}_{1}+\boldsymbol{\epsilon_{1}})+(1-\lambda)(\mathbf{a}_{2}+\boldsymbol{\epsilon_{2}})$
(5) $\displaystyle\mathbf{a}_{2}\sim{\rm
kNN}(\mathbf{a}_{1},\mathcal{A},k)\hskip 14.22636pt\text{if mixed with labeled
anomaly}$ $\displaystyle\mathbf{a}_{2}\sim{\rm
kNN}(\mathbf{a}_{1},\mathcal{H},k)\hskip 14.22636pt\text{if mixed with
unlabeled data}$
$\displaystyle\boldsymbol{\epsilon_{1}},\boldsymbol{\epsilon_{2}}\sim\mathcal{N}(\mathbf{0},\boldsymbol{\sigma})$

Here, ${\rm kNN}(\mathbf{a}_{1},\mathcal{A},k)$ and ${\rm
kNN}(\mathbf{a}_{1},\mathcal{H},k)$ refer to the selection of one of the $k$
nearest neighbors of the sample $\mathbf{a}_{1}$ from the anomaly and
unlabeled data, respectively. The choice of a nearest neighbor from the $k$ is
made randomly. An illustration of NNG-Mix is provided in Fig. 2, and the
pseudo-code for generating relevant anomalies using NNG-Mix is presented in
Algorithm 1.

Figure 2: Nearest Neighbor Gaussian Mixup makes good use of information from
both labeled anomalies and unlabeled data to generate pseudo-anomalies
effectively. Algorithm 1 Nearest Neighbor Gaussian Mixup

Input: Anomaly data $\mathcal{A}$, unlabeled data $\mathcal{H}$, generated
pseudo-anomaly $\mathcal{D}$, hyper-parameters $\alpha$, $k$ and
$\boldsymbol{\sigma}$.

1:$\mathcal{D}=\\{\\}$

2:for $i=1,\dotsc,N$ do

3: Sample anomaly $\mathbf{a_{1}}$ from $\mathcal{A}$

4: if Uniform(0, 1) $>$ 0.5 then

5: $\mathcal{M}$ = kNN($\mathbf{a_{1}},\mathcal{A},k$)

6: else

7: $\mathcal{M}$ = kNN($\mathbf{a_{1}},\mathcal{H},k$)

8: end if

9: Sample $\mathbf{a_{2}}$ from $\mathcal{M}$

10:
$\boldsymbol{\epsilon_{1}},\boldsymbol{\epsilon_{2}}\sim\mathcal{N}(\mathbf{0},\boldsymbol{\sigma})$

11: $\mathbf{a_{1}}=\mathbf{a_{1}}+\boldsymbol{\epsilon_{1}}$

12: $\mathbf{a_{2}}=\mathbf{a_{2}}+\boldsymbol{\epsilon_{2}}$

13: $\lambda$ = Beta($\alpha$, $\alpha$)

14: $\mathbf{d}=\lambda\mathbf{a_{1}}+(1-\lambda)\mathbf{a_{2}}$

15: $\mathcal{D}=\\{\mathcal{D};\mathbf{d}\\}$

16:end for

17:return $\mathcal{D}$

##  4 Experiments

We conducted experiments on ADBench han2022adbench , a comprehensive tabular
anomaly detection benchmark. This benchmark comprises $47$ widely-used real-
world Classical datasets and an additional $10$ more complex datasets from
computer vision (CV) and natural language processing (NLP) domains. These
domains cover a broad spectrum of diverse applications such as healthcare
(e.g., disease diagnosis), audio and language processing (e.g., speech
recognition), image processing (e.g., object identification), finance (e.g.,
financial fraud detection), and more. Given the diverse range of applications,
ADBench serves as an ideal benchmark for systematically evaluating the
performance of different pseudo-anomaly generation algorithms.

###  4.1 Experimental Setup

We aim to enhance the performance of anomaly detection algorithms in scenarios
with limited availability of labeled anomalies. In particular, we evaluate
both semi-supervised and supervised anomaly detection algorithms under this
setup. In our experiments, we consider scenarios containing only $10\%$,
$5\%$, or even $1\%$ of labeled anomalies. To mitigate this scarcity, we
employ various pseudo-anomaly generation algorithms, including Mixup, Curout,
CutMix, Gaussian Noise, and our proposed NNG-Mix. The use of these
augmentation algorithms enables generating $M\times$ the number of pseudo-
anomalies compared to the original labeled anomalies, significantly increasing
the number of anomalies for training. Subsequently, we incorporate these
generated pseudo-anomalies into the original training data for training the
semi-supervised and supervised anomaly detection algorithms. The expectation
is that performance will be improved with the availability of a larger volume
of anomaly data for training.

For our experiments, we choose SOTA semi-supervised algorithms DeepSAD DeepSAD
and XGBOD zhao2018xgbod , and supervised algorithms MLP
rosenblatt1958perceptron , FTTransformer gorishniy2021revisiting , and CatB
prokhorenkova2018catboost , based on their reported performances on ADBench
han2022adbench . All algorithms are used with their default hyperparameter
settings, as reported in the original papers for fair comparison.

We evaluate different algorithms separately on the Classical, CV, and NLP
datasets within ADBench to demonstrate the efficacy of pseudo-anomaly
generation across diverse data types. The training set comprises $70\%$ of
data, while the remaining $30\%$ is allocated for testing. Evaluation is
conducted using the Area Under Receiver Operating Characteristic Curve
(AUCROC) as the standardized metric for all experiments.

For the pseudo-anomaly generation methods, we set the parameters as follows:
Mixup zhang2018mixup : $\lambda\sim\text{Beta}(\alpha,\alpha)$ with
$\alpha=0.2$. Cutout devries2017cutout : Dropout ratio is sampled between
$0.1$ and $0.3$, with a randomly selected starting point for the dropout.
CutMix yun2019cutmix : Mask ratio is sampled between $0.1$ and $0.3$, with a
randomly selected starting point for the cutting and mixing process. Random
Gaussian noise: $\boldsymbol{\sigma}=0.01$. Our proposed NNG-Mix:
$\alpha=0.2$, $k=10$, and $\boldsymbol{\sigma}=0.01$ . Additionally, we
perform an ablation study to analyze the sensitivity of the parameters.

###  4.2 Results

Generating $10\times$ pseudo-anomalies using $1\%$ labeled anomalies. We
commence our experiments in a highly challenging scenario, where only a mere
$1\%$ of labeled anomalies is available. These labeled data, combined with
unlabeled data, are used to generate $10\times$ pseudo-anomalies employing
various pseudo-anomaly generation algorithms. Subsequently, we train different
semi-supervised and supervised algorithms on both original and generated data.
In the Baseline scenario, the training solely relies on the original data
containing $1\%$ labeled anomalies and unlabeled data, without integrating
pseudo-anomalies. Results from Tab. 1 to Tab. 3 present the outcomes across
Classical, CV, and NLP datasets. For each dataset type, the reported metric is
the mean AUCROC across all datasets within that type. Notably, our proposed
NNG-Mix consistently demonstrates competitive performances across all dataset
types, ranking as the best or the second best in most cases.

An interesting observation is that almost all pseudo-anomaly generation
algorithms demonstrate the ability to enhance anomaly detection performance by
introducing an augmented set of pseudo-anomalies for training. When compared
to the Baseline setup trained solely on the original data, leveraging the
pseudo-anomalies generated by NNG-Mix yields significant benefits for all
evaluated semi-supervised and supervised algorithms across various dataset
types in most cases. Specifically, for semi-supervised algorithms, there are
maximum improvements of $2.5\%$, $2.5\%$, $2.7\%$ observed in Classical, CV,
and NLP datasets, respectively. For supervised algorithms, there are maximum
improvements of $16.4\%$, $8.8\%$, $8.0\%$ observed in Classical, CV, and NLP
datasets, respectively. On Classical and NLP datasets, MLP benefits the most
from NNG-Mix, while FTTransformer exhibits the most improvement on CV
datasets. Additionally, CutMix yun2019cutmix , a conventional data
augmentation method in computer vision, showcases competitive performances on
CV datasets, with an average improvement of $2.2\%$. Similarly, Cutout
devries2017cutout , resembling random masking devlin2018bert in natural
language processing, also demonstrates competitive results on NLP datasets,
with an average improvement of $2.1\%$.

Algorithm | Baseline | Mixup | Cutout | CutMix | GN | NNG-Mix
---|---|---|---|---|---|---
$1\%$ | $10\times$ | $10\times$ | $10\times$ | $10\times$ | $10\times$
DeepSAD | 0.766 | 0.786 | 0.780 | 0.782 | 0.782 | 0.791
XGBOD | 0.815 | 0.785 | 0.798 | 0.816 | 0.837 | 0.838
MLP | 0.600 | 0.758 | 0.748 | 0.763 | 0.751 | 0.764
FTTransformer | 0.719 | 0.781 | 0.757 | 0.778 | 0.763 | 0.783
CatB | 0.851 | 0.846 | 0.836 | 0.844 | 0.859 | 0.861
Average | 0.750 | 0.791 | 0.783 | 0.797 | 0.798 | 0.807

Table 1: Results on Classical datasets with $1\%$ available labeled anomalies
and $10\times$ pseudo-anomaly generation. The best results are in red and the
second best results are in blue.

Algorithm | Baseline | Mixup | Cutout | CutMix | GN | NNG-Mix
---|---|---|---|---|---|---
$1\%$ | $10\times$ | $10\times$ | $10\times$ | $10\times$ | $10\times$
DeepSAD | 0.692 | 0.697 | 0.698 | 0.700 | 0.695 | 0.708
XGBOD | 0.725 | 0.720 | 0.749 | 0.749 | 0.744 | 0.750
MLP | 0.657 | 0.634 | 0.628 | 0.682 | 0.658 | 0.675
FTTransformer | 0.564 | 0.644 | 0.619 | 0.626 | 0.646 | 0.652
CatB | 0.769 | 0.768 | 0.776 | 0.764 | 0.757 | 0.758
Average | 0.681 | 0.693 | 0.694 | 0.704 | 0.700 | 0.709

Table 2: Results on CV datasets with $1\%$ available labeled anomalies and
$10\times$ pseudo-anomaly generation. The best results are in red and the
second best results are in blue.

Algorithm | Baseline | Mixup | Cutout | CutMix | GN | NNG-Mix
---|---|---|---|---|---|---
$1\%$ | $10\times$ | $10\times$ | $10\times$ | $10\times$ | $10\times$
DeepSAD | 0.561 | 0.581 | 0.582 | 0.581 | 0.578 | 0.586
XGBOD | 0.604 | 0.592 | 0.627 | 0.608 | 0.617 | 0.631
MLP | 0.586 | 0.654 | 0.664 | 0.660 | 0.659 | 0.666
FTTransformer | 0.495 | 0.494 | 0.476 | 0.497 | 0.487 | 0.507
CatB | 0.603 | 0.606 | 0.606 | 0.604 | 0.611 | 0.612
Average | 0.570 | 0.585 | 0.591 | 0.590 | 0.590 | 0.600

Table 3: Results on NLP datasets with $1\%$ available labeled anomalies and
$10\times$ pseudo-anomaly generation. The best results are in red and the
second best results are in blue.

Influences of the number of generated pseudo-anomalies. We then analyze the
performance of the proposed algorithm with different numbers of generated
pseudo-anomalies on Classical datasets. Specifically, we generate $1\times$,
$5\times$, and $10\times$ the number of anomalies as pseudo-anomalies using
various anomaly generation algorithms. Subsequently, we train different semi-
supervised and supervised algorithms on datasets that combine the original and
the generated data samples. As depicted in Tab. 4, for most pseudo-anomaly
generation algorithms, increasing the number of pseudo-anomalies tends to
yield improved results. Generating $1\times$ pseudo-anomalies typically
results in some improvements compared to the Baseline in most cases. The
results become notably better when generating $5\times$ pseudo-anomalies, with
the best performance achieved in the $10\times$ scenario.

Algorithm | Baseline | Mixup | Cutout | CutMix | GN | NNG-Mix
---|---|---|---|---|---|---
$1\%$ | $1\times$ | $5\times$ | $10\times$ | $1\times$ | $5\times$ | $10\times$ | $1\times$ | $5\times$ | $10\times$ | $1\times$ | $5\times$ | $10\times$ | $1\times$ | $5\times$ | $10\times$
DeepSAD | 0.766 | 0.768 | 0.779 | 0.786 | 0.765 | 0.774 | 0.780 | 0.765 | 0.775 | 0.782 | 0.766 | 0.772 | 0.782 | 0.768 | 0.779 | 0.791
XGBOD | 0.815 | 0.795 | 0.790 | 0.785 | 0.799 | 0.796 | 0.798 | 0.796 | 0.805 | 0.816 | 0.817 | 0.832 | 0.837 | 0.820 | 0.838 | 0.838
MLP | 0.600 | 0.646 | 0.745 | 0.758 | 0.647 | 0.733 | 0.748 | 0.649 | 0.739 | 0.763 | 0.666 | 0.743 | 0.751 | 0.634 | 0.743 | 0.764
FTTransformer | 0.719 | 0.730 | 0.769 | 0.781 | 0.722 | 0.754 | 0.757 | 0.720 | 0.757 | 0.778 | 0.721 | 0.754 | 0.763 | 0.721 | 0.762 | 0.783
CatB | 0.851 | 0.848 | 0.848 | 0.846 | 0.842 | 0.839 | 0.836 | 0.847 | 0.840 | 0.844 | 0.854 | 0.853 | 0.859 | 0.851 | 0.854 | 0.861

Table 4: Results on Classical datasets with $1\%$ available labeled anomalies
and $1\times$, $5\times$, $10\times$ pseudo-anomaly generation. The best
results are in red and the second best results are in blue.

Influences of the number of labeled anomalies. Here, we analyze the
performance under the availability of different amounts of labeled anomalies
on Classical datasets using NNG-Mix. With an increase in the number of labeled
anomalies, both the semi-supervised and supervised anomaly detection Baselines
exhibit improved performances. The results, presented in Tab. 5, indicate that
NNG-Mix can further enhance the Baselines with different ratios of labeled
anomalies by generating more pseudo-anomalies for training, confirming the
effectiveness of our proposed approach. For instance, applying NNG-Mix to the
DeepSAD model and generating $10$ times the number of pseudo-anomalies results
in an increase of AUCROC scores by $2.5\%$, $3.3\%$, and $3.5\%$,
respectively, in scenarios with $1\%$, $5\%$, and $10\%$ of labeled anomalies.
This observation highlights the consistent performance enhancement of the
DeepSAD model when integrated with generated pseudo-anomalies, even in
situations where a sufficient number of labeled anomalies is already
available. This indicates that NNG-Mix improves the generalization of SOTA
semi-supervised and supervised AD algorithms.

Algorithm | Baseline | NNG-mix | Baseline | NNG-mix | Baseline | NNG-mix
---|---|---|---|---|---|---
$1\%$ | $1\times$ | $5\times$ | $10\times$ | $5\%$ | $1\times$ | $5\times$ | $10\times$ | $10\%$ | $1\times$ | $5\times$ | $10\times$
DeepSAD | 0.766 | 0.768 | 0.779 | 0.791 | 0.817 | 0.828 | 0.848 | 0.850 | 0.853 | 0.865 | 0.880 | 0.888
XGBOD | 0.815 | 0.820 | 0.838 | 0.838 | 0.868 | 0.877 | 0.886 | 0.886 | 0.897 | 0.912 | 0.911 | 0.914
MLP | 0.600 | 0.634 | 0.743 | 0.764 | 0.735 | 0.775 | 0.829 | 0.853 | 0.791 | 0.838 | 0.879 | 0.891
FTTransformer | 0.719 | 0.721 | 0.762 | 0.783 | 0.749 | 0.786 | 0.816 | 0.853 | 0.787 | 0.810 | 0.859 | 0.870
CatB | 0.851 | 0.851 | 0.854 | 0.861 | 0.891 | 0.894 | 0.899 | 0.901 | 0.916 | 0.921 | 0.923 | 0.922

Table 5: Results on Classical datasets with $1\%$, $5\%$, $10\%$ available
labeled anomalies and $1\times$, $5\times$, $10\times$ pseudo-anomaly
generation using NNG-Mix. The best results for each baseline are in red.

###  4.3 Ablations

####  4.3.1 Ablations on Each Module in NNG-Mix

We conducted extensive ablation studies to investigate the role of each module
within NNG-Mix on the Classical datasets, considering scenarios with $1\%$
available labeled anomalies and $1\times$, $5\times$, $10\times$ pseudo-
anomaly generation, as shown in Tab. 6. NNG-Mix without Nearest Neighbor (NN)
and Gaussian noises (GN) corresponds to a special case of Mixup, where samples
are drawn from both the labeled anomaly data $\mathcal{A}$ and unlabeled data
$\mathcal{H}$ for mixing, with all considered as anomalies. This random mixing
of anomalies and unlabeled data can inadvertently generate anomaly data points
falling within the distribution of normal data, resulting in diminished
performance (e.g., $1.9\%$ for XGBOD and $4.9\%$ for CatB) compared to the
Baseline setup. Even when Gaussian noise is introduced before applying Mixup,
omitting Nearest Neighbor results in underwhelming performance (e.g., only
$1.3\%$ increasement for DeepSAD and even $0.8\%$ decreasement for CatB). It
becomes evident that the primary enhancement in performance is achieved by
incorporating the Nearest Neighbor search without the use of Gaussian noise.
Ultimately, the best performance is observed when Gaussian noise is added to
enlarge the potential space for pseudo-anomaly generation.

Algorithm | Baseline | w/o NN and GN | w/o NN | w/o GN | NNG-mix
---|---|---|---|---|---
$1\%$ | $1\times$ | $5\times$ | $10\times$ | $1\times$ | $5\times$ | $10\times$ | $1\times$ | $5\times$ | $10\times$ | $1\times$ | $5\times$ | $10\times$
DeepSAD | 0.766 | 0.766 | 0.770 | 0.769 | 0.767 | 0.765 | 0.779 | 0.767 | 0.779 | 0.790 | 0.768 | 0.779 | 0.791
XGBOD | 0.815 | 0.805 | 0.819 | 0.796 | 0.820 | 0.824 | 0.826 | 0.797 | 0.788 | 0.795 | 0.820 | 0.838 | 0.838
MLP | 0.600 | 0.629 | 0.687 | 0.724 | 0.625 | 0.702 | 0.706 | 0.637 | 0.738 | 0.763 | 0.634 | 0.743 | 0.764
FTTransformer | 0.719 | 0.722 | 0.760 | 0.746 | 0.730 | 0.765 | 0.750 | 0.743 | 0.760 | 0.765 | 0.721 | 0.762 | 0.783
CatB | 0.851 | 0.842 | 0.825 | 0.802 | 0.845 | 0.838 | 0.843 | 0.846 | 0.842 | 0.840 | 0.851 | 0.854 | 0.861

Table 6: Ablations on Classical datasets with $1\%$ available labeled
anomalies and $1\times$, $5\times$, $10\times$ pseudo-anomaly generation using
NNG-Mix. The best results for each baseline are in red.

####  4.3.2 Ablations on Different Number of Generated Pseudo-anomalies

In this section, we investigate the impact of the quantity of generated
pseudo-anomalies on the final AD performance. We employ NNG-Mix to create
$M\times$ number of pseudo-anomalies, where $M$ varies from
$1,5,10,20,50,100$. Generally, the performance improves with an increasing
number of generated pseudo-anomalies, as illustrated in Fig. 3 (a). However,
the performance tends to plateau when $M$ exceeds $10$.

(a) Number of Generated Pseudo-anomalies

(b) Number of Labeled Anomalies

Figure 3: Ablations on different number of generated pseudo-anomalies and
labeled anomalies.

####  4.3.3 Ablations on Different Quantities of Labeled Anomalies

Here we investigate the influence of the quantity of labeled anomalies on the
final AD performance. We use NNG-Mix to generate $10\times$ the number of
pseudo-anomalies, considering varying proportions of available labeled
anomalies: $1\%$, $5\%$, $10\%$, $20\%$, $50\%$, $100\%$. The performance of
all algorithms improves significantly with an increased number of labeled
anomalies, emphasizing the importance of supervision, as shown in Fig. 3 (b).

####  4.3.4 Ablations on Different Pollution Ratios

In this section, we explore the robustness of NNG-Mix concerning various
pollution ratios, denoted as $\boldsymbol{\gamma}$, within the unlabeled data
$\mathcal{H}$, which means that we add different numbers of anomalies in the
unlabeled data and treat them as normal data during pseudo-anomaly generation
and training. To investigate this, we generate unlabeled data $\mathcal{H}$ by
contaminating the normal data with different numbers of anomalies drawn from
all available anomaly data, excluding the designated $1\%$ labeled anomalies.
We set pollution ratios $\boldsymbol{\gamma}$ within a range of values,
including $0.0$, $0.2$, $0.4$, $0.6$, $0.8$, and $1.0$, where $1.0$ represents
the default setup used in all our previous experiments. This is also the
default setup in ADBench and further AD papers. The results, illustrated in
Fig. 4, reveal that lower pollution ratios consistently lead to performance
improvements across all algorithms. Notably, for DeepSAD, MLP, and
FTTransformer, NNG-Mix substantially enhances the Baseline setups, regardless
of the pollution ratio. In contrast, for XGBOD and CatB, when the pollution
ratio is low, the Baselines already exhibit superior performance, and the
introduction of additional pseudo-anomalies does not yield significant
benefits. NNG-Mix demonstrates its efficacy primarily when the pollution ratio
surpasses the $0.6$ threshold.

(a) DeepSAD

(b) XGBOD

(c) MLP

(d) FTTransformer

(e) CatB

Figure 4: AUCROC with different pollution ratios.

####  4.3.5 Parameter Sensitivity

In this section, we investigate the parameter sensitivity of NNG-Mix on the
Classical datasets, with $1\%$ available labeled anomalies and $10\times$
pseudo-anomaly generation, as depicted in Fig. 5. For the Nearest Neighbor
parameter $k$, we test values of $3$, $5$, $10$, and $20$. Regarding the
Gaussian noise parameter $\boldsymbol{\sigma}$, we consider choices of $0.01$,
$0.1$, and $0.3$. Additionally, for the Mixup parameter $\alpha$, we evaluate
values of $0.2$ and $2.0$ and also explore sampling $\lambda$ from the uniform
distribution. In our investigation, $5$ and $10$ emerge as well-performing
choices for the majority of algorithms when considering the $k$ parameter.
Regarding $\boldsymbol{\sigma}$, the most suitable choice varies a lot among
different algorithms, indicating distinct levels of robustness under varying
noise scales. For $\alpha$, $0.2$ emerges as a suitable choice for most
algorithms. Overall, NNG-Mix demonstrates robustness across different
parameter settings, displaying fluctuations of less than $2\%$ in most cases.

(a) $k$

(b) $\boldsymbol{\sigma}$

(c) $\alpha$

Figure 5: Parameter Sensitivity.

###  4.4 Visualization of the Generated Pseudo-anomalies

Figure 6: t-SNE visualization of pseudo-anomaly generation using Mixup
zhang2018mixup and the proposed NNG-Mix on the vertebral dataset within the
Classical dataset. Mixup often introduces unwanted noise samples within the
distribution of unlabeled data, potentially significantly hindering the
performance of AD models. In contrast, our NNG-Mix excels at generating
pseudo-anomalies by effectively utilizing information from both labeled and
unlabeled data.

Results in Fig. 6 showcase the t-SNE embedding van2008visualizing of pseudo-
anomaly generation using both Mixup and our proposed NNG-Mix on the vertebral
dataset within the Classical dataset. Application of Mixup across all training
data introduces unwanted noise samples during the training process. Employing
Mixup solely with labeled anomalies tends to overlook valuable information
from unlabeled data and also introduces samples surrounded by unlabeled data,
which are unsuitable for considering these as anomalies. In contrast, NNG-Mix
adeptly harnesses information from both labeled anomalies and unlabeled data
to effectively generate pseudo-anomalies.

##  5 Conclusions

In this paper, we investigate improving semi-supervised anomaly detection
performances from a novel view and propose the NNG-Mix algorithm for effective
pseudo-anomaly generation. Our approach maximizes the utilization of
information from both labeled anomalies and unlabeled data, resulting in
better pseudo-anomaly generation for training in comparison to other data
augmentation methods. As a result, NNG-Mix improves the generalization ability
of SOTA semi-supervised and supervised AD algorithms. We conducted extensive
experiments on $57$ datasets in ADBench and evaluated five semi-supervised and
supervised anomaly detection algorithms. Our study offers a comprehensive
comparison of various data augmentation algorithms for generating anomalies
and demonstrates the consistently superior performance of NNG-Mix compared to
various baselines across all anomaly detection algorithms. Our work presents a
novel perspective for enhancing the performance of both semi-supervised and
supervised AD algorithms. In future work, it is also interesting to
investigate pseudo-anomaly generation on graphs and videos.

## References

  * [1] Charu C Aggarwal and Charu C Aggarwal.  An introduction to outlier analysis.  Springer, 2017.
  * [2] Liron Bergman and Yedid Hoshen.  Classification-based anomaly detection for general data.  In ICLR, 2020.
  * [3] Daniel Bogdoll, Maximilian Nitsche, and J Marius Zöllner.  Anomaly detection in autonomous driving: A survey.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4488–4499, 2022.
  * [4] Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander.  Lof: identifying density-based local outliers.  In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 93–104, 2000.
  * [5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [6] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.  Autoaugment: Learning augmentation strategies from data.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 113–123, 2019.
  * [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [8] Terrance DeVries and Graham W Taylor.  Improved regularization of convolutional neural networks with cutout.  arXiv preprint arXiv:1708.04552, 2017.
  * [9] Hao Dong, Xieyuanli Chen, Simo Särkkä, and Cyrill Stachniss.  Online pole segmentation on range images for long-term lidar localization in urban environments.  Robotics and Autonomous Systems, 159:104283, 2023.
  * [10] Hao Dong, Xianjing Zhang, Jintao Xu, Rui Ai, Weihao Gu, Huimin Lu, Juho Kannala, and Xieyuanli Chen.  Superfusion: Multilevel lidar-camera fusion for long-range hd map generation.  arXiv preprint arXiv:2211.15656, 2022.
  * [11] Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy.  A survey of data augmentation approaches for nlp.  arXiv preprint arXiv:2105.03075, 2021.
  * [12] Gaëtan Frusque, Daniel Mitchell, Jamie Blanche, David Flynn, and Olga Fink.  Non-contact sensing for anomaly detection in wind turbine blades: A focus-svdd with complex-valued auto-encoder approach.  arXiv preprint arXiv:2306.10808, 2023.
  * [13] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko.  Revisiting deep learning models for tabular data.  NeurIPS, 2021.
  * [14] Hongyu Guo.  Nonlinear mixup: Out-of-manifold data augmentation for text classification.  In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
  * [15] Songqiao Han, Xiyang Hu, Hailiang Huang, Mingqi Jiang, and Yue Zhao.  Adbench: Anomaly detection benchmark.  In Neural Information Processing Systems (NeurIPS), 2022.
  * [16] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song.  Using self-supervised learning can improve model robustness and uncertainty.  Advances in neural information processing systems, 32, 2019.
  * [17] Longlong Jing and Yingli Tian.  Self-supervised visual feature learning with deep neural networks: A survey.  IEEE transactions on pattern analysis and machine intelligence, 43(11):4037–4058, 2020.
  * [18] Changchun Li, Ximing Li, Lei Feng, and Jihong Ouyang.  Who is your right mixup partner in positive and unlabeled learning.  In International Conference on Learning Representations, 2022.
  * [19] Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister.  Cutpaste: Self-supervised learning for anomaly detection and localization.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9664–9674, 2021.
  * [20] Zheng Li, Yue Zhao, Xiyang Hu, Nicola Botta, Cezar Ionescu, and George Chen.  Ecod: Unsupervised outlier detection using empirical cumulative distribution functions.  IEEE Transactions on Knowledge and Data Engineering, 2022.
  * [21] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou.  Isolation forest.  In 2008 eighth ieee international conference on data mining, pages 413–422. IEEE, 2008.
  * [22] Gabriel Michau and Olga Fink.  Unsupervised transfer learning for anomaly detection: Application to complementary operating condition transfer.  Knowledge-Based Systems, 216:106816, 2021.
  * [23] Guansong Pang, Chunhua Shen, Huidong Jin, and Anton van den Hengel.  Deep weakly-supervised anomaly detection.  In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1795–1807, 2023.
  * [24] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin.  Catboost: unbiased boosting with categorical features.  NeurIPS, 2018.
  * [25] Sridhar Ramaswamy, Rajeev Rastogi, and Kyuseok Shim.  Efficient algorithms for mining outliers from large data sets.  In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 427–438, 2000.
  * [26] Frank Rosenblatt.  The perceptron: a probabilistic model for information storage and organization in the brain.  Psychological review, 65(6):386, 1958.
  * [27] Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft.  Deep one-class classification.  In International conference on machine learning, pages 4393–4402. PMLR, 2018.
  * [28] Lukas Ruff, Robert A. Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Klaus-Robert Müller, and Marius Kloft.  Deep semi-supervised anomaly detection.  In ICLR, 2020.
  * [29] Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson.  Estimating the support of a high-dimensional distribution.  Neural computation, 13(7):1443–1471, 2001.
  * [30] Vikash Sehwag, Mung Chiang, and Prateek Mittal.  Ssd: A unified framework for self-supervised outlier detection.  arXiv preprint arXiv:2103.12051, 2021.
  * [31] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [32] Robert Vandermeulen and Clayton Scott.  Consistency of robust kernel density estimators.  In Conference on Learning Theory, pages 568–591. PMLR, 2013.
  * [33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [34] Jason Wei and Kai Zou.  Eda: Easy data augmentation techniques for boosting performance on text classification tasks.  arXiv preprint arXiv:1901.11196, 2019.
  * [35] Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al.  Contrastive training for improved out-of-distribution detection.  arXiv preprint arXiv:2007.05566, 2020.
  * [36] Suorong Yang, Weikang Xiao, Mengcheng Zhang, Suhan Guo, Jian Zhao, and Furao Shen.  Image data augmentation for deep learning: A survey.  arXiv preprint arXiv:2204.08610, 2022.
  * [37] Huaxiu Yao, Yiping Wang, Linjun Zhang, James Zou, and Chelsea Finn.  C-mixup: Improving generalization in regression.  In Proceeding of the Thirty-Sixth Conference on Neural Information Processing Systems, 2022.
  * [38] Weiren Yu, Jianxin Li, Md Zakirul Alam Bhuiyan, Richong Zhang, and Jinpeng Huai.  Ring: Real-time emerging anomaly monitoring system over text streams.  IEEE Transactions on Big Data, 5(4):506–519, 2017.
  * [39] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In CVPR, 2019.
  * [40] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In ICLR, 2018.
  * [41] Mengjie Zhao and Olga Fink.  Dynamic graph attention for anomaly detection in heterogeneous sensor networks.  arXiv preprint arXiv:2307.03761, 2023.
  * [42] Yue Zhao and Maciej K Hryniewicki.  Xgbod: improving supervised outlier detection with unsupervised representation learning.  In IJCNN, 2018.
  * [43] Yue Zhao, Xiyang Hu, Cheng Cheng, Cong Wang, Changlin Wan, Wen Wang, Jianing Yang, Haoping Bai, Zheng Li, Cao Xiao, et al.  Suod: Accelerating large-scale unsupervised heterogeneous outlier detection.  Proceedings of Machine Learning and Systems, 3:463–478, 2021.
  * [44] Yue Zhao, Zain Nasrullah, and Zheng Li.  Pyod: A python toolbox for scalable outlier detection.  JMLR, 20:1–7, 2019.
  * [45] Yingjie Zhou, Xucheng Song, Yanru Zhang, Fanxing Liu, Ce Zhu, and Lingqiao Liu.  Feature encoding with autoencoders for weakly supervised anomaly detection.  IEEE Transactions on Neural Networks and Learning Systems, 33(6):2454–2465, 2021.
  * [46] Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen.  Deep autoencoding gaussian mixture model for unsupervised anomaly detection.  In International conference on learning representations, 2018.


11institutetext: Eindhoven University of Technology (TU/e)
11email<EMAIL_ADDRESS>22institutetext: Philips
Research Eindhoven
22email<EMAIL_ADDRESS>
# Evolutionary Construction of Convolutional Neural Networks

Marijn van Knippenberg(✉) 11 Vlado Menkovski 11 Sergio Consoli 22

###### Abstract

Neuro-Evolution is a field of study that has recently gained significantly
increased traction in the deep learning community. It combines deep neural
networks and evolutionary algorithms to improve and/or automate the
construction of neural networks. Recent Neuro-Evolution approaches have shown
promising results, rivaling hand-crafted neural networks in terms of accuracy.

A two-step approach is introduced where a convolutional autoencoder is created
that efficiently compresses the input data in the first step, and a
convolutional neural network is created to classify the compressed data in the
second step. The creation of networks in both steps is guided by by an
evolutionary process, where new networks are constantly being generated by
mutating members of a collection of existing networks. Additionally, a method
is introduced that considers the trade-off between compression and information
loss of different convolutional autoencoders. This is used to select the
optimal convolutional autoencoder from among those evolved to compress the
data for the second step.

The complete framework is implemented, tested on the popular CIFAR-10 data
set, and the results are discussed. Finally, a number of possible directions
for future work with this particular framework in mind are considered,
including opportunities to improve its efficiency and its application in
particular areas.

###### Keywords:

Neuro-Evolution, Genetic Algorithms, Convolutional Autoencoders, Convolutional
Neural Networks

##  1 Introduction

Neural networks have become a popular data analysis tool in both academia and
industry, especially when tasks like image classification, natural language
processing, and speech recognition need to be addressed. They have shown to be
very adept at these tasks, which are classic problems in which we like the
computer to show “human-like” behavior. One of the challenges surrounding
neural networks which has kept researchers around the world occupied is the
matter of their design, and the possible automation of this task. Constructing
a neural network is still often seen as a somewhat “magic” skill by many: a
combination of knowledge, past experience, and intuition. Network performance
is greatly affected by its size and structure, the type and order of layers,
the choice of loss function and the way data is presented to the network.
These are all decisions that a designer has to make in what is often a lengthy
continuous cycle of re-design, re-training, and re-evaluation. This makes it
an attractive target for automation [7].

In this paper we introduce a neuro-evolution approach to the above challenge
that is more efficient than current solutions in terms of computational
resource consumption [7]. This will hopefully make this type of approach more
viable in real-life settings. It takes inspiration from a specific, recently
published study, which establishes a kind of baseline for the application of
evolutionary algorithms in this setting [10]. Implementation of the approach
also requires a look at solution efficiency and multi-criteria decision making
(MCDM), which helps reasoning about the relation between the performance
metrics of different types of neural networks [14, 9].

Both the baseline and the proposed approach are implemented and tested in
order to confirm their behavior and to draw comparisons. The results are then
inspected to establish whether the new framework has significant impact in
terms of final accuracy and consumed computational resources. These two will
form a trade-off that is controlled by the MCDM process. Finally, a number of
alternatives and future study topics are presented and considered.

##  2 Related Work

Neuro-evolution is the field of study that combines neural networks with
evolutionary algorithms in the search for innovative training methods [16].
The field has recently gained increased interest with the sharp increase of
popularity of deep neural networks. Likely the most well-known algorithm in
the field of neuro-evolution is the neuro-evolution of augmented topologies
algorithm (NEAT) [13]. Recent studies have applied evolutionary strategies to
deep neural networks in a variety of manners, such as evolving network weights
via genetic programming [9, 15], differential evolution [11], pitting networks
against each other in a tournament selection environment [10], more recent
extensions of NEAT [3, 7], and by evolving a network’s activation functions
[4]. The general “mood” of the intersection of these two fields is very much
an exploratory one. There is no clear precedent on how to apply techniques
from the meta-heuristics field to deep learning [4]. Another interesting
direction in neuro-evolution that has recently been taken under the growing
complexity of deep neural networks is that of network reduction. Evolutionary
strategies can be used to simplify networks, in the hope of reducing future
training and operating cost [12].

This work is based in large parts on a recent Neuro-Evolution study [10]. The
goal of this study was to establish the viability of genetic programming in
replacing human input during the design phase of neural networks. In the
remainder of this paper, their approach will be referred to as the “baseline
framework”. The approach is based on multiple worker processes operating in
parallel, each applying a genetic programming algorithm that continually
evolves a population of neural networks. In genetic programming, a collection
of candidate solutions, called the _population_ , is updated in steps in the
hopes of finding better candidate solutions. A candidate solution is also
known as an _individual_. An individual is considered to have “DNA”; some
simplified representation from which the solution can be reconstructed. By
mutating this DNA in different ways, an individual can be evolved into new
individual, one which hopefully has better _fitness_ , which is some measure
of the quality of the solution. In this case, each candidate solution is a
convolutional neural network, and the DNA is simply the structure of the
network. In order to apply genetic programming, a _selection method_ is
required. This is the method that is used to determine which individuals in
the population are chosen for mutation. The approach is based on the basic
process of tournament selection [8]. It is a simple and direct selection
method that is based on “rounds”. Each round, $k$ individuals are selected
from the population according to some probability distribution, and their
fitness is compared. The individual with the best fitness is copied and its
copy is mutated before inserting it into the population, while the one with
the worst fitness is removed from the population. This method of selecting
individuals is intuitive, easy to implement, and easy to adjust. An added
benefit is that it is quite easy to parallelize. In the case of the study,
$k=2$ individuals are selected uniformly at random. In order to establish a
population to select from, each worker process generates some initial
individuals. In this case, the initial individuals are very simple networks
with very low fitness scores.

The major issue of this approach is the computational resource required. To
arrive at a competitive result, many networks have to be generated, trained,
and evaluated. The authors’ main goal was to establish viability in terms of
accuracy, so their approach is fairly basic and direct. This leaves many
opportunities for improvements and further study, which is the point where
this work steps in.

##  3 Method

###  3.1 Evolving Autoencoders

The attempt of this work to reduce computation time is based on reducing the
size of the input samples that are used for training classification networks.
Reducing the input sample size has a double positive effect on the
evolutionary process: training each individual network takes less effort,
since there are less values to process for each sample, and networks can be
expected to be shallower since the input is smaller, reducing the average
training time per network in the overall evolutionary process.

Autoencoders are well-suited tools for reducing sample sizes, and in the case
of image data, a convolutional autoencoder can be used to maintain the spatial
relations of the input data in the encoded data. Since convolutional
autoencoders are simply convolutional neural networks with some extra
restraints and a different error metric, the baseline framework can also be
applied to obtain the most suitable autoencoder. The overall process is then
as follows:

  1. 1.

Perform an evolutionary process on a population of convolutional autoencoders.
Training is based on the original input and the reconstruction error of the
decoder.

  2. 2.

Pick the best autoencoder and encode the entire original input data set.

  3. 3.

Perform an evolutionary process on a population of convolutional neural
networks that classify input samples. Training is based on the encoded input
and classification error.

  4. 4.

Pick the classifier with the highest validation accuracy and append it to the
encoder used to encode the data in order to obtain the best overall network.

In order to apply the baseline framework to convolutional autoencoders, a new
selection process needs to be refined. In the baseline framework, selection
was purely based on accuracy. In the case of autoencoders, there are now two
selection criteria: reconstruction accuracy and compression ratio. We like
both of these to be as high as possible, but it is important to consider the
case where one autoencoder has a higher compression ratio, while another one
has a higher accuracy. Taking inspiration from the Non-dominated Sorting
Genetic Algorithm (NSGA)[2], autoencoders are grouped into Pareto fronts based
on dominance. If two sampled individuals are members of different fronts, the
one that is a member of the dominating front “wins”. If both individuals are
part of the same front, neither is strictly better than the other in terms of
Pareto efficiency. In this case, the individual that is furthest away the
other individuals in that front is chosen as winner. In other words, a higher
probability of innovation is rewarded if both individuals come from the same
front. The final adjustment to make to the evolutionary process is to define a
new set of mutations so actual convolutional autoencoders are evolved.

Below are listed the mutations that are made available to the evolutionary
framework for the purpose of evolving convolutional autoencoders. These are
based on the baseline framework, with some small changes, such as the
inclusion of pooling-related mutations. Note that only the encoder is mutated,
since the decoder is directly derived from it by mirroring it and replacing
each pooling layer with an up-sampling layer (see Figure 1).

  * •

Identity: The network structure is not changed in any way. In practice this
means the same network trained longer, since weights are maintained when
copying networks.

  * •

Insert convolution: This mutation inserts a convolutional layer at a random
location in the network in the encoder.

  * •

Remove convolution: This mutation removes a random convolutional layer from
the encoder.

  * •

Alter Stride: The stride length of a random convolutional layer is incremented
or decremented by 1 at random.

  * •

Insert pool: This mutation inserts a max-pooling layer at a random location in
the encoder. The initial pool size is 2x2.

  * •

Remove pool: In this case a random max-pooling layer from the encoder is
removed.

  * •

Alter filter number: The number of filters of a random convolutional layer is
adjusted.

  * •

Alter filter size: One of the dimensions of the filters of a random
convolutional layer is incremented or decremented at random

  * •

Alter pool size: The pool size of a random max-pooling layer is incremented or
decremented at random.

Figure 1: Convolutional autoencoder structure. For some input $x$, the encoder
produces an encoding $z$, and the decoder produces a reconstruction
$x^{\prime}$. The decoder mirrors the encoder, replacing pooling layers with
up-sampling layers.

On top of the mutations themselves, there is an additional constraint on each
mutation. For a mutation to be applied successfully, the output size of the
encoder must be smaller than input size of the encoder. This forces the
evolutionary process to evolve the encoder that actually compresses the data,
and helps preventing the autoencoder from learning a dictionary of the input.

At the end of the evolutionary process, one of the autoencoders has to be
chosen from the Pareto front to process the original data set to obtain a
compressed data set that is used in the second evolutionary process. A method
is needed that considers the trade-off between the compression ratio and the
accuracy of the autoencoders. This relies on MCDM, since the best autoencoder
choice is based on multiple, possibly conflicting, criteria (accuracy and
compression) to be optimized. There are many different existing algorithms
that help in picking a best solution based on multiple criteria. In this
study, the preferred method should be able to process any collection of
solutions, as it is not known beforehand what kind of autoencoders will be
involved. TOPSIS is a straight-forward and intuitive algorithm that can do
this [5].

User-provided weights are applied to each of the criteria of each solution,
allowing users to place emphasis on certain criteria. Then, the best and worst
possible solutions, called the positive ideal alternative and the negative
ideal alternative, are determined. In the context of convolutional
autoencoders, the positive ideal has an accuracy and compression ratio of
$1.0$, while the negative ideal has an accuracy and compression ratio of
$0.0$. Finally, the ratio of $L^{2}$ distance between solutions and the ideal
alternatives determines which solution is chosen.

The matter left open by this algorithm is that of the weights of criteria.
Since there are no previous results to draw conclusion from, the weights will
initially be set to equal values. The weights will depend on the relation
between the autoencoder compression ratio and the accuracy of the
classification networks.

##  4 Experiments and Results

Experiments are performed on the popular CIFAR-10 data set [6]. It is one of
the most popular data sets for the evaluation of CNNs, meaning that its use
makes for easier comparisons with other approaches. It consists of $60000$
images, split into a training set of $50000$ images, and a test set of $10000$
images. $5000$ of the training set images are held out in a validation set,
with the remaining $45000$ images constituting the actual training set. These
data splits are randomized between experiments. Each image is a $32\text{x}32$
RGB image, and is associated with one of $10$ labels, which indicate the
object in the image. The classes are evenly distributed over the training,
validation, and test sets. In all experiments, stochastic gradient descent
with a momentum of 0.9 is used to train networks. Training data is divided
into batches of 50 samples, making for 1000 batches per epoch. Each network is
trained for 25 epochs. This setup is copied from the baseline framework[10] in
order to help facilitate comparisons. Each worker process has access to an
Nvidia Tesla K80 GPU. Concurrent access to the population of networks is
facilitated purely by a shared file system; worker processes to not directly
communicate with each other. By taking advantage of POSIX
standards111http://standards.ieee.org/develop/wg/POSIX.html, minimal parallel
programming has been required [1]. The source code and accompanying
documentation is available
online.222https://github.com/marijnvk/LargeScaleEvolution

###  4.1 Baseline Framework

Initially, the baseline framework is run to confirm its performance, and to
inspect its behavior in a more resource-restricted setting. The results of
this experiment are listed in Table 1. Perhaps the most surprising result here
is that after 12 hours, 5 worker processes have reached a much higher best
classification accuracy than 250 worker processes. This may be partly because
of the more limited available mutations, which “focuses” the evolutionary
process better in the early stages in the case of five worker processes. At
the 24 hour point, the baseline has best accuracy. Nonetheless, this shows
that even a very small number of workers can be very effective at finding good
neural networks. Figure 2 illustrates the evolutionary process.

Figure 2: Results of the reproduction of the baseline framework, run with five
worker processes. Each point indicates a completely trained CNN, color-coded
by worker process. Table 1: Performance results for the baseline setup of the
framework. Accuracy values from the baseline study for the 12 and 24 hour
interval are derived from a figure and thus are not precise. The baseline
study did not report the number of generated networks.
Study | # Workers | Time elapsed (h) | Best accuracy (%) | # Networks
---|---|---|---|---
Baseline | 250 | 12.0 |  $\sim$56.6 | ?
Baseline | 250 | 24.0 |  $\sim$86.9 | ?
Baseline | 250 | 256.2 | 94.6 | ?
This study | 10 | 12.0 | 72.32 | 848
This study | 10 | 24.0 | 75.68 | 1197
This study | 1 | 12.0 | 58.7 | 190
This study | 1 | 24.0 | 59.7 | 299

###  4.2 Evolving Convolutional Autoencoders

The second experimental setting is the evolution of convolutional
autoencoders. After evolving over 900 autoencoders in a span of two days on a
single worker process, the status of the Pareto front is as in Figure 3. The
most telling structure in the population is the appearance of vertical groups
of autoencoders. These groups share the same compression ratio. Generally
those low in reconstruction accuracy tend to have fewer convolutional layers
compared to those with a higher reconstruction accuracy. Their compression
instead comes mostly from pooling layers. Evolution quickly pushes networks
towards very high compression ratios, mostly due to the insertion of multiple
pooling layers. Intuitively, however, it is the autoencoders that have a more
even trade-off between compression ratio and reconstruction accuracy that are
of most interest. These all have a fairly simple structure, usually being not
more than five layers in depth. This is due to the nature of the training
data. Images in the CIFAR-10 data set are already in compressed form when they
are presented to the autoencoders. This undoubtedly reduces the effectiveness
of the process step of evolving convolutional autoencoders.

At generation 507, an autoencoder is generated that clearly deviates from the
otherwise quite linear shape of the final Pareto front. This individual
attains a reconstruction accuracy of $70.3\%$ while reducing the input size by
two-thirds. This autoencoder is expected to give the best results when we move
to the last step of evolving image classifiers.

(a) Pareto front

(b) Full population

Figure 3: Results of the evolutionary process after generation 915. Each point
illustrates a convolutional autoencoder, plotted by encoder compression and
decoder accuracy. Fronts are marked by color. A compression of 1 means that
samples are reduced to a single value. The individual chosen by TOPSIS is
indicated by the dashed line. In the left plot, the number indicates the
generation of that individual (i.e. the number of individuals that were
generated before it).

###  4.3 Complete Framework

The final setting combines both versions of the framework into a full approach
that is supposed to outperform the baseline framework in terms of computation
time at the cost of some accuracy. A number of representative autoencoders are
picked from the Pareto font of evolved autoencoders and used to encode the
CIFAR-10 data set. Table 2 shows the chosen autoencoders and their
characteristics.

Figure LABEL:fig:cae_comp shows a representative example of runs for each of
the encoded data sets. As expected, the original data set results in the
eventual highest classification accuracy. It is the data coming from
autoencoder 507, the one which shows most promise and that progresses most
similarly to the baseline. While the results are quite similar in terms of
accuracy, the smaller, encoded data from autoencoder 507 means that
approximately $20\%$ more networks are generated, trained, and evaluated. For
more heavily compressed data sets, this increased number of generated networks
is even higher, up to nearly $50\%$. As the number of generated and evaluated
networks in the same time frame has increased, we have been able to increase
the efficiency of the method. But on the other hand, in all other cases the
encoding of data causes a significant loss of accuracy and early plateauing of
evolutionary progress. It is clear that in the encoded setting, the
evolutionary process has a significantly faster turn-over of networks. Note
that this effect can be magnified if mutations are chosen in a more efficient
way, which gives each new network a higher probability of being an improvement
in terms of accuracy.

While the gains in this setting appear rather small, if applied to the
baseline study, they would likely result in time savings on the scale of days.
As the evolutionary process on classifiers is run longer, gains increase.
After all, smaller sample sizes means that each batch is processed faster, and
reduces the need for larger networks, which pushes down the average network
size. Importantly, these results also show that the initial fair weighing of
compression ratio and accuracy of the autoencoders for constructing the Pareto
front is successful in determining the best autoencoder. Based on the results
though, it may be advisable to weigh the autoencoder accuracy somewhat heavier
since high compression rates seem to cause heavy plateauing. Note that TOPSIS
can easily be run again on the results of the evolution of autoencoders to
obtain a differently-weighed result quickly.

Table 2: Convolutional autoencoders (CAEs) sampled from the Pareto front. For
each CAE, its generation ID, compression ratio, and mean classification
accuracy over 5 runs are listed (variance in parentheses). Also listed is the
average number of networks generated when evolving classifiers with training
data encoded by this CAE. The last two CAEs do not come from the Pareto front,
but share the same compression rate with the best CAE.
CAE generation | Compression rate | Accuracy (Var) | #Generated
---|---|---|---
0 | 1.0 | 0.0 (0.0) | 105.8
507 | 0.66 | 0.7028 (0.0004) | 121.6
574 | 0.75 | 0.4289 (0.0004) | 130.4
266 | 0.83 | 0.3710 (0.0003) | 136.9
67 | 0.91 | 0.3054 (0.0003) | 141.2
611 | 0.92 | 0.2255 (0.0003) | 140.9
355 | 0.95 | 0.2123 (0.0003) | 145.0
882 | 0.97 | 0.1885 (0.0003) | 144.2
841 | 0.98 | 0.1493 (0.0002) | 146.1
130 | 0.66 | 0.5650 (0.0004) | 122.1
725 | 0.66 | 0.4271 (0.0003) | 118.4

##  5 Conclusion

A framework for evolving the structure of convolutional neural networks is
introduced. Its main weakness is clearly the large amount of computational
resources that are required to obtain networks with competitive accuracy. A
method to remedy this weakness is proposed in this paper in the form of a two-
step process. It consists of applying the baseline framework to the evolution
of convolutional autoencoders in the hope of reducing the sample size of the
input data. This makes any subsequent training of networks cheaper and helps
in limiting their size.

A significant result is the viability of much smaller number of worker
processes than previous studies demonstrated. This gives confidence in the
other results of this study, and in future work with this framework.
Experiments that investigate the impact of the new framework show the trade-
off between compressing the input data and maintaining classification
accuracy. The initial weighing of autoencoder accuracy and compression rate is
deemed successful, while some tuning may help if the framework has to be used
for other data sets. Although the experiments were necessarily limited in
scope due to the available computational resources, the results indicate that
this extended approach would result in significant running time reductions
when applied at larger scales, both in terms of input size and computational
time spent.

###  5.1 Future Work

The fundamental nature of the proposed work means that there are many more
directions to explore. Mentioned here are some the more interesting ones.

More complex data sets: As already mentioned before, the CIFAR-10 data set
used in this study consists of images that are already compressed
significantly. This naturally decreases the effectiveness of any convolutional
autoencoder that attempts to reduce the image size further. Running the
extended framework on a data set of larger images can realistically be
expected to result in a much larger impact of the step that evolves the
autoencoders.

Beating humans at their own game: The initial population for the evolutionary
process does not necessarily have to consist of very simple networks. An
interesting application is to take state-of-the-art, human-designed networks
as the individuals for the initial population.

Adaptive mutations: The availability of mutations affects the efficiency of
the overall evolutionary process. Different kinds of mutations will be more or
less likely to improve performance at different points in that process. The
process would benefit from some sort of adaptive mutation choosing process,
whereby mutations are sampled in a weighed fashion, perhaps even completely
excluding some mutations at certain points in the process.

Learning how to evolve: Given similar tasks, image classification for example,
the evolutionary process can be expected to behave somewhat similarly in terms
of which mutations are most effective at what points in the process. By adding
a layer of abstraction on top of the evolutionary process which keeps track of
this, it may be possible to learn how to best evolve networks given a task.

Constrained evolution for constrained networks: There are various situations,
most notably in embedded systems, where compact networks are very desirable.
The framework can easily be adapted to only consider a smaller solution space
that can be limited in a variety of ways by placing additional constraints on
the success of a mutation.

##  6 Acknowledgements

Research leading to these results has received funding from the EU ECSEL Joint
Undertaking under grant agreement no. 737459 (project Productive4.0) and from
Philips Research.

## References

  * [1] Breuel, T., Shafait, F.: Automlp: Simple, effective, fully automated learning rate and size adjustment. In: The Learning Workshop. vol. 4, p. 51. Utah (2010)
  * [2] Deb, K., Agrawal, S., Pratap, A., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evolutionary Computation 6(2), 182–197 (2002), https://doi.org/10.1109/4235.996017
  * [3] Desell, T.: Large scale evolution of convolutional neural networks using volunteer computing. In: Genetic and Evolutionary Computation Conference, Berlin, Germany, July 15-19, 2017, Companion Material Proceedings. pp. 127–128 (2017), http://doi.acm.org/10.1145/3067695.3076002
  * [4] Hagg, A., Mensing, M., Asteroth, A.: Evolving parsimonious networks by mixing activation functions. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2017, Berlin, Germany, July 15-19, 2017. pp. 425–432 (2017), http://doi.acm.org/10.1145/3071178.3071275
  * [5] Hwang, C., Lai, Y., Liu, T.: A new approach for multiple objective decision making. Computers & OR 20(8), 889–899 (1993), https://doi.org/10.1016/0305-0548(93)90109-V
  * [6] Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images (2009)
  * [7] Miikkulainen, R., Liang, J.Z., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., Hodjat, B.: Evolving deep neural networks. CoRR abs/1703.00548 (2017), http://arxiv.org/abs/1703.00548
  * [8] Miller, B.L., Goldberg, D.E.: Genetic algorithms, tournament selection, and the effects of noise. Complex Systems 9(3) (1995), http://www.complex-systems.com/abstracts/v09_i03_a02.html
  * [9] Morse, G., Stanley, K.O.: Simple evolutionary optimization can rival stochastic gradient descent in neural networks. In: Proceedings of the 2016 on Genetic and Evolutionary Computation Conference, Denver, CO, USA, July 20 - 24, 2016. pp. 477–484 (2016), http://doi.acm.org/10.1145/2908812.2908916
  * [10] Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y.L., Tan, J., Le, Q.V., Kurakin, A.: Large-scale evolution of image classifiers. In: Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017. pp. 2902–2911 (2017), http://proceedings.mlr.press/v70/real17a.html
  * [11] Rere, L.M.R., Fanany, M.I., Arymurthy, A.M.: Metaheuristic algorithms for convolution neural network. Comp. Int. and Neurosc. 2016, 1537325:1–1537325:13 (2016), https://doi.org/10.1155/2016/1537325
  * [12] Shafiee, M.J., Barshan, E., Wong, A.: Evolution in groups: A deeper look at synaptic cluster driven evolution of deep neural networks. CoRR abs/1704.02081 (2017), http://arxiv.org/abs/1704.02081
  * [13] Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evolutionary computation 10(2), 99–127 (2002)
  * [14] Such, F.P., Madhavan, V., Conti, E., Lehman, J., Stanley, K.O., Clune, J.: Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. CoRR abs/1712.06567 (2017), http://arxiv.org/abs/1712.06567
  * [15] Suganuma, M., Shirakawa, S., Nagao, T.: A genetic programming approach to designing convolutional neural network architectures. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2017, Berlin, Germany, July 15-19, 2017. pp. 497–504 (2017), http://doi.acm.org/10.1145/3071178.3071229
  * [16] Turner, A.J., Miller, J.F.: Neuroevolution: Evolving heterogeneous artificial neural networks. Evolutionary Intelligence 7(3), 135–154 (2014), https://doi.org/10.1007/s12065-014-0115-5