# Fixed Inter-Neuron Covariability Induces Adversarial Robustness

###### Abstract

The vulnerability to adversarial perturbations is a major flaw of Deep Neural
Networks (DNNs) that raises question about their reliability when in real-
world scenarios. On the other hand, human perception, which DNNs are supposed
to emulate, is highly robust to such perturbations, indicating that there may
be certain features of the human perception that make it robust but are not
represented in the current class of DNNs. One such feature is that the
activity of biological neurons is correlated and the structure of this
correlation tends to be rather rigid over long spans of times, even if it
hampers performance and learning. We hypothesize that integrating such
constraints on the activations of a DNN would improve its adversarial
robustness, and, to test this hypothesis, we have developed the Self-
Consistent Activation (SCA) layer, which comprises of neurons whose
activations are consistent with each other, as they conform to a fixed, but
learned, covariability pattern. When evaluated on image and sound recognition
tasks, the models with a SCA layer achieved high accuracy, and exhibited
significantly greater robustness than multi-layer perceptron models to state-
of-the-art Auto-PGD adversarial attacks without being trained on adversarially
perturbed data.

Index Terms—  Adversarial Robustness, Deep Learning, Biologically inspired

##  1 Introduction

While Deep Neural Networks (DNNs) have advanced the state of the art in many
fields of Artificial Intelligence (AI), their vulnerability to adversarial
perturbation – subtle distortions that are semantically irrelevant to humans
but can change the response of DNNs when added to their inputs [1, 2] – raises
concerns about their reliability when deployed in high-stakes real world
applications such as self-driving cars, and biometric authentication systems.
These concerns will need to be resolved for the wide-spread integration of AI
into human society to be possible.

Over the years, the development of methods to make DNNs more robust to
adversarial perturbations has become an active area of research. A large
fraction of the proposed methods rely heavily on machine learning and
statistical analysis techniques [3, 4, 5, 6, 7, 8], with the most popular and
successful of them being adversarial training in which involves training on
adversarially perturbed data [3]. Despite their effectiveness, these methods
have two major shortcommings: they trade-off accuracy on natural data for
adversarial robustness, and their effectiveness deteriorates if presented with
perturbations other than the precise type of adversarial perturbation that
they are designed to defend against [9, 10, 11], for instance a technique
designed to defend against low-magnitude perturbations applied to the whole
input may not be effective against high-magnitude perturbations applied to a
very localized region of the input [9].

In contrast, human perception is highly accurate and naturally robust to a
wide variety of such perturbations without ever being exposed to them, and
therefore it may be beneficial, as has been the case throughout the history of
neural networks, to seek inspiration from biological perceptual systems.
Following this reasoning, several biologically-inspired methods have been
proposed for defending DNNs against adversarial perturbations. These methods
generally computationalize, and integrate into DNNs, some mechanism of
biological perception that is as yet unrepresented in modern DNNs, but is
hypothesized to contribute to its robustness [12, 13, 14, 15]. The improvement
in robustness that these methods yield is relatively modest, compared to the
non-biologically inspired methods mentioned earlier, however, it usually does
not come at the cost of accuracy and tends to generalize better to other types
of perturbations.

Building on this body of work, in this paper we investigate the role of
inflexible inter-neuron covariability structures in making perception more
robust. It has been observed that in the brain the spiking activity of
individual neurons tends to be correlated [16, 17] and, the structure of this
correlation tends to be inflexible over long periods of time even if it limits
performance and learning [18]. We hypothesize that integrating such a
mechanism into DNNs may improve their robustness because it will restrict the
space of adversarial perturbations to only those that give rise to activations
that respect a fixed covariability structure. In other words, forcing the
intermediate activations to conform to a fixed covariability structure
prevents the adversarial perturbation from causing arbitrary changes to the
intermediate activations and thus misclassifications.

Integrating this behaviour into DNNs is not straightforward, because, unlike
biological neurons, neurons in a DNN output a deterministic real number, which
raises the question of how can we simulate correlated spiking in a system that
is neither stochastic nor produces spiking activity? A solution presents
itself if we consider the outputs of the artificial neuron as the frequency of
an underlying spike train. If the spiking activity of a group of neurons is
correlated, then the frequency of their spikes may also be correlated.
Therefore, we use spiking frequency as a proxy for the spikes trains and,
since the spiking frequency is represented in a DNN by the outputs of the
neurons, we impose a fixed covariability structure on the latter.

To simulate a fixed inter-neuron covariability pattern, we have developed the
Self-Consistent Activation (SCA) layer, which comprises of neurons whose
activations are consistent with each other as they conform to a fixed
covariability pattern. The SCA layer first computes the feed forward
activations for the neurons based on the input, and then iteratively optimizes
these activations to make them conform to a fixed, but learned, covariability
pattern.

We evaluated the effectiveness of the SCA layers on image and sound
classification tasks. For image classification we used MNIST [19] and Fashion-
MNIST (FMNIST) [20], and for sound classification we used SpeechCommands [21].
Compared to a Multi-Layer Perceptron (MLP), we find that the inter-neuron
correlations in models with a SCA layer are more invariant to adversarial
peturbations, thus indicating that the SCA layer does indeed make the inter-
neuron covariance structure more inflexible as we intended. Furthermore,
models with the SCA layer achieved similar if not better accuracy on the
unperturbed data, and significantly higher accuracy on adversarially perturbed
data generated by Auto-PGD [22] – a state-of-the-art white box adversarial
attack. We evaluated the accuracy of the models under adversarial
perturbations of several sizes and found that on average the model with the
SCA layer achieved an absolute improvement of 4%, 5% and 6%, and a relative
improvement of 117%, 155% and 45%, compared to the MLP model on FMNIST,
SpeechCommands and MNIST, respectively. Similar trends were observed when the
models were trained on adversarially perturbed data, with the SCA model
achieving significantly higher accuracy on the clean and adversarially
perturbed data on all datasets except MNIST.

##  2 Related Work

Non-Biologically Inspired Methods for Robustness: Most of the proposed methods
for making DNNs robust to adversarial perturbations are not inspired from
biology but rather rely on machine learning and statistical analysis
techniques. Perhaps the most successful of these methods is adversarial
training [3, 23], which adversarially perturbs each training minibatch by
using Projected Gradient Descent to modify the data in the minibatch such that
the loss is maximized. Over the years several improvements to the basic
adversarial training algorithm have been proposed, with each modifying
different parts of the model training pipeline, such as data augmentation
[24], and regularization [25]. Another popular group of methods concentrate on
creating models that are provably robust against adversarial perturbations and
are accompanied by formal guarantees of the form: with probability at least
$1-\delta$, where $\delta$ is small, the model’s output will not change if a
perturbation having norm at most $\epsilon$ is added to a given input [4, 26,
27, 28]. These methods have two major shortcommings: they trade-off accuracy
on natural data for adversarial robustness, and their effectiveness
deteriorates if presented with perturbations other than the precise type of
adversarial perturbation that they are designed to defend against [9, 10, 11].

Biologically Inspired Methods for Robustness: These methods generally involve
developing computational analogues of biological process that are absent from
common DNNs. Examples of such processes are predictive coding [12, 23],
associative memory [29] biologically constrained visual filters,
nonlinearities and stochasticity [13], foveation [14, 30, 31, 6], and non-
uniform retinal sampling and cortical fixations [15]. Perhaps most closely
related to our work are the methods based on predictive coding theory, which
posits that the brain is a hierarchical Bayesian network in which the output
of each layer is the maximum aposteriori estimate of the activations given the
output of the previous layer and the next layer [32]. Integrating predictive
coding layers within a model has been shown to improve its adversarial
robustness [12, 33]. This approach is similar to ours in so far as it
maximizes some notion of consistency between activations. Under predictive
coding the activations become consistent when the input can be reconstructed
perfectly from them, while in our work they become consistent when their
covariability structure matches the one learned during training.

##  3 The Covariability of DNN Activations

We hypothesize that the inflexible covariability structure of neuronal
activations that is observed in the animal brain contributes to the robustness
of biological vision. As a preliminary step in our investigation of this
hypothesis we determine if there is a relationship between the inflexibility
of the correlation matrix of a DNN’s activations, which we consider a proxy of
its covariability structure, and its robustness to adversarial perturbations.
To this end, we analyse the correlation structure between the neural
activations of a DNN in response to data perturbed with perturbations of
different sizes. To this end, we train two 5-layer MLP models on FMNIST, one
on clean data and the other via adversarial training, and compute the
correlation between the activations of the penultimate layer in response to
clean and adversarially perturbed images. We use $\mathbf{R}_{\epsilon}$ to
refer to the correlation matrix produced by data perturbed by perturbations of
$\ell_{\infty}$ norm $\epsilon$. We quantify the overall change in the
correlation structure as $\|\mathbf{R}_{0}-\mathbf{R}_{\epsilon}\|_{F}$, where
$\|\cdot\|_{F}$ is the Frobenius norm and plot this quantity for several
values of $\epsilon$ in Figure 1(a). We can see that the correlation structure
of the adversarially trained MLP is much more invariant to adversarial
perturbations compared to the correlation structure of the MLP trained on
clean data. It is only after the size of the perturbation become very large
does the correlation structure of the adversarially trained model begin to
change significantly. To verify that the change in the norm is not caused due
to a small number of neurons, we compute the absolute change in the
correlation of each neuron pair due to the addition of adversarial
perturbations of size 0.1, and plot the cumulative frequency curve shown in
Figure 1(b). We see that the curve for the adversarially trained model is
significantly shifted to the left of the curve for the model trained on clean
data indicating that the correlation between most, if not all, the pairs of
neurons has not changed significantly.

From these observations we can infer that the invariance of the inter-neuron
covariability structure across different perturbations of the input is related
to the adversarial robustness of the model. If this relationship is causal
then constraining the inter-neuron covariability structure should induce
adversarial robustness. To determine if this is indeed the case we design a
neural network layer that explicitly optimizes its activations to make them
conform to a fixed covariability structure. We then include this layer in a
DNN model and evaluate its robustness against state-of-the-art adversarial
attacks.

(a)

(b)

Fig. 1:  (a) The Frobenius norm of the change in the correlation matrix of the
activations of neurons in the penultimate layer of a MLP trained on clean (ST)
and adversarially perturbed (AT) FMNIST images when adversarial perturbations
of different sizes are added to the input; (b) CDF of the change in
correlation between neuron pairs when adversarial perturbation of
$\ell_{\infty}$ norm 0.1 is added.

##  4 Self-Consistent Activation Layer

We have developed the Self-Consistent (SCA) Activation Layer to simulate an
inflexible inter-neuron covariability structure. At a high-level, the SCA
layer computes its output as
${\text{SCA}(\mathbf{x})=g_{\mathcal{C}}(\mathbf{a}_{\mathbf{x}})}$ where
$\mathbf{x}\in\mathbbm{R}^{d_{\mathbf{x}}}$ is the input,
$\mathbf{a}_{\mathbf{x}}=f(\mathbf{x})\in\mathbbm{R}^{d_{\mathbf{a}}}$ is the
feed-forward activation vector, and $g_{\mathcal{C}}(\mathbf{a}_{\mathbf{x}})$
is the projection of $\mathbf{a}_{\mathbf{x}}$ onto $\mathcal{C}$, the
subspace comprised of the vectors that respect the learned covariance
structure. If we consider covariability to be a linear relationship, like
covariance, then $g_{\mathcal{C}}$ would simply be a linear projection.
However, to allow for more complex inter-neuron interaction, in this paper we
have decided to adopt the following non-linear form for $g_{\mathcal{C}}$:

$\displaystyle\arg\min_{\mathbf{a}_{\mathbf{x}}}$
$\displaystyle\|\mathbf{a}_{\mathbf{x}}-\phi(\mathbf{W}_{g}\mathbf{a}_{\mathbf{x}}+\mathbf{b}_{g})\|_{2}^{2}+\lambda\|\mathbf{x}-\mathbf{W}_{h}\mathbf{a}_{\mathbf{x}}-\mathbf{b}_{h}\|_{2}^{2},$
(1)

where $\phi=ReLU$, $\mathbf{W}_{g}\in\mathbbm{R}^{d_{\mathbf{a}}\times
d_{\mathbf{a}}}$, $\mathbf{W}_{h}\in\mathbbm{R}^{d_{\mathbf{x}}\times
d_{\mathbf{a}}}$, $\mathbf{b}_{g}\in\mathbbm{R}^{d_{\mathbf{a}}}$ and
$\mathbf{b}_{h}\in\mathbbm{R}^{d_{\mathbf{x}}}$. The first term represents the
distance between the activation and its projection onto $\mathcal{C}$, while
the second term represents the information about $\mathbf{x}$ that is not
carried by $\mathbf{a}_{\mathbf{x}}$. The latter is added as a regularizer to
prevent degenerate solutions, like $\mathbf{a}_{\mathbf{x}}=0$, in which
$\mathbf{a}_{\mathbf{x}}$ carries no information about $\mathbf{x}$, and
$\lambda$ is a scalar that controls the strength of the regularization. We set
the diagonal of $\mathbf{W}_{g}$ to zero to prevent it from becoming the
identity matrix and we perform the minimization using batch gradient descent.
The exact sequence of operations performed by the SCA layer is shown in
Algorithm 1.

Algorithm 1 SCA Layer

1:$\mathbf{u}\leftarrow f(\mathbf{x})$

2:for $t:1\rightarrow T$ do

3: $\mathbf{a}_{\mathbf{x}}\leftarrow\phi(\mathbf{u})$

4:
$J\leftarrow\|\mathbf{a}_{\mathbf{x}}-\phi(\mathbf{W}_{g}\mathbf{a}_{\mathbf{x}}+\mathbf{b}_{g})\|_{2}^{2}+\lambda\|\mathbf{x}-\mathbf{W}_{h}\mathbf{a}_{\mathbf{x}}-\mathbf{b}_{h}\|_{2}^{2}$

5:
$\mathbf{u}\leftarrow\mathbf{a}_{\mathbf{x}}-\eta\nabla_{\mathbf{a}_{\mathbf{x}}}J$

6:end for

7:$\mathbf{a}_{\mathbf{x}}\leftarrow\phi(\mathbf{u})$

##  5 Evaluation

###  5.1 Experimental Setup

####  5.1.1 Datasets

We evaluate the performance of SCA layers on image and audio classification
tasks. For image classification we use the MNIST [19] and Fashion MNIST
(FMNIST) [20] datasets, which contain 60,000 $28\times 28$ black-and-white
images of handwritten digits and 10 types of clothes, respectively. From both
MNIST and FMNIST, we used 45,000 images for training, 5000 for evaluation and
10,000 for testing. For the audio classification task we use a subset of
SpeechCommands dataset [21], which contains about 40,000 1 second, 16KHz
recordings of humans vocalizing digits 0 to 9. We use 31,000 recordings for
training, 4,000 recordings for validation and 4,000 recordings for testing.

####  5.1.2 Data Preprocessing

For the image datasets, we flatten the image into a vector, which is then
normalized by subtracting 0.5 and then dividing by 0.5. The audio data is pre-
processed by first downsampling to 8KHz. Then 128 Mel-Frequency Cepstral
Coefficients (MFCCs) are computed from a log mel spectrogram having 512 FFT
points computed over a 64 ms sliding window with a stride of 32ms. By
retaining only the first 16 MFCCs we obtain a $16\times 251$ matrix for each
1s audio recording. The matrix is then flattened, and normalized by
subtracting -0.96 and then dividing by 9.2 (the mean and standard deviation
computed over the validation set).

It is important to note here that the adversarial perturbations are computed
on and applied to the original input, prior to pre-processing. All the above
mentioned pre-processing steps are implemented using PyTorch and torchaudio,
and are fully differentiable, therefore the gradients can be propagated back
to the original input and used to compute adversarial perturbations.

####  5.1.3 Models

We compare the performance of models containing SCA layers (SCA model) to MLP
models having comparable architectures and number of parameters. The
schematics of these models are shown in Figure 2. The SCA model performs
$T=16$ optimization steps. The probability of dropout is set to 0.2 for all
models.

The models are optimized using the Adam optimizer using a learning rate of
0.001 and a batch size of 256 for up to 100 epochs. The learning rate is
halved if the loss on the validation set does not decrease for 5 epochs, and
if it does not decrease for 20 epochs the training is stopped early. All the
results presented below are averaged over 5 trials with different random
seeds.

Fig. 2: Schematics of the MLP and SCA models. $L_{.}$s are affine projections
with the superscripts representing the output dimension. The output dimension
of $L_{2}$ is set to 384 in MLPs trained on MNIST and FMNIST, and to 2048 in
MLPs trained on SpeechCommands. $\phi=ReLU$, $\sigma=\text{dropout}\circ\phi$,
$\mathcal{J}$ is the loss from eq. (1).

###  5.2 Results

####  5.2.1 Analysis of Activation Covariability Structure

To verify that SCA layers increase the invariance of the inter-neuron
correlation structure we analyse the correlation between the activations of
the penultimate layer using the method introduced in 3. Specifically, we
compute the correlation matrix $\mathbf{R}_{\epsilon}$ from the activations of
the penultimate layer of the SCA model and MLP in response to 1000 data
samples, perturbed by adversarial perturbations of $\ell_{\infty}$ norm
$\epsilon$. We compute $\mathbf{R}_{\epsilon}$ for each dataset using several
values of $\epsilon$. For each dataset and $\epsilon$ we then compute
$\|\mathbf{R}_{0}-\mathbf{R}_{\epsilon}\|_{F}$ to represent the overall change
in the correlation structure due to the addition of adversarial perturbation
of size $\epsilon$. Figure 3 shows this quantity for the SCA model and MLP on
each dataset. We see that in every case the correlation structure of the SCA
model changes more slowly than the MLP, and thus is more invariant to
adversarial perturbation. This result shows that the SCA layer indeed produces
the intended effect of constraining the covariability structure of neural
activations.

(a) MNIST

(b) FMNIST

(c) SpeechCommands

Fig. 3: The Frobenius norm of the change in the correlation matrix of the
penultimate layer activations from the SCA model and MLP due to the addition
of adversarial perturbations of different $\ell_{\infty}$ norms.

####  5.2.2 Robustness of Models Trained on Clean Data

We evaluate the adversarial robustness of the SCA models by training them on
clean data and computing their classification accuracy on adversarially
perturbed data. The adversarial perturbations are computed by running
AutoAttack [22] with different bounds on the maximum $\ell_{\infty}$ norm of
the perturbation.

The accuracy achieved by the SCA models and the baseline MLPs under
adversarial attack is shown in Table 1. We see that on all the three datasets
the SCA model is significantly more robust than the MLP to adversarial
perturbations, achieving higher accuracy on perturbations of all sizes.
Averaged across all perturbation sizes, the SCA model achieves an absolute
improvement in accuracy of 4%, 5% and 6%, and a relative improvement of 117%,
155% and 45%, compared to the MLP model on FMNIST, SpeechCommands and MNIST,
respectively. The above results clearly show that SCA layers are much more
robust to adversarial attacks than layers of perceptrons, and that their
robustness is not limited to a particular type of data but generalizes across
data complexity and modality.

Model |  Perturbation Sizes ($\ell_{\infty}$)
---|---
| 0.0 | 0.05 | 0.1 | 0.125 | 0.15 | 0.2
| MNIST
SCA | 97.9 | 88.1 | 54.8 | 32.3 | 15.7 | 1.7
MLP | 98.4 | 90.0 | 52.6 | 28.2 | 12.8 | 1.4
| FMNIST
SCA | 89.4∗ | 46.9∗ | 12.7∗ | 6.1∗ | 2.8∗ | 0.1
MLP | 88.7 | 39.2 | 7.7 | 3.1 | 1.0 | 0.0
|  Perturbation Sizes ($\ell_{\infty}$)
| 0.0 | 5e-5 | 1e-4 | 2.5e-4 | 5e-4 | 1e-3
| SpeechCommands
SCA | 90.1∗ | 45.1∗ | 32.2∗ | 10.3∗ | 2.4∗ | 0.5∗
MLP | 88.2 | 38.4 | 20.7 | 3.7 | 0.8 | 0.1
|  ∗$p<0.05$
Table 1: The accuracy (%) achieved by MLP and the SCA models

####  5.2.3 Robustness of Adversarially Trained Models

Given the SCA model is naturally more robust to adversarial perturbations, we
hypothesize that an adversarially trained SCA model will also be more robust
than an adversarially trained MLP. To verify this hypothesis, we adversarially
train MLP and SCA models using the method from [3]: for each mini-batch we
computer adversarial pertuabtions having $\ell_{\infty}$-norm at most
$\epsilon$ by performing 7 iterations of PGD with the step size $\epsilon/4$,
where $\epsilon$ is set to 0.3 for MNIST and FMNIST, and 2e-4 for
SpeechCommands.

Table 2 shows the accuracy of the adversarially trained models under Auto-PGD
attack [22]. The SCA model achieves greater accuracy than the MLP model for
all perturbation sizes on SpeechCommands, and for perturbation size greater
than 0.3 on FMNIST. Averaged across all perturbation sizes, the SCA model
achieves an absolute improvement in accuracy of 1.7% and 2.4%, and a relative
improvement of 49% and 12.6%, compared to the MLP model on FMNIST and
SpeechCommands, respectively. On the other hand, we find that the
adversarially trained MLP performs better than the SCA model under large
perturbations on MNIST. The fact that the SCA model is significantly more
robust than the MLP on more complex datasets, particularly SpeechCommands, but
not on the simpler MNIST, may indicate that, under adversarial training, the
utility of SCA layers is realized only in more complex tasks.

Model |  Perturbation Sizes ($\ell_{\infty}$)
---|---
| 0.0 | 0.2 | 0.3 | 0.325 | 0.35 | 0.375
| MNIST
SCA | 96.9∗ | 88.0∗ | 70.4 | 56.3 | 29.9 | 7.9
MLP | 94.7 | 85.8 | 74.9∗ | 64.6∗ | 34.4 | 11.0∗
| FMNIST
SCA | 70.4∗ | 51.5 | 35.1 | 22.4∗ | 7.0∗ | 2.3∗
MLP | 67.8 | 53.5∗ | 42.4∗ | 10.6 | 3.1 | 1.3
|  Perturbation Sizes ($\ell_{\infty}$)
| 0.0 | 1e-4 | 2.5e-4 | 5e-4 | 1e-3 | 2e-3
| SpeechCommands
SCA | 67.3 | 61.2∗ | 56.4∗ | 46.9∗ | 30.9∗ | 14.2∗
MLP | 66.3 | 59.6 | 54.6 | 43.6 | 26.8 | 11.4
|  ∗$p<0.05$
Table 2: The accuracy achieved by adversarially trained MLP and SCA models
under adversarial perturbations of various $\ell_{\infty}$ norms. The norm of
the perturbations used during training is 0.3 for MNIST and FMNIST models, and
2.4e-4 for the SpeechCommands models.

####  5.2.4 Impact of Number of Steps

To determine the impact of changing the number of steps, $T$, in Algorithm 1,
we train models with different values of $T$ using only clean data from
FMNIST. The accuracy of these models on clean and perturbed data is shown in
Table 3. We observe that increasing $T$ does not impact clean accuracy but
robustness significantly improves as $T$ is increased beyond 8. This indicates
that the more self-consistent the activations become the more robust the model
gets.

| Number of Steps
---|---
| 1 | 2 | 4 | 8 | 16
clean | 89.3 | 89.5 | 89.8 | 89.1 | 89.5
perturbed | 35.8 | 35.8 | 35.7 | 40.3 | 49.3
Table 3: The accuracy of SCA models with different number of self-consistency
optimizing steps ($T$) on clean and adversarially perturbed data. The size of
adversarial perturbations is set to 0.05.

##  6 Conclusion

In this paper we investigate the impact of the biological phenomenon of
inflexible inter-neuron covariability on robustness to adversarial
perturbation. To this end we develop the SCA layer, which as neural network
layer that comprises of neurons whose activations conform to a fixed, but
learned, covariability pattern. We demonstrate that DNNs with the SCA layer
tend to be significantly more robust to adversarial perturbations than
conventional MLPs without ever being trained on adversarially perturbed data,
achieving an average improvement of 102% in accuracy relative to the MLP on
image and audio data perturbed by state-of-the-art adversarial attack methods.
These results indicate that constraining the inter-neuron covariability
structure does indeed make DNNs more robust. More generally, these results
lend credence to the approach of seeking inspiration from biology for
developing better and more reliable AI systems.

## References

  * [1] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus,  “Intriguing properties of neural networks,”  in ICLR, 2014.
  * [2] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry,  “Adversarial examples are not bugs, they are features,”  Advances in neural information processing systems, vol. 32, 2019\.
  * [3] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu,  “Towards deep learning models resistant to adversarial attacks,”  arXiv preprint arXiv:1706.06083, 2017.
  * [4] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter,  “Certified adversarial robustness via randomized smoothing,”  in International Conference on Machine Learning. PMLR, 2019.
  * [5] Hadi Salman, Mingjie Sun, Greg Yang, Ashish Kapoor, and J Zico Kolter,  “Denoised smoothing: A provable defense for pretrained classifiers,”  NeurIPS, vol. 33, pp. 21945–21957, 2020.
  * [6] Muhammad A Shah and Bhiksha Raj,  “Less is more: Training on low-fidelity images improves robustness to adversarial attacks,”  in Submitted to ICLR., 2023,  under review.
  * [7] Muhammad A Shah, Raphael Olivier, and Bhiksha Raj,  “Towards adversarial robustness via compact feature representations,”  in ICASSP. IEEE, 2021.
  * [8] Raphael Olivier, Bhiksha Raj, and Muhammad Shah,  “High-frequency adversarial defense for speech and audio,”  in ICASSP. IEEE, 2021, pp. 2995–2999.
  * [9] Sander Joos, Tim Van hamme, Davy Preuveneers, and Wouter Joosen,  “Adversarial robustness is not enough: Practical limitations for securing facial authentication,”  in Proceedings of the 2022 ACM on International Workshop on Security and Privacy Analytics, 2022, pp. 2–12.
  * [10] Yash Sharma and Pin-Yu Chen,  “Attacking the madry defense model with $l\\_1$-based adversarial examples,”  arXiv preprint arXiv:1710.10733, 2017.
  * [11] Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel,  “Towards the first adversarially robust neural network model on mnist,”  arXiv preprint arXiv:1805.09190, 2018.
  * [12] Dylan M Paiton, Charles G Frye, Sheng Y Lundquist, Joel D Bowen, Ryan Zarcone, and Bruno A Olshausen,  “Selectivity and robustness of sparse coding networks,”  Journal of vision, 2020.
  * [13] Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James J DiCarlo,  “Simulating a primary visual cortex at the front of cnns improves robustness to image perturbations,”  NeurIPS, vol. 33, pp. 13073–13087, 2020.
  * [14] Aditya Jonnalagadda, William Yang Wang, B.S. Manjunath, and Miguel Eckstein,  “Foveater: Foveated transformer for image classification,” 2022.
  * [15] Manish Reddy Vuyyuru, Andrzej Banburski, Nishka Pant, and Tomaso Poggio,  “Biologically inspired mechanisms for adversarial robustness,”  NeurIPS, vol. 33, pp. 2135–2146, 2020.
  * [16] Jay A Hennig, Emily R Oby, Darby M Losey, Aaron P Batista, M Yu Byron, and Steven M Chase,  “How learning unfolds in the brain: toward an optimization view,”  Neuron, 2021.
  * [17] Patrick T Sadtler, Kristin M Quick, Matthew D Golub, Steven M Chase, Stephen I Ryu, Elizabeth C Tyler-Kabara, Byron M Yu, and Aaron P Batista,  “Neural constraints on learning,”  Nature, 2014.
  * [18] Matthew D Golub, Patrick T Sadtler, Emily R Oby, Kristin M Quick, Stephen I Ryu, Elizabeth C Tyler-Kabara, Aaron P Batista, Steven M Chase, and Byron M Yu,  “Learning by neural reassociation,”  Nature neuroscience, 2018.
  * [19] Yann LeCun, Corinna Cortes, and CJ Burges,  “Mnist handwritten digit database,”  ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2010\.
  * [20] Han Xiao, Kashif Rasul, and Roland Vollgraf,  “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,”  arXiv preprint arXiv:1708.07747, 2017.
  * [21] Pete Warden,  “Speech commands: A dataset for limited-vocabulary speech recognition,”  arXiv preprint arXiv:1804.03209, 2018.
  * [22] Francesco Croce and Matthias Hein,  “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,”  in International conference on machine learning. PMLR, 2020, pp. 2206–2216.
  * [23] Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang,  “Recent advances in adversarial training for adversarial robustness,”  arXiv preprint arXiv:2102.01356, 2021.
  * [24] Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann,  “Data augmentation can improve robustness,”  NeurIPS, 2021.
  * [25] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan,  “Theoretically principled trade-off between robustness and accuracy,”  in International conference on machine learning. PMLR, 2019, pp. 7472–7482.
  * [26] Marc Fischer, Maximilian Baader, and Martin Vechev,  “Certified defense to image transformations via randomized smoothing,”  NeurIPS, 2020.
  * [27] Aounon Kumar and Tom Goldstein,  “Center smoothing: Certified robustness for networks with structured outputs,”  NeurIPS, 2021.
  * [28] Bai Li, Changyou Chen, Wenlin Wang, and Lawrence Carin,  “Certified adversarial robustness with additive noise,”  Advances in neural information processing systems, vol. 32, 2019\.
  * [29] Dmitry Krotov and John Hopfield,  “Dense associative memory is robust to adversarial inputs,”  Neural computation, 2018.
  * [30] Yan Luo, Xavier Boix, Gemma Roig, Tomaso Poggio, and Qi Zhao,  “Foveation-based mechanisms alleviate adversarial examples,”  arXiv preprint arXiv:1511.06292, 2015.
  * [31] Jonathan M Gant, Andrzej Banburski, and Arturo Deza,  “Evaluating the adversarial robustness of a foveated texture transform module in a cnn,”  in SVRHM Workshop@ NeurIPS, 2021.
  * [32] Rajesh PN Rao and Dana H Ballard,  “Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects,”  Nature neuroscience, 1999.
  * [33] Bhavin Choksi, Milad Mozafari, Callum Biggs O’May, Benjamin Ador, Andrea Alamia, and Rufin VanRullen,  “Predify: Augmenting deep neural networks with brain-inspired predictive coding dynamics,”  NeurIPS, 2021.


[1]Corresponding author at: University of Zagreb, Faculty of Electrical
Engineering and Computing, Croatia

[orcid=0000-0003-2639-0812] [1] Conceptualization, Methodology, Software,
Validation, Data Curation, Writing - Original Draft

[orcid=0000-0001-6261-206X] Software, Data Curation, Writing - Review and
Editing

[orcid=0000-0002-4321-4557] Conceptualization, Resources, Writing - Review and
Editing, Supervision

[orcid=0000-0001-6508-1305] Resources, Data Curation, Writing - Review and
Editing, Funding acquisition

[orcid=0000-0002-4857-5351] Resources, Writing - Review and Editing,
Supervision, Project administration, Funding acquisition

# Generative adversarial network with object detector discriminator for
enhanced defect detection on ultrasonic B-scans

Luka Posilović<EMAIL_ADDRESS>Duje Medak<EMAIL_ADDRESS>Marko
Subašić<EMAIL_ADDRESS>Marko Budimir<EMAIL_ADDRESS>Sven
Lončarić<EMAIL_ADDRESS>University of Zagreb, Faculty of Electrical
Engineering and Computing, Zagreb, Croatia  Institute for Nuclear Technologies
(INETEC), Zagreb, Croatia

###### Abstract

Non-destructive testing is a set of techniques for defect detection in
materials. While the set of imaging techniques are manifold, ultrasonic
imaging is the one used the most. The analysis is mainly performed by human
inspectors manually analyzing recorded images. The low number of defects in
real ultrasonic inspections and legal issues considering data from such
inspections make it difficult to obtain proper results from automatic
ultrasonic image (B-scan) analysis. In this paper, we present a novel deep
learning Generative Adversarial Network model for generating ultrasonic
B-scans with defects in distinct locations. Furthermore, we show that
generated B-scans can be used for synthetic data augmentation, and can improve
the performance of deep convolutional neural object detection networks. Our
novel method is demonstrated on a dataset of almost 4000 B-scans with more
than 6000 annotated defects. Defect detection performance when training on
real data yielded average precision of 71%. By training only on generated data
the results increased to 72.1%, and by mixing generated and real data we
achieve 75.7% average precision. We believe that synthetic data generation can
generalize to other challenges with limited datasets and could be used for
training human personnel.

###### keywords:

non-destructive testing ultrasonic B-scan automated defect detection image
generation generative adversarial networks

A novel GAN architecture for generating high quality images with objects at
precise locations

Our proposed GAN is able to generate highly realistic data that can help
improve the object detector

An improvement of almost 5% of average precision was achieved when training on
a combined dataset of real and images generated with our GAN

##  1 Introduction

Non-destructive testing (NDT) is widely used in science and industry to
evaluate properties of materials, components or systems without causing damage
[1]. Many different methods are available such as visual examination,
ultrasonic, eddy current, to name a few. Among them, ultrasonic testing (UT)
stands out due to its versatility. Highly sensitive on most materials [2],
high signal to noise ratio [3], and ability to determine defect location and
type [2] are some of its advantages. Ultrasonic data can be represented in
several different formats suitable for analysis including A, B, or C-scans
[4]. An A-scan is an ultrasonic signal’s amplitude put as a function of time,
B-scan displays a cross-sectional view of the inspected material, and a C-scan
provides a top view of its projected features [5]. During analysis, inspectors
use all of them in order to make a decision and evaluate the data.

Automated analysis has long been used in many NDT systems. However, so far it
has been limited to classical decision-making algorithms such as amplitude
threshold [6]. Complex data such as the one from ultrasonic inspection makes
it hard to develop an automated analysis. All ultrasonic analysis is, to the
best of our knowledge, done manually by a trained human inspector. It makes
ultrasonic analysis highly reliant on the inspector’s experience. The
automated analysis could make the process much faster and reliable. There have
been some attempts in developing an automated UT analysis [7, 8, 9, 6, 5], but
very few of them involve using deep learning and modern deep convolutional
neural networks (CNNs) on B-scans. The prerequisite for using deep learning is
a large, annotated dataset. Due to the low number of flaws in real ultrasonic
inspections and legal issues considering data from such inspections available
data is limited. Data is the biggest drawback in the development of proper
automated/assisted ultrasonic analysis. This challenge can also be found in
many medical image analysis tasks [10] where, due to the rarity of some
pathology and patient privacy issues, data availability is very modest.
Furthermore, unlike medical datasets, there are no publicly available UT
datasets.

Researchers attempt to overcome this problem by using transfer learning [11]
in combination with freezing the backend CNN layers [12] which is shown to
enhance the accuracy of models. Using data augmentation is also the standard
procedure for network training. However, data augmentation methods are limited
and only slightly change some aspects of existing images (e.g. brightness
modulation). Very limited additional information can be gained by such
modifications. Synthetic data generation of high-quality images is a new type
of state-of-the-art data augmentation [13]. Generative models such as
generative adversarial networks (GANs) offer more variability and enrich the
dataset to further improve the training process.

In this work, we have presented a novel GAN architecture for generating high
quality and realistic UT B-scans. Afterward, we have used the generated images
to train an object detection neural network to detect defects in real images.
We have compared training the object detector with only real images using
traditional data augmentation and with GAN generated images. We have also
developed a more traditional method of generating new images using the
Copy/Pasting technique which proves the effectiveness of the GAN.

###  1.1 Contributions

The main contributions of this work are the following:

  * •

a novel GAN architecture for generating high-quality ultrasonic images with
objects at precise locations,

  * •

experimental demonstration that expanding the ultrasonic dataset with
generated synthetic data increases the performance of the defect detector.

###  1.2 Related work

Data availability is a major problem when using deep learning for defect
detection. B-scans are the ideal data representation for accurately detecting
defects and further estimating their depth and size. However, a B-scan usually
consists of hundreds of A-scans which further aggravates the problem of lack
of data. Developed algorithms for defect detection can be divided into three
groups related to data representation being used; A-scans [14, 15, 16, 17, 18,
19, 20, 7, 21, 22, 8, 23], B-scans [24, 25, 6, 5] and C-scans [26, 27]. The
A-scan analysis is the most researched group of all which is also related to
the data problem. Developed algorithms mostly include a combination of wavelet
transform [14, 15, 16, 17, 18, 19, 9], discrete Fourier transform [7, 21] or
discrete cosine transform [21] and a support vector machine or artificial
neural network classifier. B-scans keep the geometrical coherence of the
defect which leads to a better noise immunity [24]. However, the analysis of
B-scans can only be seen in a few works [5, 6]. In [5] two popular deep
learning object detection models, YOLOv3 and SSD, have been used for defect
detection. In [6] a deep learning detector has been tested on augmented
images, but with only three defects in the specimen block. Regarding C-scans,
in [26] a method based on the comparison of the scan with a reconstructed
reference image has been made. The method was able to detect all defects in
their dataset, but with a high number of false-positive detections. There have
also been some attempts in estimating defects from noisy measurements using
Bayesian analysis [27].

There have been some attempts in using data augmentation to enlarge the
existing dataset. As mentioned, in [6], although only three defects were
present in the test block, a copy/pasting data augmentation has been used to
enlarge the dataset for training a deep learning detector. There are many
variations on pasting and blending objects on the background in order to make
the images look as realistic as possible. For instance, it can be done using
Gaussian blur or Poisson blending [28] to smooth the edges. In [29] a
comparison between different merging techniques has been made, using a
combination of blending methods performed the best for most objects. On the
other hand in [30] authors have pasted objects on random backgrounds and
achieved improvements without any blending. Finally, generative adversarial
networks (GANs) have recently become a popular topic of research in the field
of synthetic data generation and augmentation. GANs were first conceptualized
in [31] in 2014. They can be used to generate images, video, audio, text, and
much more. The development of the GAN came a long way in a short period of
time. There are many different GAN architectures. An interesting approach to
GANs are image to image translation models. They are used for style transfer
between images [32], image inpainting [33] and even generating images from
masks [34]. One of the examples of those models is the Pix2pixGAN [34] and its
successor pix2pixHD [35]. GANs show promising results in generating realistic
images for human faces from noise with StyleGan2 [36] or converting position
mask images to street-view with Pix2pixHD. A lot of work has been done for
enlarging data sets in medical imagery. Pix2pixHD has proved to be useful in
generating skin lesion images using semantic label maps [37]. An Inception-v4
classifier [38] has been trained using real and combined real and data
generated with the Pix2pixHD. Training the classifier on a combined real and
generated data achieved a 1% improvement of the area under the ROC curve. In
[13] authors have applied the GAN framework to synthesize high-quality liver
lesion images for improved classification. In [39] authors have developed a
multi-channel GAN (M-GAN) to generate PET images from CT scans. A similar
approach with a cGAN has been made in [40, 41]. With generated data, they have
achieved a 28% reduction in average false positive per case. Generating MR
images from CT scans with paired and unpaired data has been researched in
[42]. An MR-GAN with a concept similar to CycleGAN [32] has been developed for
the purpose. In [43] a DCGAN has been employed to generate realistic brain MR
images. Data augmentation using non-convolutional GAN has been tested on three
different non-image datasets [44]. Generated data has performed even better
than real data when classifying using a Decision Tree (DT) classifier.

###  1.3 Outline

This paper is organized as follows. Section 2 gives a detailed description of
the dataset. Section 3 demonstrates the experimental procedure. Proposed GAN
architecture and copy/pasting method are presented in Section 4. Results are
shown in Section 5 and a conclusion is given in Section 6.

##  2 Dataset

Figure 1: Example of an ultrasonic B-scan with defects

The dataset was obtained by scanning six steel blocks containing artificially
created defects in the internal structure. Blocks varied in size and contained
between six to 34 defects. In total there were 68 defects. Blocks were scanned
using INETEC Dolphin scanner with phased array probes. An INETEC phased array
ultrasound transducer of central frequency of 2.25MHz was used. Data were
acquired by scanning from an angle of 45 degrees to 79 degrees with a 2-degree
increment. Blocks were also scanned with a skew of zero and 180 degrees.
INETEC SignyOne data acquisition and analysis software was used to process the
data and create B-scans (further noted as images) that were used in the
dataset. Data were converted to B-scans as-is, without pseudo-coloring, as
grayscale images. All images were then converted into patches of size 256x256
pixels and annotated by multiple human experts. There were in total 3825
images with a total of 6238 annotations. Dataset was split into train, test,
and validation subsets. The train consisted of 2278 images, validation of 379
images, and the test of 1168 images. Details of the dataset split can be seen
in Table 1. The split was made such that defects in each subset are unique,
they do not appear in other subsets. GAN and the object detector were trained
using this train/validation/test dataset split. As for the copy/pasting
method, we copied the defects from the training subset in the form of the
rectangle patches from the annotations and pasted them on the empty UT
backgrounds. There were 3400 empty UT backgrounds from all of the blocks and
4283 defect patches from the training subset.

Table 1: Number of images and annotations in each of the data subset | TRAIN | VALIDATION | TEST
---|---|---|---
Number of images | 2278 | 379 | 1168
Number of annotations | 4283 | 745 | 1210

##  3 Synthetic data generation

The acquired dataset might not be enough to properly train an object detector
to detect defects. For this reason, we propose two methods to expand our
dataset with synthetic data.

In this section, we have described the procedure of the experiment in this
work. We developed two methods for synthetic image generation and use a well-
established object detector to test the quality of the generated data. We
first start by describing the more traditional method for generating images
and proceed to describe a deep learning approach with our GAN. We then explain
the usage of the object detector in the experiment.

Our first generative method is a copy/pasting (C/P) technique. Copy/pasting is
a very logical method for enlarging the ultrasonic dataset because of the
large number of B-scans without defects. We call these images canvases because
we paste extracted defects on them. We extracted all of the defects from the
training set and pasted them on canvases in random locations. The exact method
is explained in the next section. An example of an empty image canvas and an
extracted defect can be seen in Figure 2.

Figure 2: An example of an empty canvas and a randomly extracted defect

The second method we propose is our own GAN architecture for the purpose of
generating UT B-scans. Our GAN is an image-to-image GAN, with paired position
masks as input and real images as a desired output. We make position masks
from all annotated images in the training set. An example of a paired position
mask and the real image is shown in Figure 3. Position masks on the input of
the generator serve as a location label for the desired position of the defect
on the generated image. The main novelty of our GAN is the usage of a pre-
trained object detector for training the GAN. We use the object detector as an
additional discriminator to provide information on the quality of the defect
on the generated image when compared to the real image. It is important that
the defect is positioned accurately as drawn in the position mask and that it
is merged well with the background. After training the GAN we generated new
position masks used for generating synthetic data. For generating position
masks we extract the aspect ratios of all annotations from the training set.
We then create random masks by randomly putting the white boxes with
predefined aspect ratios on black canvases. Our generated images have between
one and four defects per image.

Figure 3: A pair of GAN input image and a desired output (real image)

For estimating the quality of the generated images we used a popular object
detector that was already proven in UT defect detection on images [5], a
YOLOv3 [45] object detector. We first trained the detector using only real
images and some traditional augmentations explained in the next section. We
then tried training the object detector with images generated using the
copy/paste method. We also trained the detector with a combination of real and
generated images. We used the object detector trained on real images for
training the GAN. We generated synthetic data with our GAN and again trained
the object detector with generated images and a combination of real and
generated data. Each of the trained versions of the object detector was tested
on the same test dataset described in the previous section. Also, the same
validation set was used in all three training variations.

##  4 Methods

In this section, a detailed explanation of developed methods is given. First,
the copy/pasting method is described. Then the architecture of our proposed
GAN is described with all of its special features. In the end, a short
overview of the used object detector is given.

###  4.1 Copy/paste method

This method is a simple method that serves to illustrate the complexity of
generating synthetic data. While these images might look visually appealing,
they are not of the same quality as the ones generated by the GAN.

As mentioned in the Section 2 we have previously extracted defects from images
in the training set. We paste them on random locations on images without
visible defects. The process goes as follows. First, we randomly pick a canvas
and randomly select the defect we would like to paste on it. We then put a
threshold on a defect image. We make a binary pseudo mask by creating a binary
image from the thresholded image and dilate it for two iterations. We then use
the mask to extract only the defect from the initial defect patch image. We
randomly select the position where we will paste the defect and calculate the
compatibility of the selected defect background and the canvas on that
location. We calculate the compatibility by calculating an average value of
the background of the canvas and the defect. If these two values do not differ
by more than 5%, we accept the proposed location. If these two values differ
by more than 5% we try to select another location. We then select another
image/canvas pair and repeat the process. For each new image, we set the limit
of 100 attempts after which we just move on to generate another image.
Usually, this limit is enough to never move on, but always be able to find the
right canvas/defect and location pair. If we found the right pair, we then
proceed to paste the defect on the canvas. We first adapt the brightness of
the defect to even further match the one from the canvas. We calculate the
brightness of the location on the canvas and adapt the brightness of the
defect to it. All that is left after that is to paste the defect. We
concatenate the canvas and the defect, calculate the per pixel minimum of the
two and merge it into one resulting image.

Samples of the real image, image generated with our GAN, and image generated
with a copy/paste method are shown in Figure 5.

###  4.2 GAN

Figure 4: Simplified architecture of our DetectionGAN

The basic idea of GAN is a combination of two neural networks, a generator
that generates high-quality images from random noise and a discriminator which
tries to distinguish generated images from the real ones. The constant rivalry
between the generator and the discriminator is what makes GANs adversarial.
Mathematically, discriminator and generator play a minimax game where the goal
of the generator is to maximize the probability of discriminator labeling
generated images as real samples and discriminator has the goal of minimizing
that probability while being able to label real data as such. This neural
network configuration enables unsupervised learning of both generator and
discriminator.

We call our GAN the DetectionGAN for its specific architecture. We base it on
the Pix2pixHD implementing some of the features from it. Our GAN consists of a
U-net generator, two PatchGAN discriminators [34] that work on different
scales and a pre-trained object detector which serves as an additional
discriminator. We train the GAN with image pairs of real images and their
position masks.

Input to the generator is the position mask defining the position of the
defect. The output of the generator is connected to the discriminator and to
the object detector. There is a total of 54,409,603 parameters in the
generator. All of them are randomly initialized and trained. Unlike in
Pix2pixHD, we do not have a two-stage generator nor we upscale the position
mask in order to generate a higher resolution image.

Discriminator has a position mask image concatenated to the generated or real
image as an input. Image and mask are concatenated across channels axis. The
goal of the concatenation of the position mask and the image is to provide
information on the position of defects in the image for the discriminator.
This concatenation provides an improvement shown in the Section 5.
Discriminator gets the real and the generated image during each step as an
input. Since there are two discriminators, one of them inputs images
downscaled by a factor of two in order for them to work on different scales.
They have 1,391,554 parameters that are randomly initialized and trained.

For the additional discriminator we use a YOLO object detector during this
experiment, but any other object detector could be used. It helps generate
highly realistic images with defects in precise, desired locations. We input
the generated image and then the real image pair and compare the outputs. We
want these two outputs to be the same so that there is no difference between
the generated and real image for the detector. This way we ensure defects are
placed on the exact locations and without any artifacts. Using an object
detector as a discriminator provides a significant improvement shown in the
Section 5.

The simplified architecture of our GAN is pictured in Figure 4. Filter sizes
at each layer of the generator and discriminators are noted in the figure.

###  4.3 Object detector

Our object detector, You Only Look Once (YOLO) version 3 is mainly taken from
[5] where it proved to be able to detect defects with high average precision.
With this work, we used the proposed object detector to improve its
performance on a more realistic dataset. We train the object detector on real,
generated data and a combination of those two. We train the networks with the
same hyperparameters in order to have a fair comparison. We first trained the
detector on real data, tuned the hyperparameters to achieve the best possible
performance and used the proposed parameters for training with other data
combinations. We input images of size 416x416 pixels. We used a pre-trained
backbone and froze its parameters while training. Hence, we trained only
20,974,518 of a total number of 61,576,342 parameters.

##  5 Experimental setup and results

###  5.1 Experimental setup

In this section, we describe the experiment and parameters used to train our
GAN and the object detector. Our experiment goes as follows. We first train an
object detector with real data. We then used that neural network to train our
GAN. We generated synthetic images using the copy/pasting method and the deep
learning GAN method. We generated 200,000 synthetic images with both
copy/paste and GAN method. For each version of GAN we use the same position
masks to generate synthetic images. We again train the object detector using
the generated data and compare results.

Position masks for the input of the GAN are of size 256x256 pixels, as well as
the generated images. For training the generator we use Adam optimizer with a
first moment term of 0.5, the second one of 0.999, and a learning rate of
0.0002. One of our discriminators has an input image of 256x256 pixels, while
the other one has a downscaled image of 128x128 pixels as an input. We train
discriminators using Adam optimizer with the same parameters as the generator.
We train our GAN using a set of loss functions. For the generator, we use four
different losses. At the output of the generator, we calculate the L1 loss on
the generated image and the paired real image. For propagating discriminator
output to the generator we use the mean squared error loss. We also use the
feature map L1 loss in the generator similar to the one in [35]. We also
compare the output of the object detector when inputting the real image and
the output from the generator. We use the L1 loss to propagate it down to the
generator. For training the discriminator we use the mean square error loss,
just like it is used in Pix2pixHD. We use the pre-trained object detector and
do not train it during training the GAN. We implement a set of simple data
augmentations for training the GAN including horizontal flipping, brightness
modulation, and random cropping. These data augmentations enable us to achieve
great results in training the GAN and generating high-quality images. We train
the GAN for 800 epochs and for the last 100 epochs we linearly reduce the
learning rate to zero. Each epoch corresponds to one sweep through all the
images in the training dataset. We trained with a batch size eight. It took us
nearly 96 hours to train the GAN using a single NVIDIA RTX 2080Ti graphics
card.

For training the object detector we used the following configuration. We use
batch size eight and Adam optimizer with a learning rate 0.003. Custom anchors
were calculated on the training set for all of the training combinations. We
slightly changed only the ignore threshold hyperparameter to 0.6 from the
original YOLO implementation. We used checkpoints while training the model. An
early stopping callback was used to stop the training after the validation
loss didn’t improve for over eight epochs. We reduced the learning rate after
every two epochs with no improvement on the validation set. We also used some
basic augmentations while training all of the models. Those augmentations
include horizontal image flipping, random cropping, and HSV space modulation.
I took us 30 minutes to train the object detector using only real data, and
around 24 hours using generated data.

###  5.2 Results and discussion

The performance of the proposed approach was tested on a test subset. As
described in Section 3 we test the quality of generated images by training an
object detector on real and generated images. We used an average precision
(AP) metric for assessing the performance of an object detector on a test set.
The object threshold of YOLO was 0.001 while the non-maximum suppression
threshold was 0.5 and the intersection over union was 0.5. Generated images
used in this test were not handpicked but randomly generated.

Figure 5: Samples of (from left to right): real image, image generated with
GAN and a copy/paste image

Detailed results can be seen in Table 2. Using the C/P method for image
synthesis did not provide any improvements in the detection. When training
only on C/P images we acquire a very bad result of only 47.42% AP on the same
test set. When training on the combination of both real and C/P images we
again do not get any improvements. The reason could be that this data has some
artifacts when compared to the real images. Although visually, both images
generated with GAN and with the copy/paste method look realistic, the object
detector tends to learn wrong features and can not converge to a better model
than the one trained on real images. However, we achieved an improvement with
GAN generated images when opposed to training the object detector with only
real images. An improvement of 1% has been achieved while training only on GAN
generated images, and an improvement of almost 5% of AP was achieved when
training on a combined dataset of real and images generated with our GAN. As a
reference, experiments with two versions of the GAN without the object
detector discriminator and without position mask and image concatenation in
the discriminator were made. Both versions perform worse than our
DetectionGAN.

Table 2: Results of training the object detector on different training datasets TRAINING DATA | AP (%) | TRAINING DATA SIZE
---|---|---
Real | 70.96 | 2,278
C/P | 47.42 | 200,000
C/P + real | 69.93 | 202,278
GAN w/o conc. | 39.90 | 200,000
GAN w/o conc. + real | 69.92 | 202,278
GAN w/o yolo | 62.22 | 200,000
GAN w/o yolo + real | 69.3 | 202,278
DetectionGAN | 72.11 | 200,000
DetectionGAN + real | 75.65 | 202,278

This experiment proves that it is important to have the most realistic data as
it is possible to achieve an improvement. It illustrates the complexity of the
problem of generating synthetic data for training the object detector. Our
proposed GAN is able to generate highly realistic data that can help improve
the object detector.

##  6 Conclusion and future work

In this paper, we propose a novel generative adversarial network for
generating highly realistic B-scans (images) from position mask images. Our
GAN generates highly realistic ultrasonic images from position masks that can
be used to train an object detector. We achieved an improvement of almost 5%
while training on a combination of generated and real data. We also developed
a copy/pasting method for data augmentation that proved the complexity of
improving the average precision of an object detector with additional data. As
we didn’t cherry-pick GAN generated images all of the generated images were
proven to be of high quality.

With the increasing problem of lack of data and advances in generating high-
quality synthetic data, networks such as our GAN could be used in many science
and industry fields. In future work, our GAN should be tested using different
object detectors.

##  7 Acknowledgments

This research was co-funded by the European Union through the European
Regional Development Fund, under the grant KK.01.2.1.01.0151 (Smart UTX).

## References

  * [1] L. Cartz, Nondestructive testing: radiography, ultrasonics, liquid penetrant, magnetic particle, eddy current, ASM International, 1995.
URL https://books.google.hr/books?id=0spRAAAAMAAJ

  * [2] J. Ye, S. Ito, N. Toyama, Computerized ultrasonic imaging inspection: From shallow to deep learning, Sensors 18 (11) (2018) 3820.  doi:10.3390/s18113820.
URL https://doi.org/10.3390/s18113820

  * [3] P. Broberg, Imaging and analysis methods for automated weld inspection, Ph.D. thesis, Luleå tekniska universitet (2014).
  * [4] J. Krautkrämer, H. Krautkrämer, Ultrasonic Testing of Materials, Springer-Verlag, 1983.
URL https://books.google.hr/books?id=AvwrAAAAIAAJ

  * [5] L. Posilović, D. Medak, M. Subašić, T. Petković, M. Budimir, S. Lončarić, Flaw detection from ultrasonic images using yolo and ssd, in: 2019 11th International Symposium on Image and Signal Processing and Analysis (ISPA), IEEE, 2019, pp. 163–168.
  * [6] I. Virkkunen, T. Koskinen, O. Jessen-Juhler, J. Rinta-Aho, Augmented ultrasonic data for machine learning, arXiv preprint arXiv:1903.11399.
  * [7] I. S. Souza, M. C. Albuquerque, E. F. de SIMAS FILHO, C. T. FARIAS, Signal processing techniques for ultrasound automatic identification of flaws in steel welded joints–a comparative analysis, in: 18th World Conference on Nondestructive Testing, 2012, pp. 16–20.
  * [8] N. Munir, H.-J. Kim, S.-J. Song, S.-S. Kang, Investigation of deep neural network with drop out for ultrasonic flaw classification in weldments, Journal of Mechanical Science and Technology 32 (7) (2018) 3073–3080.  doi:10.1007/s12206-018-0610-1.
URL https://doi.org/10.1007/s12206-018-0610-1

  * [9] M. Meng, Y. J. Chua, E. Wouterson, C. P. K. Ong, Ultrasonic signal classification and imaging system for composite materials via deep convolutional neural networks, Neurocomputing 257 (2017) 128–135.
  * [10] C. Shorten, T. M. Khoshgoftaar, A survey on image data augmentation for deep learning, Journal of Big Data 6 (1) (2019) 60.
  * [11] D. Ventura, S. Warnick, A theoretical foundation for inductive transfer, Brigham Young University, College of Physical and Mathematical Sciences.
  * [12] D. Soekhoe, P. Van Der Putten, A. Plaat, On the impact of data set size in transfer learning using deep neural networks, in: International Symposium on Intelligent Data Analysis, Springer, 2016, pp. 50–60.
  * [13] M. Frid-Adar, I. Diamant, E. Klang, M. Amitai, J. Goldberger, H. Greenspan, Gan-based synthetic medical image augmentation for increased cnn performance in liver lesion classification, Neurocomputing 321 (2018) 321–331.
  * [14] F. Bettayeb, T. Rachedi, H. Benbartaoui, An improved automated ultrasonic nde system by wavelet and neuron networks, Ultrasonics 42 (1) (2004) 853 – 858, proceedings of Ultrasonics International 2003.  doi:https://doi.org/10.1016/j.ultras.2004.01.064.
URL https://doi.org/10.1016/j.ultras.2004.01.064

  * [15] S. Sambath, P. Nagaraj, N. Selvakumar, Automatic defect classification in ultrasonic ndt using artificial intelligence, Journal of Nondestructive Evaluation 30 (1) (2011) 20–28.  doi:10.1007/s10921-010-0086-0.
URL https://doi.org/10.1007/s10921-010-0086-0

  * [16] Y. Chen, H.-W. Ma, G.-M. Zhang, A support vector machine approach for classification of welding defects from ultrasonic signals, Nondestructive Testing and Evaluation 29 (3) (2014) 243–254.  arXiv:https://doi.org/10.1080/10589759.2014.914210, doi:10.1080/10589759.2014.914210.
URL https://doi.org/10.1080/10589759.2014.914210

  * [17] A. Al-Ataby, W. Al-Nuaimy, C. Brett, O. Zahran, Automatic detection and classification of weld flaws in tofd data using wavelet transform and support vector machines, Insight - Non-Destructive Testing and Condition Monitoring 52 (2010) 597–602.  doi:10.1784/insi.2010.52.11.597.
URL https://doi.org/10.1784/insi.2010.52.11.597

  * [18] V. Matz, M. Kreidl, R. Smid, Classification of ultrasonic signals, International Journal of Materials 27 (2006) 145–.  doi:10.1504/IJMPT.2006.011267.
URL https://doi.org/10.1504/IJMPT.2006.011267

  * [19] M. Khelil, M. Boudraa, A. Kechida, R. Drai, Classification of Defects by the SVM Method and the Principal Component Analysis (PCA) (09 2007).  doi:10.5281/zenodo.1060751.
URL https://doi.org/10.5281/zenodo.1060751

  * [20] M. Meng, Y. J. Chua, E. Wouterson, C. P. K. Ong, Ultrasonic signal classification and imaging system for composite materials via deep convolutional neural networks, Neurocomputing 257 (2017) 128 – 135, machine Learning and Signal Processing for Big Multimedia Analysis.  doi:https://doi.org/10.1016/j.neucom.2016.11.066.
URL https://doi.org/10.1016/j.neucom.2016.11.066

  * [21] F. Cruz, E. S. Filho, M. Albuquerque, I. Silva, C. Farias, L. Gouvêa, Efficient feature selection for neural network based detection of flaws in steel welded joints using ultrasound testing, Ultrasonics 73 (2017) 1 – 8.  doi:https://doi.org/10.1016/j.ultras.2016.08.017.
URL https://doi.org/10.1016/j.ultras.2016.08.017

  * [22] G. A. Guarneri, F. N. Junior, L. de Arruda, Weld discontinuities classification using principal component analysis and support vector machine, XI Simpósio Brasileiro de Automaçao Inteligente (2013) 2358–4483.
  * [23] J. Veiga, A. A. de Carvalho, I. Silva, J. M. A. Rebello, The use of artificial neural network in the classification of pulse-echo and tofd ultra-sonic signals, Journal of The Brazilian Society of Mechanical Sciences and Engineering - J BRAZ SOC MECH SCI ENG 27.  doi:10.1590/S1678-58782005000400007.
URL https://doi.org/10.1590/S1678-58782005000400007

  * [24] H. Cygan, L. Girardi, P. Aknin, P. Simard, B-scan ultrasonic image analysis for internal rail defect detection, in: World Congress on Railway Research, 2003.
  * [25] A. Kechida, R. Drai, A. Guessoum, Texture analysis for flaw detection in ultrasonic images, Journal of Nondestructive Evaluation 31 (2) (2012) 108–116.  doi:10.1007/s10921-011-0126-4.
URL https://doi.org/10.1007/s10921-011-0126-4

  * [26] H. Kieckhoefer, J. Baan, A. Mast, W. A. Volker, Image processing techniques for ultrasonic inspection, in: Proc. 17th World Conference on Nondestructive Testing, Shanghai, China, 2008.
  * [27] A. Dogandzic, B. Zhang, Bayesian nde defect signal analysis, Signal Processing, IEEE Transactions on 55 (2007) 372 – 378.  doi:10.1109/TSP.2006.882064.
URL https://doi.org/10.1109/TSP.2006.882064

  * [28] P. Pérez, M. Gangnet, A. Blake, Poisson image editing, in: ACM SIGGRAPH 2003 Papers, 2003, pp. 313–318.
  * [29] D. Dwibedi, I. Misra, M. Hebert, Cut, paste and learn: Surprisingly easy synthesis for instance detection, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1301–1310.
  * [30] J. Rao, J. Zhang, Cut and paste: Generate artificial labels for object detection, in: Proceedings of the International Conference on Video and Image Processing, 2017, pp. 29–33.
  * [31] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, in: Advances in neural information processing systems, 2014, pp. 2672–2680.
  * [32] J.-Y. Zhu, T. Park, P. Isola, A. A. Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2223–2232.
  * [33] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, B. Catanzaro, Image inpainting for irregular holes using partial convolutions, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 85–100.
  * [34] P. Isola, J.-Y. Zhu, T. Zhou, A. A. Efros, Image-to-image translation with conditional adversarial networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1125–1134.
  * [35] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, B. Catanzaro, High-resolution image synthesis and semantic manipulation with conditional gans, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8798–8807.
  * [36] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, T. Aila, Analyzing and improving the image quality of StyleGAN, in: Proc. CVPR, 2020.
  * [37] A. Bissoto, F. Perez, E. Valle, S. Avila, Skin lesion synthesis with generative adversarial networks, in: OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis, Springer, 2018, pp. 294–302.
  * [38] C. Szegedy, S. Ioffe, V. Vanhoucke, A. Alemi, Inception-v4, inception-resnet and the impact of residual connections on learning, arXiv preprint arXiv:1602.07261.
  * [39] L. Bi, J. Kim, A. Kumar, D. Feng, M. Fulham, Synthesis of positron emission tomography (pet) images via multi-channel generative adversarial networks (gans), in: molecular imaging, reconstruction and analysis of moving body organs, and stroke imaging and treatment, Springer, 2017, pp. 43–51.
  * [40] A. Ben-Cohen, E. Klang, S. P. Raskin, M. M. Amitai, H. Greenspan, Virtual pet images from ct data using deep convolutional networks: initial results, in: International Workshop on Simulation and Synthesis in Medical Imaging, Springer, 2017, pp. 49–57.
  * [41] A. Ben-Cohen, E. Klang, S. P. Raskin, S. Soffer, S. Ben-Haim, E. Konen, M. M. Amitai, H. Greenspan, Cross-modality synthesis from ct to pet using fcn and gan networks for improved automated lesion detection, Engineering Applications of Artificial Intelligence 78 (2019) 186–194.
  * [42] C.-B. Jin, H. Kim, M. Liu, W. Jung, S. Joo, E. Park, Y. S. Ahn, I. H. Han, J. I. Lee, X. Cui, Deep ct to mr synthesis using paired and unpaired data, Sensors 19 (10) (2019) 2361.
  * [43] K. Kazuhiro, R. A. Werner, F. Toriumi, M. S. Javadi, M. G. Pomper, L. B. Solnes, F. Verde, T. Higuchi, S. P. Rowe, Generative adversarial networks for the creation of realistic artificial brain magnetic resonance images, Tomography 4 (4) (2018) 159.
  * [44] F. H. K. d. S. Tanaka, C. Aranha, Data augmentation using gans, arXiv preprint arXiv:1904.09135.
  * [45] J. Redmon, A. Farhadi, Yolov3: An incremental improvementCite arxiv:1804.02767Comment: Tech Report.
URL https://arxiv.org/abs/1804.02767

bio/luka_bio.png Luka Posilović received his M.Sc. from the University of
Zagreb, Faculty of Electrical Engineering and Computing in 2019. He is
currently working as a young researcher in an Image Processing Group in the
Department of Electronic Systems and Information Processing and working on his
Ph.D. on the same University. His research interests include visual quality
control, deep learning object detection and synthetic image generation.

bio/duje_bio.jpg Duje Medak received his M.Sc. from the University of Zagreb,
Faculty of Electrical Engineering and Computing in 2019. He is currently
pursuing the Ph.D. degree in the same faculty while working as a researcher in
the Image Processing Group in the Department of Electronic Systems and
Information Processing. His research interests include image processing, image
analysis, machine learning and deep learning. His current research interest
includes deep learning object detection methods and their application in the
non-destructive testing (NDT) domain.

bio/marko_bio.jpg Marko Subašić is an associate professor at the Department
for Electronic Systems and Information Processing, Faculty of Electrical
Engineering and Computing, University of Zagreb, and has been working there
from 1999. He received his Ph.D. degree from the Faculty of Electrical
Engineering and Computing, University of Zagreb, in 2007. His field of
research is image processing and analysis and neural networks with a
particular interest in image segmentation, detection techniques, and deep
learning

bio/markoB_bio.png Marko Budimir received his M.Sc. of physics at University
of Zagreb, Faculty of Science in 2000., and his Ph.D. at Ecole Polytechnique
Federale de Lausanne in Switzerland in 2006. He worked at EPFL from 2006. till
2008. From 2008. he is working at the Institute of Nuclear Technology
(INETEC). He coordinated many key projects at INETEC and although he is a key
person in a company of industry sector he is still working close to the field
of science.

bio/sven_bio.jpg Sven Lončarić received his M.Sc. in 1989. He got a Fulbright
scholarship and a Ph.D. in 1994. in the University of Cincinnati, USA in the
field of image processing and analysis. He is a full professor of electrical
engineering and computer science with permanent tenure at Faculty of
Electrical Engineering and Computing, University of Zagreb, Croatia. His areas
of research interest are digital image processing and computer vision. He was
principal investigator on a number of research and development projects in
image processing and analysis including applications in medicine, automotive
industry, and visual quality inspection.