# A Literature Review on
Fetus Brain Motion Correction in MRI

Haoran Zhang1 Yun Wang2

(1Department of Computer Science, Duke University,
2Department of Biomedical Informatics, Emory University)

##  1 Introduction

Observing fetal brain development is crucial for diagnosing conditions such as
autism and other developmental anomalies [1]. Magnetic Resonance Imaging (MRI)
offers significant advantages in this context, being precise, rapid, and
exhibiting no known harmful effects in either the short or long term when
examining brain development [2]. However, the efficacy of MRI is compromised
by challenges such as rapid fetal movements and maternal breathing [3]. These
factors result in relatively low signal-to-noise ratios in the obtained
images. Moreover, unlike adults or neonates, fetuses exhibit unrestrained and
potentially large movements. Being encased within maternal tissues, the fetal
brain occupies a smaller area relative to these surrounding tissues. The above
challenges coupled with the low contrast between the fetal brain tissue and
surrounding matter, renders imaging interpretation particularly difficult [4].

MRI data acquisition occurs in the Fourier domain, known as k-space, where
each data point represents the frequency content of the image. Alterations to
even a single point in k-space can impact the entire image. Motion during MRI
scans introduces errors into k-space, resulting in blurring and/or ghosting
artifacts in the image domain [5].

Active research endeavors are underway to mitigate these motion-related
issues. Imaging techniques such as Echo Planar Imaging (EPI) [6] have shown
promise in motion reduction by employing specific strategies within EPI. Fast
snapshot imaging methods such as Single-Shot Fast Spin Echo (SSFSE) are also
commonly used to acquire thick stacks of 2D slices that can largely remove in-
plane motion [7].

In recent 15 years, the advancement of Slice to Volume Registration (SVR)
algorithms has marked a significant progression towards feasible clinical
validation of fetal brain MRI motion correction [7, 8, 9, 10, 11, 12, 13].
While demonstrated effectiveness, they are still subject to challenges like
large motions and initialization failures[14]. To tackle these challenges,
deep-learning based methods have been actively proposed like Convolutional
Neural Networks (CNNs) [15, 14, 16, 17, 18], Long Short-term Memory Network
(LSTM) [19], Transformers [20, 21], Generative Adversarial Networks (GANs)
[22, 23] and Diffusion Models [24, 25]. To gain a comprehensive understanding
of the various methodologies in this field, we have conducted a literature
review with a specific focus on algorithmic designs. Our review concentrates
on the studies on the fetal brain, intending to not only elucidate the current
state of fetal brain MRI motion correction but also offer insights into
potential future direction in the field.

##  2 Literature Search

The literature review was conducted using search terms such as "fetal brain,"
"motion correction," and "MRI." This search encompassed the period from
December 15, 2023, to January 20, 2024, focusing on the latest developments in
this rapidly evolving field.

In the subsequent sections of this paper, we will methodically examine a
variety of methodologies for correcting motion in fetal brain MRI scans. The
structure of the presentation is intentionally designed to facilitate a clear
and logical understanding, beginning with traditional methodologies and
gradually progressing to sophisticated neural network-based approaches.

These methodologies can be divided into two primary categories: i. Slice-to-
Volume Registration, and ii. Deep Learning Methods. Within each category,
various studies and techniques are examined. Some of these studies may overlap
in terms of their approaches or findings; however, we aim to differentiate
them based on their core attributes, such as network features and structures.
This progression from foundational techniques to cutting-edge advancements
will allow for a comprehensive understanding of the field. Each method will be
briefly introduced to provide a contextual framework, paving the way for more
in-depth discussion in the following sections.

##  3 Slice-to-Volume Registration

In the past 15 years, numerous studies in the field of SVR have tackled the
challenge of fetal motion through retrospective correction [10]. [4] initially
selects a low-resolution MRI stack to establish a global coordinate system,
and then aligns each subsequent stack to this reference [4]. Following the
correction of inter-slice motions, a high-resolution 3D volume is
reconstructed using scattered interpolation. [11] introduced a method for 3D
volume reconstruction, which involves co-aligning multiple 2D stacks and
employing Gaussian weighting for volumetric reconstruction. [13] developed a
two-step iterative SVR and outlier-robust SRR method, facilitating rapid high-
resolution reconstruction in standard space. [9] introduced an automated
motion correction and volumetric reconstruction framework, subsequently
releasing it as an open-source project named NiftyMIC. This framework
initially generates a high-resolution reference through Outlier-robust SRR
with L2 regularization [12], then performs slice-wise alignment between the
High-Resolution Volume and each stack, culminating in axial stack volumetric
reconstruction using Huber L2 regularization.

Figure 1: 3D SVR reconstruction pipeline for fetal brain MRI. This figure is
based on the MRI dataset from St.Thomas’ Hospital, London [10], see the
original paper at https://pubmed.ncbi.nlm.nih.gov/35834425.

The pipeline for 3D SVR fetal brain reconstruction is comprehensively depicted
in Figure 1 [10]. The process initiates with the global alignment of input
image stacks within the designated brain region, establishing a masked area
for focused analysis. This initial phase culminates in the creation of a
preliminary average 3D image, designated as Step 0 in the methodology.

Subsequently, the procedure progresses to Step 1, where each 2D slice
undergoes a process of rigid registration against the current estimate of the
3D image. This step ensures that each slice is accurately aligned in three-
dimensional space, contributing to the overall fidelity of the reconstruction.

Following this, Step 2 involves the super-resolution reconstruction of a new,
enhanced 3D image. This is achieved by integrating all the aligned slices,
thus improving the resolution and clarity of the resulting image.

The methodology adopts an iterative approach, where Steps 1 and 2 are
alternated repeatedly. This iterative process is crucial for achieving
convergence towards a stable and accurate reconstruction of the fetal brain.
Throughout these steps, there is also a quality control mechanism in place.
This mechanism involves the exclusion of any slices that are either misaligned
or exhibit signs of corruption, ensuring the integrity and reliability of the
final reconstructed image [10].

Slice-to-volume registration is defined as the process of aligning a two-
dimensional (2D) image $I$ with a three-dimensional (3D) volume $J$. The
objective is to determine a transformation function $\hat{\Theta}$ that best
aligns the 2D tomographic slice $I$ to the 3D volumetric image $J$. This
alignment is achieved by minimizing an objective function as outlined in
Equation 1:

$\hat{\Theta}=\operatorname*{arg\,min}_{\Theta}\mathcal{M}(I,J;\Theta)+\mathcal{R}(\Theta)$
(1)

In the aforementioned equation, $\mathcal{M}$ signifies the image similarity
term, commonly known as the matching criterion. This term evaluates the
similarity between the 2D image and its corresponding slice in the 3D volume,
predominantly based on intensity information or distinctive structural
features present in $I$ and $J$. Concurrently, $\mathcal{R}$ symbolizes the
regularization term, which applies constraints to guarantee a well-defined
solution. This term is particularly pivotal in governing the geometric
characteristics of the transformation model, especially crucial in scenarios
involving non-rigid registration [10].

The classification of the registration process as either rigid or non-rigid
hinges on the permissible deformations in image $I$ or its corresponding
reformed slice from $J$. In rigid slice-to-volume registration, the
transformation is confined to rigid motions, typically incorporating six
degrees of freedom. Conversely, non-rigid registration allows for more
intricate transformations, encompassing deformations or advanced linear
transformations such as affine transformations [8].

The selection of a regularizer in the function $\mathcal{R}$ is contingent
upon the specific transformation model employed. While simpler models, such as
rigid body transformations, may be estimated even in the absence of a
regularizer, more complex non-rigid models necessitate the inclusion of
$\mathcal{R}$ to ensure the realism of the transformations. In the context of
slice-to-volume registration, this regularizer can impose planarity
constraints (in situations where out-of-plane deformations are restricted) or
can limit the extent of out-of-plane deformations to yield plausible outcomes
[8]. Furthermore, when relevant, incorporating knowledge about the tissue’s
elasticity into the regularizer can enhance the model’s accuracy. The ultimate
objective is to optimize the energy as delineated in Eq. (1), identifying the
most precise $\hat{\Theta}$ that aligns the 2D and 3D images both effectively
and realistically.

While traditional iterative methods for motion correction and reconstruction
have shown effectiveness, they are hindered by two major limitations: the
inability to accurately estimate large motions, and a heavy reliance on
precisely setting initial transformation parameters [14]. Furthermore, the
similarity measures optimized in these intensity-based methods often exhibit a
highly non-convex nature. This aspect greatly increases the risk of the
optimizer being trapped in local maxima, thus restricting the capture range of
these techniques [18].

The capture range of SVR is notably limited, primarily because it depends on
iteratively optimized intensity-based similarity metrics. These metrics serve
merely as proxies for aligning slices to a reference volume, without ensuring
accurate alignment. The presence of a motion-free reference volume is not
always guaranteed. To broaden the capture range, one can employ strategies
like grid search on rotation parameters along with multi-scale registration
[26]. However, these approaches are computationally demanding due to their
reliance on iterative numerical optimization in the testing phase.
Alternatively, age-matched atlases, such as those referenced in [27], can be
used as reference volumes. Atlas-based registration methods, detailed in [26,
28], are also possible, but their computational intensity limits their
applicability in real-time scenarios.

Accurate motion correction remains a significant challenge in the field.
Supervised learning has been explored in 2D/3D registration, with some studies
proposing metric learning approaches to develop customized similarity measures
through supervised learning [29, 30].

##  4 Deep Learning Methods

Deep Learning-Based Motion Correction methods treat the entire MRI scanning
procedure as an input-output system, as depicted in Figure 2. Typically, these
methods operate within the image domain, where the training process involves
using a motion-corrupted image as input and a motion-free image as the target
label. Crucially, since the characteristics of motion artifacts vary depending
on the motion type, most studies narrow their focus to specific types of
motion [31]. During the training phase, these techniques are designed to
recognize patterns between images affected by motion and those free from
motion artifacts. When encountering a new, motion-distorted image, the system
aims to predict and produce a corrected version [32]. Such methods usually
concentrate on particular motion artifact types, predominantly categorized
into rigid and non-rigid motions.

Non-rigid or elastic motion, often due to physiological factors, can be
classified into two main groups: periodic/continuous motions (such as
respiration, cerebrospinal fluid flow, and peristalsis) and abrupt involuntary
actions (like swallowing) [31].

Conversely, rigid motion is caused by involuntary or deliberate movements of
the subject, and is frequently observed across various body parts, especially
in the brain [33]. It is more commonly found in non-compliant subjects,
including children or individuals with degenerative neurological disorders
like Parkinson’s [33]. Rigid motion is characterized by six parameters,
encompassing three translational and three rotational movements. These motion
parameters are inferable directly from the raw data by minimizing motion-
induced image quality metrics or data consistency errors. However, applying
these methods in clinical MRI settings poses challenges due to the non-convex
nature of the parameter estimation process and the extensive computation time
involved [34].

Figure 2:  A flowchart depicting CNN-based motion correction techniques from
the work in [32]. Initially, motion-corrupted images are input into a network,
which then outputs motion-free images. For training the deep learning model,
numerous pairs of motion-corrupted and motion-free images are processed
through the network. Subsequently, in the correction phase, a motion-corrected
image is produced by inputting a motion-corrupted image. This process
ultimately results in the generation of a motion-free image, see the original
paper at https://www.sciencedirect.com/science/article/pii/S2950162823000012.

###  4.1 Convolutional Neural Networks

In recent years, to increase the captured motion range and accelerate fetal
MRI volume reconstruction, neural network-based methods like CNN have been
proposed to predict the motion of fetal brain MRI 2D slices. [18] proposed a
CNN-based regression technique, termed Pose Estimation via Hierarchical
Learning (PEHL). This method is designed to facilitate real-time 2-D/3-D
registration, boasting a broad capture range and high accuracy. While
primarily focusing on X-ray attenuation as observed in CT imaging, this
approach is also adaptable for MRI applications.

[16] expanded upon the 3D pose estimation network, adapting it for use with
arbitrarily oriented objects in slice-to-volume and volume-to-volume
registration tasks. This adaptation involved the use of CNNs trained to
predict the angle-axis representation of 3-D rotations and translations, based
on image features, which is particularly relevant for fetal brain MRI.
Meanwhile, [17] introduced a multi-task learning framework for fetal MRI
stacks. This framework combines the positional data and tissue segmentation
maps of each 2D slice, thereby enhancing the effectiveness of motion
correction. The framework features a multi-output network designed to
simultaneously predict transformation parameters and segmentation results,
which are then processed through a common encoder structure and a shared
representation module to refine both the predictions and segmentation outcomes
[17].

###  4.2 Long Short-Term Memory Networks

[19] firstly proposed a deep predictive motion tracking framework utilizing
Long Short-Term Memory (LSTM) Networks. This approach marks a departure from
the static 3D pose estimation methods commonly found in [16, 18], focusing
instead on dynamic, real-time 3D motion tracking within MRI contexts. In
contrast to traditional motion tracking methodologies, their framework
directly addresses the 3D rigid motion of anatomy in the scanner or world
coordinate system, using time-sequentially acquired slice stacks.

The process begins with the extraction of spatial features from sequences of
input images using Convolutional Neural Networks (CNNs). These features are
then encoded using LSTM, which subsequently estimates objectives for the given
images. This process results in the creation of a context vector, utilized by
LSTM decoders to perform regression against angle-axis representation and
translation offset, therefore enabling the accurate prediction of 3D rigid
body motion. To prevent overfitting to either rotation or translation
parameters, the network incorporates multiple representation heads.
Additionally, the framework employs a multi-step prediction strategy, wherein
the output of a previous decoder is fed as input to the current decoder,
combined with the context vector. The networks were trained and tested on
sequences containing masked slices, which represent slices lost due to
intermittent fast intra-slice motion.

###  4.3 Transformers

The Transformer architecture, renowned for its proficiency in dynamically
highlighting relevant features in input sequences through the self-attention
mechanism, has demonstrated exceptional ability in modeling long-distance
dependencies and capturing global context [35].

Formally, the transformer uses the concept of self-attention [35], which
allows each position in the input sequence to attend to all positions in the
previous layer of the sequence. This is mathematically represented as:

$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V$
(2)

In this equation, $Q$, $K$, and $V$ represent the queries, keys, and values,
respectively. These are all vectors obtained by transforming the input
vectors. $d_{k}$ is the dimension of the key vectors. The scaling factor
$\sqrt{d_{k}}$ is used to prevent the dot products from growing too large in
magnitude, which could lead to instability in the softmax function.

The output of the attention mechanism is then passed through a series of feed-
forward neural networks. Each of these networks is applied to each position
separately and identically. This part of the transformer is represented as:

$\text{FFN}(x)=\max(0,xW_{1}+b_{1})W_{2}+b_{2}$ (3)

where $x$ is the input to the feed-forward network, $W_{1}$ and $W_{2}$ are
weight matrices, and $b_{1}$ and $b_{2}$ are bias vectors. The ReLU activation
function (denoted as $\max(0,\cdot)$) is applied element-wise. This two-layer
feed-forward network is applied to each position’s output from the attention
layer, effectively allowing the model to integrate information from different
positions of the input sequence.

This attention capability is particularly pertinent in the context of SVR for
fetal MRI, where the input comprises multiple stacks of slices. These stacks
can be conceptualized as a sequence of images, allowing for the joint
processing of multi-view information from different orientations to facilitate
the SVR task. [20] introduced the Slice-to-Volume Registration Transformer
(SVoRT), which maps multiple stacks of fetal MR slices into a canonical 3D
space. This mapping serves as an initialization step for SVR and 3D
reconstruction. Building on this, NeSVoR has further enhanced the volumetric
reconstruction process by incorporating implicit neural information [21].

###  4.4 Generative Adversarial Networks

CNNs, known for their robust feature extraction capabilities, have made
substantial contributions to reducing motion artifacts in MRI imaging.
However, a limitation arises in the form of blurred images, a consequence of
their strategy to minimize the Euclidean distance between noisy and ground-
truth images [22]. This approach, while mathematically sound in pixel space,
often falls short in terms of visual perception, leading to less-than-optimal
image quality [23].

To overcome these limitations, Generative Adversarial Networks (GANs) have
been identified as a promising technological advancement. A recent
contribution to this area is the work of [23], who introduced a novel GAN
architecture designed specifically for correcting motion artifacts in fetal
MRI images.

GANs function through a dual-network structure, comprising a Generator and a
Discriminator. These networks are trained sequentially and interactively. The
Discriminator’s primary function is to develop a loss function capable of
distinguishing between authentic and artificial images. Meanwhile, the
Generator, operating as a generative model, aims to produce artificial images
so convincingly real that they deceive the Discriminator.

In the context of [23] work, the Generator is tasked with transforming a
motion-corrupted image into a version free of motion artifacts. This is
achieved by the Generator producing a corrected image, which is then combined
with the original motion-corrupted image to create a "fake" image pair.
Conversely, a "real" image pair is formed by concatenating a ground truth
(motion-free) image with the same original motion-corrupted image. This
process of pairing the images, particularly the integration of the motion-
corrupted image, serves a crucial role. It provides auxiliary information to
the Discriminator, enabling it to more effectively identify and penalize
inaccuracies in the Generator’s output.

###  4.5 Diffusion Model

Diffusion models represent a pioneering category in generative modeling,
showcasing exceptional efficacy in capturing complex data distributions.
Although a recent development in generative learning, these models have proven
beneficial across a variety of applications. Based on differences in
approaches, they can be broadly categorized into two types: the Variational
Perspective [36, 24] and the Score Perspective [37, 38, 39, 25]. We will
examine the models within each category, including DDPMs under the Variational
Perspective and NCSNs and SDEs under the Score Perspective.

####  4.5.1 Variational Perspective

The Denoising Diffusion Probabilistic Model (DDPM) [36] is an innovative
approach in the realm of unconditional generative models. It employs two
distinct Markov chains: a forward chain that progressively distorts data into
noise, and a reverse chain that transforms noise back into data. The forward
chain is typically designed to convert any data distribution into a simple,
predefined prior distribution (such as a standard Gaussian). Conversely, the
reverse chain, powered by deep neural networks, learns to invert the forward
chain’s transformations. Generation of new data points begins by sampling a
random vector from the prior distribution, followed by ancestral sampling
through the reverse Markov chain [40].

Consider a data sample $x_{0}\sim q(x_{0})$. A forward noising process $p$
generates latent variables $x_{1}$ through $x_{T}$ by incrementally adding
Gaussian noise at each time step $t$. This process is mathematically defined
as:

$q(x_{t}|x_{t-1})=\mathcal{N}\left(x_{t};\sqrt{1-\beta_{t}}\cdot
x_{t-1},\beta_{t}\cdot\mathbf{I}\right)$ (4)

Here, $T$ denotes the total number of diffusion steps, and
$\beta_{1},\ldots,\beta_{T}\in[0,1)$ represents the variance schedule across
these steps. $\mathbf{I}$ is the identity matrix, and
$\mathcal{N}(x;\mu,\sigma)$ signifies the normal distribution with mean $\mu$
and covariance $\sigma$.

Building on the aforementioned concepts, the reverse process can be formulated
to approximate a sample from $q(x_{0})$. Starting with
$p(x_{T})=\mathcal{N}(x_{T};0,\mathbf{I})$, the reverse process can be
parameterized as follows:

$p_{\theta}(x_{0:T})=p(x_{T})\prod_{t=1}^{T}p_{\theta}(x_{t-1}|x_{t})$ (5)

####  4.5.2 Score Perspective

Models employing the Score Perspective utilize a maximum likelihood-based
estimation approach, which involves using the score function of the data’s
log-likelihood to estimate parameters in diffusion processes. Two notable
subcategories within this domain are Noise-conditioned Score Networks (NCSNs)
[38] and Stochastic Differential Equations (SDEs) [39]. NCSNs specifically
focus on estimating the derivative of the log density function for perturbed
data distributions at varying noise levels. Conversely, SDEs represent a
broader generalization of these methodologies, encompassing characteristics of
both DDPMs and NCSNs.

####  4.5.3 Application in Motion Correction

Although diffusion models have not yet been directly implemented in fetal MRI
motion correction, several related studies hold potential for such
applications. [24] developed a measurement-conditioned denoising probabilistic
model (MC-DDPM), an extension of DDPM, which demonstrated impressive
capabilities in reconstructing under-sampled MR images. Such work could be
potentially adapted for fetal brain MRI volumetric reconstruction. [25]
employed a score-based generative model that calculates the log probability of
reconstruction for the motion-corrupted MR images. Both motion parameters and
the underlying image are optimized to identify a solution that not only aligns
with the data but also possesses a high prior probability as per the
generative model’s framework.

##  5 Conclusions

The analysis of various methodologies reveals that the issue of motion
correction in MRI is inherently complex and can be approached from multiple
perspectives. While traditional methods such as SVR have proven effective,
deep learning-based approaches have demonstrated superior precision and
robustness in handling various types of motion. They excel at modeling the
intricate non-linear relationships between motion-corrupted and motion-free
images. Notably, generative models like GANs and Diffusion Models are
advancing toward more accurate reconstruction of corrupted images. However,
they often operate in a ’black-box’ manner, raising concerns about their lack
of transparency, an issue that warrants further exploration and resolution.

Emerging deep learning models, like Transformers, Generative Adversarial
Networks, and Diffusion Models, have shown promise in the field of medical
imaging. Despite their initial development for MRI reconstruction enhancement,
there is potential for their application in motion correction. Especially for
diffusion models, which generate samples through a lengthy Markov chain of
diffusion steps, are particularly noteworthy. These innovative deep-learning
models are likely to significantly enhance motion correction techniques in
future research endeavors.

## References

  * [1] Naya Juul-Dam, Jeanne Townsend and Eric Courchesne  “Prenatal, perinatal, and neonatal factors in autism, pervasive developmental disorder-not otherwise specified, and the general population”  In _Pediatrics_ 107.4 American Academy of Pediatrics Elk Grove Village, IL, USA, 2001, pp. e63–e63
  * [2] Penny Gowland  “Safety of fetal MRI scanning”  In _Fetal MRI_ Springer, 2011, pp. 49–54
  * [3] Shuzhou Jiang et al.  “MRI of moving subjects using multislice snapshot images with volume reconstruction (SVR): application to fetal, neonatal, and adult brain studies”  In _IEEE transactions on medical imaging_ 26.7 IEEE, 2007, pp. 967–980
  * [4] Francois Rousseau et al.  “Registration-based approach for reconstruction of high-resolution in utero fetal MR brain images”  In _Academic radiology_ 13.9 Elsevier, 2006, pp. 1072–1081
  * [5] Mark Hedley and Hong Yan  “Motion artifact suppression: a review of post-processing techniques”  In _Magnetic resonance imaging_ 10.4 Elsevier, 1992, pp. 627–635
  * [6] Michael K Stehling, Robert Turner and Peter Mansfield  “Echo-planar imaging: magnetic resonance imaging in a fraction of a second”  In _Science_ 254.5028 American Association for the Advancement of Science, 1991, pp. 43–50
  * [7] Sahar N Saleem  “Fetal MRI: An approach to practice: A review”  In _Journal of advanced research_ 5.5 Elsevier, 2014, pp. 507–523
  * [8] Enzo Ferrante and Nikos Paragios  “Slice-to-volume medical image registration: A survey”  In _Medical image analysis_ 39 Elsevier, 2017, pp. 101–123
  * [9] Daniel Sobotka et al.  “Motion correction and volumetric reconstruction for fetal functional magnetic resonance imaging data”  In _NeuroImage_ 255 Elsevier, 2022, pp. 119213
  * [10] Alena U Uus et al.  “Retrospective motion correction in foetal MRI for clinical applications: existing methods, applications and integration into clinical practice”  In _The British journal of radiology_ 96.1147 The British Institute of Radiology., 2023, pp. 20220071
  * [11] Kio Kim et al.  “Intersection based motion correction of multislice MRI for 3-D in utero fetal brain image formation”  In _IEEE transactions on medical imaging_ 29.1 IEEE, 2009, pp. 146–158
  * [12] Michael Ebner  “Volumetric MRI Reconstruction from 2D Slices in the Presence of Motion”, 2019
  * [13] Michael Ebner et al.  “An automated framework for localization, segmentation and super-resolution reconstruction of fetal brain MRI”  In _NeuroImage_ 206 Elsevier, 2020, pp. 116324
  * [14] Benjamin Hou et al.  “3-D reconstruction in canonical co-ordinate space from arbitrarily oriented 2-D images”  In _IEEE transactions on medical imaging_ 37.8 IEEE, 2018, pp. 1737–1750
  * [15] Benjamin Hou et al.  “Predicting slice-to-volume transformation in presence of arbitrary subject motion”  In _Medical Image Computing and Computer-Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part II 20_ , 2017, pp. 296–304  Springer
  * [16] Seyed Sadegh Mohseni Salehi, Shadab Khan, Deniz Erdogmus and Ali Gholipour  “Real-time deep pose estimation with geodesic loss for image-to-template rigid registration”  In _IEEE transactions on medical imaging_ 38.2 IEEE, 2018, pp. 470–481
  * [17] Yuchen Pei et al.  “Anatomy-guided convolutional neural network for motion correction in fetal brain MRI”  In _Machine Learning in Medical Imaging: 11th International Workshop, MLMI 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4, 2020, Proceedings 11_ , 2020, pp. 384–393  Springer
  * [18] Shun Miao, Z Jane Wang and Rui Liao  “A CNN regression approach for real-time 2D/3D registration”  In _IEEE transactions on medical imaging_ 35.5 IEEE, 2016, pp. 1352–1363
  * [19] Ayush Singh, Seyed Sadegh Mohseni Salehi and Ali Gholipour  “Deep predictive motion tracking in magnetic resonance imaging: application to fetal imaging”  In _IEEE transactions on medical imaging_ 39.11 IEEE, 2020, pp. 3523–3534
  * [20] Junshen Xu et al.  “SVoRT: iterative transformer for slice-to-volume registration in fetal brain MRI”  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , 2022, pp. 3–13  Springer
  * [21] Junshen Xu et al.  “NeSVoR: Implicit Neural Representation for Slice-to-Volume Reconstruction in MRI”  In _IEEE Transactions on Medical Imaging_ IEEE, 2023
  * [22] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou and Alexei A Efros  “Image-to-image translation with conditional adversarial networks”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1125–1134
  * [23] Adam Lim et al.  “Motion artifact correction in fetal MRI based on a Generative Adversarial network method”  In _Biomedical Signal Processing and Control_ 81 Elsevier, 2023, pp. 104484
  * [24] Yutong Xie and Quanzheng Li  “Measurement-conditioned denoising diffusion probabilistic model for under-sampled medical image reconstruction”  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , 2022, pp. 655–664  Springer
  * [25] Brett Levac, Ajil Jalal and Jonathan I Tamir  “Accelerated motion correction for MRI using score-based generative models”  In _2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI)_ , 2023, pp. 1–5  IEEE
  * [26] Vahid Taimouri et al.  “A template-to-slice block matching approach for automatic localization of brain in fetal MRI”  In _2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI)_ , 2015, pp. 144–147  IEEE
  * [27] Ali Gholipour et al.  “A normative spatiotemporal MRI atlas of the fetal brain for automatic segmentation and analysis of early brain growth”  In _Scientific reports_ 7.1 Nature Publishing Group UK London, 2017, pp. 476
  * [28] Sébastien Tourbier et al.  “Automated template-based brain localization and extraction for fetal brain MRI reconstruction”  In _NeuroImage_ 155 Elsevier, 2017, pp. 460–472
  * [29] Michael M Bronstein, Alexander M Bronstein, Fabrice Michel and Nikos Paragios  “Data fusion through cross-modality metric learning using similarity-sensitive hashing”  In _2010 IEEE computer society conference on computer vision and pattern recognition_ , 2010, pp. 3594–3601  IEEE
  * [30] Fabrice Michel, Michael Bronstein, Alex Bronstein and Nikos Paragios  “Boosted metric learning for 3D multi-modal deformable registration”  In _2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro_ , 2011, pp. 1209–1214  IEEE
  * [31] Seul Lee, Soozy Jung, Kyu-Jin Jung and Dong-Hyun Kim  “Deep learning in MR motion correction: a brief review and a new motion simulation tool (view2Dmotion)”  In _Investigative Magnetic Resonance Imaging_ 24.4, 2020, pp. 196–206
  * [32] Yuchou Chang et al.  “Deep learning-based rigid motion correction for magnetic resonance imaging: A survey”  In _Meta-Radiology_ Elsevier, 2023, pp. 100001
  * [33] Frank Godenschweger et al.  “Motion correction in MRI of the brain”  In _Physics in medicine & biology_ 61.5 IOP Publishing, 2016, pp. R32
  * [34] Kamlesh Pawar, Zhaolin Chen, N Jon Shah and Gary F Egan  “MoCoNet: Motion correction in 3D MPRAGE images using a convolutional neural network approach”  In _arXiv preprint arXiv:1807.10831_ , 2018
  * [35] Ashish Vaswani et al.  “Attention is all you need”  In _Advances in neural information processing systems_ 30, 2017
  * [36] Jonathan Ho, Ajay Jain and Pieter Abbeel  “Denoising diffusion probabilistic models”  In _Advances in neural information processing systems_ 33, 2020, pp. 6840–6851
  * [37] Amirhossein Kazerouni et al.  “Diffusion models for medical image analysis: A comprehensive survey”  In _arXiv preprint arXiv:2211.07804_ , 2022
  * [38] Yang Song and Stefano Ermon  “Generative modeling by estimating gradients of the data distribution”  In _Advances in neural information processing systems_ 32, 2019
  * [39] Yang Song et al.  “Score-based generative modeling through stochastic differential equations”  In _arXiv preprint arXiv:2011.13456_ , 2020
  * [40] Daphne Koller and Nir Friedman  “Probabilistic graphical models: principles and techniques”  MIT press, 2009


11institutetext:  University of Reading, UK
11email<EMAIL_ADDRESS>22institutetext: KTH Royal
Institute of Technology, Sweden
22email<EMAIL_ADDRESS>33institutetext: University of Catania, Italy
44institutetext: University of Cambridge, UK
44email: giuseppe.nicosia<EMAIL_ADDRESS>
# Adversarial Robustness in Deep Learning: Attacks on Fragile Neurons

✉ Chandresh Pravin 11 0000-0003-1530-0121 Ivan Martino 22 0000-0001-6306-6777
Giuseppe Nicosia 3344 0000-0002-0650-3157 Varun Ojha 11 0000-0002-9256-1192

###### Abstract

We identify fragile and robust neurons of deep learning architectures using
nodal dropouts of the first convolutional layer. Using an adversarial
targeting algorithm, we correlate these neurons with the distribution of
adversarial attacks on the network. Adversarial robustness of neural networks
has gained significant attention in recent times and highlights an intrinsic
weaknesses of deep learning networks against carefully constructed distortion
applied to input images. In this paper, we evaluate the robustness of state-
of-the-art image classification models trained on the MNIST and CIFAR10
datasets against the fast gradient sign method attack, a simple yet effective
method of deceiving neural networks. Our method identifies the specific
neurons of a network that are most affected by the adversarial attack being
applied. We, therefore, propose to make fragile neurons more robust against
these attacks by compressing features within robust neurons and amplifying the
fragile neurons proportionally.

###### Keywords:

Deep Learning Fragile Neurons Data Perturbation Adversarial Targeting
Robustness Analysis Adversarial Robustness

##  1 Introduction

Deep neural networks (DNNs) have been widely adapted to various tasks and
domains, achieving significant performances in both the real world and in
numerous research environments [11]. Previously considered state-of-the-art
DNNs have been subjected to a plethora of tests and experiments in an attempt
to better understand the underlying mechanics of how and what exactly these
learning models actually learn [13]. In doing so, we now better recognise the
strengths and more importantly the weaknesses of DNNs and have subsequently
developed better networks building on from previous architectures [15].

Adversarial attacks are one the most used methods to evaluate the robustness
of DNNs. Such methods introduce a small carefully crafted distortion to the
input of the network in an attempt to deceive the network into misclassifying
the input with a high level of confidence [9, 18]. The small distortions to
the input, termed adversarial perturbations, are hardly perceptible to humans,
even when the perturbation is amplified by several orders of magnitude [18].
This ability to fool DNNs with hardly perceptible changes in the input
highlights an intrinsic difference between artificial intelligence and true
intelligence.

There are many ways in which an adversarial perturbation can be crafted,
utilising various tools and assumptions on the target model and dataset.
Existing adversarial attacks, and methods for designing such distortions, can
be broadly categorised into white-box and black-box attacks. The distinction
between the two different types of attacks being the information that the
adversary has on the model and its parameters. With the white-box attacks, the
adversary is assumed to have complete access to the target model in question,
including model parameters and architecture [4]. Conversely, the black-box
attack is a type of perturbation designed by an adversary with no information
to the model’s parameters or architecture [14]. In this paper, we focus our
efforts at evaluating the robustness [17] of ResNet-18, ResNet-50 and
ResNet-101 networks against a simple yet effective white-box adversarial
attack, the fast gradient sign method (FGSM) attack [9]. We apply the FGSM
perturbations on the MNIST and CIFAR10 datasets for the mentioned models and
present a correlative relationship between the distribution of neurons with
high influence and targeting by an adversary. We also evaluate a method in
minimising the effects of such distortions.

With the numerous adversarial attacks formed against DNNs, there have been
equally as many defences proposed in literature [14]. The ability of a defence
model to remain unbeaten by an ever-growing selection of adversarial attacks
has proven to be difficult [14, 20]. Adversarial defences, much like
adversarial attacks, can be divided into different categories: (i) defences
focusing on gradient masking/obfuscation, whereby the network weight gradients
used by adversaries to form attacks are disguised; (ii) robust optimization
[19], where the network structure/parameters are altered to increase
adversarial defences; and (iii) adversarial example detection, where the goal
is to detect an adversarial input and process this entity differently to
ordinary inputs [15].

The goal of all adversaries is to deceive the network into predicting,
classifying, or recognising an input as a different class to its true self.
When the adversary has knowledge of the information held by the network, as is
the case for white-box attacks, it utilises this to craft a perturbation that
will exploit weaknesses within the network’s representations of the data [14].
In this paper, we propose viewing an adversarial attack as an exploitative
method that targets specific neurons within a given layer. We also draw a
relationship between the adversaries’ target neurons and neurons that show to
have higher influence on the model’s unperturbed performance.

We assume that, for a given layer, information about the input learned by the
layer through back propagation is distributed unevenly amongst individual
neurons. We propose using nodal dropout to find redundant nodes within a given
layer of a network [12]. Thus, also finding fragile neurons that carry more
information about the input [7]. We identify null neurons that once removed do
not significantly affect the overall model performance and thus considered to
carry less information about the dataset. We examine how the FGSM attack
affects different models (ResNet-18, ResNet-50 and ResNet-101) at different
stages (epochs) in learning, whilst also comparing how increasing the network
architecture affects the effectiveness of the formed attack. Therefore, we
propose to make fragile neurons more robust against these attacks by
compressing robust neurons and amplifying the fragile neurons proportionally.

Furthermore, the FGSM attack utilises a given network’s learned
representations in the form of its layer weights to calculate an effective
adversarial example. The adversarial examples can be used as a method of
evaluating the robustness [16] of the model’s composite representations. We
aim to identify the fragile and robust neurons within specific parts of the
network, and post-process them separately to investigate how they affect the
overall model’s robustness against an adversarial attack.

##  2 Related Work

Robustness analysis evaluates the defence of DNNs against malicious distortion
of its input [1, 9, 20]. There are different types of attacks available for a
potential adversary, each with their own strengths and limitations. Szegdey et
al. [18] initially proposed adversarial examples for DNNs using the Limited-
memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) algorithm, an expensive
linear search method for adversarial examples. Thereafter, the FGSM attack
proposed by Goodfellow et al. [9] has become one of the benchmarks for
adversarial attacks due to its computational process being less resource
intensive when compared to other attacks. The FGSM attack performs a pixel-
wise one step gradient update along the gradient sign direction of increasing
loss. There are several other attack methods available in the literature.
However, in this study, we focus specifically on the FGSM adversarial attack
due to its one-step gradient calculation and effective performance against
state-of-the-art DNN models.

In terms of defences against adversarial attacks, there are an equal number of
approaches proposed in literature. For every newly developed adversarial
attack, soon there have been suitable defences proposed by researchers [20].
One method of defending a DNN model is by masking the network’s parameters,
therefore making it more difficult for an adversary to exploit the network’s
learned information to generate adversarial examples. However, this method has
shown to be ineffective against many types of attacks and there exist
techniques to circumvent such defensive measures [15]. Some studies show that
adversarial examples are drawn from a different distribution to the regular
dataset [10]. Therefore, one method to defend against the effects of such
adversarial examples is to identify them and deal with the perturbed inputs to
the model separately [15]. These methods are also subjected to exploitation by
techniques that can bypass the adversarial examples detection, making such
defence methods weaker to some types of attacks [3].

In this paper we try to find a relationship between highly influential neurons
and the likelihood of being targeted by an adversary, and accordingly, propose
a method of regularising the specific neurons during post-training. As we, the
observer, propagate through the network we notice that the deconstructed,
abstract characteristics of the data begin to take a shape of salient
features, which are then assigned semantic meaning in the form of target
labels [21]. Literature on leveraging the information content of a DNN has
been used for various applications, we direct the reader to Golatkar et al.
[8] and their method of selective forgetting, in which they propose a
framework for erasing the information about a particular subset of data from
the model’s learned weights. We take inspiration from this framework and
propose that adversarial robustness is hinged on the distribution of
influential and uninfluential neurons, referred to as sets $S$ and
$S^{\prime}$ respectively within the context of this study.

We are motivated by the works of Li and Chen [12] along with related
literature in reducing network complexity by using techniques such as nodal
pruning. We leverage the idea that there exist neurons within a network that
can be classified as redundant, or uninfluential to the overall model
performance. Removing redundant neurons in some cases also shows to improve
robustness against attacks [5]. Conversely, we also consider the works of [7]
that prove the existence of multi-model neurons within networks; multi-modal
neurons being representations that hold a higher degree of influence in the
network’s understanding of the data. We investigate the correlation between
representations that show a higher influence and the highest average
concentration of an adversarial attack to these features. In consequence, we
draw attention to the nature of adversarial attacks and how such perturbations
target the model’s learned knowledge specifically.

##  3 Adversarial Attack and Defense Formulations

We consider an image classifier model $f_{\theta}$ with $L$ layers, and
trainable parameters $\theta$ that accepts an input image $x$ and its
associated true class label $y$. The model returns $\hat{y}$ as its prediction
for input $x$. The goal of the model is to reduce loss function
$\mathcal{L}(f_{\theta},x,y)$. The image $x^{\prime}=x+\delta_{\epsilon}$ is
an adversarial example produced by a distortion $\delta_{\epsilon}$ added to
image $x$, where $\epsilon$ is the perturbation magnitude.

Our objective is to minimise the difference in predictions values $\hat{y}$
obtained for unperturbed input $x$ and perturbed input $x^{\prime}$. We
examine the model’s learnable parameters $\theta_{L}\in\theta$ of layer $L$ at
various stages of the model’s training. It should be noted that while
assessing the significance of the neurons, we remove one-neuron at a time from
$\theta_{L}$. We, therefore, identified two sets of neurons indices, $S$ and
$S^{\prime}$ respectively representing (i) neuron indices within the layer $L$
showing a higher influence on the overall model performance, and (ii) neurons
indices with lower overall influence on model performance. In our work, we are
concerned with removing one-neuron at a time, removing multiple neurons from
the model $f_{\theta}$ would warrant an alternative method. We also assessed
neurons of the first layer of the network because of its high importance and
influence on features learned by subsequent layer in a network [5].

###  3.1 Attack Formulation

We formulated attack in this work using FGSM method. This method leverages a
network’s learned representations in the form of layer weights $\theta_{L}$ to
construct an efficient and effective adversarial perturbation $x^{\prime}$.
The FGSM attack is a perturbation for an input $x$ computed as:

$\delta_{\epsilon}=\epsilon\textit{ sign}(\nabla_{x}\mathcal{L}(x,y,\theta)),$
(1)

where $\nabla_{x}$ is the required gradients calculated using backpropagation.
The adversarial example therefore is $x^{\prime}=x+\delta_{\epsilon}$ [1, 9].

We find that for a 100 epoch pre-trained ResNet-50 model on the CIFAR10
dataset, a baseline model accuracy of $75.87\%$ on unperturbed input $x$ is
found. The same model applied to the CIFAR10 dataset with an FGSM attack,
using a perturbation magnitude of $\epsilon=0.01$, results in an accuracy of
$58.88\%$. If we consider the same ResNet-50 architecture trained equally for
100 epochs, with the input dimensions adjusted to comply with the MNIST
dataset, the baseline model accuracy on unperturbed MNIST dataset is
$99.42\%$. While the model accuracy is found to be $79.4\%$ when perturbed
with an $\epsilon=0.34$ attack. These are examples of the FGSM attack
performance against CIFAR10 and MNIST datasets on the ResNet-50 DNN model.

If we consider a metric to assess the complexity of a given dataset, such as
the cumulative spectral gradient (CSG) method [2], we notice that the CSG
complexity measure for the for the CIFAR10 dataset is $1.00$ and MNIST dataset
is $0.11$. As we may expect, the FGSM attack is more effective on more complex
data (e.g., CIFAR10) compared to less complex data (e.g., MNIST). This can be
realised from the perturbation magnitude $\epsilon$ required for the model
performance to decrease proportionally. For example, to decrease performance
by approximately $20\%$, a lower value of $\epsilon$ (small perturbation) is
required for CIFAR10 and a higher value of $\epsilon$ (large perturbation) is
required for MNIST.

###  3.2 Defence Formulation

To better understand how to form a suitable defence against an adversarial
attack, we may consider how an adversary can form an effective attack. With
the FGSM attack, a single step in the parameter space is taken in the
direction of increasing loss. The perturbation is calculated using the
network’s weights to perturb the input data features in the direction of an
incorrect class. Then it is natural to consider that this informed way of
creating adversarial perturbations may, even with relatively low magnitudes,
affect the neurons that are more influential to the model’s performance (e.g.,
set of highly influencing neurons $S$).

We aim to show this effect of adversarial perturbations experimentally by
comparing the output of the layer-wise convolution for original input $x$ and
perturbed input $x^{\prime}$ computed using pre-trained parameters $\theta$.
We expect the original model prediction $f(x,y,\theta)$ and the model
prediction on perturbed input $f(x^{\prime},y,\theta)$ to be not equal. In our
defence formulation, we aim to modify the model’s layer parameters
$\theta_{L}$ as $\theta^{\prime}_{L}$ such that a potential adversary is
forced to distribute the attack strength throughout the layer. We propose that
this will make the model’s layer $\theta^{\prime}_{L}$ more robust against an
adversarial attack.

###  3.3 Fragile and Null Kernels Identification

We identified fragile neurons (kernels) $S$ and null neurons (kernels)
$S^{\prime}$ by dropping the kernels out systematically one-by-one and
measuring the variance in model performance. Fig. 1 show model’s performance
for each kernel along the x-axis being dropped. The indices of fragile kernels
$S$ are indicated with blue circled symbols and are below the mean performance
line indicated in red, which is computed over each kernel’s effect on the
model’s accuracy. The dropping of these fragile kernels has a higher influence
on the model’s performance when compared to the dropping of the null kernels
indicted with black star symbol shown above mean performance line.

Figure 1: Evaluated ResNet-50 model trained for 10 epochs. Fragile kernels $S$
shown in blue below mean performance line in red and null kernels $S^{\prime}$
are shown in black star above mean line in red.

##  4 Adversarial Targeting Algorithm

We assuming that the parameters $\theta_{L,S^{\prime}}$ of null kernels
$S^{\prime}$ in layer $L$, carry within them some noise that render the
overall influence of these kernels on the model performance to be lower than
the fragile kernel $S$, we propose to filter parameters
$\theta_{L,S^{\prime}}$ to remove noise. We assume the distribution of the
noise in matrix $K_{L,\bar{S}}$ to be Gaussian noise. For this, we can use the
works of Gavish and Donoho [6] to recover a lower rank matrix from noisy data
and retaining only the most important features. The filtering of
$\theta_{L,S^{\prime}}$ produces the modified parameters
$\theta^{\prime}_{L,S^{\prime}}$. The filtered parameters relating to null
kernels $\theta^{\prime}_{L,S^{\prime}}$, are said to be more robust if the
probability of predicting the true class using modified model parameters
$\theta^{\prime}$ is higher than $\theta$ as per:

$P(\hat{y}=y|x^{\prime},\theta^{\prime})>P(\hat{y}=y|x^{\prime},\theta).$ (2)

We compose a matrix $K_{L,S^{\prime}}$ by stacking flattened null kernel
parameters $\theta^{\prime}_{L,S^{\prime}}$ and compress $K_{L,S^{\prime}}$ to
remove noise or redundant information, thus increasing the influence of these
null kernels $S^{\prime}$ on the model’s overall performance. While filtering
the null kernels $S^{\prime}$, we proportionally amplify fragile kernel $S$.
This is to maintain relative magnitude of the local features within the
network and propagate the essential representations to deeper layers of the
network better.

###  4.1 Filtering of Null Kernels $S^{\prime}$

We decompose the null kernel’s matrix $K_{L,S^{\prime}}$ using singular value
decomposition (SVD) and reduce the complexity of the representations by
clamping all values below a filtering threshold $\tau$. We apply this method
only to the first convolutional layer because of its susceptibility to any
distortions having a higher influence on the network’s performance [5]. We use
SVD to decompose our null kernel matrix $K_{L,\bar{S}}$ into its respective
eigenvalues $\Sigma$ and eigenvectors matrices $U$ and $V$ as:

$K_{L,S^{\prime}}=U\Sigma V^{T}.$ (3)

We then compute a truncated matrix of singular values $\widetilde{\Sigma}$ by
clamping all singular values to be at most equal to threshold value $\tau$ as
per:

$\widetilde{\sigma_{i}}=\arg\min(\sigma,\tau),$ (4)

where $\sigma$ is the diagonal of $\Sigma$ and $\widetilde{\sigma_{i}}$ is the
row upto which the matrix $\sigma$ is truncated. The thresholding value $\tau$
for $m$-by$n$ matrix is given as:

$\tau=\lambda(\beta)\cdot\sqrt{n}\varepsilon,$ (5)

where $\beta=m/n$, $\varepsilon$ is the noise level within the matrix, and the
term $\lambda(\beta)$ is expressed as [6]:

$\lambda(\beta)=\sqrt{2(\beta+1)+\frac{c_{1}\beta}{(\beta+1)+\sqrt{\beta^{2}+c_{2}\beta+1}}},$
(6)

where constants $c_{1}$ and $c_{2}$ respectively are 8 and 14.

We then find the noise level value $\varepsilon$ in (5) experimentally through
a systematic search method using a sample set of the parameters. As the final
filtering step, we reconstruct the filtered weight matrix
$\widetilde{K}_{L,\bar{S}}$ by using the clamped singular values and
corresponding eigenvectors as:

$\widetilde{K}_{L,\bar{S}}=U\widetilde{\Sigma}V^{T}.$ (7)

###  4.2 Amplification of Fragile Kernels $S$

The amplification of fragile kernels parameters matrix $K_{L,S}$ by a scaling
factor of $\alpha$ computed using (3) and (7) as per:

$\widetilde{K}_{L,S}=\alpha K_{L,S},$ (8)

where scaling factor of $\alpha$ is

$\alpha=1+||K_{L,\bar{S}}-\widetilde{K}_{L,\bar{S}}||_{2}.$ (9)

The aim of this process is to amplify the features within fragile kernels $S$,
such that a greater magnitude of adversarial perturbation is required to vary
such kernels.

###  4.3 Adversarial Targeting of Fragile and Null Kernels

We assess the robustness of the fragile kernels $S$ and null kernels
$S^{\prime}$ by our robustness targeting algorithm shown in Algorithm 1. The
FSGM attack for varied range of perturbations $\epsilon$ is used to compute
the evaluated first convolutional layer’s outputs $\hat{y}_{x}$ and
$\hat{y}_{x^{\prime}}$. The mean difference between each kernel in the output
of $\hat{y}_{x}$ and $\hat{y}_{x^{\prime}}$ are calculated and compared to see
which is highest, indicating a greater average concentration of the attack.

Algorithm 1 Adversarial targeting

1:Initialise $f()\rightarrow f_{L}()$ $\triangleright$ $f_{L}()$ is the $L$-th
layer of full network $f()$

2:Compute indices of fragile kernels $S$ and null kernels $S^{\prime}$ as per
Sec 3.3

3:$S_{attack}=\\{\\}$ $\triangleright$ an empty list to store examples that
attacks $S$

4:for perturbation $\epsilon\in\mathbb{R}$ do $\triangleright$ where
$\epsilon$ is perturbation magnitude

5: attack = FGSM($f_{L},\epsilon)$

6: $S_{\text{count}}=0$

7: for ($x,y$) in ($X_{test},Y_{test}$) do

8: $x^{\prime}=\textit{attack}(x,y)$ $\triangleright$ create an adversarial
example $x^{\prime}$ for input $x$ and level $y$

9: $\hat{y}_{x}=f_{L}(x)$ $\triangleright$ output of $L$-th layer on
unperturbed input $x$

10: $\hat{y}_{x^{\prime}}=f_{L}(x^{\prime})$ $\triangleright$ output of $L$-th
layer on perturbed input $x^{\prime}$

11: $\textbf{d}=||\hat{y}_{x}-\hat{y}_{x^{\prime}}||_{2}$ $\triangleright$
Euclidean distance $\textbf{d}=(d_{1},\ldots,d_{k})$ between $\hat{y}_{x}$ and
$\hat{y}_{x^{\prime}}$

12: $S_{f}=(\sum_{j}^{|S|}d_{j,S})/|S|$ $\triangleright$ Average of distances
$d_{j,S}$ of all $S$ select from d

13: $S_{n}=(\sum_{j}^{|S^{\prime}|}d_{j,S^{\prime}})/|S^{\prime}|$
$\triangleright$ Average of distances $d_{j,S^{\prime}}$ of all $S^{\prime}$
select from d

14: if $S_{f}>S_{n}$ then

15: $S_{\text{count}}=S_{\text{count}}+1$ $\triangleright$ increase counter of
attacks for fragile kernels

16: end if

17: end for

18: $S_{attack}\leftarrow S_{\text{count}}$ $\triangleright$ add
$S_{\text{count}}$ to the list $S_{attack}$

19:end for

##  5 Results and Discussion

In first series of experiments, we use the two sets $S$ and $S^{\prime}$
obtained as per Fig. 1 on the ResNet-50 model and apply them to Algorithm 1
using the CIFAR10 dataset, resulting in Fig. 2 and the MNIST dataset,
resulting in Fig. 3:

For Fig. 2, we measure the robustness of ResNet-50 models and compare the
percentage of examples attacking fragile kernels $S$ and the model accuracy
against FGSM attack. In Fig. 2 (Left), we notice that as the number of
training epochs increases, the model’s accuracy also increases for both the
unperturbed ($\epsilon=0$) and perturbed ($\epsilon>0$) examples. In Fig. 2
(Right), using the results from the adversarial targeting Algorithm 1, we also
notice that the percentage of examples attacking fragile kernels $S$ is higher
for highly perturbed examples. However, for smaller perturbation magnitudes,
100 epoch model is more robust. This suggests that as the model becomes more
robust (from epoch 10 to 100), the percentage of examples attacking fragile
kernels $S$ and null kernels $S^{\prime}$ tends to distribute equally.

Figure 2: Left: ResNet-50 model trained on the CIFAR10 dataset for epochs 10,
50 and 100 against the FGSM attack, with $\epsilon$ increasing linearly,
marked by dots. Right: ResNet-50 model trained on the CIFAR10 dataset for
epochs 10, 50 and 100 against the FGSM attack, with attack magnitude
increasing logarithmically, marked by star symbols. Epoch 10, 50, 100
respectively indicated in colors grey, green, violet.

After applying our framework proposed in sections 4.1 and 4.2 using
$\varepsilon$ value of $0.015$ to the first convolutional layer $\theta_{L}$,
resulting in filtered layer parameters $\theta^{\prime}_{L}$, we observe the
difference in attack distribution between original model and modified model
using Algorithm 1.

We apply the parameter filtering framework to a ResNet-50 model trained on the
MNIST for 10 epochs. The results of which is shown in Fig. 3. In this
experiment, although the number of fragile kernels $S$ are $37\%$ of the total
kernels within the layer, these kernels show a larger average distance between
the outputs of the original layer $\theta_{L}$ and modified layer
$\theta^{\prime}_{L}$ for almost $89\%$ of the tested input examples on
original model. Furthermore, as the attack strength is increased by increasing
$\epsilon$, the average magnitude of the attack on kernels $S$ also increased.
However, our method of filtering parameters $\theta^{\prime}_{L}$ kept the
percentage of tested examples attacking fragile kernels $S$ lower than the
original model.

Figure 3: Concentration of the adversarial attack on fragile kernels $S$ for
both the original model with parameters $\theta_{L}$ and the modified model,
with $\theta^{\prime}_{L}$ in a ResNet-50 model trained on the MNIST dataset
for 10 epochs, using the methods proposed in Sections. 4.1 and 4.2.

We observe from Fig. 4 how the influence of kernels in the first convolutional
layer varies during the training process while we systematically drop and
assess the kernels. In Fig. 4, red circles are the kernels that carry a higher
influence through all stages of model training. We notice that as we change
the model from ResNet-18 to ResNet-50 and ResNet-101, the number of
influential fragile kernels increases on the CIFAR10 dataset. This is as we
may expect, model architectures with greater complexities are able to learn
the important features from the dataset faster than shallower model
architectures. We notice from Fig. 4, that the average model performance of
the kernels in $\theta_{L}$ increases to a limit for models trained on the
CIFAR10 dataset and shows to increase and then decrease for the models trained
on the MNIST dataset. This characteristic invites a separate set of
experiments to better understand how model overfitting affects nodal dropouts.

Figure 4: Variance of model performance to individual kernels being dropped
out within the first convolutional layer. Red circles indicate fragile kernels
that remain fragile throughout the all training epochs, whereas red crosses
indicate kernels that are observed as fragile for the specific training epoch
length.

##  6 Conclusion

In this study we show how an FGSM attack targets specific neurons within the
first convolutional layer of ResNet-18, ResNet-50 and ResNet-101 models
trained on both the CIFAR10 and NNIST datasets. To prove this property, we
first identify fragile kernels $S$ and null kernels $S^{\prime}$ sets within
the evaluated layer using an iterative dropout method and measuring the
variance in model performance. We use the kernel indices of $S$ and
$S^{\prime}$ to evaluate the highest average distance between the outputs of
the layer using the original input $x$ and perturbed example $x^{\prime}$. In
doing so, we find that for a ResNet-50 model trained on the CIFAR10 dataset
for 50 epochs, the number of fragile kernels $S$ account to $37\%$ of the
total number of kernels in the layer yet show to have a higher average
difference for approximately $89\%$ of the examples evaluated.

We also show how the robustness against the FGSM attack, and the targeting of
fragile kernels $S$ varies as the model is trained, thus showing a correlation
between a model becoming more robust and the targeting of fragile kernels.
Furthermore, we propose a layer parameter filtering algorithm that improves
robustness in a model by removing information from null kernels $S^{\prime}$
and amplifying the information in $S$. This simple method, despite only being
applied to the first convolutional layer, improves the robustness of a model
with less training. It should be noted that, although our study focuses on the
first convolutional layer only due to the layer being highly influence over
the model’s performance, other layers can also be evaluated using this
proposed framework.

## References

  * [1] Akhtar, N., Mian, A.: Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access 6, 14410–14430 (2018)
  * [2] Branchaud-Charron, F., Achkar, A., Jodoin, P.M.: Spectral metric for dataset complexity assessment. In: IEEE CVPR (2019)
  * [3] Carlini, N., Wagner, D.: Adversarial examples are not easily detected: Bypassing ten detection methods. In: Proc 10th ACM Workshop Artif Intell Secur. pp. 3–14 (2017)
  * [4] Carlini, N., Wagner, D.: MagNet and “efficient defenses against adversarial attacks” are not robust to adversarial examples. arXiv:1711.08478 (2017)
  * [5] Cheney, N., Schrimpf, M., Kreiman, G.: On the robustness of convolutional neural networks to internal architecture and weight perturbations. arXiv:1703.08245 (2017)
  * [6] Gavish, M., Donoho, D.L.: The optimal hard threshold for singular values is $4/\sqrt{3}$. IEEE Trans Inf Theory 60(8), 5040–5053 (2014)
  * [7] Goh, G., et al.: Multimodal neurons in artificial neural networks. Distill 6(3) (2021)
  * [8] Golatkar, A., Achille, A., Soatto, S.: Eternal sunshine of the spotless net: Selective forgetting in deep networks. In: IEEE CVPR. pp. 9304–9312 (2020)
  * [9] Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: ICLR (2015)
  * [10] Grosse, K., Manoharan, P., Papernot, N., Backes, M., McDaniel, P.: On the (statistical) detection of adversarial examples. arXiv:1702.06280 (2017)
  * [11] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
  * [12] Li, B., Chen, C.: First-order sensitivity analysis for hidden neuron selection in layer-wise training of networks. Neural Process Lett 48(2), 1105–1121 (2018)
  * [13] Papernot, N., et al.: The limitations of deep learning in adversarial settings. In: IEEE Eur Symp Secur Priv. pp. 372–387 (2016)
  * [14] Ren, K., Zheng, T., Qin, Z., Liu, X.: Adversarial attacks and defenses in deep learning. Eng 6(3), 346–360 (2020)
  * [15] Silva, S.H., Najafirad, P.: Opportunities and challenges in deep learning adversarial robustness: A survey. arXiv:2007.00753 (2020)
  * [16] Stracquadanio, G., Ferla, A.L., Felice, M.D., Nicosia, G.: Design of robust space trajectories. In: Bramer, M., et al (eds.) Proc of AI-2011, the 31st SGAI International Conference on Innovative Techniques and Applications of Artificial Intelligence, Cambridge, UK, December. pp. 341–354. Springer (2011)
  * [17] Stracquadanio, G., Nicosia, G.: Computational energy-based redesign of robust proteins. Comput. Chem. Eng. 35(3), 464–473 (2011)
  * [18] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. In: ICLR (2014)
  * [19] Umeton, R., Stracquadanio, G., Sorathiya, A., Liò, P., Papini, A., Nicosia, G.: Design of robust metabolic pathways. In: Stok, L., et al (eds.) Proceedings of the 48th Design Automation Conference, DAC 2011, San Diego, California, USA, June 5-10. pp. 747–752. ACM (2011)
  * [20] Yuan, X., He, P., Zhu, Q., Li, X.: Adversarial examples: Attacks and defenses for deep learning. IEEE Trans Neural Netw Learn Syst 30(9), 2805–2824 (2019)
  * [21] Zhou, B., Bau, D., Oliva, A., Torralba, A.: Interpreting deep visual representations via network dissection. IEEE Trans Pattern Anal Mach Intell (2018)