1]organization=the BRain and Artificial INtelligence Lab (BRAIN LAB), School
of Automation, Northwestern Polytechnical University, city=Xi’an,
postcode=710072, country=China

[1]

2]organization=Remote Sensing Technology Institute (IMF), German Aerospace
Center (DLR), city=Wessling, postcode=82234, country=Germany
3]organization=University POLITEHNICA of Bucharest (UPB), city=Bucharest,
postcode=060042, country=Romania

[1]Corresponding author

# Physically Explainable CNN for SAR Image Classification

Zhongling Huang<EMAIL_ADDRESS>[  Xiwen Yao  Ying Liu  Corneliu
Octavian Dumitru  Mihai Datcu  Junwei Han  [  [

###### Abstract

Integrating the special electromagnetic characteristics of Synthetic Aperture
Radar (SAR) in deep neural networks is essential in order to enhance the
explainability and physics awareness of deep learning. In this paper, we first
propose a novel physically explainable convolutional neural network for SAR
image classification, namely physics guided and injected learning (PGIL). It
comprises three parts: (1) explainable models (XM) to provide prior physics
knowledge, (2) physics guided network (PGN) to encode the knowledge into
physics-aware features, and (3) physics injected network (PIN) to adaptively
introduce the physics-aware features into classification pipeline for label
prediction. A hybrid Image-Physics SAR dataset format is proposed for
evaluation, with both Sentinel-1 and Gaofen-3 SAR data being experimented. The
results show that the proposed PGIL substantially improve the classification
performance in case of limited labeled data compared with the counterpart
data-driven CNN and other pre-training methods. Additionally, the physics
explanations are discussed to indicate the interpretability and the physical
consistency preserved in the predictions. We deem the proposed method would
promote the development of physically explainable deep learning in SAR image
interpretation field.

###### keywords:

explainable deep learningphysical modelSAR image classificationprior knowledge

##  1 Introduction

Synthetic Aperture Radar (SAR) can work in all-day all-weather conditions as
an active microwave sensing technology. Different from the optical remote
sensing images close to the visual understanding system of human eyes, SAR
images reflect the electromagnetic characteristics of objects and terrain. In
order to understand SAR images in a more comprehensive way, the artificial
intelligence approaches should pay close attention on not only the visual
information, but also the physical properties of SAR.

SAR image classification is a basic task, aiming to assign the semantic label
to each SAR image patch. Some conventional theory-driven approaches were
explored to extract the hand-crafted features based on the expertise of SAR,
e.g., the statistic model based Leng et al. (2020); Gao et al. (2017) and the
physical model based methods Leng et al. (2019). These model based approaches
have strong interpretability, yet the feature selection and classifier design
are time-consuming and lack flexibility. As a comparison, the data-driven deep
learning approaches can build an end-to-end system to learn the hierarchical
features automatically and predict the semantic labels simultaneously without
human intervention, superior to the pure model-based methods on SAR image
classification tasks Huang et al. (2017); Chen et al. (2016).

Nevertheless, the current data-driven solutions for SAR image classification
are still facing several challenges. The first is the contradiction between
the data-hungry deep learning approaches and the expensive cost in manual
annotation for SAR. At present, some pre-training related methods are
popularized to tackle the issue, such as transfer learning and self-supervised
learning. The transfer learning methods utilize the models pre-trained on
other data domains (like natural images, optical remote sensing imagery, etc)
via fine-tuning Huang et al. (2017), domain adaptation Huang et al. (2020b),
meta-learning Fu et al. (2022), etc. The self-supervised learning usually
takes the current domain data without annotations to optimize a designed
contrastive or pretext task, obtaining the pre-training model for the
downstream classification Wen et al. (2021); Ren et al. (2021).

Despite the good performance of transfer learning and self-supervised learning
on SAR image classification, the prediction of most deep models is short of
physical explanation. In consideration of the special physical characteristics
underlying SAR images, it is important to develop the hybrid approaches that
blend the deep learning algorithms with physical models in SAR domain to keep
the prediction consistent with physics and expert knowledge, which still
remains a big challenge in the current researches. At first, such feature
fusion methods that combine the CNN features with the hand-crafted description
of SAR images were proposed Zhang et al. (2020); Wang et al. (2021); Sun et
al. (2020), but there still exist limitations on providing explanations and
physical insights of SAR. More advanced hybrid approaches are required to
embed the prior scientific knowledge from physical model into the deep neural
networks de Bézenac et al. (2019), particularly in SAR image classification
domain, where the relevant studies are at the initial stage with a rising
trend Huang et al. (2020a).

To meet the above challenges in SAR image classification, we propose a novel
physically explainable CNN that blends the data-driven and model-driven
approach to perform impressive generalization with limited labeled data and
achieves physically explainable predictions. Our motivations are two-fold, as
depicted in Fig. 1. Firstly, we intend to build a physics-inspired data-driven
model for SAR image classification such as Daw et al. (2020); Park and Park
(2019); Svendsen et al. (2018) in other research fields, which embeds the
knowledge prior of physical model into the neural network. Secondly, inspired
by self-supervised learning where the semantic feature embeddings are learned
without supervision, we set a surrogate task based on the physical model to
leverage the unlabeled SAR images and further support the main classification
task. The proposed method, namely physics guided and injected learning (PGIL),
is composed of three modules:

Figure 1: A surrogate task $f_{PGN}$ is built based on the physical
information of SAR $y_{phy}$ derived from the explainable model $f_{XM}$.
Thus, the prior knowledge of physical model is embedded as feature
representation $F_{PA}$, which is successively injected into the main
classification task via $f_{PIN}$ to learn the semantic label $y_{tgt}$.

  1. 1.

Explainable Model (XM): We adopt the explainable model to generate the
abstract physical representation for each SAR image patch in semantic level.

  2. 2.

Physics Guided Network (PGN): We propose a novel unsupervised learning neural
network based on the designed surrogate task under the guidance of the
abstract physical representation. The knowledge prior in the explainable model
is converted as feature embeddings with PGN, aware of physical properties of
SAR.

  3. 3.

Physics Injected Network (PIN): As for the main classification task, PIN is
proposed to introduce the physics-aware features into the popular CNN
pipeline. It captures more comprehensive representations and maintains the
physical consistency of features, so as to prevent overfitting effectively
with a few labeled samples available.

For evaluation, a hybrid Image-Physics dataset format is proposed, equipped
with both SAR image patches and the corresponding physical scattering
mechanisms. Sufficient experiments are conducted mainly on the Sentinel-1 sea-
ice classification dataset and also the Gaofen-3 SAR data to demonstrate the
effectiveness of each module in the proposed method. The results show that the
proposed PGIL exhibits remarkable generalization performance compared with the
counterpart CNN architecture with supervised learning, transfer learning, and
self-supervised learning in case of limited labeled data. More importantly,
the physical explanations are discussed to demonstrate how the prior knowledge
constrains the network from training and how the physics consistency is
maintained in the predictions.

The contributions are summarized as follows:

  1. 1.

A novel physically explainable deep learning method is proposed for SAR image
classification that deeply integrates the data-driven and theory-driven
approaches.

  2. 2.

By establishing a novel surrogate task based on explainable physical models,
an unsupervised physics guided network is optimized to learn general features
aware of prior knowledge.

  3. 3.

A hybrid Image-Physics dataset formation is proposed for evaluation, which
combines the image and physics information of SAR in a concise way.

  4. 4.

We analyze the physics awareness of features, the good generalization on
limited labeled data, and the explainability as well as the physics
consistency of the predictions by sufficient experiments and discussions.

The rest of this paper is organized as follows. Section 2 reviews the
background knowledge of physical models applied in this paper. Section 3
presents the physics guided and injected learning (PGIL) neural network for
SAR image classification. The experiments and discussions are given in Section
4. Finally, Section 5 provides the conclusions.

##  2 Background

In this section, we introduce the background of explainable theory-driven
models for SAR applied in the following proposed method.

(a)

(b)

Figure 2: Two different physical models of SAR. (a) shows the 2-dimensional
H/$\alpha$ classification space Cloude et al. (1997) to demonstrate the
scattering mechanisms for full-polarized SAR data. (b) indicates the time-
frequency analysis model in HDEC-TFA method Huang et al. (2021a) where the
backscattering variations in different range and azimuth bandwidths of SAR
targets are characterized.

The first is the target decomposition model for PolSAR data to represent the
target scattering by several basic scattering mechanisms. One of the well-
known methods is the Cloude-Pottier decomposition for full-polarized SAR
Cloude et al. (1997), with the entropy $H$ and the angle $\alpha$ calculated
from coherency matrix. An $H/\alpha$ plane is separated into nine zones to
depict different scattering characteristics of full-pol SAR data, as shown in
Fig. 2 (a). The scattering mechanism classification result can be obtained via
the complex Wishart classifier proposed in Jong-Sen Lee et al. (1999).
Afterwards, the Cloude-Pottier decomposition model has been improved for dual-
polarized SAR images Ji and Wu (2015). In this paper, we employ the Cloude-
Pottier decomposition for both full- and HH/HV dual-pol SAR data Cloude et al.
(1997); Ji and Wu (2015), and the scattering mechanism classification results
are obtained by SNAP software.

The polarimetric decomposition is no longer available in single channel SAR
image data. The second one we introduce in this paper is based on the time-
frequency analysis model for single-polarized SAR data, as shown in Fig. 2(b).
The 2-dimensional short-time Fourier transform based time-frequency analysis
on complex-valued high resolution SAR data characterizes the backscattering
intensity variations of targets with different range and azimuth bandwidths,
denoted as sub-band scattering pattern Huang et al. (2021a). Given a specific
target with the position of $(x_{0},y_{0})$, and a segment $s$ centered in
$(x_{0},y_{0})$. The sub-band scattering pattern of target $(x_{0},y_{0})$ is
defined as

$r(x_{0},y_{0},f_{r},f_{a})=abs(\mathrm{FFT}^{-1}\\{\mathrm{FFT}(s)\cdot
w(f_{r},f_{a})\\}(x_{0},y_{0})),$ (1)

where $w(f_{r},f_{a})$ represents a series of bandpass filters centered on
frequency pairs $\\{(f_{r},f_{a})\\}$ in both range and azimuth directions.
The details can be found in Huang et al. (2021a). Fig. 2(b) gives some
examples of the extracted sub-band scattering pattern for different targets. A
learning based HDEC-TFA method was proposed in Huang et al. (2021a) to
classify the scattering patterns.

The above two different physical models will be applied in our proposed
method. Fig. 3 (a) shows the visualized result of H/$\alpha$-Wishart
classification Cloude et al. (1997) on full-polarized SAR data, where the
labels are consistent with Fig. 2 (a). Fig. 3 (b) is the HDEC-TFA result Huang
et al. (2021a) on single channel (HH) SAR image, where the given classes and
colorization are in accordance with Huang et al. (2021a).

(a)

(b)

Figure 3: Given a SAR image $x$, (a) shows the H/$\alpha$-Wishart
classification result Cloude et al. (1997) on full-polarized SAR data, and (b)
is the HDEC-TFA result Huang et al. (2021a) on single channel (HH) SAR data.

##  3 Physics Guided and Injected Learning

###  3.1 Overview

In most patch-wise SAR image classification methods, the processed SAR
amplitude images denoted as $x_{I}$ are considered other than the original
complex product $x$ to predict the image label $y_{tgt}$. Thus, the intrinsic
electromagnetic characteristics of SAR is not considered but desired. To this
end, our proposed physics guided and injected learning (PGIL), as summarized
in Fig. 1, leverages the visual friendly image data and the underlying prior
knowledge in physical model. The basic motivation is to embed the physics
knowledge into the neural network effectively.

Three main modules are included in PGIL, that are explainable models (XM),
physics guided learning network (PGN), and physics injected learning network
(PIN). XM offers prior knowledge of physical model. PGN convert the prior
knowledge into feature embedding, which is successively fused in PIN for label
prediction. The overall framework is depicted in Fig. 4.

XM acts on complex SAR image data $x$, where the explainable descriptor
$y_{phy}$ is obtained to represent the physical scattering properties of SAR
image:

$f_{XM}:x\to y_{phy}.$ (2)

$y_{phy}$ plays a major role in establishing the surrogate task of PGN for
optimization, therefore referred to physics guided signal. PGN follows an
unsupervised learning manner and outputs the feature embedding $F_{PA}$ aware
of prior physical knowledge, namely physics-aware features. The mapping
function $f_{PGN}$ is written as:

$f_{PGN}:\\{x,y_{phy}\\}\to F_{PA}.$ (3)

Finally, PIN is proposed to complete the main classification task where the
physics-aware feature $F_{PA}$ is injected, denoted as

$f_{PIN}:\\{x,F_{PA}\\}\to y_{tgt}.$ (4)

Figure 4: The physics guided and injected learning (PGIL) for SAR image
classification.

###  3.2 Explainable Models

As introduced in Section 2, the physics-based H/$\alpha$-Wishart Cloude et al.
(1997); Ji and Wu (2015) and HDEC-TFA Huang et al. (2021a) models serving as a
part of XM, are adopted to obtain the scattering mechanism of target in SAR
image $x$, denoted as $\mathcal{L}(x)$. The discrete physical scattering
labels $\mathcal{L}(x)$ either depict different zones in H/$\alpha$ plane
Cloude et al. (1997); Ji and Wu (2015), or refer to different targets with
diverse scattering variation patterns (Fig. 2(b)) Huang et al. (2021a).

Compared with SAR image label $y_{tgt}$, $\mathcal{L}(x)$ is too physics-
specific to offer the semantic information. In view of optimizing PGN to
obtain the feature embedding $F_{PA}$ aware of physics knowledge as well as
being semantic distinctive, we additionally employ the Latent Dirichlet
Allocation (LDA) on $\mathcal{L}(x)$ to output the topic mixture $y_{phy}$,
denoted as $y_{phy}=\mathcal{G}(\mathcal{L}(x))$.

In LDA topic modeling, the document formulated with bag-of-words
representation is characterized by a distribution over latent topics, and each
topic is represented by a distribution over words in the vocabulary. The
corpus is gathered from dataset to train the LDA model in an unsupervised
pattern and the generative process is explainable. The details can be found in
the related literature Blei et al. (2003); Rasiwasia and Vasconcelos (2013).
We redefine the essential variables of LDA on the basis of SAR scattering
characteristics as follows.

  * •

word vector: Randomly crop an area with the size of 8$\times$8 from
$\mathcal{L}(x)$ as $\mathcal{L}_{i}$ and calculate the normalized histogram
of $\mathcal{L}_{i}$ as the word vector.

  * •

vocabulary: With randomly generated word vectors, apply the $k$-means
algorithm Hartigan and Wong (1979) to obtain the vocabulary $V$ with $N_{v}$
cluster centers.

  * •

document: For each SAR image patch, the set of scattering word vector is
gathered by tiling $\mathcal{L}(x)$ with a step-size of 4. The document is
given by the frequency of each word in vocabulary $V$.

  * •

corpus The corpus is collected for training the LDA model that formed as a
matrix with the size $N_{v}\times N_{d}$, where $N_{d}$ is the number of
document.

Finally, $y_{phy}$ is obtained as the topic mixture (namely Bag of Topics,
BoT) of $\mathcal{L}(x)$,

$\mathcal{G}(\mathcal{L}(x))=\\{\varphi_{1},\varphi_{2},...,\varphi_{K}\\},$
(5)

where $\varphi_{k}$ denotes the score of the $k$th topic. Generally, the
summation of $\varphi_{k}$ equals 1.

###  3.3 Physics Guided Network

The role of PGN lies in embedding the prior physics knowledge in a neural
network, so as to extract the physics-aware features with semantic
discrimination beneficial to classification. The optimization of PGN is
motivated by the pretext task setting in self-supervised learning Misra and
Maaten (2020).

Tănase et al. Tănase et al. (2017) pointed that the topic semantics of
scattering properties are close to human semantics used for basic land-cover
types. Correspondingly, we propose to build a surrogate task under the
following assumption: the SAR image features and the topic mixture of physical
scattering labels should share common attributes in semantic level. In other
word, the physics descriptor $y_{phy}$ can be partly represented by high-level
deep features extracted from SAR image $x_{I}$.

We apply the first three residual blocks of ResNet-18 He et al. (2016) as the
SAR image feature extractor, denoted as $\mathcal{F}_{img}(x_{I})$. Since the
weak relationship between physics-specific topics $y_{phy}$ and image-specific
features, we design the physics mapping layer (PML) denoted as
$\mathcal{F}_{pml}$ to narrow the knowledge gap. The PML is composed of a
convolution module and a fully-connected layer, mapping the image
representations to physics topic space, denoted as
$\phi_{I}=\mathcal{F}_{pml}(\mathcal{F}_{img}(x_{I}))$ where
$\phi_{I}\in\mathbb{R}^{K}$.

The following objective function describes the soft semantic relations between
them, that

$L_{sr}(y_{phy},\phi_{I})=-\sum_{k\in\bm{K}_{vis}}y_{phy}^{k}\cdot\phi_{I},$
(6)

where $\bm{K}_{vis}$ denotes the topics that can be represented by features
from SAR vision domain. Equa. (6) is a relaxed constraint, where only the
related semantics are considered to be similar. As a comparison, the hard
constraint is

$L_{sr}(y_{phy},\phi_{I})=-\sum_{k\in\bm{K}_{vis}}y_{phy}^{k}\cdot\phi_{I}+\sum_{k\notin\bm{K}_{vis}}y_{phy}^{k}\cdot\phi_{I},$
(7)

where the unrelated semantics ($k\notin\bm{K}_{vis}$) are additionally
required to be highly different.

We choose the soft constraint in our method considering the semantic gap
between SAR physics knowledge and the visual perception of image data. The
follow-up experiments will discuss the differences of soft and hard
constraints.

In order to simplify the gradient descent optimization, we modify Equa. (6) as

$L_{sr}(y_{phy},\phi_{I})=-\sum_{k=1}^{K}\left(y_{phy}^{k}\log\frac{e^{\phi_{I}^{k}}}{\sum_{i}e^{\phi_{I}^{i}}}\right)\cdot\delta_{k},$
(8)

where $\delta_{k}$ is the activation term with a value of 1 or 0. We choose
the locations where $y_{phy}^{k}\geqslant\alpha$ as activated ones
($\delta_{k}=1$) with a probability of $p_{a}$, otherwise deactivated
($\delta_{k}=0$). The parameter $\alpha$ filters the remarkable attributes in
$y_{phy}$, that is, only the significant semantic topics are considered as
possibly related. The probability $p_{a}$ decides only part of the semantic
topics are selected to be related.

###  3.4 Physics Injected Network

The physics injected network (PIN) is designed to inject the physics-aware
features obtained from the unsupervised PGN into the traditional deep neural
network. The injected features provide abundant prior information for the deep
network training and are adapted to satisfy the classification task as far as
possible.

To decide which layer is appropriate to be taken for the injected physics-
aware features is crucial. In our work, we select the output of ResBlk-3 as
the physics-aware features to be injected, that is,
$F_{PA}=\mathcal{F}_{img}(x_{I})$. The decision will be discussed in the
following experiments.

We propose a simple injection strategy that adding the transformed physics-
aware features to the mid- and high-level layers of the traditional
classification network successively. The blue modules in Fig. 4 are the
conventional ResNet-18 classification networks, denoted as
$\mathcal{F}_{rs18}$. The transform layers, shown in the red module in Fig. 4,
are designed to convert the physics-aware features to the same size of the
destination, denoted as $\mathcal{F}_{tr}$. The transform module is composed
of a 1$\times$1 convolution layer and an upsample layer if needed, for channel
and feature size transformation respectively. The final output of the physics
injected neural network is written as
$Inj(\mathcal{F}_{rs18}(x_{I}),\mathcal{F}_{tr}(F_{PA}))$.

In the classification case, the cross entropy softmax loss function is widely
used. Here, we denote the CE loss as

$L_{ce}(x_{I},y_{tgt})=-\sum_{i}y_{tgt}^{i}\log c_{i},$ (9)

where
$c_{i}=Inj(\mathcal{F}_{rs18}(x_{I}^{i}),\mathcal{F}_{tr}(\mathcal{F}_{img}(x_{I}^{i})))$.

In order to ensure the physics-aware features to be more adaptive to the
classification task, we add the small weighted soft constraint $L_{sr}$ in
Equa. (8) as an regularization term and fine-tune the PGN slightly in
supervised classification training. The total loss function is written as

$L_{ce}(x_{I},y_{tgt})+\lambda L_{sr}(y_{phy},\phi_{I}).$ (10)

##  4 Experiments

In this section, we firstly introduce the hybrid Image-Physics data format and
the experimented datasets. Then, we conduct several experiments to prove the
effectiveness of our proposed method with sufficient discussions.

###  4.1 Dataset and Experimental Setup

Most SAR image classification datasets, like OpenSARUrban Zhao et al. (2020),
only provide the processed SAR amplitude images $x_{I}$ for a better visual
understanding. For the purpose of leveraging the underlying physics knowledge
in SAR and meanwhile preventing large storage space for complex data $x$, we
propose the hybrid Image-Physics (Img-Phy) data format to integrate $x_{I}$
and $\mathcal{L}(x)$ in a concise way, to accomplish the proposed PGIL method.

Figure 5: The hybrid Img-Phy data examples. Left: sea-ice classification
dataset with manual annotations. The Img part is the Sentinel-1 grounded range
detected (GRD) product and the Phy part is the H/$\alpha$-Wishart result from
the single-look complex (SLC) product. Right: the Gaofen-3 SAR image patches
in urban areas. Phy-1: the HDEC-TFA result Huang et al. (2021a), Phy-2: the
H/$\alpha$-Wishart result Jong-Sen Lee et al. (2004), which are also given in
Fig. 3.

We mainly evaluate our method on a sea-ice classification dataset acquired by
Sentinel-1, as shown in Fig. 5 (Left). The Sentinel-1 Interferometric Wide
(IW) SAR data in polar region is downloaded 111https://scihub.copernicus.eu/,
both single-look complex (SLC) and multi-looked Ground Range Detected High
resolution (GRDH) product (HH channel). Seven sea-ice types are annotated with
the patch size of 256$\times$256 for GRDH image, serving as $x_{I}$. Besides,
the dual-polarized SLC data is processed by SNAP software
222https://step.esa.int/main/download/snap-download/ to obtain the H/$\alpha$
labels, serving as $\mathcal{L}(x)$. To ensure $\mathcal{L}(x)$ almost covers
the same area with $x_{I}$, some essential operations like multi-looking,
grounded range projection, are required. Since the different pixel spacing
between the processed SLC and GRDH, the Phy data is not square anymore, which
is stored as a matrix with the size of about 187$\times$139\. For better
visualization, Fig. 5 shows the resized square Phy patch in RGB format, where
each color represents a scattering label.

For a better evaluation, especially in case of limited labeled data, we
randomly select 45, 35, 25, 15, and 5 samples from each class for training,
denoted as train-45, train-35, train-25, train-15, and train-5, respectively.
The test set is fixed, as shown in Table 1.

Table 1: The sea-ice classification dataset of Sentinel-1. class | YI | GL | FY | WT | IB | FI | OI
---|---|---|---|---|---|---|---
train | train-45, train-35, train-25, train-15, train-5
test | 1399 | 51 | 129 | 406 | 154 | 76 | 1107

In addition, a Gaofen-3 SAR scene image covering a wide urban area is
experimented 333https://www.ietr.fr/GF3/, shown in Fig. 5 (Right). Seven land
cover and land use classes are annotated. The training/test details are listed
in Table 2. We present two different types of $\mathcal{L}(x)$ introduced
before, that are Phy-1 (the HDEC-TFA result Huang et al. (2021a) on single-
polarized HH channel data) and Phy-2 (the H/$\alpha$-Wishart result Jong-Sen
Lee et al. (2004) on full-polarized data). In the following experiments, we
will discuss the physics guided learning results with different Phy data.

Table 2: The land-use and land-cover classification dataset of Gaofen-3. class | WT | AL | MF | AP | HD | MD | MU
---|---|---|---|---|---|---|---
train | 20 | 20 | 20 | 10 | 20 | 20 | 20
test | 157 | 80 | 650 | 34 | 231 | 871 | 775

When using H/$\alpha$-Wishart result of the polarimetric SAR as the Phy data,
the obtained $\mathcal{L}(x)$ is with $N_{s}=9$ classes of physical scattering
characteristics, corresponding to nine zones in H/$\alpha$ plane shown in Fig.
2(a). While for HDEC-TFA result of the single channel (HH) SAR image, $N_{s}$
is set to 15 according to Huang et al. (2021a). In the topic modeling for
physics guided signals generation, the vocabulary size $N_{v}$ and the topic
number $K$ of the LDA model are set to 500 and 175, respectively. Note that
the topic number $K$ is a critical parameter in the algorithm, that will be
under discussions in the following experiments. To determine $\delta_{k}$ in
Equation (8), we set $\alpha$ and $p_{a}$ to 0.1 and 0.9, respectively. The
following discussions will illustrate the strategy of parameter setting.

The physics guided learning is optimized by stochastic gradient descent (SGD)
with a fixed learning rate of 0.05, and the momentum is set to 0.9 by default.
All Img-Phy pairs in the dataset are fed into the PGN for training, lasting
200 epochs in total. The physics injected learning only takes annotated data
to train $\mathcal{F}_{rs18}$ and $\mathcal{F}_{tr}$. The initial learning
rate is set to 0.001 and the cosine annealing strategy is applied to decrease
the learning rate to $10^{-8}$ in the last 3 epochs of 50 in total. The soft
constraint regularization term in Equa. (10) is weighted by $\lambda$ set to
0.1.

All experiments are conducted on a workstation of 64 bit Linux operating
system, with 64G RAM and NVIDIA RTX 3090 graphics card of 24GB GDDR6X VRAM
clocked at 1700 MHz.

###  4.2 Unsupervised Physics Guided Learning

The PGN learns physics-aware features $F_{PA}$ from all Img-Phy pairs. To
evaluate the discriminative ability of $F_{PA}$ in semantic domain, we train a
support vector machine (SVM) on $F_{PA}$ to predict $y_{tgt}$. In this
section, we will firstly discuss how the topic number $K$, and the activation
strategy $\delta_{k}$ effect the discriminative physics-aware feature learning
based on the sea-ice classification dataset of Sentinel-1. Then, the
characteristics of physics-aware features, as well as the differences between
hard and soft constraint, are analyzed. At last, we additionally evaluate the
effectiveness of the proposed physics guided learning approach on Gaofen-3 SAR
data covering a wide urban area.

####  4.2.1 Hyperparameter Discussion

Firstly, we set $\alpha$ and $p_{a}$ to 0.1 and 0.9, respectively, and discuss
the topic number $K$. The classification results are shown in Table 3, where
the highest overall accuracy or F1-score are marked in red. We check out six
different values of $K$ (25, 50, 100, 150, 175, 200) and train the SVM
classifier on 5 different training sets. It can be observed from Table 3 that
a larger $K$ almost leads to a better result. Since the fully-connected layer
in the PML module is determined by the topic number, a large $K$ would
introduce plenty of parameters and increase the computation load.
Consequently, we decide $K=175$ for a better trade-off.

Table 3: The SVM performance (Overall Accuracy / F1-score ($\%$)) of physics-aware features in case of different topic numbers. Parameter setting: $\alpha=0.1,p_{a}=1.0$ $K$ | train-45 | train-35 | train-25 | train-15 | train-5
---|---|---|---|---|---
25 | 77.57 / 69.34 | 77.21 / 69.34 | 77.24 / 68.85 | 77.33 / 68.85 | 74.14 / 66.64
50 | 81.88 / 75.98 | 80.70 / 74.29 | 80.85 / 74.66 | 78.51 / 72.05 | 78.00 / 70.78
100 | 83.20 / 78.54 | 81.88 / 76.26 | 81.88 / 75.52 | 80.34 / 74.00 |  78.45 / 71.46
150 | 84.14 / 78.79 | 83.53 / 78.39 | 82.33 / 76.28 | 82.00 / 75.47 | 77.81 / 70.86
175 | 84.18 / 79.24 | 83.62 / 78.51 | 82.03 / 76.95 | 81.19 / 75.05 | 78.39 / 71.53
200 | 82.66 / 77.20 | 82.51 / 76.99 |  82.45 / 76.78 | 82.03 / 76.20 | 78.12 / 72.29

The topic number $K$ decides the shared attributes space $\mathbb{R}^{K}$
where the image representation is mapped. The assumption is proposed to build
a bridge between $\phi_{I}\in\mathbb{R}^{K}$ and $\\{\varphi_{k}\\}$. A larger
$K$ ensures more fine-grained physics attributes, so that the soft constraint
in Equa. (6) can be more precise. We calculate the sparsity of BoT vector
$\bm{\varphi}$, defined as

$sparsity=1-\frac{||\bm{\varphi}||_{0}}{K},$ (11)

where $||\cdot||_{0}$ denotes the L0 norm. Fig. 6 plots the sparsity of BoT
representation in case of different topic numbers. We find the fine-grained
physics attributes lead to a more sparse representation of BoT encoding. The
BoT sparsity is also highly relate to the physics-aware feature performance.
Intuitively, a sparsity greater than 0.985 can be regarded as a good choice
with the topic number $K$ no less than 150.

Figure 6: The topic sparsity of LDA model in case of different topic number
$K$.

Next, we will discuss how to fix the activated physics attributes $\delta_{k}$
in Equa. (8). $\delta_{k}$ is determined by $\alpha$ and $p_{a}$, where
$\alpha$ filters the prominent attributes in $y_{phy}$ as the candidates, and
$p_{a}$ randomly select the potential attributes to calculate the constraint.
Table 4 shows the SVM classification results of different $\alpha$, with
$K=175$ and $p_{a}=0.9$. When $\alpha$ equals 0, some unconsidered attributes
($\phi_{k}$ close to 0) may be included to guide the network to learn the
insignificant features. It is better to set $\alpha$ as a small value but
greater than 0, e.g. 0.1 in our case.

Table 4: The SVM performance (Overall Accuracy / F1-score ($\%$)) of physics-aware features in case of different $\alpha$. Parameter setting: $p_{a}=0.9,K=175$ $\alpha$ | train-45 | train-35 | train-25 | train-15 | train-5
---|---|---|---|---|---
0 | 84.05 / 79.15 | 83.32 / 78.05 | 82.69 / 77.32 | 81.55 / 75.55 | 78.20 / 71.46
0.1 | 84.18 / 79.24 | 83.62 / 78.51 | 82.03 / 76.95 | 81.19 / 75.05 | 78.39 / 71.53
0.2 | 83.96 / 78.98 | 83.20 / 77.93 | 82.90 / 77.54 | 82.42 / 76.43 | 78.09 / 71.44
0.3 | 81.58 / 76.11 | 82.21 / 76.55 | 81.55 / 75.69 | 79.38 / 73.20 | 75.89 / 70.32

Table 5 discusses the values of $p_{a}$ in the context of $\alpha=0.1$ and
$K=100$. $p_{a}$ is the probability for randomly selecting the potential
attributes in $y_{phy}$. The value of $p_{a}$ from 0.5 to 1.0 indicates the
constraint becomes rigid. The result shows it is better to choose a greater
values of $\alpha$, which demonstrates a majority of remarkable attributes
should be considered. Here, we choose $\alpha=0.9$ in our experiments.

To summarize, the topic number $K$ is the most important hyperparameter in PGN
learning which can be determined by the sparsity of BoT representation.
$\delta_{k}$ controls a relax activation of physics attributes, decided by a
relatively casual value of $\alpha$ and $p_{a}$. We recommend to set $\alpha$
to 0.1 or 0.2, and $p_{a}$ to 0.9 or 0.8, respectively.

Table 5: The SVM performance (Overall Accuracy / F1-score ($\%$)) of physics-aware features in case of different $p_{a}$. Parameter setting: $\alpha=0.1,K=100$ $p_{a}$ | train-45 | train-35 | train-25 | train-15 | train-5
---|---|---|---|---|---
1.0 | 83.20 / 78.54 | 81.88 / 76.26 | 81.88 / 75.52 | 80.34 / 74.00 | 78.45 / 71.46
0.9 | 83.35 / 78.09 | 83.02 / 77.37 |  82.48 / 76.00 | 81.13 / 74.74 | 79.23 / 71.87
0.8 | 83.87 / 78.73 |  82.93 / 77.51 | 82.45 / 76.64 | 81.10 / 74.60 | 78.24 / 71.84
0.7 | 83.17 / 77.93 | 82.93 / 77.66 | 81.58 / 76.30 | 79.74 / 73.70 | 78.27 / 71.83
0.6 | 82.18 / 76.81 | 81.52 / 76.54 | 80.67 / 75.25 | 79.74 / 73.65 | 77.39 / 71.32
0.5 | 80.34 / 74.39 | 80.79 / 74.71 | 79.86 / 73.17 | 79.02 / 72.58 | 75.80 / 68.96

####  4.2.2 The Physics-Aware Features Discussion

The PGN is comprised of the ResNet-13 backbone and the 2-layer PML module. The
ResNet-13 backbone $\mathcal{F}_{img}$ extracting hierarchical features from
$x_{I}$ is basically image specific, with the higher-level features becoming
closer to semantic meaning. The PML module transforms $\mathcal{F}_{img}(x)$
to the physics attributes space $\mathbb{R}^{K}$, building the semantic
relation between image representation and physics knowledge. The physics-aware
features are expected to be discriminative in the classification semantic
domain and also with physics awareness.

We analyze the outputs of different layers in PGN to demonstrate the semantic
discrimination and the physics awareness of features. The SVM classification
results are used to indicate the semantic discrimination, as shown in the
first row in Table 6. The features from ResBlk-3 reach the highest
classification accuracy of 84.18%, followed by an overall accuracy of 82.39%
for the features from the convolution layer in PML module. The results
demonstrate how the feature discrimination in semantic level changes with the
physics guided neural network. Note that the physics BoT encoding only
achieves 61.53% in classification, which indicates $y_{phy}$ is highly physics
specific and the semantic gap is truly existed between $y_{phy}$ and
$y_{tgt}$. Even so, $y_{phy}$ can guide the PGN to learn the discriminative
features close to $y_{tgt}$ successfully, with the designed objective
function. Intuitively, the first row in Fig. 7 displays the annotated labels
with different colors, indicating the feature discrimination in semantic
level. We can observe that the BoT encodings are confused in understanding the
semantic labels, since the SAR images in the same class may have different
physics attributes. After the physics guided learning, the feature
discrimination in semantic level has been improved from PML-fc to ResBlk-3
layer. Due to the lower level of ResBlk-2, the features in Fig. 7(a) are not
as discriminative as those of ResBlk-3 in Fig. 7(b).

Table 6: The SVM classification results (OA / F1-score), and the physics awareness analysis of features (quantified by silhouette score) from different layers in PGN. The soft and hard constraints in Equa. (6) and (7) are both explored. $F_{PA}$ | Constraint | ResBlk-2 | ResBlk-3 | PML-conv | PML-fc | BoT
---|---|---|---|---|---|---
SVM result OA / F1-score ($\%$) | soft | 79.05 / 72.75 | 84.18 / 79.24 | 82.39 / 76.84 | 77.90 / 71.74 | 61.53 / 51.86
hard | 78.45 / 71.55 | 82.99 / 78.05 | 81.85 / 75.90 | 77.45 / 71.40 | 61.53 / 51.86
Physics Awareness Silhouette score | soft | 0.0157 | 0.0351 | 0.0904 | 0.1087 | 0.4766
hard | 0.0181 | 0.0380 | 0.0949 | 0.1202 | 0.4766

Additionally, we demonstrate the physics awareness of features by
visualization and quantitative metrics, shown in the second row of Fig. 7 and
Table 6, respectively. Given the BoT encodings of physics attributes, we apply
a $k$-means algorithm to cluster the BoT representations into $k$ classes as
the color identification in the second row of Fig. 7. Thus, (f)(g)(h)(i)(j)
indicate the feature discrimination in physics level, that is, the samples
with the same color have similar physics attributes. We can observe how the
features become physics aware with the help of PML module. Table 6 also lists
the silhouette coefficient of features which reflect the separation between
clusters. The silhouette coefficient values between -1 and 1, and a higher
silhouette score indicates better discrimination of physics information in
this feature space, that is, stronger physics awareness of features. It
gradually decreases from physics topic space to image feature space, but still
keep a positive value of 0.0351 in ResBlk-3. As a comparison, the silhouette
score of features in ResBlk-3 of the traditional CNN learning model is
-0.0085, indicating physics unawareness. Hence, we assert the PGN is able to
learn the physics-aware features.

Figure 7: The feature visualization of training data in different layers.
Features in (a)(f), (b)(g), (c)(h), (d)(i), are from ResBlk-2, ResBlk-3, PML-
conv, and PML-fc, respectively. (e)(j) are BoT representations.
(a)(b)(c)(d)(e) are marked with true labels. (f)(g)(h)(i)(j) are marked with
$k$-means cluster labels of BoT representations.

####  4.2.3 Hard and Soft Constraint Discussion

We discuss the soft and hard constraint objective functions defined in Equa.
(6) and (7). The soft constraint only emphasizes the common semantics from
physics and vision domain to be highly similar, while the hard constraint
additionally restricts the specific ones to be different. The SVM
classification results in Table 6 indicate that the soft constraint guides the
PGN to learn more general features which achieve better performance in
classification. The listed silhouette coefficients of hard constraint in Table
6 are larger than those of soft constraint, which demonstrate the the learned
features by hard constraint are more physics specific but less semantic
discriminative. The discussion explains the semantic gap between the physics
attributes and the image features of SAR, encouraging us to find a trade-off
in learning the physics-aware features.

####  4.2.4 Generalization Analysis of PGN

We additionally take the other Gaofen-3 SAR data, as shown in Fig. 5 (right),
to demonstrate the effectiveness and generalization ability of the proposed
PGN, besides the polarimetric characteristics derived from H/$\alpha$-Wishart
method Jong-Sen Lee et al. (1999). In our previous work Huang et al. (2021a),
we verified that the HDEC-TFA method could automatically discover the time-
frequency properties of SAR target in high resolution SAR images, especially
for some man-made targets with characteristic scattering behaviors. It is
based on the physics meanings of time-frequency analysis on complex SAR data,
which reveals the scattering variation on different azimuth angles and range
bandwidths. We apply different physics information on PGN to illustrate our
proposed method can be integrated with various physical models.

Table 7: The SVM classification results of features in physics guided learning with different physics information, compared with the supervised end-to-end CNN training. (Overall Accuracy / F1-score ($\%$)) Phy Data | None (CNN) | HDEC-TFA (Phy-1) |  H/$\alpha$-Wishart (Phy-2)
---|---|---|---
Result | 52.43 / 48.51 | 68.76 / 62.32 | 62.54 / 58.71

The SAR image in urban city with dense buildings and man-made targets is more
complicated than polar area. With very limited annotation, it is difficult for
supervised CNN training to learn generalized and discriminative features.
Table 7 shows the CNN only achieves an accuracy and F1-score of 52.43% and
48.51%, respectively, with severe overfitting. The SVM classification is
utilized to evaluate the physics-aware features by PGN with different Phy
data. As recorded in Table 7, the classification result of physics-aware
features guided by Phy-1 (HDEC-TFA) signals improves 16.33% in accuracy than
CNN training result, 6.22% better than guided by H/$\alpha$-Wishart scattering
characteristics. It also verifies the effectiveness of HDEC-TFA learning
approach to extract significant physics properties under circumstance of no
polarimetric information available.

Figure 8: The physics guided learning result with different physics
information, compared with the CNN result on Gaofen-3 SAR image.

Fig. 8 visualizes the annotated ground truth, the test results of CNN, PGN
with Phy-1 and Phy-2 on two classes, High-density residential area and
Airport. The scattering in high-density residential area is strong and
extremely complicated so that the visual interpretation is difficult. However,
the physical characteristics in this area, as shown in Fig. 3, are more
distinctive than other regions, especially in HDEC-TFA case. Consequently, PGN
with Phy-1 (HDEC-TFA) achieves the best prediction result on high-density
residential area with minimum false negative samples. There are 4 airports in
the SAR image, as shown in Fig. 8, where only 10 annotated samples in
Airport-1 are used for training. The PGN with Phy-1 can remarkably predict the
remaining 3 unseen airports with only 3 false negative samples, which is much
superior than traditional data-driven CNN model.

###  4.3 Supervised Physics Injected Learning

In this section, we discuss the effectiveness of PIN module on sea-ice
classification dataset in Table 1 (train-45 as the training data). The
physics-aware representation $F_{PA}$ is considered as an off-the-shelf
feature from PGN to be injected. The baseline is the traditional CNN method,
that is, training $\mathcal{F}_{rs18}$ with only labeled amplitude SAR images.
We both test the training from the scratch strategy and the transfer learning
strategy using a SAR image pre-trained model proposed in Huang et al. (2021b).
The classification results are shown in Table 8, where the retraining accuracy
is only 74.55% and the transfer learning result is 82.57%. It shows that the
limited labeled data is insufficient to train a very deep neural network from
scratch. After the physics-aware features injection, however, the retraining
results improve more than 10%, as shown in Table 8.

Table 8: The ablation study of physics injected learning. (OA ($\%$)) inj-2 | inj-3 | inj-4 | retrain (%) | pre-trained (%)
---|---|---|---|---
- | - | - | 74.55 $\pm$ 1.05 | 82.57 $\pm$ 0.82
$\surd$ | - | - | 82.56 $\pm$ 2.36 | 81.56 $\pm$ 1.89
- | $\surd$ | - | 84.52 $\pm$ 0.39 | 85.19 $\pm$ 0.75
- | - | $\surd$ | 82.15 $\pm$ 1.18 | 84.05 $\pm$ 0.89
$\surd$ | $\surd$ | - | 82.11 $\pm$ 0.31 | 84.15 $\pm$ 0.83
- | $\surd$ | $\surd$ | 82.77 $\pm$ 1.47 | 83.73 $\pm$ 1.35
$\surd$ | $\surd$ | $\surd$ | 84.96 $\pm$ 0.36 | 86.40 $\pm$ 0.35

We discuss the different locations where the physics-aware features are
injected, including the ResBlk-2, ResBlk-3, and ResBlk-4, denoted as inj-2,
inj-3, and inj-4 in Table 8, respectively. With the same depth of $F_{PA}$ and
the features of ResBlk-3, the injection in ResBlk-3 reaches the best
performance of single-injection strategy, marked in blue. The results indicate
that the obtained physics-aware feature is with abstract meanings, but still
has semantic gap with the semantic features for target task. We also find that
the multi-layer injection can improve the classification most in both
retraining and transfer learning cases.

The unsupervised PGN training costs about 4.45h, with the batchsize of 300 and
training epochs of 200. Afterwards, the supervised PIN only takes 13.25
minutes for training, with the batchsize of 100 and training epochs of 100.

###  4.4 Ablation Study

In order to demonstrate the effectiveness of each module in the proposed
method, we conduct the detailed ablation study of each module on sea-ice
dataset (train-45 as the training data).

As shown in Table 9, the baseline is set as retraining the CNN model from
scratch, achieving an overall accuracy of 74.55% with strong overfitting. As a
contrast, the proposed method reaches 85.27% in average, obtaining about
10.72% improvements. We discuss the effectiveness of the following four parts:

  1. 1.

Explainable Models (XM): Based on the existing physical scattering labels,
generating the abstract BoT encodings with LDA as the physics-guided signals.

  2. 2.

Physics Guided Network (PGN): Learning the physics-aware features guided by
physics BoT representations.

  3. 3.

Physics Injected Network (PIN): Retraining the CNN with injecting physics-
aware features.

  4. 4.

Self-Adaptive Learning (SAL): Fine-tuning the physics guided network during
the PIN training with a combined loss.

Table 9 shows the ablation experiments, excluding each part above-mentioned
separately. It is clear that the XM and PGN play the most important roles in
the proposed method. Generating an appropriate physics guided signal $y_{phy}$
in XM remarkably effects the quality of injected features learned by PGN, with
the accuracy increasing from 80.61% to 84.96%. The PGN module ensures the
injected knowledge more related to the target task. Otherwise, directly
injecting the BoT representation in Equa. (5) only has an accuracy of 78.78%
in average, and the proposed PGN contributes 6.18% improvement. The PIN
learning makes further efforts on fusing the physics knowledge and vision
features together, which has an improvement of about 2.37%. The SAL part which
makes the physics-aware features more adaptive to the target task slightly
improves the result.

Table 9: The ablation study of different modules. (OA ($\%$)) XM | PGN | PIN | SAL | description | result (%)
---|---|---|---|---|---
Baseline | retrain CNN |  74.55 $\pm$ 1.05
- | $\surd$ | $\surd$ | - |  use $\mathcal{L}(x)$ as $y_{phy}$ |  80.61 $\pm$ 1.18
$\surd$ | - | $\surd$ | - | inject Equa. (5) |  78.78 $\pm$ 1.67
$\surd$ | $\surd$ | - | - | fine-tune PGN |  82.59 $\pm$ 1.47
$\surd$ | $\surd$ | $\surd$ | - |  optimizing Equa. (9) |  84.96 $\pm$ 0.36
$\surd$ | $\surd$ | $\surd$ | $\surd$ |  optimizing Equa. (10) | 85.27 $\pm$ 0.28

Additionally, we compare some self-supervised learning methods in computer
vision field Wu et al. (2018); Chen et al. (2020) and also for PolSAR data Ren
et al. (2021) with the proposed PGIL, since they all establish pretext tasks
for unsupervised learning the feature embeddings. NPID Wu et al. (2018)
learned the optimal feature via instance-level discrimination, while SimCLR
Chen et al. (2020) conducted the contrastive learning based on data-
augmentation, both focusing on image contents. MI-SSL Ren et al. (2021) was
proposed for PolSAR land cover classification, learning discriminative high-
level features between multi-modal representations of PolSAR data. In order to
adapt MI-SSL method to our case, we changed the SSL input of multi-modal
features for full-polarized SAR data to our Img-Phy pairs. The results are
listed in Table 10. Although NPID and SimCLR perform well in natural image
classification, such as ImageNet, the results demonstrate the pretext tasks of
instance-level discrimination and contrastive learning via data-augmentation
almost fail in SAR image classification. MI-SSL performs better for
comparison, because it considers the multiple representations of SAR data. Our
proposed PGIL reaches the best among them.

Table 10: The comparison with self-supervised learning methods on sea-ice classification dataset. (OA ($\%$)) Method | Description | result (%)
---|---|---
NPID Wu et al. (2018) |  instance-level discrimination |  66.56 $\pm$ 2.95
SimCLR Chen et al. (2020) |  data-augmented contrastive learning |  74.02 $\pm$ 0.78
MI-SSL Ren et al. (2021) |  self-supervised for PolSAR data |  77.75 $\pm$ 1.41
PGIL | Proposed |  85.27 $\pm$ 0.28
Figure 9: The confusion matrix and the classification metrics (precision,
recall, f1-score, and overall accuracy) of CNN learning, PGN + SVM, and PGIL
performances are shown, respectively.

###  4.5 Interpretability Discussion

In this section, we use the sea-ice classification case to demonstrate the
physics explainability of the proposed method. The explainability lies in the
following two aspects. Firstly, the topic modeling for physics information
provides explainable representations for each SAR image patch. Secondly, the
PGN and PIN maintain the explainable physics consistency of features to learn
reasonable results and prevent overfitting during automatic training.

####  4.5.1 Physics Explanation of $y_{phy}$

The PGN optimization is driven by the physics guided signals $y_{phy}$,
denoted as $\mathcal{G}(\mathcal{L}(x))=\\{\phi_{1},\phi_{2},...,\phi_{K}\\}$
in Equa. (5). The LDA topic modeling processing $\mathcal{G}$ explains the
physical scattering characteristics $\mathcal{L}(x)$ by a combination of
topics. Each latent topic is represented by a set of specific words, that is,
the physical scattering characteristics distribution in a small area of SAR
image (defined as "word" in LDA). The weight assigned ($\phi_{i}$) describes
the probability of the SAR image patch belonging to the topic $i$. This
benefits the understanding of hidden semantic structure between scattering
labels of a large-scale SAR image area at an aggregate level.

Figure 10: The averaged physics BoT representation of training data for each
class. The water bodies and the floating ice have very similar physics
attributes.

Fig. 10 presents the averaged physics topic distribution of training data for
some selected sea-ice classes, with topic number $K=175$. Since we have
discussed previously that most SAR patches are with highly sparse physics
topic representations, the classes whose distribution concentrated in fewer
topics, such as iceberg and glaciers shown in Fig. 10, are more
characteristic. The iceberg is mainly represented by topic-73 weighted 0.6 and
topic-119 weighted 0.13. The word distribution of each topic is also given in
Fig. 10, where each word can be explained by the physical scattering
properties. In this sea-ice hybrid Phy-Img dataset, word-6 and word-7 are
mostly random surface and Bragg surface scattering, respectively, and word-50
is a mixture of these two scattering properties.

Also, we list the four dominated topics and the topic weights of water bodies
and floating ice in Fig. 10. The result indicates the physics attributes of
water bodies and the floating ice are similar in semantic level. The inference
can be proved from the semantic definition of floating ice – any form of ice
found floating in water 444The semantic definition of floating ice, that is to
say, the SAR image patch with floating ice probably includes water bodies.
Additionally, there are similar topics combinations in water bodies, floating
ice, and young ice, shown in the zoom-in region of Fig. 10. The young ice has
another specific topic of 62, indicating this class could have two kinds of
representative physics attributes.

####  4.5.2 Explanation of PGIL Results

Since the PGN is trained with the explainable BoT physics encoding, the above
inferred information can explain the discrimination of physics-aware features
in Fig. 7(b). With more characteristic topic distribution of glaciers and
icebergs, their physics-aware features are the most discriminative, and the
SVM classification results show the two classes achieve the best F1-score of
0.909 and 0.911, respectively, as shown in Fig. 9. In addition, the previous
analysis explains the similar feature distribution of water bodies and
floating ice shown in Fig. 7(b), and the features of young ice have two
different characteristics, one of which is close to water bodies and floating
ice. As a result, we can observe in the confusion matrix that the most
misclassified test samples in water bodies are predicted as young ice and
floating ice, with a number of 57 and 37, respectively. Among the 11 false
negative samples in the floating ice class, 9 are classified to water bodies
and 2 to young ice.

(a)

(b)

Figure 11: The feature visualization of CNN supervised training result, the
physics guided learning result, and the result after physics injected
learning.

Two cases are presented in Fig. 11 to demonstrate how the PIN preserves the
physics consistency of features. We visualize the features of CNN supervised
training, physics-aware features from unsupervised PGN, and the features after
injection in PIN by t-sne Maaten and Hinton (2008). As shown in Fig. 11(a),
the CNN feature of instance 996 and 496 in glacier class are far away from the
majority due to the different image contents. Based on the similar physics
attributes, the middle figure in Fig. 11(a) shows the two samples are close to
most glacier data in physics-aware features. In the visualization of features
after physics injected learning, we can observe sample 496 and 996 are still
close to each other due to the similar visual representation, and also, they
maintain the closeness with sample 494, 793, and 1193 as they were in physics-
aware features. As a result, we infer the physics injected learning can
preserve the physics consistency during the network training. Another example
is about sample 5013 and 4224, shown in Fig. 11(b). They have similar image
features in CNN training while different from the other first year ice
samples. The physics-aware features in the middle figure reveals that sample
5013 and 4224 have their own physics characteristics. The physics constraints
existed in physics-aware features are, i.e, sample 5013 having similar physics
properties with 4628, 5606, 5906, and sample 4224 being very close to sample
3925 from old ice class. PIN continues this kind of constraint, as shown in
the right figure of Fig. 11(b). In a word, the proposed PGN and PIN represent
the SAR images from a more comprehensive perspective than the traditional
supervised CNN learning.

####  4.5.3 The Inspiration from Explainability

In addition, the explainability we discussed before can inspire us to improve
our deep learning algorithm in the future work. For example, the above
analysis indicates the floating ice has a similar physics BoT representation
with water bodies so that the physics-aware features of them are not well
discriminative in semantic level, as shown in Fig. 11. On the contrary, they
can be discriminative based on the visual contents. The final PIN result shows
injecting physics-aware features could not improve the performance of
recognizing water bodies and floating ice, because their physics knowledge is
not as helpful as other classes. We can see in Fig. 9 that the true positive
samples of water bodies and floating ice in PGIL result are fewer than those
in CNN training result. Thus, it inspires us to re-think the physics injected
learning strategy that the constraint of physics consistency should be relaxed
in such classes. In our future work, we will further improve the physics
injected learning method in this direction to achieve a better result.

##  5 Conclusion

In this paper, we propose a novel physics guided and injected learning neural
network for SAR image classification with limited labeled data, to explore the
potential of physically explainable deep learning. Three components of PGIL
are explainable model, physics guided network, and physics injected network.
The prior knowledge in explainable models is encoded into the physics-aware
features via unsupervised PGN learning, and then is injected in the
classification pipeline through PIN, supervised by limited labeled data. The
hybrid Img-Phy dataset format is proposed for evaluation and abundant
experiments are conducted on Sentinel-1 and Gaofen-3 SAR data. The results
demonstrate the semantic discrimination and the physics awareness of the
learned features by PGN, as well as the good generalization. Additionally, we
discuss the interpretability of the guided signals in the established
surrogate task to prove the results are with physical constraint. The
advantages of the proposed PGIL are (1) the unsupervised PGN is a plug-and-
play module which can be integrated to any deep learning framework for
physics-aware feature injection; (2) the physics knowledge injection is
capable of preserving the physics consistency in the prediction and preventing
overfitting in case of limited labeled data; (3) the results are explainable
with the help of the distinct physical models and expertise to a certain
extent, that inspire us to further improve the deep learning model in the
right direction. The sea-ice dataset and source code are publicly in
https://github.com/Alien9427/XAI4SAR-PGIL.

##  6 Acknowledgment

This work was supported in part by the National Natural Science Foundation of
China under Grant 62101459, U20B2068, and in part by the China Postdoctoral
Science Foundation under Grant BX2021248, and the Fundamental Research Funds
for the Central Universities under Grant G2021KY05104.

## References

  * de Bézenac et al. (2019) de Bézenac, E., Pajot, A., Gallinari, P., 2019.  Deep learning for physical processes: incorporating prior scientific knowledge.  Journal of Statistical Mechanics: Theory and Experiment 2019, 124009.  URL: https://doi.org/10.1088/1742-5468/ab3195, doi:10.1088/1742-5468/ab3195.
  * Blei et al. (2003) Blei, D.M., Ng, A.Y., Jordan, M.I., 2003.  Latent dirichlet allocation.  Journal of Machine Learning Research 3, 993–1022.
  * Chen et al. (2016) Chen, S., Wang, H., Xu, F., Jin, Y.Q., 2016.  Target classification using the deep convolutional networks for SAR images.  IEEE Transactions on Geoscience and Remote Sensing 54, 4806–4817.
  * Chen et al. (2020) Chen, T., Kornblith, S., Norouzi, M., Hinton, G., 2020\.  A simple framework for contrastive learning of visual representations, in: III, H.D., Singh, A. (Eds.), Proceedings of the 37th International Conference on Machine Learning, PMLR. pp. 1597–1607.  URL: https://proceedings.mlr.press/v119/chen20j.html.
  * Cloude et al. (1997) Cloude, S.R., Member, S., Pottier, E., 1997.  An Entropy Based Classification Scheme for Land Applications of Polarimetric SAR 35, 68–78.
  * Daw et al. (2020) Daw, A., Thomas, R.Q., Carey, C.C., Read, J.S., Appling, A.P., Karpatne, A., 2020\.  Physics-guided architecture (pga) of neural networks for quantifying uncertainty in lake temperature modeling, in: Proceedings of the 2020 siam international conference on data mining, SIAM. pp. 532–540.
  * Fu et al. (2022) Fu, K., Zhang, T., Zhang, Y., Wang, Z., Sun, X., 2022\.  Few-shot sar target classification via metalearning.  IEEE Transactions on Geoscience and Remote Sensing 60, 1–14.  doi:10.1109/TGRS.2021.3058249.
  * Gao et al. (2017) Gao, G., Ouyang, K., Luo, Y., Liang, S., Zhou, S., 2017.  Scheme of parameter estimation for generalized gamma distribution and its application to ship detection in sar images.  IEEE Transactions on Geoscience and Remote Sensing 55, 1812–1832.  doi:10.1109/TGRS.2016.2634862.
  * Hartigan and Wong (1979) Hartigan, J.A., Wong, M.A., 1979\.  Algorithm as 136: A k-means clustering algorithm.  Journal of the Royal Statistical Society. Series C (Applied Statistics) 28, 100–108.  URL: http://www.jstor.org/stable/2346830.
  * He et al. (2016) He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: Proc. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  * Huang et al. (2020a) Huang, Z., Datcu, M., Pan, Z., Lei, B., 2020a.  Deep sar-net: Learning objects from signals.  ISPRS Journal of Photogrammetry and Remote Sensing 161, 179 – 193.  URL: http://www.sciencedirect.com/science/article/pii/S0924271620300162, doi:https://doi.org/10.1016/j.isprsjprs.2020.01.016.
  * Huang et al. (2021a) Huang, Z., Datcu, M., Pan, Z., Qiu, X., Lei, B., 2021a.  HDEC-TFA: An Unsupervised Learning Approach for Discovering Physical Scattering Properties of Single-Polarized SAR Image.  IEEE Transactions on Geoscience and Remote Sensing 59, 3054–3071.  URL: https://ieeexplore.ieee.org/document/9169671/, doi:10.1109/TGRS.2020.3014335.
  * Huang et al. (2021b) Huang, Z., Dumitru, C.O., Pan, Z., Lei, B., Datcu, M., 2021b.  Classification of Large-Scale High-Resolution SAR Images With Deep Transfer Learning.  IEEE Geoscience and Remote Sensing Letters 18, 107–111.  URL: https://ieeexplore.ieee.org/document/8966281/, doi:10.1109/LGRS.2020.2965558.
  * Huang et al. (2017) Huang, Z., Pan, Z., Lei, B., 2017.  Transfer learning with deep convolutional neural network for SAR target classification with limited labeled data.  Remote Sens. 9.  doi:10.3390/rs9090907.
  * Huang et al. (2020b) Huang, Z., Pan, Z., Lei, B., 2020b.  What, Where, and How to Transfer in SAR Target Recognition Based on Deep CNNs.  IEEE Transactions on Geoscience and Remote Sensing 58, 2324–2336.  doi:10.1109/TGRS.2019.2947634, arXiv:1906.01379.
  * Ji and Wu (2015) Ji, K., Wu, Y., 2015.  Scattering Mechanism Extraction by a Modified Cloude-Pottier Decomposition for Dual Polarization SAR , 7447--7470doi:10.3390/rs70607447.
  * Jong-Sen Lee et al. (1999) Jong-Sen Lee, Grunes, M.R., Ainsworth, T.L., Li-Jen Du, Schuler, D.L., Cloude, S.R., 1999\.  Unsupervised classification using polarimetric decomposition and the complex wishart classifier.  IEEE Transactions on Geoscience and Remote Sensing 37, 2249--2258.  doi:10.1109/36.789621.
  * Jong-Sen Lee et al. (2004) Jong-Sen Lee, Grunes, M.R., Pottier, E., Ferro-Famil, L., 2004\.  Unsupervised terrain classification preserving polarimetric scattering characteristics.  IEEE Trans. Geosci. and Remote Sens. 42, 722--731.  doi:10.1109/TGRS.2003.819883.
  * Leng et al. (2019) Leng, X., Ji, K., Zhou, S., Xing, X., 2019\.  Ship detection based on complex signal kurtosis in single-channel sar imagery.  IEEE Transactions on Geoscience and Remote Sensing 57, 6447--6461.  doi:10.1109/TGRS.2019.2906054.
  * Leng et al. (2020) Leng, X., Ji, K., Zhou, S., Xing, X., 2020\.  Fast shape parameter estimation of the complex generalized gaussian distribution in sar images.  IEEE Geoscience and Remote Sensing Letters 17, 1933--1937.  doi:10.1109/LGRS.2019.2960095.
  * Maaten and Hinton (2008) Maaten, L.v.d., Hinton, G., 2008\.  Visualizing data using t-sne.  Journal of machine learning research 9, 2579--2605.
  * Misra and Maaten (2020) Misra, I., Maaten, L.v.d., 2020\.  Self-supervised learning of pretext-invariant representations, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  * Park and Park (2019) Park, J., Park, J., 2019.  Physics-induced graph neural network: An application to wind-farm power estimation.  Energy 187, 115883\.  URL: https://www.sciencedirect.com/science/article/pii/S0360544219315555, doi:https://doi.org/10.1016/j.energy.2019.115883.
  * Rasiwasia and Vasconcelos (2013) Rasiwasia, N., Vasconcelos, N., 2013\.  Latent dirichlet allocation models for image classification.  IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 2665--2679.
  * Ren et al. (2021) Ren, B., Zhao, Y., Hou, B., Chanussot, J., Jiao, L., 2021\.  A mutual information-based self-supervised learning model for polsar land cover classification.  IEEE Transactions on Geoscience and Remote Sensing 59, 9224--9237.  doi:10.1109/TGRS.2020.3048967.
  * Sun et al. (2020) Sun, R., Sun, X., Chen, F., Pan, H., Song, Q., 2020\.  An artificial target detection method combining a polarimetric feature extractor with deep convolutional neural networks.  International Journal of Remote Sensing 41, 4995--5009.  doi:10.1080/01431161.2020.1727584.
  * Svendsen et al. (2018) Svendsen, D.H., Martino, L., Campos-Taberner, M., García-Haro, F.J., Camps-Valls, G., 2018.  Joint gaussian processes for biophysical parameter retrieval.  IEEE Transactions on Geoscience and Remote Sensing 56, 1718--1727.  doi:10.1109/TGRS.2017.2767205.
  * Tănase et al. (2017) Tănase, R., Bahmanyar, R., Schwarz, G., Datcu, M., 2017\.  Discovery of semantic relationships in polsar images using latent dirichlet allocation.  IEEE Geoscience and Remote Sensing Letters 14, 237--241.  doi:10.1109/LGRS.2016.2636663.
  * Wang et al. (2021) Wang, C., Su, W., Gu, H., 2021\.  A joint change detection method on complex-valued polarimetric synthetic aperture radar images based on feature fusion and similarity learning.  International Journal of Remote Sensing 42, 4864--4881.  doi:10.1080/01431161.2021.1899332.
  * Wen et al. (2021) Wen, Z., Liu, Z., Zhang, S., Pan, Q., 2021.  Rotation awareness based self-supervised learning for sar target recognition with limited training samples.  IEEE Transactions on Image Processing 30, 7266--7279.  doi:10.1109/TIP.2021.3104179.
  * Wu et al. (2018) Wu, Z., Xiong, Y., Yu, S.X., Lin, D., 2018.  Unsupervised feature learning via non-parametric instance discrimination, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  * Zhang et al. (2020) Zhang, J., Xing, M., Xie, Y., 2020.  FEC: A Feature Fusion Framework for SAR Target Recognition Based on Electromagnetic Scattering Features and Deep CNN Features.  IEEE Transactions on Geoscience and Remote Sensing , 1--14URL: https://ieeexplore.ieee.org/document/9127111/, doi:10.1109/tgrs.2020.3003264.
  * Zhao et al. (2020) Zhao, J., Zhang, Z., Yao, W., Datcu, M., Xiong, H., Yu, W., 2020.  OpenSARUrban: A Sentinel-1 SAR Image Dataset for Urban Interpretation.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 13, 187--203.  doi:10.1109/JSTARS.2019.2954850.


11institutetext: IRIT, Université de Toulouse, CNRS, Toulouse INP, UT3,
Toulouse, France 11email: {mael.madon,georges.da-costa,jean-
<EMAIL_ADDRESS>
# Characterization of different user behaviors for demand response in data
centers

Maël Madon  Georges Da Costa  Jean-Marc Pierson

###### Abstract

Digital technologies are becoming ubiquitous while their impact increases. A
growing part of this impact happens far away from the end users, in networks
or data centers, contributing to a rebound effect. A solution for a more
responsible use is therefore to involve the user. As a first step in this
quest, this work considers the users of a data center and characterizes their
contribution to curtail the computing load for a short period of time by
solely changing their job submission behavior.

The contributions are: (i) an open-source plugin for the simulator Batsim to
simulate users based on real data; (ii) the exploration of four types of user
behaviors to curtail the load during a time window namely delaying, degrading,
reconfiguring or renouncing to their job submissions. We study the impact of
these behaviors on four different metrics: the energy consumed during and
after the time window, the mean waiting time and the mean slowdown. We also
characterize the conditions under which the involvement of users is the most
beneficial.

###### Keywords:

Demand response User involvement User-aware Reproducible research Parallel
workload Data center

##  1 Introduction

Digital technologies are increasingly contributing to global warming for
instance through mining of their components, transport along their supply
chains or electricity consumed during their use phase. A recent review of
estimates [6] puts this impact at 1.0–1.7 GtCO2e in 2020, ie., 1.8%-2.8% of
global greenhouse gas emissions. Authors also argue that although progress in
energy efficiency of these technologies will probably continue, it will likely
be outbalanced by growth in usage, leading to an overall increase of the
carbon footprint. This so-called “rebound effect” seems difficult to fight
within our research area (scheduling and distributed computing) where the
focus is on energy optimization that must be effortless to end-users. On the
contrary, we argue that users of digital technologies must be brought back
into the loop, made aware of their impact and empowered to mitigate it.

Involving the user for environmental-aware scheduling in data centers has two
aspects. One is to consider user requests for more environment-friendly
services (eg., guarantees, green labels) and try to achieve them. The other is
to consider the users as a lever for flexibility in the scheduling, ie., they
accept to compromise occasionally on their quality of service to allow some
optimizations. The degradation can be spatial [9] (reducing the amount of
resources allocated for the jobs), temporal [15] (delaying their execution) or
both [8].

This paper proposes an experimental analysis of such user levers in a context
of demand response management by investigating the following question: from
the users’ perspective, what is the room for manoeuvre to curtail the load on
the data center for a short period of time?

The rest of the article is organized as follows. Section 2 presents our data
center model and lists the user behaviors studied for demand response. Section
3 describes the experimental setup for characterizing these behaviors. The
results are presented in Section 4 while Section 5 provides a discussion on
the results and the limitations of the study. Section 6 discusses the related
works. Finally, we conclude in Section 7 and provide perspectives for future
works.

##  2 Model

Data centers are viewed as good candidates to participate in demand response
programs [16]. Large consumers of electricity, they also have a more flexible
load than other industrial facilities. Demand response consists of adapting
the electricity consumption in response to the availability of production. For
example, some electricity markets have Coincident Peak Pricing programs where
industrial consumers are charged a very high price during the time window when
the most electricity is requested overall in the grid. These peak pricing
events last typically 15 minutes [17] or one hour [14] but are only known
afterwards, eg., at the end of the month. The electricity supplier would only
send warnings to the consumer that a peak load event may happen in the next
few hours.

In our model, a demand response event will be represented by a time window of
few hours (called “demand response window”) during which the objective is to
reduce electricity consumption. The event is supposed unknown in advance. In
order to characterize the efficiency of different user behaviors to react to
such demand response event, we consider a data center to which users can
submit their jobs. At the interface between the two is the RJMS (Resource and
Job Management System), the scheduler in charge of job placement and resource
management. In this section, we describe the different components of our
system.

###  2.1 Data center

In the data center, we only take into account the energy consumption of the
multi-core homogeneous machines. The power of a machine is $P_{off}$,
$P_{son}$ or $P_{soff}$ if the machine is switched off, switching on or
switching off, respectively. When a machine is switched on, its power is equal
to $P_{idle}+N*P_{core}$ with $P_{idle}$ the power drawn by an idle machine,
$N$ the number of cores in use (ie., with a job running on it) and $P_{core}$
the power drawn by each core.

A job is completely defined by its submission time, execution time and number
of requested cores that we denote by size in the rest of this paper. The
scheduler decides the starting time for the job and the machine it will be
executed on. Note that the scheduler in our model only execute jobs on single
machines. We suppose perfect communication without latency.

###  2.2 Scheduler

The scheduler is a bin-packing scheduler with shutdown (same as Guyon et al.
[8, 9]). It is a greedy algorithm trying to schedule (“pack”) all the jobs in
the least possible machines and shut down idle machines. To do so, it
maintains and updates two data structures: a queue of waiting jobs and a list
of switched-on machines. The queue of jobs is sorted by decreasing size order
– and by increasing submission time (first come first served) in case of a
tie. The list of machines is sorted by increasing order of available cores.
Every time one (or more) job is submitted or finishes, the scheduler goes
through the job queue in order and tries to find for each job the smallest
machine where it fits. If no machine is found that way, a new machine (if any)
is powered on and the job is scheduled on this machine. After that, we
immediately shut down all idle machines.

###  2.3 Users

During the demand response window, users are asked to make an effort to
curtail the load in the data center. They do so by adopting different
behaviors:

  * •

rigid: replay jobs as in the original workload; Baseline for comparison.

  * •

renounce: do not submit jobs originally submitted during the window.

  * •

delay: delay all job submissions to the end of the window.

  * •

degrad: divide the size of the jobs by two, rounded up. The execution time
stays the same. Note that the rounding implies that when only one core is
requested for a job, the job remains unchanged.

  * •

reconfig: also divide the size by two, rounded up, but increase the execution
time to match the original computing mass. We make the hypothesis of perfect
speedup, ie., a job executing on one core completes in exactly twice the time
than on two cores.

##  3 Experimental setup

###  3.1 Software used for simulation

To simulate our system, we use Batsim [4], an open-source infrastructure and
resource management system simulator111Batsim: https://batsim.org/ based on
SimGrid222SimGrid: https://simgrid.org with the energy plugin
https://simgrid.org/doc/latest/Plugins.html?highlight=energy#host-energy. We
implemented the bin-packing scheduler for Batsim in C++. We also developed a
plugin called “batmen” to interact with simulated users and receive their job
submissions. For the purpose of this study, users replay an input workload
trace except in the demand response window where they act according to their
behavior. We therefore implemented five user classes corresponding to the five
behaviors. Our code is open source333code repository:
https://gitlab.irit.fr/sepia-pub/mael/batmen. With this simulation tool, we
designed and conducted an experimental campaign whose main details are given
below444All scripts are available to reproduce our results:
https://gitlab.irit.fr/sepia-pub/open-science/demand-response-user.

###  3.2 Workload

We replay a real public workload trace containing the information about the
submitting user for each recorded job. We chose the 2-year trace from
MetaCentrum (national grid of the Czech republic), available in the Parallel
Workload Archive555METACENTRUM-2013-3.swf available at
https://www.cs.huji.ac.il/labs/parallel/workload/l˙metacentrum2/index.html.
The platform is very heterogeneous and underwent majors changes during the
logging period[12] so we perform the following selection:

  1. 1.

We truncate the workload to keep only 6 months (June to November 2014) where
no major change was performed in the infrastructure and we remove all the
clusters whose nodes have more than 16 cores;

  2. 2.

From this truncated workload, we remove all jobs with an execution time
greater than one day and all jobs with a size greater than 16. It leaves us
with a workload manageable with machines of a usual size, and without more
than one day of inertia.

###  3.3 Platform

The first selection step keeps a total of 6304 cores. The second selection
step exclude 2.7% of jobs from the truncated workload, representing 73.7% of
the mass (in core-hour). Consequently, we create a simulated platform adapted
to this load with $6304*(1-0.737)/16=$ 104 homogeneous 16-core machines. Power
constants ($P_{idle}=100W$, $P_{core}=7.3125W$, $P_{off}=9.75$, $P_{son}=100W$
and $P_{soff}=125W$) for the servers and time to switch on ($T_{son}=150s$)
and switch off ($T_{son}=6s$) are measurements in Taurus Grid’5000 cluster
from existing work [9].

###  3.4 Experimental campaign

Figure 1: Descriptive statistics for the 105 experiments. Red lines
corresponds to the infrastructure (1664 cores). (a) number of jobs submitted
in window; (b) computing mass (in core-hour) in window; (c) computing mass in
window by number of submitted jobs (1-hour window); (d) computing mass in
window by weekday (1-hour window)

We conducted an experimental campaign consisting of 105 experiments (the
number of weekdays between Jun 1, 2014 and Oct 23, 2014). For each experiment,
we vary the input workload corresponding to three full days of data center
operation. We make the demand response event arise at 16:00 on day 2, chosen
to be a weekday. This choice is justified by a characterization of 26 years’
coincident peak pricing data [14], given that the MetaCentrum trace also
displays diurnal and weekday/weekend patterns. We study two lengths for the
demand response window: one and four hours. We also tried other starting times
(drawn at random) and other window lengths (0.5 and 2 hours) but decided not
to report their results here as they are not leading to different conclusions.

The simulation starts one day before the event and stops one day after, to
ensure the infrastructure runs at nominal load on day 2 and has absorbed the
event by the end of day 3 (the selected jobs in the workload have an execution
time lower than one day). In each experiment we simulate the five user
behaviors on the two windows with the user behavior “rigid” as a baseline.
Descriptive statistics on the experiments are displayed in Fig. 1.

The campaign launched in parallel on a 2 x 8-core Intel Xeon E5-2630 v3
machine finished in less than two hours, based in France, and ran 2 times in
total, this has a carbon footprint of around 50 g CO2e (calculated using
green-algorithms.org v2.1 [13]).

##  4 Results

###  4.1 Energy metrics

Figure 2: Energy consumed in each simulation. Y-axis: energy consumed (in kWh)
during the demand response window. X-axis: computing mass (in core-hour)
during the demand response window for the baseline behavior.

We recall our research question: by intervening only on the user’s side, what
energy gains can be expected by adapting one’s behavior for a few hours? Fig.
2 displays the energy consumed during the demand response window for every
experiment and every behavior. Values are scattered by the total load of the
infrastructure during the window for the baseline behavior. We note an almost
linear relationship between infrastructure load and consumed energy.
Deviations from the linear line are due to situations favoring a more or less
good packing from the scheduler inside the 16-core machines. “renounce” and
“delay” behaviors perform identically for this metric: Users of both behaviors
stop submitting inside the demand response window, resulting in a lower energy
consumption compared to the baseline. This gain is the best we can expect.
Behaviors “degrad” and “reconfig” display similar results. In addition, one
would expect a positive correlation between the load of the platform and the
relative energy gains of the four behaviors compared to the baseline. It would
translate into an increasing distance between the colored dots and the blue
dots in the graphs, as the load increases. Counter-intuitively, this does not
seem to happen.

(a) 1-hour demand response window (b) 4-hour demand response window

Figure 3: Energy metrics per behavior relatively to the baseline behavior. The
green triangle in the box plots indicates the mean.

The experimental campaign showing very scattered results, Fig. 3 displays the
relative energy gains for each experiment as box plots. We can read for
example that “renounce”, the most radical behavior, allows energy savings of
up to 33% in the window for a one-hour window, and 53% for a four-hour window.
The savings do not go up to 100% because jobs that were already there before
the window are still running in the infrastructure, which consumes energy.

In addition to the energy consumed within the window, Fig. 3 shows the impact
of the different behaviors on the energy consumed after the demand response
event ie., from 17:00 or 20:00 on day2 (depending on the window length) to
24:00 on day3. For this second metric, “delay” performs very differently
compared to “renounce”. All the jobs within the window get postponed,
resulting in an extra power consumption at the end of the window: +0.3% (resp.
+3.4%) on average for a 1-hour (resp. 4-hour) window. This behavior remains
neutral with respect to overall energy consumption (within \+ after the
window).The behavior “reconfig”, which also keeps a constant mass of jobs
compared to the baseline, allows some optimizations. Up to 10% overall energy
consumption could be saved because the reconfigured jobs “fit better in the
holes” left by the available cores in the switched on machines. “Degrad”
performs unsurprisingly better in all respects, the users having accepted to
reduce the mass of job submitted.

Finally, we notice that the bigger the window, the better the energy gains.
This is due to inertia of the system: with a longer window, a behavior on the
submitted jobs has more time to make a difference compared to the residual
jobs that are still running in the infrastructure.

###  4.2 Perceived impact on the scheduling

(a) Mean waiting time (in seconds) (b) Mean slowdown (dimensionless)

Figure 4: Scheduling metric distribution for the 105 experiments, baseline
behavior.

We use two usual metrics: mean waiting time and mean slowdown. Waiting time is
the time a user has to wait until her job starts running:
$waiting_{time}=starting_{time}-submission_{time}$. Slowdown expresses this
extended completion time as a function of execution time:
$slowdown=(finish_{time}-submission_{time})/execution_{time}$. For each
experiment, we take the average waiting time (resp. slowdown) on all jobs
submitted between the beginning of the demand response window and the end of
the experiment (same period as metric energy_in + energy_after). Fig. 4 shows
these results for the “rigid” behavior. We observe that for half of the
experiments, the mean waiting time is below one hour (3600s) and the mean
slowdown below 25. These are experiments with an unsaturated infrastructure
and a queue of waiting job often empty. On the other hand, there are also
cases of high congestion (eg., the seven outliers at more than 6 hours mean
waiting time).

The results for the other behaviors are plotted in Fig. 5, as a percentage of
gain/loss compared to the baseline. Specifically for the behavior “delay”, we
provide both corrected and uncorrected metrics. The uncorrected slowdown and
waiting time are calculated in relation to the new (delayed) submission times
while the corrected ones use the original submission times (from the
baseline). Note also that for the behavior “renounce” some jobs have been
canceled, thus the mean waiting time and slowdown is calculated on a subset of
the jobs compared to the other behaviors. From Fig. 5 it can be observed that
the behaviors “renounce”, “degrad” and “reconfig” (in this order) affects the
scheduling positively on average. This is not surprising as the first two
behaviors reduce the total mass of jobs to compute and the third allows a
better packing. Yet, the scheduling gets worsened in a significant number of
cases (around 50% for “reconfig” and 25% for “degrad” and “renounce”), due to
bad choices of the scheduler.

The behavior “delay” stands out from the others as it affects the scheduling
negatively in most cases, even for the uncorrected metrics. It gets even worse
when including the extra waiting time from the delayed job in the calculation
of the corrected metrics. In fact, it is preferable in terms of waiting time
and slowdown that the job submissions are spread out throughout the time.

(a) 1-hour demand response window (b) 4-hour demand response window

Figure 5: Scheduling metrics per behavior relatively to the baseline behavior

##  5 Discussion

###  5.1 The fluid-residual ratio: an explanation of the results

Figure 6: Example of fluid and residual mass. (Thursday Jun 26 2014)

As seen previously in Fig. 2, the achievable energy savings in the demand
response window cannot be explained by the infrastructure load during that
window. In fact, it is possible that the load is very high because of a large
mass of job submitted before the window, although the load on which the users
have an influence is the mass submitted during the window. We call these two
quantities the residual mass, submitted outside of the window, and the fluid
mass, submitted inside the window (Fig. 6).

Figure 7: Energy gains in function of the fluid-residual ratio. Only one plot
for the behaviors “renounce” and “delay” because they are identical for this
metric.

Users, by accepting to “renounce to” or “delay” their jobs allow to cut the
energy consumption due to the fluid mass, which is roughly proportional to the
mass itself, as we saw before. In other terms, the gains during the window are
at most equal to the proportion of fluid mass in that window. This is exactly
what we see in Fig. 7 displaying the energy gains as a function of the fluid-
residual ratio. The red line indicates the best possible gains, which are
almost achieved by “renounce” and “delay” behavior (the non-linearity of the
energy model explaining the gap).

In some cases however, these behaviors don’t realize that gain: In the
saturation cases, when many jobs are waiting in the queue. The removal of the
fluid mass is compensated by the execution of the awaiting residual mass.

For the “degrad” behavior, gains are expected to be of half the fluid mass at
most, as users divide their submitted mass by two. In practice, the results
are even more scattered and further away from their optimal (magenta line).
This is partially due to the saturation effect, but also to rounding (eg., a
job with an original size of 3 will be submitted with size 2 and a job with
size 1 remains unchanged) and imperfect packing. The analysis for the behavior
“reconfig” is similar, with even less expected gains. Some experiments even
make negative gains: they are due to the greedy and non-clairvoyant scheduler
taking bad decisions for the future, like switching off a machine just before
the submission of new jobs.

###  5.2 Pros and cons of each behavior

The behavior renounce performs the best for all the metrics studied (summary
in Table 1). We saw that it actually reaches the optimal energy gains during
the window for unsaturated cases. This rank is not surprising considering the
sacrifice required from the user. Yet, we think that such a behavior is often
overlooked in similar studies and argue that environmentally aware users or
users provided with a proper incentive would do it. Moreover, some jobs
running in data centers today might not be indispensable.

On the other end of the spectrum, the behavior reconfig seems to be the most
acceptable to the users as it does not decrease the mass initially submitted
and provides better waiting time and slowdown than “delay” for both the jobs
within and after the window. “Reconfig” is a good trade-off to achieve some
optimizations with a low effort from the user, especially in combination with
bin-packing schedulers and on/off policies (see [9]).

Delay also keeps the mass constant, which ranks it second behind “reconfig” in
terms of acceptability. Same as “renounce”, it reaches the optimal energy
gains during the window. However, it introduces an overhead in overall energy
consumption and slowdown compared to the baseline behavior. Note that this
overhead would probably be less important in real life due to users adapting
their behavior if they experience congestion in the infrastructure. This is
the limit of blindly replaying past workload traces in simulations, as pointed
by Feitelson [5].

Finally, the behavior degrad ranks second or third in all the categories of
Table 1. It remains an interesting trade-off between simply renouncing to a
job and reconfiguring it at constant mass. We can think about optional
features in an application that can be cut off if needed (eg., recommendations
for e-commerce, alternative paths for mapping apps).

behavior | energy in | energy overall | scheduling metrics | acceptability
---|---|---|---|---
renounce | 1st | 1st | 1st | 4th
delay | 1st | 4th | 4th | 2nd
degrad | 3rd | 2nd | 2nd | 3rd
reconfig | 4th | 3rd | 3rd | 1st
Table 1: Summary ranking of the four behaviors with regards to their impact on
energy consumption and scheduling metrics. The column “acceptability” is
opinion-based, it reflects the size of the effort asked from the user.

All in all, having different user behaviors remains one lever for energy
saving among others. It has the particularity of having some latency, which
makes it not optimal in a context of demand response without prediction.
Taking into account these behaviors inside the scheduler seems essential to
make the best of their potential and go beyond the fluid-residual limit. For
example in combination with malleable applications [3] or contracts with the
data center operator specifying the degradation the user is willing to accept
[2]. Nevertheless, latency is not critical in other contexts, and involving
the user appears as the main path towards a sufficient [11] use of our
technologies, if not the only.

###  5.3 Limitations

##### Model simplifications

In our data center simulations, we do not take into account the latency and
bottleneck effects in the communications. Also, we suppose perfect speedup in
the model, ie., a job executed on two cores will take exactly twice longer
than the same job executed on four cores. Finally, we accounted only for the
energy consumption of the CPUs, and neglected others like memory, network or
cooling. Hopefully, the powerful simulation tools that we use (Batsim and
Simgrid) will help us to overcome these simplifications in future works.

##### Methodological limitations

We see three major threats to the validity of our method to answer the
research question. First, we study only one scheduler (bin-packing) while
results with other common schedulers (FCFS, easy-backfilling…) would have been
of interest. Second, we use only one input trace (MetaCentrum) which comes
from a research infrastructure and not a production cloud, and we perform a
selection from it (see 3.2) that might make us miss the big picture. Finally,
our study includes all the limitations related to the use of a simulation,
especially when dealing with human behaviors which are unpredictable.

##  6 Related works

Among the large body of work on energy-aware scheduling in data centers, some
authors have studied strategies involving the users. Some works aim at
providing guarantees to their users (“green offers” [7], “green SLA” [10, 1])
and commit to fulfilling them by classical methods (self-supply of renewable
energy [10], geo-distributed data centers with variable PUE and energy mix [7,
1]).

More related to this paper, some works study user flexibility as a lever for
energy efficiency. For example Guyon et al. [9] give to the users the choice
between three execution modes (big, medium, little) for their jobs. Small
execution modes request fewer resources but take longer to complete. They
achieve gains through spatial consolidation with a bin-packing algorithm.
Orgerie et al. [15] save energy through thermal-aware scheduling and smart
resource switch off by letting the users choose between different submission
times on the basis of energy consumption estimations for each of the
alternatives. A combination of both spatial and thermal consolidation is
proposed in an other work by Guyon et al. [8] or in the All4Green project [2],
where user involvement is leveraged through contracts between the energy
supplier, the data center and the user. The latter work, also in a context of
demand response, is the closest to our approach. However, it integrates demand
response mechanisms affecting the user with mechanisms transparent to them
(use of batteries, precooling, geographical workload migration) so much so
that the contribution of each user behavior to the final results is difficult
to identify.

The originality of our work is to focus on the user behaviors which allows to
provide a characterization of them. To the best of our knowledge, we are also
the only ones to consider the behavior of simply renouncing to job
submissions. It is a radical behavior but to be considered in a sufficiency
approach.

##  7 Conclusion and future works

In this paper, we study four different ways for a user of a data center to
curtail her load for a certain period of time by changing submission behavior.
These behaviors are delaying, degrading, reconfiguring or renouncing to the
jobs during the time period. We show experimentally through simulation on real
world data that these behaviors have a certain latency for decreasing the load
on the infrastructure. Indeed they cannot decrease the load due to jobs that
are already running on the infrastructure. Therefore, we define two
quantities, the fluid and residual mass, and discuss the experimental results
according to the ratio of these two quantities. We also discuss the pros and
the cons of each behavior to the light of their energy saving potential,
impact on scheduling and acceptability to the user. We hope that this work
will pave the way for studies involving the user more intensely.

Future work will focus on (i) improving the data center model to deal with the
model simplifications listed in Subsection 5.3, (ii) proposing schedulers
capable of leveraging the efforts made by the user (eg., through “green SLA”),
(iii) elaborating on the user model to more realistically account for
submission patterns and response to feedback from the infrastructure (as
proposed by Feitelson [5]) and (iv) going beyond the limited scope of demand
response to reason on the sustainability of the infrastructure as a whole.

####  7.0.1 Acknowledgements

Experiments presented in this paper were carried out using the Grid’5000
testbed, supported by a scientific interest group hosted by Inria and
including CNRS, RENATER and several Universities as well as other
organizations (see https://www.grid5000.fr). This work was partly supported by
the French Research Agency under the project Energumen (ANR-18-CE25-0008) and
DataZero2 (ANR-19-CE25-0016).

## References

  * [1] Amokrane, A., Langar, R., Zhani, M.F., Boutaba, R., Pujolle, G.: Greenslater: On Satisfying Green SLAs in Distributed Clouds. IEEE Transactions on Network and Service Management 12(3), 363–376 (Sep 2015). https://doi.org/10.1109/TNSM.2015.2440423
  * [2] Basmadjian, R., Botero, J.F., Giuliani, G., Hesselbach, X., Klingert, S., De Meer, H.: Making Data Centers Fit for Demand Response: Introducing GreenSDA and GreenSLA Contracts. IEEE Transactions on Smart Grid 9(4), 3453–3464 (Jul 2018). https://doi.org/10.1109/TSG.2016.2632526
  * [3] Dupont, B., Mejri, N., Da Costa, G.: Energy-aware scheduling of malleable HPC applications using a Particle Swarm optimised greedy algorithm. Sustainable Computing: Informatics and Systems 28, 100447 (Dec 2020). https://doi.org/10.1016/j.suscom.2020.100447
  * [4] Dutot, P.F., Mercier, M., Poquet, M., Richard, O.: Batsim: A Realistic Language-Independent Resources and Jobs Management Systems Simulator. In: 20th Workshop on Job Scheduling Strategies for Parallel Processing. Chicago, United States (May 2016)
  * [5] Feitelson, D.G.: Resampling with Feedback — A New Paradigm of Using Workload Data for Performance Evaluation. In: Dutot, P.F., Trystram, D. (eds.) Euro-Par 2016: Parallel Processing. pp. 3–21. Lecture Notes in Computer Science, Springer International Publishing, Cham (2016). https://doi.org/10.1007/978-3-319-43659-3_1
  * [6] Freitag, C., Berners-Lee, M., Widdicks, K., Knowles, B., Blair, G.S., Friday, A.: The real climate and transformative impact of ICT: A critique of estimates, trends, and regulations. Patterns 2(9) (Sep 2021). https://doi.org/10.1016/j.patter.2021.100340
  * [7] Garg, S.K., Yeo, C.S., Buyya, R.: Green Cloud Framework for Improving Carbon Efficiency of Clouds. In: Jeannot, E., Namyst, R., Roman, J. (eds.) Euro-Par 2011 Parallel Processing. pp. 491–502. Lecture Notes in Computer Science, Springer, Berlin, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23400-2_45
  * [8] Guyon, D., Orgerie, A.C., Morin, C.: Energy - Efficient IaaS-PaaS Co-Design for Flexible Cloud Deployment of Scientific Applications. In: 2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD). pp. 69–76 (Sep 2018). https://doi.org/10.1109/CAHPC.2018.8645888
  * [9] Guyon, D., Orgerie, A.C., Morin, C., Agarwal, D.: Involving users in energy conservation: A case study in scientific clouds. International Journal of Grid and Utility Computing 10(3), 272–282 (Jan 2019). https://doi.org/10.1504/IJGUC.2019.099667
  * [10] Haque, M.E., Le, K., Goiri, Í., Bianchini, R., Nguyen, T.D.: Providing green SLAs in High Performance Computing clouds. In: 2013 International Green Computing Conference Proceedings. pp. 1–11 (Jun 2013). https://doi.org/10.1109/IGCC.2013.6604503
  * [11] Hilty, L.: Computing Efficiency, Sufficiency, and Self-sufficiency: A Model for Sustainability? In: LIMITS 2015, First Workshop on Computing within Limits. s.n., Irvine, CA, USA (Jun 2015). https://doi.org/10.5167/uzh-110766
  * [12] Klusáček, D., Tóth, Š., Podolníková, G.: Real-Life Experience with Major Reconfiguration of Job Scheduling System. In: Desai, N., Cirne, W. (eds.) Job Scheduling Strategies for Parallel Processing. pp. 83–101. Lecture Notes in Computer Science, Springer International Publishing, Cham (2017). https://doi.org/10.1007/978-3-319-61756-5_5
  * [13] Lannelongue, L., Grealey, J., Inouye, M.: Green Algorithms: Quantifying the Carbon Footprint of Computation. Advanced Science 8(12), 2100707 (2021). https://doi.org/10.1002/advs.202100707
  * [14] Liu, Z., Wierman, A., Chen, Y., Razon, B., Chen, N.: Data center demand response: Avoiding the coincident peak via workload shifting and local generation. Performance Evaluation 70(10), 770–791 (Oct 2013). https://doi.org/10.1016/j.peva.2013.08.014
  * [15] Orgerie, A., Lefèvre, L., Gelas, J.: Save Watts in Your Grid: Green Strategies for Energy-Aware Framework in Large Scale Distributed Systems. In: 2008 14th IEEE International Conference on Parallel and Distributed Systems. pp. 171–178 (Dec 2008). https://doi.org/10.1109/ICPADS.2008.97
  * [16] Wierman, A., Liu, Z., Liu, I., Mohsenian-Rad, H.: Opportunities and challenges for data center demand response. In: International Green Computing Conference. pp. 1–10 (Nov 2014). https://doi.org/10.1109/IGCC.2014.7039172
  * [17] Zarnikau, J., Thal, D.: The response of large industrial energy consumers to four coincident peak (4CP) transmission charges in the Texas (ERCOT) market. Utilities Policy 26, 1–6 (Sep 2013). https://doi.org/10.1016/j.jup.2013.04.004