# Towards Instance-adaptive Inference for Federated Learning

Chun-Mei Feng1 Kai Yu1∗ Nian Liu2 Xinxing Xu1 Salman Khan2,3 Wangmeng Zuo4
1Institute of High Performance Computing (IHPC),
Agency for Science, Technology and Research (A*STAR), Singapore
2Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE
3Australian National University, Canberra ACT, Australia
4Harbin Institute of Technology, Harbin, China
<EMAIL_ADDRESS>yu<EMAIL_ADDRESS><EMAIL_ADDRESS>
https://github.com/chunmeifeng/FedIns Corresponding author.

###### Abstract

Federated learning (FL) is a distributed learning paradigm that enables
multiple clients to learn a powerful global model by aggregating local
training. However, the performance of the global model is often hampered by
non-i.i.d. distribution among the clients, requiring extensive efforts to
mitigate inter-client data heterogeneity. Going beyond inter-client data
heterogeneity, we note that intra-client heterogeneity can also be observed on
complex real-world data and seriously deteriorate FL performance. In this
paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client
data heterogeneity by enabling instance-adaptive inference in the FL
framework. Instead of huge instance-adaptive models, we resort to a parameter-
efficient fine-tuning method, i.e., scale and shift deep features (SSF), upon
a pre-trained model. Specifically, we first train an SSF pool for each client,
and aggregate these SSF pools on the server side, thus still maintaining a low
communication cost. To enable instance-adaptive inference, for a given
instance, we dynamically find the best-matched SSF subsets from the pool and
aggregate them to generate an adaptive SSF specified for the instance, thereby
reducing the intra-client as well as the inter-client heterogeneity. Extensive
experiments show that our FedIns outperforms state-of-the-art FL algorithms,
e.g., a $6.64\%$ improvement against the top-performing method with less than
$15\%$ communication cost on Tiny-ImageNet.

##  1 Introduction

The availability of large-scale data dramatically promotes the development of
deep learning models. However, as these abundant data tend to be distributed
across many devices due to logistical and privacy concerns, decentralized
training is often required to train the deep neural network [1]. As a
promising distributed learning paradigm, federated learning (FL) can train
global models in a distributed manner without sharing local private data [25,
30, 29]. During this process, each client first trains a local model on their
private data and then sends the model parameters to the server for aggregation
and distribution back to the client [30].

Figure 1: Illustration of inter- and intra-client data heterogeneity with
t-SNE visualizations (see (a) and (b)) on DomainNet and their effect on
accuracy of different FL algorithms (see (c) and (d)). Please refer to the
_Suppl._ for details on the settings to increase data heterogeneity. One can
see that both inter- and intra-client data heterogeneity degrades FL
performance. While FedBN [27] is able to alleviate the effect of inter-client
data heterogeneity, both FedAvg [30] and FedBN [27] are limited in handling
intra-client data heterogeneity.

However, each client collects the local data in its own manner (_e.g_.,
different devices, different ways). Consequently, the data distribution among
different clients is heterogeneous (see Fig. 1 (a)), resulting in serious
performance degradation of conventional algorithms [4] (_e.g_., FedAvg [30]).
Many FL algorithms have been suggested to solve the problem of inter-client
data heterogeneity for making them more robust in the non-i.i.d. setting [25,
16, 27, 24, 9] (see Fig. 1 (c)).

Actually, even for an individual client, the data may be collected by
different devices [7], under different environmental conditions, _etc_.
Therefore, intra-client data heterogeneity can also be observed on complex
real-world data [3] (see Fig. 1 (b)), and give rise to serious degradation of
FL performance (see Fig. 1 (d)). A naive solution is to build a specific model
for each instance [42]. However, such a scheme results in significant
challenges in the training, storage and communication of the instance-adaptive
models. Instead, we resort to the parameter-efficient fine-tuning, _e.g_.,
prompt tuning [14], of pre-trained models. As for prompt tuning, we can simply
freeze the backbone of pretrained models, only learn and communicate a small
number of learnable prompts between the server and the client [6, 36, 35, 14].
Nonetheless, instance-wise modeling remains an unstudied issue for parameter-
efficient fine-tuning. Additionally, in comparison to the original pre-trained
model, prompt tuning also introduces additional parameters and increases the
computation cost in the inference stage [28].

To handle the intra-client data heterogeneity, instead of instance-adaptive
models, this paper resorts to instance-adaptive inference, and presents a
novel FL algorithm, _i.e_., FedIns, built upon pre-trained models. For
parameter-effective fine-tuning, following [28], we scale and shift the deep
features (SSF) to fine-tune the pre-trained model in the local training phase
and merge them into the original pre-trained model weights by
reparameterization in the inference phase. To enable instance-adaptive
inference, we train a SSF pool for each client and aggregate them on the
server, and thus low storage and communication costs can still be maintained.
For example, compared with FedAvg [30], FedIns only has less than $15\%$
communication costs. Our federated SSF pool is aggregated from multiple groups
of local SSF, which implicitly add client-relevant knowledge to the federated
SSF pool and dynamically guide the client to handle the corresponding response
in an instance-wise fashion. During inference, for a given instance, we
dynamically find the best-matched SSF subsets from the pool and aggregate them
to generate an adaptive SSF specified for the instance. As such, the model can
reduce both inter- and intra-client heterogeneity, achieving a $6.64\%$ gain
against FedAvg on CIFAR-100.

In a nutshell, our contributions are three-fold:

  * •

A novel FL algorithm, _i.e_., FedIns, is presented to handle intra-client data
heterogeneity, which has been overlooked by the existing FL literature.

  * •

Federated SSF is proposed and extended by allowing each client to have an SSF
pool, and instance-adaptive inference is fulfilled by dynamically finding and
aggregating the best matched SSF subsets for each test instance.

  * •

Extensive experimental results show FL performance can be effectively improved
by alleviating the intra- and inter-client data heterogeneity.

##  2 Related Work

Data Heterogeneity in FL. Generally, federated learning algorithm aims to
obtain an aggregated model that minimizes training losses for all clients. The
classic FL algorithm, FedAvg [30] simply sends the local model to the server
for aggregation to learn a global model. However, as each device generates its
own local data, data heterogeneity across different clients occurs, making
FedAvg becomes a sub-optimal solution for FL [26, 40, 20, 8].

Existing FL methods usually alleviate this problem in two aspects. One is to
improve local training. For example, FedProx adds a proximal term to the
objective function of the local model to tackle heterogeneity [25].
Karimireddy et al. used control variates to correct the client-drift caused by
data heterogeneity [16]. Li et al. kept all the batch normalization in the
local to alleviate the heterogeneity of local data across clients [27].
Inspired by contrastive learning, Li et al. [24] corrected the local clients
by computing similarity between model representations to handle the
heterogeneity issue. Gao et al. used an auxiliary local drift variable to
bridge the gap between the local and the global model parameters, thereby
alleviating the data heterogeneity [9]. Mendieta et al. alleviated data
heterogeneity by promoting local learning generality rather than proximal
restriction [31]. The non-i.i.d. problem caused by data heterogeneity can also
be alleviated by addressing catastrophic forgetting from the server to the
client, where each local and global communication is regarded as a new task
[39, 19].

Besides improving local training, another is to improve the server aggregation
process for alleviating data heterogeneity. For example, Yurochkin et al.
replaced classical aggregation schemes by matching neuron aggregation in local
models based on a Bayesian non-parametric approach [41]. Analogously, Wang et
al. created a normalized averaging method as an alternative to the average
update mechanism [38]. To sum up, to handle the data heterogeneity across
clients, existing methods either limit the impact of local updates on the
server (_e.g_., by regularizing and personalizing the design of clients to
correct the update direction of locals [25, 16, 27, 24, 9, 31]), or modify the
aggregation scheme [41, 38].

However, in many real-world applications, local data in each client may be
collected by different devices and from different environments. Consequently,
there may be multiple mixed unknown sub-domains within one client [3], _i.e_.,
intra-client data heterogeneity. As discussed in Sec. 1, both intra-client and
inter-client data heterogeneity will lead to the performance degradation of
FL. Nonetheless, existing methods mainly focus on inter-client data
heterogeneity, leaving intra-client heterogeneity remain an uninvestigated
issue. Given this, this work presents a novel FedIns algorithm to handle
intra-client data heterogeneity, which can also be readily deployed to
alleviate inter-client data heterogeneity.

Parameter-efficient Fine-tuning. In the recent few years, we have witnessed
the promising performance of pre-training models in many downstream tasks.
Accordingly, in the field of FL [32, 2], pre-trained models can also serve as
a strong baseline. However, the number of parameters of pre-trained models is
usually very large [37, 11], and simply fine-tuning the full model undoubtedly
yields a huge amount of communication cost in FL algorithms. Therefore, some
works have attempted to explore large-scale pre-training models in a
parameter-efficient fine-tuning manner [17]. In this way, the backbone network
is frozen during training, and only a small number of parameters can be
learned or tuned to “understand” downstream tasks. In particular, prompt [21,
14, 22], adapter [13, 12, 33], and SSF [28] have been suggested to leverage
the representation abilities of large-scale pre-training models to achieve
good performance on downstream tasks by fine-tuning a few trainable
parameters.

Among these parameter-efficient fine-tuning methods, most prompt and adapter-
based methods [13, 15] introduce additional parameters and computational costs
in the inference stage. In addition, prompt tuning is sensitive to the number
of prompts; _e.g_., different tasks need to adopt different numbers of
prompts, and an inappropriate number will reduce the accuracy or increase the
redundancy of the calculation. In contrast, SSF [28] does not bring extra
parameters and FLOPs during the inference phase since it only adds learnable
parameters during the training phase and merges them into the original pre-
trained model weights via reparameterization after training. Despite these
progress, it remains not studied for the effectiveness of SSF in a distributed
framework with intra- and inter-client data heterogeneity. Therefore,
different from the above centralized works, in this paper we focus on
exploring how to make SSF work for decentralized FL frameworks to effectively
handle data heterogeneity.

$\bm{\delta}^{z}$$\bm{\delta}_{g}^{z+1}$${\mathbf{FI}}^{z}$${\mathbf{FI}}_{g}^{z+1}$

Figure 2: Overall pipeline of our proposed FedIns. (a) Pre-train the backbone
on the massive data offline and fine-tune a small number of learnable
parameters on local data in a parameter-efficient manner. (b) For federated
SSF, we train an SSF for each client and aggregate the SSFs from all clients
to obtain the global SSF (see Sec. 3.2). (c) As for FedIns, we train a
federated SSF pool for each client and aggregate the SSF pools from all
clients to obtain the global SSF pool (see Sec. 3.3).

##  3 Methodology

###  3.1 Federated Learning

As growing privacy concerns arise, federated learning has received intensive
recent interest by training and deploying deep neural network models in a
distributed manner [30]. Suppose there are $K$ local clients, where each of
them has their local dataset $\mathcal{D}^{k}$. The distributed paradigm FL
aims to learn a global model $\bm{w}$ over the whole training data
$\mathcal{D}=\left\\{\mathcal{D}^{1},\mathcal{D}^{2},\ldots,\mathcal{D}^{K}\right\\}$
using a central server without exchanging local private data. Formally, such a
process can be expressed as

${\arg\underset{\bm{w}}{\min}}\mathcal{L}(\bm{w})=\sum_{k=1}^{K}\frac{|\mathcal{D}^{k}|}{|\mathcal{D}|}\mathcal{L}_{k}(\bm{w}),$
(1)

where $|\mathcal{D}|$ denotes the number of samples in $\mathcal{D}$, and
$\bm{w}$ denotes the model parameters. $\mathcal{L}_{k}(\bm{w})$ is the
empirical loss of client $k$ which can be expressed as

$\mathcal{L}_{k}(\bm{w})=\mathbf{E}_{\left(\mathbf{x},\mathbf{y}\right)\in\mathcal{D}^{k}}\ell_{k}\left(\mathbf{x};\bm{w}\right),\vspace{-1pt}$
(2)

where $\ell_{k}$ denotes the local loss term, _e.g_., cross-entropy loss, and
$\mathbf{x}$ denote a sample of client $k$.

###  3.2 Learning Federated SSF

In federated learning, the local data $\mathcal{D}^{k}$ of a client may not be
sufficient to train a large scale deep network. Pre-trained models can thus be
introduced to compensate for the deficiency of local data [32, 2]. However,
pre-trained model usually has a large number of model parameters. Direct fine-
tuning the full model consequently gives rise to significant communication
costs between the server and the client. In this work, we adopt a recent
parameter-efficient fine-tuning paradigm, _i.e_., SSF [28], which trains only
a small number of learnable parameters and brings no inference overhead by
reparameterizing them into the original pre-trained model weights. In the
following, we will introduce federated SSF, which incorporates SSF into the FL
framework.

In SSF, given a pre-trained model with parameters $\bm{\theta}$, we remodulate
features by insert SSF with the scale $\bm{\gamma}$ and shift $\bm{\beta}$
factors after each operation (OP) [28], _i.e_., multi-head self-attention
(MSA), MLP and layer normalization (LN), _etc_. During the fine-tuning phase,
the model parameters of SSF can be represented as
$\bm{w}\\!=\\!\\{\bm{\gamma},\bm{\beta},\bm{h},\bm{\theta}\\}$, where $\bm{h}$
is the parameters of the classification head. In particular, the pre-trained
weights are kept frozen, and only the SSF and classification head are kept
updated. Once the training is accomplished,
$\\{\bm{\gamma},\bm{\beta},\bm{h}\\}$ can then be merged into $\bm{\theta}$ to
obtain the updated model parameter $\bm{\theta}^{\prime}$.

As shown in Fig. 2 (b), when incorporating SSF into FL, we only require to
update the client-specific SSF and classification head, _i.e_.,
$\bm{\delta}=\\{\bm{\gamma},\bm{\beta},\bm{h}\\}$. Thus, Eq. (1) can be
modified as,

$\bm{\delta}_{g}={\arg}~{}\underset{\bm{\delta}}{\min}\mathcal{L}(\bm{\delta})=\sum_{k=1}^{K}\frac{|\mathcal{D}^{k}|}{|\mathcal{D}|}\mathcal{L}_{k}(\bm{\delta}).$
(3)

Federated SSF only requires a small number of parameters $\bm{\delta}_{k}$ of
local clients to be updated and communicated with the server, and thus is both
parameter- and communication-efficient for FL.

Local Update Step: Assume there are $Z$ rounds of communication with $T$ local
updates per round. The clients are optimized using the following update rules
with a learning rate of $\eta_{k}$ for each communication round
$z=\left\\{1,2,...,Z\right\\}$:

$\bm{\bm{\delta}}_{{k}}^{z,t+1}\leftarrow\bm{\bm{\delta}}_{{k}}^{z,t}-\eta_{k}\nabla\ell_{k}\left(\mathbf{x}^{k};\bm{\bm{\delta}}_{{k}}^{z,t}\right),$
(4)

where $t$ denotes the $t$-th update of the local clients.

Server Update Step: The server performs aggregation every round by receiving
the updated parameters of all participated clients after the local updates
within each round. Formally, we have

$\bm{\bm{\delta}}_{{g}}^{z+1}\leftarrow\sum_{k=1}^{K}\frac{|\mathcal{D}^{k}|}{|\mathcal{D}|}\bm{\bm{\delta}}_{{k}}^{z},\vspace{-4pt}$
(5)

where $\bm{\bm{\delta}}_{{g}}^{z+1}$ denotes the global updated parameters of
round $z+1$. Then, we can obtain a robust global model parameterized by
$\bm{\bm{\delta}}_{{g}}$ after $Z$ rounds of communication without disclosing
any local private data.

When the training is accomplished, we can re-parameterize the SSF by merging
it into the original parameter space (_i.e_., model weight $\bm{\theta}$). As
a result, federated SSF is not only efficient in terms of communication costs,
but also does not introduce any extra parameters during the inference phase.

###  3.3 FedIns

In this subsection, we further present FedIns for handling intra-client data
heterogeneity. In many complex real-world scenarios, the data in a local
client may contain multiple unknown mixed sub-domains [3]. Despite its merit
on communication cost, federated SSF is still limited in alleviating the
intra-client data heterogeneity issue. To address this issue, we extend
federated SSF to SSF pool for enabling instance-adaptive inference in the FL
framework, resulting in our FedIns algorithm. As shown in Fig. 2 (c), we train
an SSF pool for each client, and aggregate them into the federated SSF pool on
the server. For a given instance, we dynamically find the best-matched SSF
subsets from the pool and aggregate them to generate an adaptive SSF for
instance-adaptive inference. In the following, we will introduce FedIns in
more detail.

$\bm{\bm{q}}_{k}(\mathbf{x})$$\mathbf{k}_{k}^{1}$$\mathbf{k}_{k}^{2}$$\mathbf{k}_{k}^{3}$$\mathbf{k}_{k}^{M}$$\bm{\bm{\delta}}(\mathbf{x})$

Figure 3: Illustration of instance-adaptive inference. For a given instance,
we use the pre-trained model to generate a query, and then dynamically select
the $C$ best-matched SSFs, which are then aggregated to form an instance-
adaptive SSF for enabling instance-adaptive inference.

To extend federated SSF, we allow each client have an SSF pool
$\bm{\bm{\Delta}}_{k}=\left\\{\bm{\bm{\delta}}_{k}^{1},\bm{\bm{\delta}}_{k}^{2},\cdots,\bm{\bm{\delta}}_{k}^{M}\right\\}$
of a set of $\bm{\bm{\delta}}_{k}^{m}$s, where $M$ denotes the size of pool.
Furthermore, we introduce a pool of learnable Keys
$\mathbf{K}_{k}=\left\\{\mathbf{k}_{k}^{1},\mathbf{k}_{k}^{2},\cdots,\mathbf{k}_{k}^{M}\right\\}$
corresponding to $\bm{\bm{\Delta}}_{k}$. Taking both $\bm{\bm{\Delta}}_{k}$
and $\mathbf{K}_{k}$ into account, the learnable parameters of FedIns of each
client can be expressed as,

$\mathbf{FI}_{k}=\left\\{\mathbf{K}_{k},\bm{\bm{\Delta}}_{k}\right\\}.$ (6)

Then, the local update in FedIns can be written as,

$\mathbf{FI}_{{k}}^{z,t+1}\leftarrow\mathbf{FI}_{{k}}^{z,t}-\eta_{k}\nabla\ell_{k}\left(\mathbf{x}^{k};\mathbf{FI}_{{k}}^{z,t}\right).\vspace{-2pt}$
(7)

After $T$ local updates, the server update can be expressed as,

${\mathbf{FI}}_{g}^{z+1}\leftarrow\sum_{k=1}^{K}\frac{|\mathcal{D}^{k}|}{|\mathcal{D}|}{\mathbf{FI}}_{k}^{z}.\vspace{-1pt}$
(8)

After $Z$ rounds of communication, we can obtain the global model of FedIns,
which is also parameterized by an SSF pool $\bm{\bm{\Delta}}_{g}$ and a set of
keys $\mathbf{K}_{g}$. The detailed FedIns algorithm is given in Algorithm 1.
In contrast to federated SSF, during the inference phase, FedIns allow to
dynamically find the best matched SSF subsets from the SSF pool for a given
instance, thereby making instance-adaptive inference feasible.

1

Input: Local datasets of $K$ clients:
$\mathcal{D}^{1},\mathcal{D}^{2},\ldots,\mathcal{D}^{K}$, local updates $T$,
communication rounds $Z$, pre-trained model parameters $\bm{\theta}$,
learnable parameters
${\mathbf{FI}_{k}}=\left\\{\mathbf{K}_{k},\bm{\bm{\Delta}}_{k}\right\\}$,
learning rate $\eta$, hyperparameter $c$, and $M$;

Output: The final global model $\bm{w}$;

2 // ServerExecution:

3 Initialize global SSF pool
$\bm{\bm{\Delta}}_{k}=\left\\{\bm{\bm{\delta}}_{k}^{1},\bm{\bm{\delta}}_{k}^{2},\cdots,\bm{\bm{\delta}}_{k}^{M}\right\\}$
with Keys of
$\mathbf{K}_{k}=\left\\{\mathbf{k}_{k}^{1},\mathbf{k}_{k}^{2},\cdots,\mathbf{k}_{k}^{M}\right\\}$;

4for  _each communication round $z\in\left\\{1,2,...,Z\right\\}$_ do

5 for  _each client $k\in\left\\{1,2,...K\right\\}$ in parallel_ do

6 ${\mathbf{FI}}_{k}^{z}\leftarrow{\mathbf{FI}}_{g}^{z}$;

7 Local updating with regard to each input instance $\mathbf{x}$:
${\mathbf{FI}}_{g}^{z,t+1}\leftarrow\text{LocalUpdate}(k,\mathbf{x},{\mathbf{FI}}_{g}^{z,t})$;

8 end for

9

10
${\mathbf{FI}}_{g}^{z+1}\leftarrow\sum_{k=1}^{K}\frac{|\mathcal{D}^{k}|}{|\mathcal{D}|}{\mathbf{FI}}_{k}^{z}$;

11

12 end for

13return ${\mathbf{FI}}_{g}^{z+1}$

14// LocalUpdate ($k$, $\mathbf{x}$, $\mathbf{FI}_{{k}}^{z,t}$):

15for  _each local epoch $t\in\left\\{1,2,...T\right\\}$_ do

16
$\mathbf{FI}_{{k}}^{z,t+1}\leftarrow\mathbf{FI}_{{k}}^{z,t}-\eta_{k}\nabla\ell_{k}\left(\mathbf{x}^{k};\mathbf{FI}_{{k}}^{z,t}\right)$;

17

18 end for

return $\mathbf{FI}_{{k}}^{z,t+1}$

Algorithm 1 FedIns

###  3.4 Instance-Adaptive Inference

When the training of FedIns is accomplished, the learned federated SSF pool
$\bm{\bm{\Delta}}_{g}$ and the corresponding global Keys $\mathbf{K}_{g}$ will
be distributed to each client. As shown in Fig. 3, to enable instance-adaptive
inference using FedIns, we resort to generating an instance-adaptive SSF from
$\bm{\bm{\Delta}}_{g}$ and $\mathbf{K}_{g}$ during the inference phase.

To this end, for a given instance $\mathbf{x}$ of a client, we first use the
pre-trained model to generate a query $\bm{\bm{q}}_{k}(\mathbf{x})$ which has
the same dimension as the keys. Based on the cosine similarity between
$\bm{\bm{q}}_{k}(\mathbf{x})$ and each keys in $\mathbf{K}_{g}$, we select the
$C$ best-matched SSFs, _i.e_.,
$\left\\{\bm{\bm{\delta}}_{g}^{m_{1}},\bm{\bm{\delta}}_{g}^{m_{2}},\cdots,\bm{\bm{\delta}}_{g}^{m_{C}}\right\\}$
from $\bm{\bm{\Delta}}_{g}$ corresponding to the $C$ most similar keys. The
instance-adaptive SSF of $\mathbf{x}$ can then be given by,

$\bm{\bm{\delta}}(\mathbf{x})=\sum_{c=1}^{C}\frac{1}{C}\bm{\bm{\delta}}_{g}^{m_{c}}.$
(9)

We note that $\bm{\bm{\delta}}(\mathbf{x})$ is determined by $\mathbf{x}$ and
is instance-adaptive. Thus, instance-adaptive inference can be fulfilled by
merging $\bm{\bm{\delta}}(\mathbf{x})$ into $\bm{\theta}$ to obtain the
instance-adaptive model parameter $\bm{\theta}^{\prime}(\mathbf{x})$.

##  4 Experiments

###  4.1 Experimental Setup

Implementation Details. We implement our method with Pytorch on one NVIDIA RTX
3090Ti GPU. During the federated training, all participants adopt the same
hyperparameter settings, _e.g_., $M=25$, $C=3$. Both collaborative and local
updating use the stochastic gradient descent (SGD) optimizer with a batch size
of $32$ and a learning rate of $0.01$.

Table 1: Accuracy $\%$ of state-of-the-art FL methods on two scenarios, including Label Shift: CIFAR-100 [18] and Tiny-ImageNet, and Feature Shift: DomainNet [34], where # Com.cost is the communication cost. w.Full indicates that the local model of a FL algorithm is fully fine-tuned. The arrow ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\uparrow}$ and ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow}$ indicate improvements and decrements compared with FedAvg (w.Full), respectively. Detailed analyses are provided in Sec. 4.2. Method |  # Com.cost |  DomainNet |  CIFAR-100 |  Tiny-ImageNet
---|---|---|---|---
SOLO | $-$ | 62.18($17.84$) $\downarrow$ | 33.81($47.41$) $\downarrow$ | 17.17($61.85$) $\downarrow$
FedAvg${\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}_{2017}}$ (w.Full) [30] |  $85.80$ M |  $80.02$($0.00$) $\downarrow$ |  $81.22$($0.00$) $\downarrow$ |  $79.02$($0.00$) $\downarrow$
FedProx${\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}_{2020}}$ [25] |  $85.80$ M |  $78.73$($1.29$) $\downarrow$ |  $81.56$($0.34$) $\uparrow$ |  $79.57$($0.55$) $\uparrow$
SCAFFOLD${\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}_{2020}}$ [16] |  $85.80$ M |  $80.31$($0.29$) $\uparrow$ |  $78.02$($3.20$) $\downarrow$ |  $76.76$($2.26$) $\downarrow$
FedBN${\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}_{2021}}$ [27] |  $85.76$ M |  $80.07$($0.05$) $\uparrow$ |  $81.24$($0.02$) $\uparrow$ |  $79.82$($0.80$) $\uparrow$
MOON${\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}_{2021}}$ [24] |  $85.80$ M |  $81.65$($1.64$) $\uparrow$ |  $81.92$($0.70$) $\uparrow$ |  $81.38$($2.18$) $\uparrow$
FedDC${\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}_{2022}}$ [9] |  $85.80$ M |  $79.92$($0.10$) $\downarrow$ |  $78.44$($2.78$) $\downarrow$ |  $79.81$($0.79$) $\uparrow$
FedIns (w. SSF Pool) |  5.35 M |  82.34($\underline{\textbf{2.32}}$) $\uparrow$ |  84.11($\underline{\textbf{2.89}}$) $\uparrow$ |  86.29($\underline{\textbf{7.27}}$) $\uparrow$
FedIns (Ours) |  5.35 M |  83.12($\underline{\textbf{3.10}}$) $\uparrow$ |  84.83($\underline{\textbf{3.61}}$) $\uparrow$ |  86.79($\underline{\textbf{7.77}}$) $\uparrow$

Datasets. Experiments are conducted on two scenarios [27, 23], including Label
Shift: CIFAR-100 [18], and Tiny-ImageNet111https://www.kaggle.com/c/tiny-
imagenet., and Feature Shift: DomainNet [34]. To simulate the FL scenario, we
use Dirichlet distribution to split the training data of CIFAR-100 and Tiny-
ImageNet into multiple non-i.i.d. clients, respectively, and evaluate the
performance on the test data [24]. DomainNet consists of data from six
different domains with the same class labels. Following [27], we deploy the
data from each domain as a specific client. To demonstrate the effectiveness
of our proposed approach, each client in our experiments has only a small
number of images, _i.e_., DomainNet is divided into six clients with only $26$
images for each of them; CIFAR-100 is divided into five clients with $123$,
$209$, $168$, $222$, and $278$ images, respectively; and Tiny-ImageNet is
divided into five clients with $322$, $360$, $343$, $466$, and $509$ images,
respectively.

Baselines. We compare our method, FedIns, with various state-of-the-art FL
algorithms, including: (1) FedProx [25], which alleviates the data
heterogeneity by applying a proximal term to the local objective function; (2)
SCAFFOLD [16], which corrects the client-drift by a series of control
variates; (3) MOON [24], which corrects the local update by computing the
similarity between model representations; (4) FedBN [27], which alleviates the
client-shift by using batch normalization on each local client; (5) FedDC [9],
which bridges the gap between the local and global model parameter by an
auxiliary local drift variable and the classical FL algorithm; and (6) FedAvg
[30], which trains a global model by averaging parameters from all the
participating clients. In comparison, we also add (7) SOLO, where participants
train a model on each client and their private data without FL. For a fair
comparison, we use ViT-B/16 [5] pre-trained on ImageNet-21K as the backbone of
all the methods in our experiments.

###  4.2 Comparison with State-of-the-arts

To assess the effectiveness of our method, FedIns, we compared it to the above
state-of-the-art FL algorithms, which aim to alleviate the data heterogeneity.
For a fair comparison, all methods are retrained using the same model with
their best hyperparameters. Specifically, for MOON, the hyperparameter of
$\mu$ is set to $1$. For FedProx [25], the hyperparameter to control the
weight of its proximal term is set to $0.001$. For FedDC, we set the
hyperparameter of $\alpha$ to $0.01$. Here, all the competing methods are
trained with $300$ communication rounds and $10$ local epochs for each round.
As the results shown in Table 1, our FedIns consistently outperform baselines
in all settings, achieving state-of-the-art results in most cases. In
particular, on DomainNet, CIFAR-100, and Tiny-ImageNet datasets, our method
improves the accuracy of FedAvg from $80.02\%$, $81.22\%$, and $79.02\%$ to
$\textbf{83.12}\%$, $\textbf{84.83}\%$, and $\textbf{86.79}\%$, respectively.
These results indicate the robustness of our method in the two scenarios,
_i.e_., Label Shift and Feature Shift. The FedAvg (w.Full) [30] is prone to
overfitting, even it uses the pre-trained model locally when local data is
limited. However, the parameter-efficient method of freezing the backbone
network is consistent with the mechanism of personalized FL which is also an
effective way to improve FL performance. More importantly, the number of
training parameters and communication cost of our method are only $15\%$ of
those of FedAvg (w.Full) [30]. It is worth noting that FedProx [25] and
SCAFFOLD [16] perform worse than FedAvg [30], _i.e_., DomainNet: $80.02$$\%$
$\rightarrow$ 78.73$\%$ and CIFAR-100: $81.22$$\%$ $\rightarrow$ 78.02$\%$.
This is primarily due to the fact that correcting local updates through
proximal terms and control variates will result in some deviations when there
are only a few training datasets on the local clients. Differently, FedDC
dynamically bridges the gap between the local model and the global model in
the parameter aggregation stage, making the global model able to alleviate the
data heterogeneity [9]. Nonetheless, even compared with FedDC [9], which is
the latest work aiming to address the statistical heterogeneity, our method
still achieves a $8.75\%$ improvement on Tiny-ImageNet, (_i.e_., $79.81$$\%$
$\rightarrow$ 86.79$\%$).

In contrast, the results of SOLO are lower than those of other FL algorithms,
although it is also fine-tuned on a pre-trained model, which demonstrates the
benefits of FL. These classical FL algorithms only consider the inter-client
data heterogeneity, ignoring the intra-client data heterogeneity, which will
also degrade the FL performance. This finding confirms our core idea that the
performance of FL algorithms can be improved by simultaneously relieving the
inter- and intra-client data heterogeneity. In the next section, we analyze
the communication efficiency of FedIns and the influence of FedIns on
inter-/intra-client data heterogeneity and local epoch.

Figure 4: Accuracy comparison of different state-of-the-art FL algorithms in terms of different communication rounds on two scenarios, including Label Shift: CIFAR-100 [18] and Tiny-ImageNet, and Feature Shift: DomainNet [34] datasets. Table 2: Ablation studies with regard to the key components of FedIns on the three datasets, where #New.Param. represents the new introduced parameters of each method, ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow}$ indicates decrements compared with our full model _SSF Pool_ (Ours). Detailed analyses are provided in Sec.4.5. Variation | Prompt | SSF | Pool | #New.Param. |  DomainNet |  CIFAR-100 |  Tiny-ImageNet
---|---|---|---|---|---|---|---
Prompt | ✓ | $-$ | $-$ |  $0.10$ M |  $79.03$($4.09$) $\downarrow$ |  $80.25$($4.58$) $\downarrow$ |  $81.02$($5.77$) $\downarrow$
Prompt Pool | ✓ | $-$ | ✓ |  $2.32$ M |  $80.17$($2.95$) $\downarrow$ |  $81.39$($3.44$) $\downarrow$ |  $81.98$($4.81$) $\downarrow$
_SSF_ | $-$ | ✓ | $-$ |  $0.00$ M |  $80.75$(${{2.37}}$) $\downarrow$ |  $83.81$(${{1.02}}$) $\downarrow$ |  $84.32$(${{2.47}}$) $\downarrow$
_SSF Pool_ (Ours) | $-$ | ✓ | ✓ |  $5.16$ M |  $83.12$(${{0.00}}$) $\downarrow$ |  $84.83$(${{0.00}}$) $\downarrow$ |  $86.79$(${{0.00}}$) $\downarrow$

###  4.3 Communication Efficiency Analysis

Our primary goal is to investigate how to efficiently alleviate inter- and
intra-client heterogeneity and make instance-adaptive models feasible for FL,
leveraging FL to perform well with less local data, a smaller number of
parameters, and lower communication costs. Therefore, we evaluate the
communication efficiency of the proposed method in terms of communication
cost, number of parameters, and different numbers of communication rounds. As
shown in Table 1, FedIns only requires $5.35$ M of communication, which is
$15\%$ of the others. Similarly, the number of learnable parameters of FedIns
is only $5.35$ M. Therefore, our method has high efficiency in both local
client updates and server communication. On the contrary, although the
classical federated algorithms MOON [24] and FedBNs [27] can also achieve good
accuracy, they require a large amount of local computation and incur high
communication costs. In Fig. 4, we compare the accuracy of all methods under
different communication rounds to demonstrate the superior communication
efficiency of our method. We note that for all competing methods, the local
epoch of each method is fixed at $10$. As can be seen from this figure, FedIns
reached stability in the $10$-th round, while the other methods needed to
reach stability after $20$ rounds. This indicates that our FedIns dynamically
guides the local model to alleviate the intra-client data heterogeneity,
enabling the instance-adaptive model feasible in FL.

###  4.4 Effect of Statistical Heterogeneity

As we mentioned before, our proposed method can improve the performance of FL
algorithms by alleviating both inter- and intra-client data heterogeneity.
Therefore, here we explore whether the proposed method remains effective as
inter- and intra-client data heterogeneity increases. First, we examine the
influence of inter-client data heterogeneity on our algorithm by changing the
concentration parameter $\beta$ of the Dirichlet distribution. The smaller
$\beta$ values, the higher inter-client data heterogeneity [24]. Fig. 5 (a)
shows the classification accuracies of $\beta$ values vary in
$\\{0.1,0.5,1,1.5,2,2.5,3,3.5,4,4.5,5\\}$. As can be seen from the figure, the
classification accuracies of all the FL algorithms increase with the
increasing $\beta$ value. FedIns, on the other hand, consistently has the
highest classification accuracy and is least affected by $\beta$. However,
with the increase in inter-client data heterogeneity, the performance of
FedAvg [30] will rapidly decline. On the contrary, FedIns is least affected by
heterogeneity and still holds an accuracy of $85.13$$\%$ when $\beta$ = $0.1$.
Then, we fix $\beta$ = $0.2$, and randomly convert some local images to other
styles to control the intra-client data heterogeneity [10].

Similarly, the bigger $\gamma$ values, the higher intra-client data
heterogeneity. Fig. 5 (b) shows the classification accuracies of $\gamma$
values vary in $\\{2,4,6,...,20\\}$. Similar to Fig. 5 (a), the classification
accuracy scores of all the FL algorithms decrease with the increasing $\gamma$
value. These baseline methods, in particular, show the most marked downward
trend. Nonetheless, when $\gamma$ = $20$, our proposed method only decreases
the accuracy from $75.63$$\%$ to ${64.15}$$\%$. In contrast, the classical FL
algorithm FedBN, which aims to solve the data heterogeneity problem, decreases
the accuracy from $68.53$$\%$ to 31.69$\%$ when $\gamma$ = $20$.

$\beta$$\gamma$

Figure 5: Accuracy comparison of different state-of-the-art FL algorithms in
terms of various (a) inter-client heterogeneity and (b) intra-client
heterogeneity on Tiny-ImageNet, where the smaller $\beta$ values, the higher
inter-client heterogeneity; the larger $\gamma$ values, the higher intra-
client heterogeneity.

###  4.5 Ablation Study

Effectiveness of Core Designs. We first analyze the key components of FedIns
on three datasets to evaluate the effectiveness of our core designs. The
evaluated components include SSF pool in our model, and SSF, which is the
parameter-efficient mechanism in our model. To this end, we built four derived
ablation models, where _Prompt_ represents that the SSF of our method is
replaced by the similar parameter-efficient mechanism prompts [14], _Prompt
Pool_ represents employing the prompt pool mechanism on _Prompt_ , _SSF_
indicates that the FedIns model removes the SSF pool mechanism, and only
retains the parameter-efficient mechanism of SSF in the federated framework,
and _SSF Pool_ represents our FedIns that preserves the full components.
Following [14], we add the prompt embeddings with a size of $10\times 12\times
768$. Table 2 summarizes the classification accuracy of all ablation models.
As compared with _Prompt_ , we observed that _SSF_ improves the performance
from $81.02\%$ to $\textbf{84.32}\%$ on the Tiny-ImageNet dataset, indicating
that SSF can provide better performance than prompt since prompt is often
sensitive to data and tasks and needs to be carefully designed for each
client. Additionally, SSF achieves zero overhead by reparameterizing them into
the original pre-trained model weights at the inference stage, while prompt-
based methods introduced additional parameters, _i.e_., _Prompt_ with $0.10$ M
and _Prompt Pool_ with $2.32$ M parameters. This supports our design of using
SSF to fine-tune the local model and communicate with the server. _SSF Pool_
(Ours) equipped with all components, produces the best classification
accuracies, improving the results from $81.02\%$ to $\textbf{86.79}\%$ on the
Tiny-ImageNet dataset. In general, our core designs can help to alleviate
federated data heterogeneity and enhancing the performance of FL algorithms.

Pool Size Analysis. Here, we focus on analyzing the effect of the pool size on
model effectiveness. We can freely select the size with the greatest
performance even though the number of network parameters increases with pool
size because the number of parameters for SSF is small, _i.e_., $M$ $=$ $1$
only requires $0.20$ M parameters. Additionally, these parameters can be
incorporated into the original pre-trained weights by model reparameterization
at the inference stage, thereby avoiding additional parameters for the
downstream tasks. We record the performance of different pool sizes on the
three datasets in Fig. 6 (a). It can be seen from the figure that as the pool
size increases, the performance of FedIns on the three datasets will improve.
When $M$ $=$ $25$, our method yields the best classification performance while
only requiring $5.35$ M parameters. When $M$ is greater than $25$, the
performance degrades due to a large amount of redundancy, which affects the
update of the local client by instance-adaptive models. It is worth noting
that our model always preserves a higher classification accuracy than the
baseline, _i.e_., when $M=5$, Ours preserve the results of Acc. $=$
$\textbf{86.50}\%$ $vs.$ FedBN: Acc. $=$ $79.81\%$.

$M$$c$

Figure 6: Ablation studies versus (a) Pool size analysis and (b) Best-matched
number analysis on DomainNet, CIFAR-100, and Tiny-ImageNet datasets.

Best-matched Number Analysis. As the selected SSF values for each input will
affect the update of the local model, we need to choose an appropriate best-
matched number $C$ to train a good SSF pool. In Fig. 6 (b), we record the
classification accuracy of $c$ values varying in $\\{1,3,6,...,24\\}$. As can
be seen from this figure, we observe that our proposed method obtains the
highest classification accuracy when $C$ $=$ $3$ and the accuracy decreases
monotonically on the CIFAR-100 when the value of $C$ increases, while
remaining invariant on the Tiny-ImageNet dataset. However, even at the lowest
point, our method still produces higher classification accuracy than the
baselines in the two scenarios, _i.e_., Label Shift and Feature Shift (see
Table 1). Therefore, this study reveals the effectiveness of our proposed SSF
pool mechanism.

##  5 Conclusion

In this work, we re-examine the problem of data heterogeneity in FL, and find
that there is not only inter-client data heterogeneity but also intra-client
heterogeneity in many complex real-world scenarios, which also significantly
degrades FL performance. To address this issue, we propose a new parameter-
efficient fine-tuning FL algorithm, FedIns, which makes it possible to use
instance-adaptive inference for FL, thus enabling dynamically guided locals to
alleviate intra-client data heterogeneity. Unlike existing approaches, FedIns
trains an SSF pool for each client and aggregates them into the federated SSF
pool on the server. For a given instance, FedIns dynamically finds the best-
matched SSF subsets from the pool, and aggregates them to generate an adaptive
SSF for instance-adaptive inference. In addition, FedIns introduces only a
small number of learnable parameters with the help of the large-scale pre-
training model, and greatly reduces the communication cost. Extensive
experiments show the superiority of FedIns in handling the inter- and intra-
client data heterogeneity issue.

Acknowledgements: This work was supported by the National Research Foundation,
Singapore under its AI Singapore Programme (AISG Award No: AISG2-TC-2021-003)
and Agency for Science, Technology and Research (A*STAR) through its RIE2020
Health and Biomedical Sciences (HBMS) Industry Alignment Fund Pre-Positioning
(IAF-PP) (grant no. H20C6a0032).

## References

  * [1] Ismaeel Al Ridhawi, Safa Otoum, Moayad Aloqaily, and Azzedine Boukerche.  Generalizing ai: challenges and opportunities for plug and play ai solutions.  IEEE Network, 35(1):372–379, 2020.
  * [2] Hong-You Chen, Cheng-Hao Tu, Ziwei Li, Han-Wei Shen, and Wei-Lun Chao.  On pre-training for federated learning.  arXiv preprint arXiv:2206.11488, 2022.
  * [3] Ziliang Chen, Jingyu Zhuang, Xiaodan Liang, and Liang Lin.  Blending-target domain adaptation by adversarial meta-adaptation networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2248–2257, 2019.
  * [4] Enmao Diao, Jie Ding, and Vahid Tarokh.  Heterofl: Computation and communication efficient federated learning for heterogeneous clients.  arXiv preprint arXiv:2010.01264, 2020.
  * [5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [6] Chun-Mei Feng, Bangjun Li, Xinxing Xu, Yong Liu, Huazhu Fu, and Wangmeng Zuo.  Learning federated visual prompt in null space for mri reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8064–8073, 2023.
  * [7] Chun-Mei Feng, Yunlu Yan, Huazhu Fu, Li Chen, and Yong Xu.  Task transformer network for joint mri reconstruction and super-resolution.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VI 24, pages 307–317. Springer, 2021.
  * [8] Chun-Mei Feng, Yunlu Yan, Shanshan Wang, Yong Xu, Ling Shao, and Huazhu Fu.  Specificity-preserving federated learning for mr image reconstruction.  IEEE Transactions on Medical Imaging, 2022.
  * [9] Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu.  Feddc: Federated learning with non-iid data via local drift decoupling and correction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10112–10121, 2022.
  * [10] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel.  Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.  arXiv preprint arXiv:1811.12231, 2018.
  * [11] Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, et al.  Pre-trained models: Past, present and future.  AI Open, 2:225–250, 2021.
  * [12] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig.  Towards a unified view of parameter-efficient transfer learning.  arXiv preprint arXiv:2110.04366, 2021.
  * [13] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019.
  * [14] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  arXiv preprint arXiv:2203.12119, 2022.
  * [15] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder.  Compacter: Efficient low-rank hypercomplex adapter layers.  Advances in Neural Information Processing Systems, 34:1022–1035, 2021.
  * [16] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh.  Scaffold: Stochastic controlled averaging for federated learning.  In International Conference on Machine Learning, pages 5132–5143. PMLR, 2020.
  * [17] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  In International Conference on Machine Learning, pages 5583–5594. PMLR, 2021.
  * [18] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [19] Gihun Lee, Minchan Jeong, Yongjin Shin, Sangmin Bae, and Se-Young Yun.  Preservation of the global knowledge by not-true distillation in federated learning.  In Advances in Neural Information Processing Systems, 2022.
  * [20] Gihun Lee, Yongjin Shin, Minchan Jeong, and Se-Young Yun.  Preservation of the global knowledge by not-true self knowledge distillation in federated learning.  arXiv preprint arXiv:2106.03097, 2021.
  * [21] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  arXiv preprint arXiv:2104.08691, 2021.
  * [22] Haoran Li, Chun-Mei Feng, Tao Zhou, Yong Xu, and Xiaojun Chang.  Prompt-driven efficient open-set semi-supervised learning.  arXiv preprint arXiv:2209.14205, 2022.
  * [23] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He.  Federated learning on non-iid data silos: An experimental study.  In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 965–978. IEEE, 2022.
  * [24] Qinbin Li, Bingsheng He, and Dawn Song.  Model-contrastive federated learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10713–10722, 2021.
  * [25] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith.  Federated optimization in heterogeneous networks.  Proceedings of Machine learning and systems, 2:429–450, 2020.
  * [26] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang.  On the convergence of fedavg on non-iid data.  arXiv preprint arXiv:1907.02189, 2019.
  * [27] Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou.  Fedbn: Federated learning on non-iid features via local batch normalization.  arXiv preprint arXiv:2102.07623, 2021.
  * [28] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang.  Scaling & shifting your features: A new baseline for efficient model tuning.  arXiv preprint arXiv:2210.08823, 2022.
  * [29] Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang Liang, Qiang Yang, Dusit Niyato, and Chunyan Miao.  Federated learning in mobile edge networks: A comprehensive survey.  IEEE Communications Surveys & Tutorials, 22(3):2031–2063, 2020\.
  * [30] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
  * [31] Matias Mendieta, Taojiannan Yang, Pu Wang, Minwoo Lee, Zhengming Ding, and Chen Chen.  Local learning matters: Rethinking data heterogeneity in federated learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8397–8406, 2022.
  * [32] John Nguyen, Kshitiz Malik, Maziar Sanjabi, and Michael Rabbat.  Where to begin? exploring the impact of pre-training and initialization in federated learning.  arXiv preprint arXiv:2206.15387, 2022.
  * [33] Xing Nie, Bolin Ni, Jianlong Chang, Gaomeng Meng, Chunlei Huo, Zhaoxiang Zhang, Shiming Xiang, Qi Tian, and Chunhong Pan.  Pro-tuning: Unified prompt tuning for vision tasks.  arXiv preprint arXiv:2207.14381, 2022.
  * [34] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406–1415, 2019.
  * [35] Timo Schick and Hinrich Schütze.  It’s not just size that matters: Small language models are also few-shot learners.  arXiv preprint arXiv:2009.07118, 2020.
  * [36] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  arXiv preprint arXiv:2010.15980, 2020.
  * [37] Yang Shu, Zhi Kou, Zhangjie Cao, Jianmin Wang, and Mingsheng Long.  Zoo-tuning: Adaptive transfer from a zoo of models.  In International Conference on Machine Learning, pages 9626–9637. PMLR, 2021.
  * [38] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor.  Tackling the objective inconsistency problem in heterogeneous federated optimization.  Advances in neural information processing systems, 33:7611–7623, 2020.
  * [39] Chencheng Xu, Zhiwei Hong, Minlie Huang, and Tao Jiang.  Acceleration of federated learning with alleviated forgetting in local training.  arXiv preprint arXiv:2203.02645, 2022.
  * [40] Haolin Yuan, Bo Hui, Yuchen Yang, Philippe Burlina, Neil Zhenqiang Gong, and Yinzhi Cao.  Addressing heterogeneity in federated learning via distributional transformation.  In European Conference on Computer Vision, pages 179–195. Springer, 2022.
  * [41] Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni.  Bayesian nonparametric federated learning of neural networks.  In International conference on machine learning, pages 7252–7261. PMLR, 2019.
  * [42] Yuliang Zou, Zizhao Zhang, Chun-Liang Li, Han Zhang, Tomas Pfister, and Jia-Bin Huang.  Learning instance-specific adaptation for cross-domain segmentation.  arXiv preprint arXiv:2203.16530, 2022.

## Contents

The following items are included in our supplementary material:

  * •

Detailed illustration of Fig. 1 (c) and (d) in Section A.

  * •

Exemplars of the converted style for the intra-client data heterogeneity in
Section B.

##  Appendix A Detailed Illustration of Fig. 1 (c) and (d)

We redisplay Fig. 1 (c) and (d) in Fig. A1 in details. As can be seen from
this figure, the accuracy % comparisons of FedAvg [30], FedBN [27], and FedIns
in terms of various (a) inter-client heterogeneity and (b) intra-client
heterogeneity on DomainNet are recorded. We use the hyperparameter $\beta$ of
Dirichlet distribution to control the inter-client heterogeneity and $\gamma$
to control the intra-client heterogeneity. The smaller $\beta$ values, the
higher inter-client heterogeneity; the larger $\gamma$ values, the higher
intra-client heterogeneity, where $\gamma$ is the number of the converted
styles [10]. We provide the exemplars of the converted styles in Sec. B.

$\beta$$\gamma$

Figure A1: Accuracy comparisons of FedAvg [30], FedBN [27], and FedIns in
terms of various (a) inter-client heterogeneity and (b) intra-client
heterogeneity on DomainNet, where $\beta$ is hyperparameter of Dirichlet
distribution, the smaller $\beta$ values, the higher inter-client
heterogeneity; $\gamma$ is the number of the converted styles (see Sec. B for
details), the larger $\gamma$ values, the higher intra-client heterogeneity.

##  Appendix B Exemplars of the Intra-client Data Heterogeneity

As we mentioned, we randomly convert some local images to other styles to
control the intra-client data heterogeneity [10]. Here, we show some exemplars
of the converted images on the client clipart under DomainNet dataset in
Figure. A2. As can be seen from this figure, different styles show great
differences, thereby making the local data preserve higher intra-client data
heterogeneity and being able to be used to demonstrate the effectiveness of
our method.

Figure A2: Visualization of the convert images with various style on DomainNet
to control the intra-client data heterogeneity.


∎

11institutetext:  ✉Xiang Ma
11email<EMAIL_ADDRESS>
22institutetext: 1 Institute of High Energy Physics, Chinese Academy of
Sciences, Beijing, 100049, China. 33institutetext:
2 University of Chinese Academy of Science, Beijing, 100049, China.
44institutetext:
3 Southwest Jiaotong University, Chengdu, 610092, China.

# The design and implementation of GECAM satellite payload performance
monitoring software

Peng Zhang1,3 Xiang Ma1,2* Yue Huang1,2 Shaolin Xiong1,2 Shijie Zheng1,2
Liming Song1,2 Ge Ou1,2 Yanqi Du3 Jing Liang3 Hong Wu3

(Received: date / Accepted: date)

###### Abstract

Background The Gravitational wave high-energy Electromagnetic Counterpart All-
sky Monitor (GECAM) is primarily designed to spot gamma-ray bursts
corresponding to gravitational waves. In order to achieve stable observations
from various astronomical phenomena, the payload performance need to be
monitored during the in-orbit operation.
Method This article describes the design and implementation of GECAM satellite
payload performance monitoring (GPPM) software. The software extracts the
payload status and telescope observations (light curves, energy spectrums,
characteristic peak fitting of energy spectrums, etc) from the payload data.
Considering the large amount of payload status parameters in the engineering
data, we have designed a method of parameter processing based on the
configuration tables. This method can deal with the frequent changes of the
data formats and facilitate program maintenance. Payload status and
performance are monitored through defined thresholds and monitoring reports.
The entire software is implemented in python language and the huge amount of
observation data is stored in MongoDB.
Conclusion The design and implementation of GPPM software have been completed,
tested with ground and in-orbit payload data. The software can monitor the
performance of GECAM payload effectively. The overall design of the software
and the data processing method can be applied to other satellites.

###### Keywords:

GECAM Payload Data process Python

††journal: RDTM

##  1 Introduction

The first detection of gravitational wave signal was made on December 14, 2015
and was announced on February 11, 2016 GW 0 . The direct detection of
gravitational wave has started a revolutionary era of gravitational wave
astronomy GW 1 ; GW 2 . Gravitational wave events occur randomly, and each
event contains valuable information and significant opportunities for
discovery. The detection of high-energy electromagnetic counterparts of
gravitational wave events as an important method for studying gravitational
wave sources and fundamental physical laws has become the research hotspot of
astronomy.

The Gravitational wave high-energy Electromagnetic Counterpart All-sky Monitor
(GECAM) satellite is a full-sky monitoring telescope which can detect high-
energy electromagnetic counterparts of gravitational wave events GECAM system
design . GECAM also monitors other burst events, such as gamma-ray bursts
(GRBs), the high-energy radiation of fast radio bursts, and magnetar bursts.
GECAM measures the energy spectrum, light curves, and location of these burst
events. GECAM is composed of two small satellites which operate in the low
earth orbit. The two satellites operate in opposite orbital phase and opposite
geocentric directions to obtain all-sky view, while each satellite has 25
Gamma-Ray Detectors (GRDs) and 8 Charged Particle Detectors (CPDs) GECAM
payload . GRDs are applied to detect gamma-rays in the energy range from 6 keV
to 5 MeV. GRDs are comprised of $\rm LaBr_{3}$:Ce scintillator, SiPM array and
preamplifier; the large dynamic energy range of GRDs are achieved by the high-
gain and low-gain ADC channels of the preamplifier GECAM GRD 1 ; GECAM GRD 2 .
The location of GRB is calculated from the photon counts from different
pointing modes of GRDs GECAM localization method . CPDs are insensitive to
gamma rays but are designed to detect energetic charged particles of energies
from 300 keV to 5 MeV. CPDs can be used to exclude interference from space
charged particle bursts during data analysis. The telescope observation data
and payload status (voltage, current, temperature, etc) of the GRDs and CPDs
are processed by five data acquisition devices (DAQs).

The GECAM has been successfully launched in December 2020. The GECAM mission
consists of six systems: Satellite System; Launch Vehicle System; Launch Site
System; Tracking, Telemetry and Command System (TT&C); Ground Support System
and Scientific Application System. The Ground Support System is developed by
National Space Science Center, Chinese Academy of Sciences (NSSC), which
receives, pre-processes, manages and archives satellite data. Scientific
Application System links scientific research and satellite operations, which
monitors the performance of the payload to ensure the stable operation of the
instruments and implements the calibration, production, storage, distribution
and rapid processing of scientific data GECAM sci system ; GECAM data analysis
. GECAM satellite payload performance monitoring (GPPM) software is a part of
Scientific Application System. The software firstly process payload raw data
to generate data products, and then analyse the parameters of the data
products to monitor the payload performance. When the satellite is in orbit,
the raw data processing is implemented by the Ground Support System. The raw
data processing method processes the raw data which generated from payload
ground test. Payload raw data is unpacked and reorganized to the data products
which are used for functional and interface testing by the various subsystems
of Scientific Application System.

In this paper, we describes the process of designing and implementing of the
GPPM software. First, general information and structure of the software are
presented. Second, the detailed implementation of each module of the software
is described. Finally, the testing process for the functionality of the
software is illustrated.

##  2 Software overview

The main task of GPPM software is to monitor the performance of payload. The
flowchart of the software is shown in Figure. 1. The software consists of
three modules: data processing module, data analysis module and data display
module. When satellite is in ground test, the data processing module of this
software unpacks and reorganizes the payload raw data to level 0 data product.
During in orbit performing, the data processing is implemented by the NSSC.
The correspondence between raw data and data product is shown in Table. 1.
Data analysis module extracts payload status and telescope observations from
payload data, then determines anomalies from the thresholds for crucial
monitoring objects, and finally stores the observations and anomalies in the
database. User analyzes the payload performance from anomalies and monitoring
reports in the data display module. The software uses 17 and 582 data tables
in MySQL mysql introduction  and MongoDB mongodb introduction  for data
storage, respectively. Regular data is stored in MySQL, such as lists of
files, thresholds for anomalies, anomalies informations, etc. The observations
data are stored in MongoDB. MongoDB is more efficient than MySQL when querying
for large amounts of data mysql mongodb compare . From satellite launch to
July 2021, MongoDB have stored 124 GByte of data. The whole software is
implemented in python 3.6 python introduction . The functions of the three
modules of the software are described below:

Data processing module unpacks the payload raw data according to pre-defined
format. The parameters in the raw data are converted to physical values
according to calculation formula and saved to the data product. While the
satellite’s payload had been tested on the ground, payload data management
software generated large number of scientific and engineering raw data for
payload tests. The data processing module unpacks, converts and reorganizes
these payload raw data to produce data products for analysing the performance
of the payload. When satellite operates in orbit, satellite transmits
scientific and engineering raw data via telemetry and data transmission
channels. When Ground Support System receives the payload raw data, it
processes these raw data to data products and transfers to Scientific
Application System via virtual private network (VPN) for further processing.

The function of data analysis module is to extract the payload status and
telescope observation data (such as light curve and energy spectrum) of each
detector from the data products. For crucial monitoring objects, this module
needs to decide whether the objects is abnormal based on the defined
threshold.

The data display module generates payload performance monitoring reports for
the past period at regular intervals and creates web pages for viewing
historical data, anomalies, monitoring reports etc.

Table 1: Correspondence between processed raw data and generated data product Processed raw data | Generate | Corresponding data product | Category of data product
---|---|---|---
Engineering raw data | $\rightarrow$ | Payload engineering data product | Engineering data product
Orbital data product
Event raw data | $\rightarrow$ | Event data product | Scientific data product
Binned raw data | Binned data product
Figure 1: The flowchart of the GPPM software

##  3 Module design

###  3.1 Data processing module

The data processing module queries the binary raw data required to process
from the MySQL for every ten minutes, and multi-threads unpacks the raw data
files in parallel to generate data products in FITS111FITS format is a common
data storage format in the astronomical field, for details see the web site of
https://fits.gsfc.nasa.gov/ format. The binary raw data consists of several
packages. The length of each package is 884 bytes; the first 12 bytes contain
information including the start flag and the type of package; the rest 872
bytes are the data field. Raw data are divided into three types: event raw
data, engineering raw data and binned raw data.

The data field of each package in the event raw data consists of the DAQ
number (2 bytes) and 145 events (the length of each event is 6 bytes). The
arrival time of each event is recorded using a local time counter with a time
resolution of 0.1 microsecond. There are six types of events in total.
Physical events contain time, energy channel (4096 channels in total) and dead
time (time consumed by DAQ to process a single photon) of the photon. Other
types of events are recorded at one second intervals to aid in the calculation
of full arrival time of each photon. Arrival time, energy, dead time and
information used to assist with time calculation of each photon are stored in
the event data product.

The binned raw data consists of time binned raw data and energy binned raw
data. The time binned raw data records the photon counts and sum of dead time
every 50 milliseconds. The energy binned raw data records photon counts and
sum of dead time per second, the photon counts are the number of photons in
each channel after the original 4096 energy channels have been merged into 128
channels. The photon counts and dead time information extracted from time
binned raw data and energy binned raw data are saved as time binned data
product and energy binned data product, respectively.

Engineering raw data records information of payload status every second, such
as temperature, current, voltage and photon counts (both processed and
downlinked) of the detectors. The satellite’s attitude orbit information is
also recorded in engineering raw data. The data field of each package from the
engineering raw data consists of 511 parameters, while the individual
parameters are stored by one or more bits. We need to extract the original
value of the parameter based on the number of bits in the corresponding
position of the package, and then convert it to the physical value using the
defined formula. The status of the payload is saved to payload engineering
data product. The attitude and orbit data of the satellite are saved to
orbital data product. For the engineering raw data that contains large amount
of parameters needs to be processed, we have designed a method of parameter
processing according to the configuration table. If changes are made to data
format, parameter processing can be quickly updated by adjusting the
configuration table directly without modifying the program code. The flowchart
of the method is shown in the Figure 2. The specific process flow is described
below:

Figure 2: The flowchart of the engineering raw data processing

  1. 1.

The first step is to read the description table of engineering raw data format
file which is stored by the CSV222CSV files store tabular data (numbers and
text) in plain text form file, see Table. 2 for an example. The first and
second columns represent the parameter name and the number of bits occupied by
the parameters, respectively. The third column represents the type of data
that the parameter needs to convert. The last column represents the formula
required for the calculation after the parameter is converted.

  2. 2.

The second step is to read the binary corresponding bits according to the
second column, and convert it to the data type described in the third column.
The final value is obtained according to the calculation formula of the last
column. If the formula is not empty, it is converted to expression execution
using python’s built-in eval function. When the calculation formula of some
parameters is complex and cannot be described by string, we need to pre-define
its calculation function in the program and start with the character formula
as the function name.

  3. 3.

The third step is to read the description table of engineering data product
format table which is also stored in the CSV file, see Table. 3 for an
example. Data products in FITS format use several Header Data Units (HDUs) to
distinguish data categories. We choose to store data in HDUs using data
arrays. The second and third columns in Table. 3 indicate the name and format
of each column in the data array, respectively. The fourth column indicates
the unit of data. The last column indicates that the program can get the
corresponding parameter value from the engineering raw data via the parameter
name.

  4. 4.

Finally, the values corresponding to parameters are obtained from engineering
raw data according to Table. 3, and the descriptions in this table are further
used to generate engineering data products in the corresponding format.

Table 2: Example of engineering raw data parsing format table Parameter name | Bit | Parameter type | Calculation
---|---|---|---
Telemetry package identification | 16 | Hexadecimal | -
Local accumulation of seconds of pulse time | 40 | Binary | formula_local_time(value)
Receive command count | 8 | Decimal | -
Physical Package Delivery Count | 8 | Decimal | value*1024
+30V voltage | 8 | Decimal | (value/255)*2*1050/50
Temperature sensitive resistor telemetry of GRDA | 12 | Decimal | formula_warm_resistance(value)
Digital temperature telemetry of GRDA | 12 | Decimal | formula_digital_temperature(value)
Rate meter for high gain processing data of GRDA | 16 | Decimal | value*4
Rate meter for high gain downlink data of GRDA | 16 | Decimal | value*4
Rate meter for high low processing data of GRDA | 16 | Decimal | value*4
Rate meter for high low downlink data of GRDA | 16 | Decimal | value*4
J2000 coordinate system position X | 32 | Float | -
J2000 coordinate system position Y | 32 | Float | -
J2000 coordinate system position Z | 32 | Float | -
Table 3: Example of the specification table of engineering data product format |  |  |  | Parameter of raw data from Table. 2
---|---|---|---|---
HDU name | Data name | Data format | Data unit | Parameter name
GRD01 | TIME | D | s | Local accumulation of seconds of pulse time
GRDA_Resist | E | - | Temperature sensitive resistor telemetry of GRDA
GRDA_Temp | E | degree | Digital temperature telemetry of GRDA
GRDA_HGainCnt1 | J | - | Rate meter for high gain processing data of GRDA
GRDA_HGainCnt2 | J | - | Rate meter for high gain downlink data of GRDA
GRDA_LGainCnt1 | J | - | Rate meter for low gain processing data of GRDA
GRDA_LGainCnt2 | J | - | Rate meter for low gain downlink data of GRDA
Figure 3: The flowchart of the data analysis module Table 4: Example of engineering parameter table in mongoDB database |  |  | Data of data product from Table. 3
---|---|---|---
Table name | Column name | Column type | HDU name | Data name
dataMonitor_a_grdgcllj | _id | Integer | GRD01 | TIME
grd1_high_rate | Integer | GRD01 | GRDA_HGainCnt1
grd2_high_rate | Integer | GRD02 | GRDB_HGainCnt1
grd3_high_rate | Integer | GRD03 | GRDC_HGainCnt1
grd4_high_rate | Integer | GRD04 | GRDD_HGainCnt1
| grd5_high_rate | Integer | GRD05 | GRDE_HGainCnt1

###  3.2 Data analysis module

Data analysis module regularly queries whether there are unprocessed data
products in MySQL every ten minutes. Current module is multi-threaded to
process data products, and save the required observation data to the MongoDB.
Meanwhile, this module determines whether the data is in an abnormal state
using defined thresholds and meanwhile saves the abnormal information to
MySQL. The range of exception thresholds corresponding to the data values is
also stored in the MySQL. The flowchart of data analysis module is shown in
Figure 3.

Scientific data product includes event data product and binned data product.
For the scientific data product, we extract the light curve (time scale is one
second) and energy spectrum of each detector from event data product and
binned data product. The sum of the dead times per second is extracted from
time binned data product. We determine detector anomalies by whether the
photon counts in the light curve exceed the defined threshold. In order to
measure the performance of the detector in several dimensions, we extract the
light curves of electronic noise (channel ¡ 200) and extract light curves of
the split channels (channel ¡ 300, 300 $\sim$ 1000, 1000 $\sim$ 3000 and ¿
3000) from event data product; the time scale of these light curves are 1
second. Gaussian function is applied fit the characteristic peak (37.4 KeV,
511 KeV and 1470 KeV) in the energy spectrum, while the shift in the
characteristic peak reflects the change in the detector’s energy
response333The 37.4 keV and 1470 keV intrinsic gamma-ray lines of $\rm
LaBr_{3}$:Ce can be resolved from in-flight backgrounds. The galactic 511keV
gamma-ray line also can be resolved GECAM GRD 2 . Counts of noisy light curve
and fitting parameters of the characteristic peaks that exceeded the threshold
are considered anomalies. The extracted data and anomalies will be saved in
the database.

The engineering data product includes orbital data product and payload
engineering data product. The satellite’s orbit, attitude, time and speed per
second are stored in orbital data product. The angle between payload z-axis
and geocentric are calculated based on the satellite’s attitude and orbit.
From the satellite’s attitude, orbit and time information, we calculate the
angle between the normal direction of satellite’s solar panel and solar
incidence, and whether the satellite is in the solar irradiation zone. The
payload engineering data product stores the status parameters of each detector
such as electric current, voltage, temperature, etc. We need to extract the
corresponding values of each parameter every second and determine whether the
parameter is in an abnormal state based on the defined threshold. Since the
536 parameters in the payload engineering data product need to be stored using
144 MongoDB data tables, we have also designed a method for automatic
engineering parameter extraction and saving based on configuration tables. We
first extract the corresponding values from the payload engineering data
products according to the parameters in Table. 3 and determine the exceptions
according to the defined thresholds. We then obtain the corresponding
engineering parameters from the engineering data product according to the HDU
name and data name in the format of the MongoDB data table, the format is
shown on Table. 4. Lastly, the parameters are saved in the corresponding data
table.

###  3.3 Data display module

In order to facilitate users to view data, we build a web system to provide
data query and display. We can view all the historical observation data of the
telescope and the abnormal payload information found in the data analysis
module. This module generates monitoring reports at regular intervals for the
past period (the exact time period can be modified in this module) based on
the data stored in the MongoDB. The monitoring report contains status
parameters for all detectors such as temperature, bias voltage, baseline, etc.
The total photon counts, the photon counts in different energy ranges, energy
spectrum, dead time, and the satellite position per second are also shown in
the monitoring report. The monitoring report also contains overall information
of the satellite such as attitude, orbit, velocity, position, etc. Payload
performance can be analysed by anomaly records and monitoring reports.

The operation of different users can be controlled by permissions in this
module. All users can view the observation data of telescope at different time
intervals, such as the light curve, the energy spectrum and the fitted
parameters of the characteristic peaks of the energy spectrum. Administrators
will authorise users, modify the thresholds for determining parameter
anomalies in the data product analysis module, and the time for generating
monitoring reports.

##  4 Software test

The data processing module unpacks and reorganizes the binary raw data to data
product during the payload ground test, and this module is involved in
experiments such as mechanics experiments, temperature experiments, load
experiments, etc. Data analysis module extracts payload status and telescope
observations from data products, and determines anomalies based on defined
thresholds. The observation data and anomalies are used to analyse how the
payload performs. Figure 4 shows the light curves of a GRB event
GRB210619B444https://gcn.gsfc.nasa.gov/gcn3/30264.gcn3 that was detected by
GECAM. Figure 5 gives an example of energy spectrum. The monitoring reports
generated by the data display module at regular intervals are shown in the
Figure 6. We can intuitively analyse the impact of the detector’s status on
the observation performance from the monitoring reports. The light curves,
energy spectrums and payload monitoring report are consistent with the
theoretical expectations.

Until July 2021, the satellite has been tested in orbit for 7 months. We have
counted the number of various data products processed during this period, as
shown in Table 5. We test the performance of the software in processing one
hour of data products. The CPU used in both testing and operating environment
is an Intel(R) Xeon(R) Silver 4210 with frequency of 2.2 GHz. Performance test
results are shown in Table 6, and the processing time for each category of
data product is less than the 5 minutes required by the software design.

Tests using ground tests and in-orbit operational data have demonstrated that
the software functions correctly. Performance tests show that the software’s
performance meets the software’s design requirements.

##  5 Conclusion

The GPPM software is designed to monitor the performance of GECAM payload. The
software unpacks and reorganizes the payload raw data to data product. Payload
status and telescope observations from data product are extracted and saved to
MongoDB. Payload anomalies are obtained by testing whether the extracted data
exceeds defined threshold value. Considering that the large number of
parameters need to be processed in engineering raw data and data product, we
have designed a method of processing parameters according to configuration
table which makes the program easier to maintain. Regularly generated
monitoring reports based on historical data in the database can be used to
analyse correlation of payload parameters and anomalies.

Tested with ground and in-orbit payload data, the functionality and
performance of the software meets the design requirements. The software
enables the performance of the payload to be monitored effectively and
guarantees the proper functioning of GECAM. The software is more flexible and
practical, and can be applied to other satellite projects with appropriate
modifications.

###### Acknowledgements.

This work is supported by the Strategic Priority Research Program of Chinese
Academy of Sciences (Grant Nos. XDA15360300, XDA15360302 and XDB23040400).

Table 5: Statistics on the number of data products processed during the in-orbit testing Category | Data product | Total file number | Total file size(Mbyte)
---|---|---|---
Engineering | Payload engineering data product | 5729 | 19451
Orbital data product | 4918 | 916
Scientific | Event data product | 7852 | 4696045
Binned data product | 11187 | 551818
Table 6: Performance of data product processing Category | Data product | File size | Time consuming(s)
---|---|---|---
Engineering | Payload engineering data product | 4 MByte | 16.98
Orbital data product | 217 KByte | 20.38
Scientific | Event data product | 958 MByte | 70.57
Binned data product | 60 MByte | 32.20
Figure 4: Light curves of the three GRD with one second time bin. Each light
curve is extracted from the whole energy band, and the time is relative to UTC
2021-06-19T23:55:00. Figure 5: Energy spectrums of the three GRD. The time
range for each energy spectrum is from UTC time 2021-06-19T22:00:00 to
2021-06-19T23:00:00.

## References

  * (1) Abbott B P, Abbott R, Abbott T D, et al. Observation of gravitational waves from a binary black hole merger[J]. Physical review letters, 2016, 116(6): 061102.
  * (2) Mészáros P, Fox D B, Hanna C, et al. Multi-messenger astrophysics[J]. Nature Reviews Physics, 2019, 1(10): 585-599.
  * (3) Mostafá M. The Astrophysical Multi-messenger Observatory Network[J]. Nature Reviews Physics, 2020, 2(9): 446-448.
  * (4) HAN X B, ZHANG K K, HUANG J, et al. GECAM satellite system design and technological characteristic[J]. SCIENTIA SINICA Physica, Mechanica & Astronomica, 2020, 50(12): 129507.
  * (5) LI X Q, Wen X, AN Z H, et al. The GECAM and its payload[J]. SCIENTIA SINICA Physica, Mechanica & Astronomica, 2020, 50(12): 129508.
  * (6) Lv P, Xiong S L, Sun X L, et al. A novel gamma-ray detector for GECAM[C]//International conference on Technology and Instrumentation in Particle Physics. Springer, Singapore, 2017: 3-7.
  * (7) Zhang D, Li X, Xiong S, et al. Energy response of GECAM gamma-ray detector based on LaBr3: Ce and SiPM array[J]. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 2019, 921: 8-13.
  * (8) LIAO J Y, LUO Q, ZHU Y, et al. The localization method of GECAM and simulation analysis[J]. SCIENTIA SINICA Physica, Mechanica & Astronomica, 2020, 50(12): 129510.
  * (9) Chen W, SONG L M, ZHENG S J, et al. Introduction of the scientific application system of GECAM[J]. SCIENTIA SINICA Physica, Mechanica & Astronomica, 2020, 50(12): 129512.
  * (10) SONG X Y, XIONG S L, LUO Q, et al. Introduction to gamma-ray burst data analysis algorithm and software tools for GECAM[J]. SCIENTIA SINICA Physica, Mechanica & Astronomica, 2020, 50(12): 129511.
  * (11) Official website of Mysql. https://www.mysql.com/
  * (12) Official website of Mongodb. https://www.mongodb.com/
  * (13) Jose B, Abraham S. Performance analysis of NoSQL and relational databases with MongoDB and MySQL[J]. Materials today: PROCEEDINGS, 2020, 24: 2036-2043.
  * (14) Millman K J, Aivazis M. Python for scientists and engineers[J]. Computing in Science & Engineering, 2011, 13(2): 9-12.

Figure 6: Example of payload performance monitoring report. The figures in the
left column represent some monitoring objects of the GRD01 detector, from top
to bottom are temperature and bias voltage adjustment, baseline, light curves,
light curve in different channel ranges, dead time, respectively. The figures
in the right column show the monitoring objects of the overall satellite
status, from top to bottom are angular velocity, source of the time signal
that marks the second, +30V electric current, longitude and latitude, angle
between payload Z-axis and geocentric, satellite charging information,
trajectory of satellite (gradient colour represents time change, and red area
indicates the South Atlantic Anomaly), respectively.