# Real-Time Compressed Sensing for Joint Hyperspectral Image Transmission and
Restoration for CubeSat ††thanks: This study was supported in part by the
National Science and Technology Council (NSTC), Taiwan, under grants MOST
112-2221-E-006 -157 -MY3 and 111-2221-E-006 -210; partly by the Higher
Education Sprout Project of Ministry of Education (MOE) to the Headquarters of
University Advancement at National Cheng Kung University (NCKU). We thank to
National Center for High-performance Computing (NCHC) of National Applied
Research Laboratories (NARLabs) in Taiwan for providing computational and
storage resources. ††thanks: (Corresponding author: Chih-Chung Hsu.) ††thanks:
C.-C. Hsu, C.-Y. Jian, C.-M. Lee and G.-L. Chen are with Institute of Data
Science and Department of Statistics, Miin Wu School of Computing, National
Cheng Kung University, Tainan, Taiwan (R.O.C<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>E.-S. Tu is with Department of
Computer Science, National Cheng Kung University, Tainan, Taiwan (R.O.C.)

Chih-Chung Hsu, , Chih-Yu Jian, Eng-Shen Tu, Chia-Ming Lee and Guan-Lin Chen
https://rcts01.github.io/

###### Abstract

This paper addresses the challenges associated with hyperspectral image (HSI)
reconstruction from miniaturized satellites, which often suffer from stripe
effects and are computationally resource-limited. We propose a Real-Time
Compressed Sensing (RTCS) network designed to be lightweight and require only
relatively few training samples for efficient and robust HSI reconstruction in
the presence of the stripe effect and under noisy transmission conditions. The
RTCS network features a simplified architecture that reduces the required
training samples and allows for easy implementation on integer-8-based
encoders, facilitating rapid compressed sensing for stripe-like HSI, which
exactly matches the moderate design of miniaturized satellites on push broom
scanning mechanism. This contrasts optimization-based models that demand high-
precision floating-point operations, making them difficult to deploy on edge
devices. Our encoder employs an integer-8-compatible linear projection for
stripe-like HSI data transmission, ensuring real-time compressed sensing.
Furthermore, based on the novel two-streamed architecture, an efficient HSI
restoration decoder is proposed for the receiver side, allowing for edge-
device reconstruction without needing a sophisticated central server. This is
particularly crucial as an increasing number of miniaturized satellites
necessitates significant computing resources on the ground station. Extensive
experiments validate the superior performance of our approach, offering new
and vital capabilities for existing miniaturized satellite systems.

Index Terms— deep learning, hyperspectral image, compressed sensing,
hyperspectral restoration, real-time applications.

##  I Introduction

Hyperspectral remote sensing benefits tasks including environmental
monitoring, matter recognition, urban planning, agriculture, and surveillance.
Deep learning (DL), growing in popularity for vision tasks, has significantly
impacted hyperspectral remote sensing applications [1, 2, 3, 4], demonstrating
superior performance over conventional optimization-based approaches [5, 6, 7,
8, 9]. Hyperspectral images (HSI) contain hundreds of spectral bands, both
visible and invisible, resulting in a substantial data volume. Efficient
remote sensing is essential for transmitting HSI to ground receivers for
applications in matter recognition, urban planning, and agriculture. The
increase in Low Earth Orbit satellites (LEOS) based on CubeSat/miniaturized
technology has led to bandwidth challenges due to substantial data
transmission. Hyperspectral compressed sensing (HCS) is thus widely used for
concurrent sensing and compression of HSI [10, 11, 12, 13].

Hyperspectral sensors capture data in stripe-like sequences, forming HSI in
the along-track direction, a method known as pushbroom sensing. However,
satellite movement, orography, or IFOV (Instantaneous Field of View) issues
mean that spatial correlation between contiguous slices may not always be
guaranteed. Thus, we must perform geometric correction and geographic data
projection to achieve spatially correlated images. Given its high
computational load, this pre-processing is often deferred to post-transmission
at the ground station. Moreover, limited memory in miniaturized satellites
restricts the stacking of multiple spectral slices. Stripe-like encoding, as
proposed in DCSN [1], offers a solution by encoding fewer slices, emphasizing
the need for efficient and precise stripe-like hyperspectral data transmission
from miniaturized satellites to ground receivers.

Conventional data compression strategies like JPEG, JPEG2000, and video codecs
could theoretically reduce HSI data volume. Several approaches have been
proposed, utilizing principal component analysis (PCA) and codecs such as
JPEG2000 [14] and X264 [15]. However, given the prevalent use of pushbroom
sensing for acquiring HSI (stripe-like data) and the limited storage of
miniaturized satellites, traditional image/video compression techniques may
lack the necessary computational resources [14, 15], rendering these methods
incompatible for real-time compressed sensing in such satellites.

The power and computational limitations in CubeSats necessitate the adoption
of lightweight and efficient HCS. HCS can perform concurrent compression and
sensing, significantly easing computational and storage demands in
miniaturized satellites. Therefore, an ultra-lightweight HCS is crucial.
Spectral compressive acquisition (SpeCA) [10] leverages spatial and spectral
redundancies to create random projection matrices for low-dimensional data
extraction. Novel HCS methods utilizing $L_{1}$-norm and total-variation (TV)
regularization have been introduced [11, 12] to exploit global spatial-
spectral correlations in HSI data while maintaining local smoothness. A
spatial/spectral compressed encoder (SPACE), based on low-rank representation
[16], transforms HCS into an image fusion process by separately compressing
spatial and spectral data. The recently improved SPACE variant, All-Addition
Hyperspectral Compressed Sensing (AAHCS), enhances both spectral and spatial
qualities of reconstructed HSI data [17]. However, these optimization-based
HCS methods often require lengthy computational times due to the necessity of
eigendecomposition. Although AAHCS demonstrates fast sensing and high
performance through all-addition operations in the encoder [17], real-time
compressed sensing is hampered by the required eigendecomposition, which is
computationally intensive ($\mathcal{O}(n^{3})$) and thus challenging for on-
board processing in miniaturized satellites. Moreover, the stripe-like HSI
acquired through pushbroom sensing in moderate satellites does not align well
with these optimization-based HCS designs, potentially suppressing
performance.

Deep learning offers an effective and efficient solution for hyperspectral
compressed sensing (HCS) challenges. The recently introduced Deep Compressed
Sensing Network (DCSN) addresses stripe-like HSI data processing challenges in
miniaturized satellites [1]. DCSN features a non-square kernel within a three-
layer convolutional neural network (CNN), creating a lightweight encoder that
swiftly compresses HSI data before transmission to the ground station. For
data recovery, DCSN employs a complex decoder leveraging Multi-scale Feature
Fusion Block (MFB) and Aggregation (MFA). Despite DCSN’s state-of-the-art
performance at lower sampling rates (approximately 1%), achieving real-time
HCS is hindered by delays from multi-layer CNNs with non-linear activation
functions. Conventional activation functions, such as ReLU, demand additional
power and computational resources [18]. The hardware implementation of these
functions requires multiple units to approximate non-linear models, leading to
increased computation time [18, 19, 20]. Consequently, the LeakyReLU [21]
activation function used in DCSN’s encoder may not be practical for resource-
constrained devices, like edge devices [22, 23].

Minimizing the computational complexity of HSI data decoding is crucial. The
surge in edge computing, which involves processing on mobile or low-power
embedded devices, is increasingly popular due to its ability to reduce central
server load [24, 25, 26]. Quick reconstruction of compressed hyperspectral
data on these devices, including geometric correction and stitching,
alleviates decompression burdens on central servers. However, deploying
sophisticated decoders like those in DCSN [1] on edge devices is challenging.
Additionally, the iterative decoding approach of AAHCS [17] precludes real-
time processing on such devices. As the number of LEOS is expected to rise,
simplifying the decoder’s computational demands becomes essential for real-
time applications.

A critical yet often overlooked issue in hyperspectral compressed sensing
(HCS) is the striping effect caused by anomalies in hyperspectral sensor’s
electrical units [27]. While robust HCS methods like DCSN [1], AAHCS [17],
SPACE [16], and SpeCA [10] typically focus on channel noise during
transmission, they seldom address the striping effect. This phenomenon, often
handled as a data imputation challenge and addressed through HSI inpainting
[27, 28, 29, 30, 31], is especially prevalent in devices like NASA’s EO-1
satellite’s Hyperion imager [32]. Addressing this striping effect in HCS is
essential for enhancing the robustness of sensing techniques in miniaturized
satellites, aiming to minimize stripe artifacts in the reconstructed HSI.

Figure 1: The performance and computational complexity (decoding and encoding)
comparison between the proposed RTCS Network and other peer methods in terms
of frame-per-second (FPS) and Spectral angle mapper (SAM), Peak-Signal-to-
Noise (PSNR), and Root-Mean-Squared-Error (RMSE), where the DCSN [1] and our
RTCS are evaluated in Jetson TX2, the other methods are evaluated on a
personal computer, and the sampling rate is ranged in from 0.5% to 5%.

This manuscript presents the Real-time Compressed Sensing (RTCS) network, a
system innovatively designed for the efficient transmission and restoration of
hyperspectral data. It addresses challenges such as rapid sensing, stripe
effect mitigation, and noise-resistant transmission. Figure 1 demonstrates how
RTCS adeptly balances efficiency and effectiveness in real-time applications.
In our evaluations, conducted on the Jetson TX2 platform, RTCS and DCSN [1]
displayed notable performance, while other HCS methods were tested on a
personal computer due to their incompatibility with Arm-based architectures
(refer to Section III). In contrast to DCSN’s [1] reliance on multi-layer
nonlinear activations, which can impede real-time processing, RTCS employs a
straightforward linear projection. This approach effectively captures
essential spatial and spectral features, resonating with the success of linear
models in optimization-based HCS [16, 17]. Furthermore, the encoder in RTCS is
tailored for hardware efficiency and compatible with quantization, unlike the
high-precision requirements of traditional optimization-based methods.

With the burgeoning number of LEOS, real-time decoding on edge devices is
becoming increasingly crucial. Deeper networks typically result in slower
inference due to inter-layer dependencies. To address this, our proposed
decoder features a reduced depth to facilitate faster inference. It
incorporates a novel two-streamed network with a Cross-Scale Feature (CSF)
module, which efficiently captures a range of features across different
scales. We employ grouped convolution to significantly cut down on the number
of parameters and computational complexity, taking advantage of the redundancy
in HSI data spectra. This innovation enables the decoder to support real-time
HSI reconstruction on edge devices. Additionally, we introduce a Spectral
Angle Mapper-aware (SAM) Loss, aiming to enhance spectral quality by guiding
spectral signature reconstruction. The RTCS, with its task-specific training
approach, demonstrates exceptional resistance to stripe effects commonly
encountered in miniaturized satellites. Moreover, the unique two-streamed
network design, combined with RTCS’s streamlined architecture and reduced
parameter count, significantly boosting its adaptability and training
efficiency for various hyperspectral sensors.

The primary contributions of this paper are as follows:

  * •

Simple Linear Projection and Edge Device Compatibility in encoder: RTCS
introduces a unique linear projection in its encoder, precisely aligning with
the principles of compressed sensing. This projection not only adheres to the
integer-8 standard, making it highly suitable for edge devices, but also
offers significant power savings and computational reduction, deviating from
traditional optimization approaches.

  * •

Innovative Two-Streamed Network in decoder: The novel two-streamed network
architecture in RTCS employs a lightweight design that effectively balances
performance and robustness. This innovative approach enhances the system’s
capabilities without imposing additional computational complexity,
exemplifying an advanced integration of efficiency and functionality.

  * •

Task-Specific Training for Enhanced Capabilities: The RTCS leverages a task-
specific training policy, enabling it to excel not only in compressed sensing
for efficient transmission but also in reducing stripe effects in decoded
hyperspectral images. This marks the first instance of seamlessly integrating
compressed sensing and tensor inpainting within a single model, achieving
unparalleled performance across extensive datasets.

The remainder of this paper is organized as follows. Section II introduces the
proposed RTCS architecture design. Section III experimentally demonstrates the
superiority of RTCS over benchmark methods. Finally, Section IV presents the
conclusions drawn from this study.

##  II Proposed Method

Figure 2: Overview of the proposed RTCS framework for hyperspectral data
processing in CubeSat platforms, illustrating the compression and restoration
pipeline. The process includes stripe-like sensing, encoding on a low-power
chip, and efficient decoding on edge devices. The system is designed for real-
time operation, emphasizing minimizing memory usage and computational load for
resource-constrained satellite systems.

As illustrated in Figure 2, we present a novel transmission framework designed
to minimize the computational complexity inherent in the encoding and decoding
processes. This enables the swift decoding of hyperspectral data on edge
devices, thereby alleviating the computational burden on central servers. Such
a framework gains prominence as the constellation of Low Earth Orbit (LEO)
satellites, including CubeSats, expands. It allows central servers to allocate
their computational power more efficiently to demanding tasks, like geometric
correction or hyperspectral data integration, rather than spending it on
decoding. Thus, the ability to perform encoding and decoding in real-time is
crucial.

The Real-Time Compressed Sensing (RTCS) method presented in this study is
specifically designed for use in compact satellite systems. It introduces a
stripe-based compressed sensing technique that is compatible with the sensing
capabilities of modern miniaturized satellites and is optimized for low memory
usage. This approach leverages simple matrix operations to facilitate real-
time onboard satellite computations. The RTCS’s encoder operates through a
linear transform-based method that is both hardware-friendly and conducive to
integer operations, contrasting with traditional Hyperspectral Compressed
Sensing (HCS) that relies on high-precision floating-point operations. As a
result, our encoder is well-suited for edge devices with limited processing
power. The RTCS’s support for stripe-like HCS significantly reduces memory
demands on miniaturized satellites, requiring only four hyperspectral stripes
for effective compression and immediate transmission. Even on satellites with
stringent resource limitations, RTCS maintains its capability to encode
hyperspectral data efficiently.

Ground station decoding speed is a critical factor. With an increasing fleet
of Low Earth Orbit Satellites (LEOS), the central server may be inundated with
compressed hyperspectral data awaiting decoding and integration. The RTCS
method we propose circumvents the need for server-centric decoding, instead
facilitating the reconstruction of hyperspectral data directly on edge devices
within ground stations. This expedites data reconstruction and conserves
computational resources for more critical HSI tasks. For this purpose, we
introduce a Cross-Scale Feature (CSF) network decoder that employs a dual-
stream, shallow architecture, effectively extracting multi-scale features
while maintaining computational efficiency. This innovation significantly
reduces the workload on central servers.

Moreover, we propose a Spectral Angle Mapper-aware (SAM) Loss to guide the
network in preserving spectral signatures, thus enhancing spectral quality. A
task-specific training policy is subsequently introduced to improve the
robustness of our RTCS, allowing for real-time compressed sensing and reduced
stripe effects during the decoding phase. This feature renders the proposed
RTCS more practical. Due to the relatively few parameters of the proposed
RTCS, relatively few training samples are required to match the promising
performance, enhancing feasibility when re-training the RTCS for further
applications.

In the ensuing sections, we thoroughly discuss the effectiveness and
efficiency of the proposed encoder and decoder in our RTCS.

###  II-A Real-Time Compressed Sensing

The proposed encoder, though simple in design, proves effective by utilizing a
single matrix multiplication, similar to the traditional optimization-based
HCS that relies on random projection. This approach draws inspiration from the
linear subspace/basis concept in optimization-based HCS, which is known to
preserve most of the HSI’s energy. This forms the basis of our hypothesis that
a linear operation in the decoder could suffice for efficient compressed
sensing. Specifically, the encoder in our RTCS system consists of a single
convolution operation without any activation function, significantly reducing
computational complexity and latency. We employ a stripe-like input to align
with the pushbroom scanning mechanism typical in moderate LEOS, as indicated
in DCSN [1]. However, our measurement matrix design in compressed sensing
diverges from the encoder in DCSN [1], where a multi-layer approach might lead
to considerable encoding delay for hyperspectral stripes and fail to represent
a measurement matrix accurately. We contend that single matrix multiplication
can act as an efficient encoder, fully meeting the requirements of a
measurement matrix in compressed sensing.

Let the original high-resolution (HR) HSI be denoted as
${\bm{X}}_{o}\in\mathbb{R}^{B\times H\times W}$, where $B$, $H$, and $W$
represent the number of bands, height, and width of the HRHSI, respectively.
The stripe-like HSI, sensed by the pushbroom mechanism of the LEOS, is defined
as ${\bm{X}}={\bm{R}}{\bm{X}}_{o},$ where $\bm{R}$ is the sampling matrix used
to acquire the stripe-like HSI ${\bm{X}}\in\mathbb{R}^{B\times h\times w}$,
$h$ and $w$ represent the height and width of the stripe-like HSI. Our
compressed sensing in the proposed encoder could be done simply by

$\bm{Z}=\bm{\Psi}\bm{X},$ (1)

where $\bm{\Psi}$ is the measurement matrix formed by convolutional and
learnable coefficients. Let us explain in detail how the convolution benefits
the HCS. Suppose that a low-dimensional representation ${\bm{Z}}$ of the HSI
could be compressively sensed by a single learnable convolution kernel $W$ via

$Z(m,n)=\sum_{j=-\frac{k_{w}}{2}}^{\frac{k_{w}}{2}}\sum_{i=-\frac{k_{h}}{2}}^{\frac{k_{h}}{2}}X(i,j)\cdot
W(m-i,n-j),$ (2)

where $k_{w}$ and $k_{h}$ indicate the convolutional kernel’s width and
height. It is well-known that the convolutional could be expressed as matrix
multiplication. Consider a $3\times 3$ data matrix ${\bm{X}}$ and a $2\times
2$ convolution kernel ${\bm{W}}$, the convolution operation (stride $s=1$, no
padding $p=0$) is equivalent to:

${\bm{Z}}={\bm{X}}\ast{\bm{W}}=\begin{bmatrix}x_{1}&x_{2}&x_{3}\\\
x_{4}&x_{5}&x_{6}\\\ x_{7}&x_{8}&x_{9}\\\
\end{bmatrix}\ast\begin{bmatrix}w_{1}&w_{2}\\\ w_{3}&w_{4}\\\ \end{bmatrix}.$
(3)

This can be reformulated as a matrix multiplication by rearranging ${\bm{W}}$
with zero-padding and shifting to form ${\bm{W}}_{m}$ and vectorizing
${\bm{X}}$ into its column vector form ${\bm{X}}_{v}$:

$\displaystyle{\bm{Z}}$ $\displaystyle={\bm{W}}_{m}{\bm{X}}_{v}$ (4)
$\displaystyle=\begin{bmatrix}w_{1}&w_{2}&0&w_{3}&w_{4}&0&0&0&0\\\
0&w_{1}&w_{2}&0&w_{3}&w_{4}&0&0&0\\\ 0&0&0&w_{1}&w_{2}&0&w_{3}&w_{4}&0\\\
0&0&0&0&w_{1}&w_{2}&0&w_{3}&w_{4}\end{bmatrix}\begin{bmatrix}x_{1}\\\ x_{2}\\\
x_{3}\\\ x_{4}\\\ x_{5}\\\ x_{6}\\\ x_{7}\\\ x_{8}\\\ x_{9}\end{bmatrix}.$

In hyperspectral image compression, exploiting spatial and spectral
redundancies is essential. Traditional methods typically reshape 3D
hyperspectral data ${\bm{X}}\in\mathbb{R}^{B\times H\times W}$ into a 2D
matrix ${\bm{X}}_{2D}\in\mathbb{R}^{B\times(HW)}$ to achieve dimensionality
reduction. Linear transformations, for example, efficiently capture spectral
information and preserve energy, a principle that can be extended to single
convolution layers in deep learning frameworks. This strategy is particularly
well-suited for the constraints of miniaturized satellite imaging systems,
notably those utilizing pushbroom sensing. It effectively reduces the space
complexity that is otherwise inherent in compressing the full tensor.

Figure 3: The diagram of convolutional operation as matrix multiplication for
HCS in our RTCS encoder.

A concern is that the typical kernel size $3\times 3$ has a small receptive
field in the spatial dimension for HSI. As shown in the convolution example in
(3), the compact representation via convolution is determined by only four
coefficients $\bm{W}$, while traditional optimization-based HCS has a larger
number of coefficients for dimensionality reduction. Therefore, a larger
kernel size, such as $9\times 9$, is introduced to capture the long-range
dependency in the spatial domain, resulting in better energy preservation. The
kernel size is an example of the effective receptive field. To satisfy the
requirements of pushbroom sensing in the miniaturized satellites, we use
$9\times 1$ kernel in the encoder to enable the compressed sensing in
hyperspectral stripes. Assuming that the input HSI of size $B\times H\times W$
is projected to its compact representation
$\bm{Z}=\bm{X}\bm{\Psi},\in\mathbb{R}^{b\times h\times w}$, where $b\ll B,h\ll
H$, and $w\ll W$, $B\times k_{h}\times k_{w}$ coefficients are required to
project the HSI to a single feature map, and thus, $b\times B\times
k_{h}\times k_{w}$ coefficients are needed to generate the $b$ feature maps
for HSI. The complete matrix multiplication diagram of the convolution
operation in the proposed encoder is depicted in Figure 3, illustrating how
convolutional kernels capture the spatial and spectral dependency
simultaneously. Specifically, each column vector in the kernel matrix is used
to project the row vector of the HSI within a local window determined by
kernel size $k_{h}\times k_{w}$ to the lower dimensional space, where the
spectral redundancy is fully explored by $B\times k_{h}\times k_{w}$
coefficients. As the parameter space of the linear convolution is sufficiently
large, it should be able to project the HSI to a compact linear subspace.

It is crucial to discuss the computational complexity of the proposed encoder
as well. The computational complexity of the proposed encoder, primarily based
on matrix multiplication, is $\mathcal{O}(b(Bk_{w}k_{h})(hw))$ with stride
$s=1$ and padding $p=0$. In contrast, the major time cost of the most
efficient HCS method, AAHCS [17], lies in the basis computation via
eigendecomposition, where the complexity can be
$\mathcal{O}(B^{3}+(WH)B(WH))$. In AAHCS [17], the vectorized representation
${\bm{X}}_{\text{v}}$ of HSI ${\bm{X}}$ transforms the $B\times H\times W$
matrix into a $B\times(WH)$ matrix and performs eigendecomposition on the
square matrix ${\bm{U}}={\bm{X}}_{\text{v}}{\bm{X}}_{\text{v}}^{T}$ to obtain
the basis $\bm{\Psi}$. Although the optimization-based approach [17, 33]
claims that the basis matrix $\bm{\Psi}$ can be calculated from pre-collected
data, their experiments are evaluated by data-dependent basis (i.e., the basis
should be calculated for each HSI individually), resulting in
eigendecomposition being required onboard the miniaturized satellite during
HCS for optimal performance. In comparison to the computational complexity of
basis decomposition in AAHCS [17], $\mathcal{O}(B^{3}+(WH)B(WH))$, the matrix
${\bm{U}}$ demands $\mathcal{O}((WH)B(WH))$. When $w\ll W$, $h\ll H$, and
$b\ll B$, the computational complexity of the proposed encoder can be
significantly reduced. For instance, with a sampling rate of $\approx 1\%$ for
our encoder, we set the size of compressed HSI to $b=27,h=32,w=1$ (cf. Section
III) for the HSI sized at $172\times 128\times 4$, resulting in computational
complexity of approximately $3.95\times 10^{6}$, while the matrix ${\bm{U}}$
in AAHSC costs $45.088\times 10^{6}$. Furthermore, convolutional operations
can be highly parallelized in practice, significantly accelerating the
computing process and minimizing onboard computational effort in miniaturized
satellites. In summary, the proposed encoder is highly efficient and
effective.

###  II-B Hardware-Friendly Decoding and Restoration

Figure 4: The overview of the proposed hardware-friendly and effective decoder
for HSI compressed sensing and restoration. Figure 5: Network architecture of
the proposed two-streamed CSF-Net, where k(c)n(c) represents the number of
kernels and feature maps respectively.  Figure 6: The comparison between
grouped and standard convolution lies in their treatment of feature maps for
reducing the number of parameters and computational complexity.

As previously stated, decoding compressed HSI ${\bm{Z}}$ in edge devices can
effectively reduce the workload of central servers at ground stations. Toward
this goal, we propose a novel network structure for efficient decoding of
compressed HSI. While reducing the depth and number of kernels in a neural
network is an effective way to accelerate inference, overly simplifying the
network can detrimentally affect performance, despite achieving faster
inference speeds. For instance, in real-time applications, the MobileNet-based
architecture [34] is favored due to its computational efficiency. However,
there is still a noticeable performance gap between MobileNet and earlier
conventional CNN designs like SENet [35] and ResNet [36]. Specifically,
MobileNet’s best top-1 accuracy on the ImageNet validation set [37] is
$75.2\%$, compared to $78.57\%$ for ResNet-152 [36]. In the context of HCS,
particularly during the decoding phase, achieving promising performance is as
crucial as ensuring speed.

In this section, we highlight the benefits of our proposed network
architecture for the decoder, which is fast, hardware-friendly, and effective.
This architecture is illustrated in the Figure 4. Consider the decoder’s input
as the compressed HSI ${\bm{Z}}\in\mathbb{R}^{b\times h\times w}$. Our two-
stream-based network architecture is designed to learn features with varying
receptive fields efficiently. Initially, two stem convolution layers extract
low-level features ${\bm{Z}}_{stem}$ and $\bm{\zeta}_{stem}$ from $\bm{Z}$. In
one branch, we implement a modified version of the decoder from DCSN [1],
halving its depth and reducing parameters to decrease computational
complexity. In the other branch, we propose a Fast Residual-Dense-Block-based
Network (FRDB-Net), employing grouped convolution and larger kernels yet
maintaining a low parameter size. A Multi-Scale Feature Fusion (MFF) strategy
is introduced, combining feature representations from both branches through
element-wise addition. Subsequently, using an upsampling block, the low-
spatial-resolution (LSR) features are progressively upsampled to high-spatial-
resolution (HSR) features. Finally, the HSI is reconstructed from the low-
spectral-resolution (LSPR) features using a spectral upsampler via
convolution. Detailed descriptions follow in the subsequent paragraphs.

Cross-Scale Feature Network. We first discuss the significant advantages
conferred by the two-streamed network architecture in HCS. A direct reduction
in network depth can lead to substantial performance loss, as shallower
networks often struggle to learn rich feature representations. This is due, in
part, to the larger equivalent receptive field of deeper networks compared to
shallower ones. Our proposed two-streamed architecture aims to overcome the
limitations associated with shallower networks. It comprises two distinct sub-
networks utilizing $3\times 3$ and $9\times 9$ convolutional kernels to
capture features with different receptive fields. Feature aggregation from
these sub-networks is achieved through element-wise addition using a MFF
strategy. This approach allows for sharing cross-scale feature information,
which is then progressively upsampled in both spatial and spectral dimensions
to reconstruct the HSI.

The top line of Figure 4 presents the first branch, the Improved Multi-
Receptive-field Aggregation Network (IMFA-Net), while the bottom line
illustrates the second branch, our newly proposed FRDB-Net with a larger
kernel size. In developing the IMFA and FRDB blocks, we opted for a single
residual connection to minimize memory usage, addressing the issue of feature
map reuse seen in DCSN [1]. Our experiments demonstrate that this streamlined
approach with just one residual connection is both more effective and
efficient.

Specifically, the intermediate feature $\bm{Z}_{\text{m}}$ could be extracted
via

$\bm{Z}_{m}=f_{\text{u}}(f_{\text{IMFA}}(f_{\text{s}_{1}}({\bm{Z}})))+f_{\text{FRDB}}(f_{\text{s}_{2}}({\bm{Z}})),$
(5)

where $f_{\text{FRDB}}$ and $f_{\text{IMFA}}$ are the backbone networks of the
reduced IMFA and the proposed FRDB, $f_{\text{s}_{1}}$ and $f_{\text{s}_{2}}$
are the stem convolution layers for IMFA and FRDB. In IMFA-Net and the
proposed FRDB-Net, we stack $n_{f}$ IMFA and FRDB blocks, respectively, where
$n_{f}=8$ in this study. Since the number of the feature maps of
$f_{\text{IMFA}}$ is less than that of $f_{\text{FRDB}}$, an extra convolution
layer $f_{\text{u}}$ is used to project the feature map to the desired size.
Then, the HSR feature map $\bm{Z}_{h}$ could be learned based on the spatial
upsampler via

$\bm{Z}_{\text{h}}=f_{\text{up}}(\text{Bilinear}(\bm{Z}_{\text{m}})),$ (6)

where Bilinear stands for the bilinear interpolation function and
$f_{\text{up}}$ is a $3\times 3$ convolution layer. By repeating the function
6, the feature map with higher spatial resolution could be obtained. Finally,
the high spectral resolution HSI could be recovered from the HSR feature map
via

$\bm{X}^{*}=f_{\text{rec}}(\bm{Z}_{\text{h}}),$ (7)

where $f_{\text{rec}}$ is the convolution layer for mapping 96-channeled
feature maps to $B$-channeled ones. Note that the kernel size of most
convolution layers is $3\times 3$, except for the proposed FRDB-Net. The rest
of the question is the design of the reduced IMFA and the proposed FRDB
networks. More specifically, a larger kernel size used in the proposed FRDB
could result in heavy computational effort, how to effectively reduce the
complexity of the FRDB is essential.

Lightweight Design. In this subsection, we introduce the base blocks for IMFA-
Net and FRDB-Net, which are both effective and efficient. The detailed network
architectures of these blocks are depicted in Figure 5. As illustrated in
Figure 4, each block in both IMFA-Net and FRDB-Net comprises three stacked
base blocks with a shortcut connection linking the input and output feature
maps (Figure 4(a)). Within each base block, the input feature map is condensed
into a $c_{s}$-channeled feature map and merged with the input feature map,
resulting in a $c_{s}+N_{\text{Base}}$-channeled feature map (Figure 5(b)(c)).
Following DCSN [1], we set $N_{\text{Base}}$ to 64 and reduce $c_{s}$ to 16
from 32, decreasing computational complexity.

For the FRDB-Net, we draw inspiration from [1, 38] and use a residual dense
block (RDB) to construct its base block. We identify that feature map reuse is
the primary complexity factor in the RDB feed-forward process. Utilizing
depthwise separable convolution, as in MobileNet, effectively reduces
parameters and computational complexity. Additionally, grouped convolution,
employed in ResNeSt, further decreases complexity with an optimal number of
groups $c_{g}$. We apply grouped convolution in our FRDB base block to achieve
a balance between parameter count and complexity. Specifically, we set
$c_{g}=N_{\text{Base}}=c_{s}=96$, reducing the computational complexity in the
last layer of the FRDB base block (Figure 5(b)) by a factor of 5. This
approach shows that grouped convolution can lower computational complexity
without compromising performance.

The 2D convolution exploits spatial domain local correlation for complexity
reduction, just as the high redundancy between HSI bands permits further
computational efficiency. Figure 6 illustrates the efficacy of grouped
convolution in this context. Given the higher redundancy among successive HSI
bands than distant ones, grouped convolution naturally leverages this
redundancy, significantly reducing computational complexity and parameter
count without sacrificing performance. For example, with a $172\times
128\times 4$ stripe-like HSI, one feature map can be derived from 43 bands
(172/ 4) using grouped convolution when $c_{g}=4$, efficiently exploring local
redundancy in the spectral domain. The proposed decoder, incorporating IMFA-
Net, capitalizes on cross-scale feature interaction, ensuring promising
performance.

###  II-C Task-Specific Optimization

To boost efficiency, especially in the grouped convolution of our proposed
FRDB-Net, we have significantly reduced the number of channels in the feature
maps of the RTCS. This reduction could potentially lead to a slight decrease
in performance, particularly in terms of spectral quality, due to the
constrained parameter space. For instance, the proposed method comprises
approximately $6.291M$ parameters, in contrast to the $11.951M$ parameters in
DCSN [1]. DCSN [1] employs solely the $\ell_{1}$ loss for training, which
might not fully address the variance across different bands, reflecting
diverse values. This could mean that the $\ell_{1}$ loss might inadvertently
prioritize minimizing bands with higher values, potentially compromising
spectral fidelity in certain bands.

Recognizing the paramount importance of spectral index in HCS, we introduce a
SAM loss function to guide the proposed RTCS towards superior spectral index
outcomes more effectively. The SAM loss is defined as:

$\ell_{\text{SAM}}=\frac{1}{N}\sum_{n=1}^{N}\cos^{-1}\left(\text{clip}\left(\frac{\bm{X}_{n}^{T}\bm{X}^{*}_{n}+\epsilon}{\|\bm{X}_{n}\|_{2}\cdot\|\bm{X}_{n}^{*}\|_{2}+\epsilon}\right)\right)$
(8)

where $\bm{X}_{n}$ and $\bm{X}^{*}_{n}$ represent the $n$-th spectral vector
of the ground truth and the reconstructed HSI, respectively. The SAM loss
calculates the angular difference between these spectral vectors, using the
arc-cosine of the normalized dot product to measure cosine similarity. We add
a small constant $\epsilon$ to the denominator to prevent division by zero,
thus enhancing numerical stability. The clip function constrains the
similarity range between -1 and 1, ensuring stable arc-cosine calculations. By
focusing on angular differences instead of magnitudes, this loss provides a
robust metric for assessing the spectral fidelity of the reconstructed HSI
compared to the original. It emphasizes the orientation of spectral signatures
over their amplitude, facilitating a more accurate comparison of spectral
characteristics.

$\ell_{\text{T}}(\bm{X},\bm{X}^{*})=\lVert\bm{X}-\bm{X}^{*}\rVert+\alpha\ell_{\text{SAM}}(\bm{X},\bm{X}^{*}),$
(9)

where $\alpha$ is the weighting factor to control the balance between
$\ell_{1}$ and the proposed SAM loss. Furthermore, the stripe effect is often
presented in the existing hyperspectral image datasets due to sensor errors or
electrical exceptions. We also introduce this phenomenon into the training
process via random masking on the hyperspectral stripes. In this fashion, we
could enable the proposed RTCS to restore the complete hyperspectral stripes
from the missing one during the decoding stage. Specifically, let the binary
mask be $\bm{M}\in\mathbb{R}^{B\times H\times W}$, the masked HSI could be

$\bm{X}_{\text{mask}}=\bm{X}\odot\bm{M}.$ (10)

In our proposed method, the use of a mask is crucial, yet its design remains
relatively simple. The diverse characteristics of the stripe effect make it
impractical to capture all possible types during the training process. To
address this, we have implemented a basic mask design that involves randomly
masking parts of successive bands within a set of hyperspectral stripes. Based
on our research, we found that the likelihood of a hyperspectral stripe being
affected by stripe effects is about 0.2, and the maximum number of consecutive
bands impacted by stripe effects is also 0.2. This strategic approach
effectively equips the proposed RTCS technique to reconstruct hyperspectral
stripes while reducing the impact of the stripe effect. It ensures that the
RTCS can effectively reconstruct hyperspectral data using only partially
available compressed information.

In order to guide the proposed RTCS in effectively restoring the HSI with
stripe effect, the total loss function used for data imputation could be

$\displaystyle\ell_{\text{Aug.}}$
$\displaystyle=\ell_{\text{T}}(\bm{X},\bm{X}^{*})$ (11)
$\displaystyle+\lVert\bm{X}_{\text{mask}}-\bm{X}^{*}_{\text{mask}}\rVert+\alpha\ell_{\text{SAM}}(\bm{X}_{\text{mask}},\bm{X}^{*}_{\text{mask}}).$

Finally, the robust RTCS could be trained with the proposed total loss
function $\ell_{\text{Aug.}}$. The mask selection and design used in
$\ell_{\text{Aug.}}$ are discussed in the experiments.

##  III Experimental Results

###  III-A Experimental Settings

Dataset descriptions. The dataset used in this study was collected from AVIRIS
[39] sensor, which was pre-processed in the same way as DCSN [1]. We collected
the HSI data between 2008-2012, and the original HSI data was partitioned into
256 $\times$ 256 sub-images, resulting in 1735 HSI sub-images. We then applied
the k-means to cluster the HSI sub-images into five types, including City,
Mountain, Forest, Farm, and Other types, and manually corrected any
misclassified HSI sub-images. We selected 200 HSI sub-images from each class,
resulting in a total of 1200 HSI sub-images as the dataset. The dataset was
split into training, validation, and testing sets, consisting of 1000, 100,
and 100 HSI sub-images. The dataset will be released.

The size of the stripe-like HSI is designed to be $172\times 128\times 4$ in
the training phase, while $172\times 256\times 4$ stripe-like HSI is used in
the evaluation phase, leading to that only very limited memory usage is
required for storing four hyperspectral stripes for compressed sensing. But
the proposed RTCS still works well for a single hyperspectral stripe-based
compressed sensing. To evaluate the performance, we stack the decoded HSI
stripes to restore the HSI sub-image with $172\times 256\times 256$ since the
optimization-based approach requires full 3D-tensor as the input, followed by
measuring their objective quality by the metrics in [1], including Peak
signal-to-noise ratio (PSNR), Root-mean-square error (RMSE), and the Spectral
Angle Mapper (SAM).

Hyper-parameter settings. For the training phase of our RTCS model, the AdamW
optimizer [40, 41], an improved version of traditional Stochastic Gradient
Descent (SGD) algorithm, is employed to train our network, where ($\beta_{1}$,
$\beta_{2}$) are (0.9, 0.999), and the epsilon $\epsilon$ is
$1\textrm{e}\\!-\\!8$. Considering the possible diversity of HSI datasets,
including exhibiting extremely high variability due to their rich spectral
information and complex spatial features, the AdamW optimizer is crucial to
enhance the generalization ability of the model. The initial learning rate is
$1\textrm{e}\\!-\\!4$ with the step-wise learning rate decaying every 1,000
epochs with scaling factor $1/2$, and the batch size is set to $10$. The
number of total epochs of the proposed RTCS in the training phase is 5,000.
The sampling rate $s_{r}$ of the proposed RTCS is determined by the size of
the compact representation $\bm{Z}$ of the HSI $\bm{X}$. Specifically, the
spatial size of the HSI is reduced to be $\frac{1}{16}$ (i.e., $\frac{1}{4}$
for width and height) for the target sampling rate $s_{r}\leq 1\%$, while the
number of the feature maps $b$ of the compressed HSI $\bm{Z}$ could be
calculated based on $b=\lfloor B\times s_{r}\times 16\rfloor$. While the
sampling rate $s_{r}\geq 1\%$, the spatial reduction is $\frac{1}{4}$ and the
number of channels of $\bm{Z}$ could be $b=\lfloor B\times s_{r}\times
4\rfloor$.

TABLE I: Performance comparison between the proposed RTCS and other HCS
methods under different sampling rates (SR) $\approx 0.5\%$, $1\%$, and $5\%$.
The result in bold denotes the best method under the same sampling rate. Star
(*) denotes the method failed at SR=$0.5\%$. The data marked with *SS are
evaluation metrics that use only single stripe data.

Method | SR | Test set (PSNR$\uparrow$ / RMSE$\downarrow$ / SAM$\downarrow$)
---|---|---
C-type | M-type | F-type | L-type | O-type | Averaged
AAHCS*[17] | $1\%$ | 31.295 / 220.932 / 7.738 | 33.247 / 65.114 / 3.609 | 31.611 / 91.802 / 4.809 | 31.337 / 115.610 / 3.920 | 31.611 / 91.802 / 4.809 | 32.978 / 107.917 / 4.287
$5\%$ | 43.400 / 66.755 / 2.171 | 44.056 / 23.598 / 1.215 | 44.403 / 23.854 / 1.103 | 44.286 / 33.319 / 1.033 |  44.403 / 23.854 / 1.103 | 44.784 / 32.820 / 1.202
SPACE*[16] | $1\%$ | 26.275 / 437.212 / 8.861 | 22.073 / 389.708 / 6.077 | 22.389 / 446.935 / 8.131 | 21.340 / 879.260 / 6.902 | 29.445 / 245.843 / 2.441 | 24.304 / 479.792 / 6.483
$5\%$ | 38.607 / 56.614 / 2.171 | 38.021 / 26.525 / 1.597 | 37.511 / 27.172 / 1.485 | 37.901 / 34.258 / 1.500 | 39.092 / 27.860 / 0.898 | 38.226 / 34.486 / 1.530
SpeCA*[10] | $1\%$ | 15.092 / 590.184 / 22.785 | 19.122 / 216.356 / 13.100 | 13.867 / 421.521 / 19.745 | 12.814 / 535.711 / 22.127 | 23.255 / 168.216 / 7.714 | 16.830 / 386.398 / 17.094
$5\%$ | 27.942 / 180.709 / 6.627 | 29.225 / 68.090 / 3.873 | 27.615 / 83.614 / 4.178 | 27.424 / 117.731 / 4.795 | 32.417 / 67.002 / 1.800 | 28.924 / 103.429 / 4.255
| $0.5\%$ | 14.854 / 429.016 / 12.990 | 15.929 / 213.824 / 11.357 | 16.182 / 286.819 / 13.775 | 16.824 / 301.670 / 12.180 | 19.161 / 218.439 / 8.197 | 16.590 / 289.954 / 11.700
TenTV [12] | $1\%$ | 16.921 / 356.217 / 10.692 | 17.612 / 169.388 / 8.749 | 18.195 / 210.931 / 9.709 | 18.891 / 220.237 / 8.822 | 19.484 / 164.420 / 5.856 | 18.220 / 224.239 / 8.766
| $5\%$ | 22.553 / 208.414 / 6.793 | 23.177 / 97.187 / 5.101 | 24.149 / 108.601 / 4.867 | 24.924 / 111.323 / 4.411 | 25.313 / 84.591 / 2.916 | 24.023 / 122.023 / 4.818
| $0.5\%$ | 21.885 / 1949.291 / 12.710 | 18.691 / 338.328 / 12.089 | 18.361 / 359.933 / 11.898 | 16.375 / 586.444 / 14.185 | 21.779 / 387.668 / 7.313 | 19.418 / 724.333 / 11.639
H-LSS[13] | $1\%$ | 24.120 / 324.823 / 8.691 | 21.803 / 244.465 / 9.023 | 22.491 / 197.237 / 8.262 | 20.687 / 329.950 / 9.498 | 25.405 / 229.222 / 4.799 | 22.901 / 265.139 / 8.054
| $5\%$ | 27.622 / 194.043 / 6.247 | 28.720 / 79.231 / 4.730 | 26.769 / 104.582 / 5.315 | 26.292 / 129.837 / 5.716 | 32.007 / 70.704 / 2.574 | 28.282 / 115.679 / 4.916
| $0.5\%$ | 29.398 / 152.900 / 3.733 | 30.588 / 59.821 / 2.466 | 30.674 / 69.437 / 2.454 | 31.340 / 83.893 / 2.465 | 32.714 / 54.669 / 1.357 | 30.943 / 84.144 / 2.495
DCSN[1] | $1\%$ | 31.446 / 127.919 / 2.745 | 32.450 / 48.979 / 1.878 | 32.372 / 58.523 / 1.897 | 32.986 / 69.670 / 1.879 | 34.263 / 47.262 / 1.064 | 32.703 / 70.471 / 1.893
| $5\%$ | 35.938 / 86.090 / 1.101 | 37.651 / 30.632 / 0.797 | 36.873 / 39.761 / 0.832 | 37.251 / 47.666 / 0.757 | 38.726 / 31.610 / 0.467 | 37.288 / 47.152 / 0.791
| $0.5\%$ |  31.601 / 120.754 / 2.377 |  32.034 / 50.676 / 1.680 |  32.583 / 54.021 / 1.590 |  33.617 / 61.818 / 1.415 |  34.862 / 44.466 / 0.867 |  32.939 / 66.353 / 1.586
RTCS | $1\%$ |  36.036 / 74.521 / 1.66 |  36.113 / 31.814 / 1.215 |  36.129 / 33.908 / 1.169 |  36.952 / 40.374 / 1.092 |  37.455 / 37.788 / 0.734 |  36.537 / 43.681 / 1.176
| $5\%$ | 47.795 / 17.187 / 0.608 | 45.535 / 9.571 / 0.578 | 44.479 / 11.840 / 0.611 | 44.367 / 16.573 / 0.620 | 43.376 / 22.100 / 0.497 | 45.110 / 15.455 / 0.583
RTCS *SS | $1\%$ | 36.767 / 67.353 / 1.754 | 35.571 / 32.479 / 1.352 | 35.797 / 33.791 / 1.274 | 37.119 / 38.141 / 1.178 | 37.894 / 31.161 / 0.784 | 36.629 / 40.585 / 1.269

###  III-B Performance Evaluation

To illustrate the superiority of RTCS, seven state-of-the-art HCS methods are
selected as baselines, including AAHCS[17], SPACE[16], SpeCA[10], TenTV[12],
Hyper-LSS (H-LSS) [13], and the prior art, DCSN[1].

We follow the standard HSI-based metrics used in DCSN [1], including PSNR,
RMSE, and SAM, to evaluate the performance of the proposed and other peer
methods, where PSNR and RMSE are used to evaluate the spatial fidelity while
SAM is adopted to verify the degree of spectral preservation of the
reconstructed HSI.

####  III-B1 Objective Quality Assessment

To demonstrate the robustness and generalizability of our RTCS framework, we
carried out cross-type dataset evaluations. The RTCS was trained on one type
from the dataset and then tested across four different types. As shown in
Table I, the performance of RTCS is benchmarked against other methods using
metrics like SAM, PSNR, and RMSE. Across all these indices, RTCS consistently
surpasses other methods, proving its robustness even when applied to a single
hyperspectral stripe without relying on a large memory buffer, as evidenced by
the last row in the Table I.

Significantly, RTCS shows superior performance compared to other deep-
learning-based HCS methods, such as DCSN [1], despite being trained on a
smaller dataset. While DCSN uses 2,300 training samples, RTCS requires only
1,000, highlighting the efficiency of our model in a limited-data scenario.
This difference indicates that more complex network structures might need
larger datasets for similar performance levels. By simplifying the network
architecture, RTCS not only accommodates but excels in few-shot learning
scenarios, demonstrating its practicality and efficiency.

Furthermore, as the sampling rate increases, for instance, to $5\%$, most HCS
methods, including RTCS, AAHCS [17], and DCSN [1], show significant
performance improvements, as seen in the Table I. In AAHCS , self-similarity
regularization in the spatial domain effectively aids the restoration task,
yielding comparable PSNR and RMSE for the reconstructed HSI. However, RTCS
outperforms AAHCS in terms of SAM, suggesting superior preservation of
spectral signatures by RTCS.

Consider a scenario where bandwidth is severely limited due to an increase in
the number of miniaturized satellites. In such cases, a viable solution might
involve reducing the sampling rate of the proposed RTCS even further. As shown
in the Table I, the RTCS demonstrates considerable performance at an
approximate sampling rate of 0.5%, while the performance of DCSN [1] markedly
declines at such low rates. It is worth noting that some partial optimization-
based HCS methods, such as AAHCS [17] and SPACE [16], couldn’t assess
performance at this low sampling rate due to significant memory usage and
prolonged decoding times. Consequently, the proposed RTCS stands out,
especially at lower sampling rates ($s_{r}$), achieving state-of-the-art
performance compared to learning-based HCS methods like DCSN. Furthermore,
optimization-based HCS methods generally struggle to deliver satisfactory
results at very low sampling rates.

When comparing the performance of the proposed RTCS with other advanced HCS
methods, RTCS distinctly surpasses them. Notably, RTCS effectively manages the
stripe effect and transmission noise associated with pushbroom scanning,
delivering promising results. Additionally, RTCS is fast, lightweight, and
hardware-friendly, making it suitable for real-time applications without
compromising its leading performance. In Section III-C, we will delve into the
robustness of the proposed RTCS, encompassing aspects such as restoration with
stripe effect, generalizability, and resistance to transmission noise.

###  III-C Robustness Evaluation

While the performance of the proposed RTCS achieves state-of-the-art in the
ideal case, the robustness and stability evaluation of the proposed RTCS for
real-world scenarios of the HCS and data transmission could be more essential.
We conduct the robustness evaluation in this subsection, including the cases
of channel noise during transmission, stripe effect due to partially and
temporarily broken sensors, and few-shot compatibility.

Figure 7: The visualized examples of (a) hyperspectral image (HS) for three
missing types, including (b) partially missing (PM), (c) band-wise missing
(BM), and (d) completely missing (CM).

####  III-C1 Restoration with Missing Data

As longer-running sensors at the miniaturized satellite in space, a partial
spectrum could be missing during compressed sensing, known as the stripe
effect of HSI. Most approach treats this could be resolved by post-processing,
known as HSI inpainting techniques [27]. With the currently accessible data
portal of HSI, some stripe-type stripe effects in the multiple successive
bands could be found in [27], which is also shown in the HSI inpainting
approach ADMM-ADAM [27]). In this study, we show that the proposed RTCS can
recover the complete HSI from the corrupted HSI using the proposed loss
function $\ell_{\text{Aug.}}$.

To evaluate the robustness of the stripe effect, three types of stripe effect
of HSI are studied in this experiment, as follows:

  * •

PM (Partially missing): Some of the lines of the HSI stripe are sometimes
missing at continuous bands due to unexpected electrical errors on the
miniaturized satellite, as an example shown in the Figure 7(a). It is the
lightest missing type.

  * •

BM (Band-wise missing): The sensor for a range of the bands is partially
broken, resulting in partially corrupted data in each HSI stripe, as shown in
the regular pattern of Figure 7(b).

  * •

CM (Completely missing): The sensor for a range of the bands is completely
broken, resulting in no information in the continuous bands, as shown in the
Figure 7 (c).

Only PM and CM are adopted for $\ell_{\text{Aug.}}$ with the randomly selected
stripes and the continuous bands in the training phase. In the testing phase,
the fixed missing pattern for three types is generated by the fixed random
seed to ensure the fairness of the performance comparison between the proposed
and other HCS methods. Note that the BM is not involved in the training phase;
thus the results of the BM in the testing phase could be regarded as the
generalizability evaluation.

TABLE II: Robustness evaluation of the different stripe effect types for the
proposed RTCS and other peer methods. The result in bold denotes the best
method under the same setting of missing bands. (sampling rate $\approx 1\%$)

Method | Missing bands | Missing type (PSNR$\uparrow$ / RMSE$\downarrow$ / SAM$\downarrow$)
---|---|---
Band-wise missing | Partially missing | Completely missing | Averaged
AAHCS[17] | 50-60 | 31.547 / 1138.582 / 7.091 | 31.613 / 1158.320 / 6.174 | 31.666 / 1133.931 / 4.309 | 31.608 / 1143.611 / 5.858
50-80 | 28.639 / 1107.729 / 9.758 | 29.146 / 1147.370 / 7.674 | 29.000 / 1079.546 / 4.267 | 28.928 / 1111.548 / 7.233
SpeCA[10] | 50-60 | 14.610 / 445.688 / 20.848 | 15.196 / 428.369 / 18.752 | 14.880 / 447.976 / 21.431 | 14.895 / 440.667 / 20.343
50-80 | 11.565 / 563.434 / 25.992 | 12.751 / 510.790 / 22.620 | 10.945 / 616.859 / 29.180 | 11.753 / 563.694 / 25.907
TenTV[12] | 50-60 | 17.784 / 274.052 / 11.458 | 18.199 / 228.665 / 8.955 | 16.988 / 325.028 / 14.194 | 17.657 / 275.915 / 11.535
50-80 | 16.802 / 338.644 / 14.992 | 17.798 / 244.089 / 9.713 | 13.946 / 489.591 / 23.134 | 16.182 / 357.441 / 15.946
H-LSS[13] | 50-60 | 20.719 / 423.186 / 14.204 | 21.860 / 384.037 / 8.756 | 22.158 / 403.553 / 16.952 | 21.579 / 403.592 / 13.304
50-80 | 19.553 / 433.793 / 17.562 | 21.153 / 388.866 / 9.073 | 20.434 / 509.740 / 24.233 | 20.373 / 444.133 / 16.956
SPACE[16] | 50-60 | 26.69 / 243.598 / 11.740 | 28.546 / 135.776 / 5.634 | 24.899 / 298.634 / 14.563 | 26.711 / 226.002 / 10.645
50-80 | 22.540 / 320.169 / 15.180 | 26.119 / 172.501 / 6.638 | 17.632 / 452.930 / 22.559 | 23.430 / 315.2 / 14.792
DCSN[1] | 50-60 | 26.562 / 133.927 / 4.543 | 28.681 / 102.926 / 3.173 | 25.443 / 149.647 / 5.379 | 26.895 / 128.833 / 4.365
50-80 | 22.169 / 216.315 / 8.867 | 26.110 / 139.823 / 4.447 | 19.314 / 288.096 / 12.701 | 22.531 / 214.744 / 8.671
RTCS | 50-60 | 33.628 / 57.824 / 1.798 | 35.261 / 48.993 / 1.396 | 33.318 / 59.726 / 1.881 | 34.069 / 55.514 / 1.691
50-80 | 32.450 / 64.309 / 2.079 | 35.157 / 49.389 / 1.408 | 33.058 / 61.972 / 2.002 | 33.555 / 58.556 / 1.829

Table II presents the performance comparison between the proposed RTCS and
other HCS methods for three stripe effect types. Note that the sampling rate
$s_{r}$ in the experiments is trained and evaluated on $\approx 1\%$. All the
stripe effects could result in significant performance degradation for most of
the HCS methods, except for the proposed RTCS. An even worse case is assumed
that the partial sensors for the specific bands are completely broken (i.e.,
CM), resulting in a discontinuity at a range in the spectral domain. In this
case, the spectral signature (i.e., SAM) of the reconstructed HSI is
significantly degraded for most of the HCS methods under the stripe effect.
However, the proposed RTCS still remains strong, even in successive 30 bands
completely missing. In BM type, the proposed RTCS also presents promising
performance, implying that the proposed RTCS has the ability to deal with the
new missing types which are not observed in the training phase. While the DCSN
achieves promising performance on the pure HCS, the stripe effect still
significantly harms the effectiveness due to the lack of the proposed
$\ell_{\text{Aug}}$. In this way, the compressed hyperspectral stripes could
be well reconstructed based on the spectral and spatial context, greatly
reducing the stripe effect of the reconstructed HSI based on the decoded and
restored hyperspectral stripes. Figure 8 reveals the visualized results using
the false color of the reconstructed HSI based on the proposed RTCS and other
peer methods in CM-type distortion, showing that the proposed RTCS presents
more reliable and high-spectral-spatial quality of the reconstructed HSI under
stripe effect scenarios.

Figure 8: The false color representation of the visualized hyperspectral image
of the (a) ground truth, and the reconstructed counterpart under completely
missing (CM) bands from 50-80 using (b) the proposed RTCS, (c) SpeCA [10], (d)
DCSN [1], (e) H-LSS [10], (f) TenTV [12] , and (g) AAHCS [17] and followed by
inpainting the stripe effect using ADMM-ADAM [27].

As for visualized results comparison, the false RGB image of HSI could be
synthesized by normalizing the band indices 13, 25, and 61. Figure 9
illustrates the false RGB image visualized from the reconstructed HSI of the
different HCS methods under the 50-80 bands completely missing (i.e., CM), in
which the red spectrum is also missing. As a result, the visualized results of
the proposed RTCS are significantly better than that of other peer methods,
implying the spectral signature of the reconstructed HSI using the proposed
RTCS is promising. The state-of-the-art HCS methods, AAHCS [17] and DCSN [1],
fail to reconstruct the visually pleasing results, implying the stripe effect
could greatly influence the compressed signal $\bm{Z}$. Although the state-of-
the-art inpainting method is adopted to recover the complete HSI from the
corrupted one, the lower spatial and spectral quality cannot provide rich
enough information for further remote-sensing-related analysis. Since the
results reported in ADMM-ADAM [27] is evaluated in the pure and high-quality
HSI with stripe effect, it is hard to guarantee such high-quality of the
decoded HSI under a very low sampling rate (e.g., $\approx 1\%$), leading to
unstable performance. In contrast, the proposed RTCS successfully reconstructs
the missing parts from the compressed representation directly, leading to a
promising visual quality of the decoded HSI and enabling the other downstream
applications possible, even when some sensors are partly/completely broken.

Figure 9: The false color representation of the visualized hyperspectral image
of the (a) ground truth, and the reconstructed counterpart under noised data
with SNR = 15dB and sampling rate $\approx 1\%$, using (b) noised data (c) the
proposed RTCS, (d) DCSN [1], (e) TenTV [12], (f) AAHCS [17], (g) SPACE [16],
(h) H-LSS [10].

As one can argue that the missing types could vary due to different types of
sensors, the proposed RTCS requires to re-train the network, consisting of
encoder and decoder, to enable the capability of HSI inpainting of the new
missing type. Notably, the proposed RTCS is few-shot-compatible (cf. the
following section), leading the cost of fine-tuning the RTCS for new types of
stripe effects could be low and easy.

TABLE III: Performance comparison under different numbers of the training
samples (%) between the proposed RTCS and other HCS methods (sampling rate
$\approx 1\%$).

Method |  Training samples | Test set (PSNR$\uparrow$ / RMSE$\downarrow$ / SAM$\downarrow$)
---|---|---
C-type | M-type | F-type | L-type | O-type | Averaged
DCSN[1] | 10% | 28.038/176.860/4.287 | 28.705/71.969/3.044 | 28.833/84.082/3.101 | 29.533/103.691/3.302 | 30.576/70.463/1.865 | 29.137/101.413/3.120
20% | 29.062/165.511/3.984 | 29.428/68.925/2.772 | 28.540/84.651/3.009 | 29.247/105.912/3.224 | 30.432/72.712/1.793 | 29.342/99.542/2.957
RTCS | 10% | 34.264/92.628/2.053 | 34.030/39.627/1.611 | 34.102/44.867/1.522 | 34.763/51.488/1.522 | 35.664/42.866/0.932 | 34.565/54.295/1.528
20% | 34.457/89.947/1.999 | 34.347/38.111/1.516 | 34.528/42.481/1.430 | 35.322/48.756/1.385 | 35.990/41.314/0.887 | 34.929/52.122/1.444

Figure 10: The performance comparison of the DL-based HCS methods on varying
amounts of the training samples.

####  III-C2 Generalizability Evaluation

The data-driven approach usually suffers performance degradation issues from
insufficient training samples. Compared to the optimization-based HCS methods,
the data collection in the DL-based HCS methods is usually tedious and time-
consuming. Since the number of parameters of the proposed RTCS is relatively
small, the proposed RTCS should be able to train well using a relatively
small-scale training set without overfitting. In this Section, we reduce the
number of training data by randomly sampling $10\%$ to $40\%$ from the
original training set (say, 1,000 HSI) to evaluate the performance of the same
test set (say, 100 HSI). As shown in the Table III, the proposed RTCS could
obtain the promising performance using only $10\%$ training samples, i.e., 100
HSI only, while the DCSN [1] requires $100\%$ training set to have similar
performance. As claimed in the ADMM-ADAM theory [27], a huge amount of the HSI
collection is infeasible, so they have developed ADMM-ADAM using just small
training data consisting of 400 HSI. However, we show that the proposed RTCS
is more effective, even with only 100 HSI as the training set. As shown in the
Figure 10, the proposed RTCS improves smoothly when the number of training
samples increases to the $30\%$ of the original dataset, implying that the
performance of our RTCS becomes promising using just a small amount of the
training data.

Besides data collection is time-consuming for DL-based HCS methods, however,
the proposed RTCS only requires a tiny amount of the training samples, making
our RTCS useful and practical. Once a new type of HSI sensor appears, the
proposed RTCS requires only small data, making the HCS training and deployment
more effective and efficient.

Our RTCS was developed on a proprietary dataset in the initial phase, laying
the groundwork for a foundational model in hyperspectral imaging (HSI)
compressed sensing. This foundational model facilitates rapid domain
adaptation through fine-tuning with a limited subset of a target dataset—a
method that proves to be computationally economical.

To evaluate the framework’s performance, we engaged in further experimentation
using an auxiliary dataset referenced as [1]. The RTCS framework was subjected
to training on the designated training subset of this dataset, with its
effectiveness subsequently gauged on the test subset. In an endeavor to
validate the model’s applicability, only 10% of the [1] training samples were
employed to fine-tune the pre-established RTCS model, which substantiated its
efficacy on the test subset. The DCSN dataset [1] encompasses 2300 HSI
samples, encapsulating a variety of landscapes, including City (C-type), Land
(L-type), Farm (F-type), and Mountainous (M-type) terrains. Training the RTCS
on the entirety of this dataset is projected to enhance performance; however,
the collection of comprehensive datasets for novel target HSI applications is
often constrained by practicality and resource availability.

The few-shot compatibility embedded within our RTCS enables the efficient
training of the network on novel datasets with a minimal quantity of samples,
mirroring the process of task-specific fine-tuning seen with foundational
models. Illustrated in the Table V, the RTCS achieves state-of-the-art
performance with a mere 1% of the training samples, equivalent to 23 HSI
images. This advancement highlights the significance and viability of Few-shot
Compatibility in practical settings, where extensive data collection and model
training are typically arduous. Crucially, this feature permits the swift and
efficacious deployment of our RTCS across a spectrum of HSI types, employing
only a handful of training samples and thereby promoting extensive
adaptability and practicality for CubeSat HSI applications.

TABLE IV: SNR performance comparison under sampling rate $\approx 1\%$ and
$5\%$ between the proposed RTCS and other state-of-the-art HCS methods. The
result in bold denotes the best method under the same sampling rate.

Method | SR | SNR (PSNR$\uparrow$ / RMSE$\downarrow$ / SAM$\downarrow$)
---|---|---
25dB | 30dB | 35dB | 40dB
AAHCSD[17] | $1\%$ | 24.426 / 1181.978 / 7.574 | 27.231 / 1173.048 / 5.760 | 29.471 / 1169.741 / 4.908 | 31.016 / 1168.419 / 4.533
$5\%$ | 25.995 / 1455.559 / 9.553 | 28.296 / 1250.833 / 5.783 | 30.976 / 1177.976 / 3.880 | 33.420 / 1173.690 / 2.959
SPACE[16] | $1\%$ | 24.880 / 151.329 / 6.730 | 27.539 / 88.070 / 3.149 | 28.616 / 112.770 / 4.843 | 29.456 / 108.174 / 4.585
$5\%$ | 28.452 / 91.104 / 4.404 | 32.060 / 58.909 / 2.847 | 34.875 / 44.118 / 2.087 | 36.576 / 38.130 / 1.755
SpeCA[10] | $1\%$ | 14.032 / 446.873 / 19.291 | 15.582 / 409.135 / 17.873 | 16.300 / 394.734 / 17.352 | 16.666 / 389.423 / 17.172
$5\%$ | 22.163 / 209.673 / 10.392 | 24.340 / 156.942 / 7.613 | 26.048 / 129.245 / 6.039 | 27.326 / 114.161 / 5.104
TenTV[12] | $1\%$ | 18.486 / 258.820 / 10.333 | 18.498 / 237.224 / 9.359 | 18.321 / 229.991 / 9.024 | 18.248 / 226.777 / 8.875
$5\%$ | 22.388 / 156.861 / 6.539 | 23.240 / 138.504 / 5.607 | 23.721 / 128.602 / 5.130 | 23.925 / 124.294 / 4.926
H-LSS[13] | $1\%$ | 22.564 / 264.779 / 8.367 | 22.726 / 246.871 / 8.227 | 22.900 / 268.792 / 8.335 | 22.861 / 250.298 / 8.185
$5\%$ | 26.245 / 137.323 / 5.631 | 27.318 / 123.489 / 5.244 | 27.803 / 118.771 / 5.042 | 28.129 / 118.016 / 4.976
DCSN[1] | $1\%$ | 31.308 / 80.501 / 2.382 | 32.153 / 74.007 / 2.079 | 32.501 / 71.657 / 1.959 | 32.635 / 70.858 / 1.915
$5\%$ | 36.301 / 50.942 / 1.029 | 36.820 / 48.958 / 0.899 | 37.009 / 48.325 / 0.847 | 37.075 / 48.119 / 0.828
RTCS | $1\%$ | 35.649 / 48.190 / 1.309 | 36.254 / 45.041 / 1.216 | 36.470 / 43.995 / 1.184 | 36.542 / 43.672 / 1.173
$5\%$ | 40.264 / 26.433 / 0.974 | 42.137 / 21.451 / 0.777 | 43.006 / 19.623 / 0.704 | 43.332 / 19.018 / 0.679

TABLE V: Performance comparison between the proposed RTCS and other HCS
methods under a sampling rate (SR) of $\approx 1\%$ for the dataset in [1].
The result in bold denotes the best method under the same sampling rate, and
RTCS (1%) denotes that the RTCS was fine-tuned by only 1% training samples of
the entire dataset in [1].

Method |  Dataset in [1] (PSNR$\uparrow$ / RMSE$\downarrow$ / SAM$\downarrow$)
---|---
C-type | M-type | F-type | L-type | Averaged
AAHCS*[17] | 30.259 / 169.090 / 6.767 | 31.496 / 37.446 / 3.617 | 31.340 / 59.915 / 4.375 | 28.768 / 49.172 / 4.733 | 31.121 / 46.418/ 3.950
SPACE*[16] | 24.129 / 613.661 /7.207 | 29.161 / 140.415 / 3.743 | 29.674 / 64.151 / 3.121 | 27.727 / 209.757 / 4.446 | 28.855 / 158.751 / 3.892
SpeCA*[10] | 9.299 / 784.867 / 42.863 | 15.377 / 234.735 / 21.510 | 11.701 / 407.530 / 33.036 | 14.024 / 225.772 / 22.006 | 14.585 / 274.191 / 23.659
TenTV[12] | 20.208 / 570.255 / 26.247 | 18.533 / 260.221 / 22.972 | 20.401 / 248.994 / 18.714 | 18.824 / 314.248 / 25.523 | 18.835 / 277.403 / 22.930
H-LSS[13] | 20.798 / 228.967 / 9.241 | 23.942 / 93.679 / 6.206 | 21.552 / 73.796 / 9.898 | 23.578 / 331.702 / 8.761 | 23.518 / 123.890 / 7.016
DCSN[1] | 34.274 / 65.120 / 2.016 | 33.729 / 30.620 / 1.631 | 35.908 / 17.408 / 1.380 | 35.566 / 21.558 / 1.408 | 34.195 / 29.526 / 1.594
RTCS (1% / 23 HSIs) | 34.278 / 66.858 / 1.849 | 32.499 / 33.231 / 1.830 | 35.694 / 13.562 / 1.418 | 33.984 / 35.138 / 1.530 | 34.114 / 37.197 / 1.657
RTCS (10% / 230 HSIs) | 35.160 / 62.948 / 1.656 | 34.074 / 29.049 / 1.471 | 36.835 / 12.420 / 1.235 | 35.407 / 30.906 / 1.259 | 34.573 / 29.259 / 1.428

TABLE VI: The run-time and computational complexity comparison of the decoders
of the proposed RTCS and other methods under different computing platforms.

Method | Computing platform
---|---
GPU (GTX 1660) | CPU (i9-9900) | Jetson TX2 GPU | Denver 2 64-bit CPU
#Params | FLOPs | Run-time | PSNR$\uparrow$ / RMSE$\downarrow$ / SAM$\downarrow$ | Run-time | Run-time | Run-time
AAHCS[17] | - | - | - | 32.978 / 107.917/4.287 | 274.15 | - | -
SpeCA[10] | - | - | - | 16.830 / 386.398 / 17.094 | 0.257 | - | -
DCSN[1] | 11.95M | 0.866G | 0.035 | 32.703 / 70.471 / 1.893 | 0.431 | 0.033 | 3.85
DCSN-TVM[1] | 11.95M | 0.866G | 0.023 | 32.703 / 70.471 / 1.893 | 0.383 | 0.025 | 2.119
RTCS | 6.291M | 0.297G | 0.023 | 36.537 / 43.681 / 1.176 | 0.392 | 0.024 | 3.497
RTCS-TVM | 6.291M | 0.297G | 0.011 | 36.537 / 43.681 / 1.176 | 0.243 | 0.019 | 2.011

TABLE VII: The performance comparison of the proposed RTCS with/without
integer-8-based encoder.

Method | Test set (PSNR$\uparrow$ / RMSE$\downarrow$ / SAM$\downarrow$)
---|---
C-type | M-type | F-type | L-type | O-type | Averaged
FP32 | 36.036 / 74.521 / 1.66 | 36.113 / 31.814 / 1.215 | 36.129 / 33.908 / 1.169 | 36.952 / 40.374 / 1.092 | 37.455 / 37.788 / 0.734 | 36.537/43.681/1.176
INT8 (QAT) | 34.992 / 84.096 / 1.973 | 34.914 / 36.454 / 1.434 | 34.801 / 40.819 / 1.472 | 35.353 / 48.415 / 1.400 | 34.891 / 46.213 / 1.024 | 34.990/51.200/1.461
INT8 (PAQ) | 34.952 / 84.748 / 1.993 | 34.844 / 36.734 / 1.448 | 34.714 / 41.221 / 1.492 | 35.280 / 49.002 / 1.419 | 34.858 / 46.616 / 1.038 | 34.929/51.664/1.478

####  III-C3 Transmission Noise

In order to evaluate the robustness of the proposed method, we follow the
training process suggested in Denoised AutoEncoder[42, 43] and DCSN [1] to add
the Gaussian noise with the 30dB Signal-to-Noise-Ratio (SNR) to learn the
robust RTCS model. The rest hyperparameters of the proposed RTCS are identical
to the default settings. Table IV draws the performance comparison between the
proposed RTCS and other state-of-the-art HCS methods under sampling rates
$\approx 1\%$ and $\approx 5\%$. As a result, the proposed RTCS significantly
outperforms the other HCS methods in terms of SAM, PSNR, and RMSE. Remarkably,
the Robust DCSN (RDCSN) [1] also adopts the Gaussian noise during the training
phase to make the model could robust to noise. However, the proposed RTCS
still achieves a notable performance improvement, implying that the
architecture of the proposed RTCS is better than that of DCSN [1] naturally.
The results in the Figure 9 has shown the efficacy of the proposed RTCS.
Compared with other HCS methods that have varying degrees of chromatic
aberration, the reconstruction results of RTCS are closer to the ground-truth.

###  III-D Computational Complexity Analysis

We compare the proposed RTCS to the other state-of-the-art HCS methods for
computational complexity and run-time of encoding and decoding. Note that each
result is measured with the same hardware. In the general computational
comparison, a personal computer equipped with an i9-9900 CPU, 128GB Memory,
and a low-end graphics processing unit (GPU) is used to measure the run-time
of HCS for an HSI sized of $172\times 256\times 256$. Although the proposed
RTCS is a stripe-like HCS, the run-time of the RTCS and DCSN [1] is measured
in the whole HSI sized of $172\times 256\times 256$ since the optimization-
based HCS methods work for the whole HSI only. In order to verify the
efficiency of the proposed RTCS, we adopt NVIDIA Jeston TX2 to measure the
run-time of the proposed RTCS and DCSN [1]. Note that the optimization-based
HCS methods were developed based on Matlab, which is incompatible with ARM-
based architecture; thus, those methods cannot run in Jetson TX2. We take the
average of the run-time by repeatedly executing the inference 30 times.

In the Table VI, the proposed RTCS has the lowest parameter space as well as
the computational complexity compared to other optimization- and DL-based
approaches. In the run-time comparison, the proposed RTCS achieves the fastest
inference time since the depth of the network is relatively shallow and the
number of parameters is low, making real-time compressed sensing and decoding
possible. Although AAHCS [17] achieves good performance as illustrated in
Section III-B1 under higher sampling rate $s_{r}\approx 5\%$, the decoding
process of the proposed RTCS is 3000x faster than that of AAHCS [17], implying
our RTCS is more feasible for real-time hyperspectral stripes decoding.
Moreover, with the sampling rate $\approx 1\%$, the proposed RTCS
significantly outperforms the peer methods for all quality indices, as well as
achieves the lowest computational complexity. On the one hand, the encoder of
the proposed RTCS only requires a matrix multiplication, making the cost of
the computation and memory usage could be minimized in the miniaturized
satellite. On the other hand, the decoder of the proposed RTCS could be
deployed on edge devices due to its computational cheapness, which could
greatly reduce the overhead of the central server and make more critical HSI
applications possible.

More specifically, due to the real-time decoding requirement of the proposed
RTCS, we deploy the proposed RTCS to the low-end device, NVIDIA Jetson TX2,
with the TVM compiler [44] for optimizing the efficiency of the DL-based
approach (i.e., DCSN and our RTCS) for the specific hardware. Clearly, the
proposed RTCS still achieves a very fast inference in the embedding platform,
i.e., 19 Milliseconds (ms), while the DCSN in TX2 requires 25 ms for
inference. In the DL-based HCS methods without TVM-optimization, the proposed
RTCS still meets the real-time requirements (say, $1/0.024=41.6$ FPS), while
DCSN is relatively hard to achieve such a fast inference process due to deeper
network. Even in the ARM-based CPU in TX2, the inference time of the proposed
RTCS is only 2 seconds, whereas the optimization-based HCS methods, including
AAHCS [17], SpeCA [10], and SPACE [16], failed to decode the HSI due to
limited memory and insufficient computing power. Therefore, we could conduct
that the proposed RTCS is efficient, as well as a state-of-the-art HCS method
for miniaturized satellites.

We also further verify that the proposed RTCS is hardware friendly. The
proposed encoder could deploy the HCS in the lightweight chip without any
hardware accelerator, such as a GPU or intelligent processing unit (IPU), in a
real-time sense. Although only one matrix multiplication is required in the
encoder of the proposed RTCS, the floating-point with 32-bit precision (FP32)
is still required, the same as the optimization-based HCS methods. Most low-
end computing chips, such as Kneron KL720/FL520, require integer-based
inference to maximize efficiency. Therefore, we further study the model
quantization for our encoder to enable integer computation for saving power-
consuming purposes by using quantization-aware training (QAT) and post-
quantization (PQ) [45]. As shown in the Table VII, the quantized encoder of
RTCS maintains promising performance while being more hardware-friendly.”

Compared to the traditional optimization-based approach, the floating point
operation is strictly required in the miniaturized satellite, leading to a
higher computational cost inside the computational chip. As claimed in AAHCS
[17], the basis could be calculated first and only all-addition operations
required in the miniaturized satellite. However, the performance is unknown
since the experiments were conducted on a data-dependant basis. In this
fashion, the basis decomposition is still needed, so a number of floating
point operations are required, which is computationally expensive. As a
result, the encoder of the proposed RTCS could be state-of-the-art as well as
computationally cheap.

TABLE VIII: Ablation study under sampling rate $\approx 1\%$.

L1 | SAM | IMFA | FRDB | PSNR$\uparrow$ | SAM$\downarrow$ | Flops / #Params
---|---|---|---|---|---|---
✓ |  | ✓ |  | 34.330 | 2.094 | 0.302G/6.434M
✓ | ✓ | ✓ |  | 35.677 | 1.320 | 0.302G/6.434M
✓ | ✓ |  | ✓ | 34.837 | 1.435 | 0.291G/6.0753M
✓ |  |  | ✓ | 35.037 | 1.936 | 0.291G/6.0753M
✓ |  | ✓ | ✓ | 35.222 | 1.876 | 0.297G/6.291M
| ✓ | ✓ | ✓ | 9.123 | 1.576 | 0.297G/6.291M
✓ | ✓ | ✓ | ✓ | 36.537 | 1.176 | 0.297G/6.291M

###  III-E Ablation Study

The proposed RTCS primarily consists of two distinct branches designed to
aggregate features with varying equivalent receptive fields. In order to
assess the efficacy of the dual-stream network architecture, we examine the
performance of each individual stream within the proposed RTCS. We increase
the number of base blocks, denoted as $n_{f}$, to 16 in order to bring the
parameter space of each branch closer to that of the proposed RTCS. Table VIII
presents an ablation study of the proposed RTCS to validate the effectiveness
of both the two-streamed architecture and the SAM loss.

In the first stream, the IMFA-Net-based HCS, which can be considered an
enhanced version of DCSN [1], demonstrates a high PSNR but limited spectral
quality, as indicated by the SAM value. Comparing IMFA-Net with the original
design of DCSN [1], it becomes evident that more channels in the IMFA-Net
decoder are necessary to restore the high-quality spectral signature from
compressed HSI effectively. When the SAM loss is considered during the
training phase, spectral quality is further improved, suggesting that the
proposed SAM loss is better suited for preserving HSI signatures in HCS tasks.
Owing to the simplified network of IMFA-Net, its performance surpasses that of
DCSN, mainly when dealing with a small training set.

Likewise, the performance of the FRDB-Net-based HCS can be enhanced when both
loss functions are utilized concurrently. In the proposed dual-stream
architecture, significant performance improvements are achieved as a result of
the judicious aggregation of multiple receptive fields. Although the number of
convolutional kernels is considerably reduced through the use of grouped
convolution, leading to diminished performance in the HSI spectral signature,
the proposed SAM loss can guide network learning toward spectral preservation,
resulting in a superior SAM index for the proposed RTCS.

In conclusion, the proposed dual-stream network architecture combined with the
SAM loss demonstrates effectiveness, efficiency, robustness, and hardware
compatibility, making it a valuable solution for the task at hand.

##  IV Conclusion

This study proposed an effective, efficient, and robust hyperspectral
compressed sensing (HCS) network, termed real-time compressed sensing (RTCS),
for significantly lower computational resources required in the encoder side
at the miniaturized satellite, as well as hardware-friendly decoder inside the
receiver based on edge devices. A very computationally cheap encoder with only
one matrix multiplication with integer-8 precision, has been proposed to
compressively sense the stripe-like hyperspectral image (HSI), making HCS on
CubSat possible and practical. A two-streamed-based decoder, consisting of two
shallow and effective sub-networks, has been proposed to reduce the
computational complexity to meet the requirements of edge devices by
judiciously aggregating the middle-level and semantic features by the larger
and regular convolutional kernel sizes for two sub-networks.

Benefiting from the lower number of parameters in the proposed RTCS, the
proposed method only requires small data, making its high adaptation power for
new datasets. A novel task-specific training policy with Spectral Angle
Mapper-aware (SAM) loss has been proposed to improve the quality of the
spectral signature preservation and enable the proposed RTCS to restore the
complete HSI even if partial information of the HSI is missing or noisy.
Comprehensive experimental results demonstrated the superiority of the
proposed RTCS. Moreover, the proposed RTCS is the first research to tackle the
joint issue of the HSI restoration and compressed sensing in edge devices
only, leading to our RTCS being practical and useful in real-world HCS
applications.

## References

  * [1] C.-C. Hsu, C.-H. Lin, C.-H. Kao, and Y.-C. Lin, “Dcsn: Deep compressed sensing network for efficient hyperspectral data transmission of miniaturized satellite,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 59, no. 9, pp. 7773–7789, 2020.
  * [2] Z. Zhu, J. Hou, J. Chen, H. Zeng, and J. Zhou, “Hyperspectral image super-resolution via deep progressive zero-centric residual learning,” _IEEE Transactions Image Processing_ , vol. 30, pp. 1423–1438, 2021.
  * [3] W. Wang, W. Zeng, Y. Huang, X. Ding, and J. Paisley, “Deep blind hyperspectral image fusion,” in _Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV)_ , Oct. 2019, pp. 4149–4158.
  * [4] J. Yao, D. Hong, J. Chanussot, D. Meng, X. Zhu, and Z. Xu, “Cross-attention in coupled unmixing nets for unsupervised hyperspectral super-resolution,” in _Proc. European Conf. Computer Vision (ECCV)_ , 2020, pp. 208–224.
  * [5] N. Keshava and J. F. Mustard, “Spectral unmixing,” _IEEE Signal Processing Magazine_ , vol. 19, no. 1, pp. 44–57, Jan. 2002.
  * [6] J. M. Bioucas-Dias, A. Plaza, G. Camps-Valls, P. Scheunders, N. Nasrabadi, and J. Chanussot, “Hyperspectral remote sensing data analysis and future challenges,” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 1, no. 2, pp. 6–36, Jun. 2013.
  * [7] D. Stein, S. Beaven, L. Hoff, E. Winter, A. Schaum, and A. Stocker, “Anomaly detection from hyperspectral imagery,” _IEEE Signal Processing Magazine_ , vol. 19, no. 1, pp. 58–69, Jan. 2002.
  * [8] C.-I. Chang, C.-C. Wu, C. S. Lo, and M.-L. Chang, “Real-time simplex growing algorithms for hyperspectral endmember extraction,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 48, no. 4, pp. 1834–1850, Apr. 2010.
  * [9] C.-H. Lin, R. Wu, W.-K. Ma, C.-Y. Chi, and Y. Wang, “Maximum volume inscribed ellipsoid: A new simplex-structured matrix factorization framework via facet enumeration and convex optimization,” _SIAM Journal on Imaging Sciences_ , vol. 11, no. 2, pp. 1651–1679, Jun. 2018.
  * [10] G. Martín and J. M. Bioucas-Dias, “Hyperspectral blind reconstruction from random spectral projections,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 9, no. 6, pp. 2390–2399, Jun. 2016.
  * [11] M. F. Duarte and R. G. Baraniuk, “Kronecker compressive sensing,” _IEEE Transactions on Image Processing_ , vol. 21, no. 2, pp. 494–504, Feb. 2012.
  * [12] Y. Wang, L. Lin, Q. Zhao, T. Yue, D. Meng, and Y. Leung, “Compressive sensing of hyperspectral images via joint tensor Tucker decomposition and weighted total variation regularization,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 14, no. 12, pp. 2457–2461, Dec. 2017.
  * [13] L. Zhang, W. Wei, Y. Zhang, H. Yan, F. Li, and C. Tian, “Locally similar sparsity-based hyperspectral compressive sensing using unmixing,” _IEEE Transactions on Computational Imaging_ , vol. 2, no. 2, pp. 86–100, Jun. 2016.
  * [14] Q. Du and J. E. Fowler, “Hyperspectral image compression using JPEG2000 and principal component analysis,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 4, no. 2, pp. 201–205, Apr. 2007.
  * [15] C. Kwan and J. Larkin, “New results in perceptually lossless compression of hyperspectral images,” _Journal of Signal and Information Processing_ , vol. 10, no. 03, pp. 96–124, Aug. 2019.
  * [16] C.-H. Lin, J. M. Bioucas, T.-H. Lin, Y.-C. Lin, and C.-H. Kao, “A new hyperspectral compressed sensing method for efficient satellite communications,” in _Proceedings of the 11th IEEE Sensor Array and Multichannel Signal Processing Workshop (SAM)_ , 2020.
  * [17] C.-H. Lin and T.-H. Lin, “All-addition hyperspectral compressed sensing for metasurface-driven miniaturized satellite,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–15, 2022.
  * [18] T.-J. Yang and V. Sze, “Design considerations for efficient deep neural networks on processing-in-memory accelerators,” in _IEEE International Electron Devices Meeting (IEDM)_. IEEE, 2019, pp. 22–1.
  * [19] A. Wuraola and N. Patel, “Resource efficient activation functions for neural network accelerators,” _Neurocomputing_ , vol. 482, pp. 163–185, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0925231221017094
  * [20] G. González-Díaz_Conti, J. Vázquez-Castillo, O. Longoria-Gandara, A. Castillo-Atoche, R. Carrasco-Alvarez, A. Espinoza-Ruiz, and E. Ruiz-Ibarra, “Hardware-based activation function-core for neural network implementations,” _Electronics_ , vol. 11, no. 1, p. 14, 2021.
  * [21] A. L. Maas, A. Y. Hannun, A. Y. Ng _et al._ , “Rectifier nonlinearities improve neural network acoustic models,” in _Proc. icml_ , vol. 30, no. 1. Atlanta, Georgia, USA, 2013, p. 3.
  * [22] G. Ascia, V. Catania, S. Monteleone, M. Palesi, D. Patti, and J. Jose, “Analyzing networks-on-chip based deep neural networks,” in _Proceedings of the 13th IEEE/ACM International Symposium on Networks-on-Chip_ , New York, USA, Oct. 2019, pp. 1–2.
  * [23] B. Wang, J. Zhou, W.-F. Wong, and L.-S. Peh, “Shenjing: A low power reconfigurable neuromorphic accelerator with partial-sum and spike networks-on-chip,” _arXiv preprint arXiv:1911.10741_ , 2019.
  * [24] B. Varghese, N. Wang, S. Barbhuiya, P. Kilpatrick, and D. S. Nikolopoulos, “Challenges and opportunities in edge computing,” in _IEEE International Conference on Smart Cloud (SmartCloud)_. IEEE, 2016, pp. 20–26.
  * [25] K. Cao, Y. Liu, G. Meng, and Q. Sun, “An overview on edge computing research,” _IEEE access_ , vol. 8, pp. 85 714–85 728, 2020.
  * [26] W. Z. Khan, E. Ahmed, S. Hakak, I. Yaqoob, and A. Ahmed, “Edge computing: A survey,” _Future Generation Computer Systems_ , vol. 97, pp. 219–235, 2019.
  * [27] C.-H. Lin, Y.-C. Lin, and P.-W. Tang, “Admm-adam: A new inverse imaging framework blending the advantages of convex optimization and deep learning,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–16, 2021.
  * [28] L. Zhuang and J. M. Bioucas-Dias, “Fast hyperspectral image denoising and inpainting based on low-rank and sparse representations,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 11, no. 3, pp. 730–742, 2018.
  * [29] W. He, H. Zhang, H. Shen, and L. Zhang, “Hyperspectral image denoising using local low-rank matrix recovery and global spatial–spectral total variation,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 11, no. 3, pp. 713–729, 2018.
  * [30] D. Cerra, R. Müller, and P. Reinartz, “Unmixing-based denoising for destriping and inpainting of hyperspectral images,” in _IEEE Geoscience and Remote Sensing Symposium_. IEEE, 2014, pp. 4620–4623.
  * [31] J. D’Errico, “Inpainting nan elements in 3-d,” _MATLAB Central File Exchange_ , 2008.
  * [32] E. Middleton, P. Campbell, F. Huemmrich, Q. Zhang, Y.-B. Cheng, D. Landis, S. Ungar, L. Ong, and N. Pollack, “Eo-1/hyperion: Nearing twelve years of successful mission science operation and future plans,” in _IEEE Geoscience and Remote Sensing Symposium_ , 2012.
  * [33] G. Martín, J. Bioucas-Dias, and A. Plaza, “HYCA: A new technique for hyperspectral compressive sensing,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 53, no. 5, pp. 2819–2831, May 2015.
  * [34] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan _et al._ , “Searching for mobilenetv3,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 1314–1324.
  * [35] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [36] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , Neveda, USA, Jun. 2016, pp. 770–778.
  * [37] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 25, 2012.
  * [38] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “Esrgan: Enhanced super-resolution generative adversarial networks,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , Munich, Germany, Sep. 2018, pp. 63–79.
  * [39] AVIRIS Free Standard Data Products. [Online]. Available: http://aviris.jpl.nasa.gov/html/aviris.freedata.html
  * [40] I. Loshchilov and F. Hutter, “Fixing weight decay regularization in adam,” _CoRR_ , vol. abs/1711.05101, 2017. [Online]. Available: http://arxiv.org/abs/1711.05101
  * [41] I. Loshchilov and F. H., “Decoupled weight decay regularization,” in _International Conference on Learning Representations_ , 2019. [Online]. Available: https://openreview.net/forum?id=Bkg6RiCqY7
  * [42] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” in _Proceedings of the 25th international conference on Machine learning_ , 2008, pp. 1096–1103.
  * [43] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol, “Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion,” _Journal of machine learning research_ , vol. 11, pp. 3371–3408, Dec. 2010.
  * [44] T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Yan, H. Shen, M. Cowan, L. Wang, Y. Hu, L. Ceze _et al._ , “$\\{$TVM$\\}$: An automated $\\{$End-to-End$\\}$ optimizing compiler for deep learning,” in _13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)_ , 2018, pp. 578–594.
  * [45] H. Wu, P. Judd, X. Zhang, M. Isaev, and P. Micikevicius, “Integer quantization for deep learning inference: Principles and empirical evaluation,” _arXiv preprint arXiv:2004.09602_ , 2020.

|  Chih-Chung Hsu (S’11-M’14-SM’20) received his B.S. degree in Information
Management from Ling-Tung University of Science and Technology, Taiwan, in
2004, and his M.S. and Ph.D. degrees in Electrical Engineering from National
Yunlin University of Science and Technology and National Tsing Hua University
(NTHU), Taiwan, in 2007 and 2014, respectively. From 2014 to 2017, Dr. Hsu was
a postdoctoral researcher with the Institute of Communications Engineering at
NTHU. He served as an assistant professor with the Department of Management
Information Systems at National Pingtung University of Science and Technology
from February 2018 to 2021 and has been affiliated with the Institute of Data
Science at National Cheng Kung University since 2021. His research interests
primarily focus on computer vision and machine/deep learning, with
applications in image and video processing. He has published several papers in
top-tier journals and conference proceedings, including IEEE TPAMI, IEEE TIP,
IEEE TMM, IEEE TGRS, ACM MM, IEEE ICIP, IEEE IGARSS, and IEEE ICASSP,
including the best student paper award from the IEEE ICIP in 2019. Dr. Hsu
became a Senior Member of the Institute of Electrical and Electronics
Engineers (IEEE) in October 2020. He received the Best Young Professional
Member Award from IEEE Tainan Section in 2023. Dr. Hsu leads the Advanced
Computer Vision Lab (ACVLab), winning over 20 grand challenges at top-tier
conferences over the years, such as the 3rd place award at the Learning to
Drive Challenge from the IEEE International Conference on Computer Vision
(ICCV), where he was also an invited speaker. He secured the 3rd place award
in the Visual Inductive Priors for Data-Efficient Computer Vision Challenge
and 1st place in the COV19D challenge at the European Conference on Computer
Vision (ECCV) in 2020 and 2022.
---|---
|  Chih-Yu Jian earned a bachelor’s degree and graduated in 2018 from the
Department of Information Management at Pingtung University of Science and
Technology. He obtained his master’s degree from the same department in 2020.
From September 2020 to November 2022, he served as a master’s researcher at
Chang Gung Medical Foundation Hospital. Since December 2022, he has been
working as a master’s researcher at Far Eastern Memorial Hospital. His main
interests are focused on computer vision and machine/deep learning, as well as
the applications of medical imaging and natural language processing. He has
published papers in IEEE ICASSP, IEEE VCIP, and NeuroImage Clinical. Over the
years, he has won several significant challenges with his team at top
conferences, such as securing third place in the Learning to Drive Challenge
at the IEEE International Conference on Computer Vision (ICCV) and first place
in the COV19D Challenge.
---|---
|  Eng-Shen (James) Tu is a research assistant in the Software Engineering and
Intelligent Test Automation Lab at National Cheng Kung University in Tainan,
Taiwan. Currently, he is working towards his Bachelor’s degree in Computer
Science and Information Engineering at the same university. Though still early
in his academic journey, James continues to explore and deepen his
understanding of his chosen fields, always keen to contribute to new knowledge
and innovative solutions in the domain of Data Science and Software
Engineering.
---|---
|  Chia-Ming Lee received his B.S. degree at the Department of Statistics and
Information Science, Fu Jen Catholic University (FJCU), Taiwan, in 2023. He is
currently a M.S. student with the Advanced Computer Vision Laboratory (ACVLab)
at the Institute of Data Science, National Cheng Kung University (NCKU),
Tainan, Taiwan. His research interests mainly focus on computer vision, deep
learning and their application. He have awarded the Jury Prize at Visual
Inductive Priors Workshop from the IEEE International Conference on Computer
Vision (ICCV). He secured the 1st place in the AI-enabled Medical Image
Analysis Workshop and COVID-19 Diagnosis Competition held at the IEEE
International Conference on Acoustics, Speech, Signal Processing (ICASSP). He
received the Top Paper Award in the Social Media Prediction Challenge at ACM
Multimedia (ACMMM).
---|---
|  Guan-Lin Chen received the B.S. degree from the Department of Financial
Engineering and Actuarial Mathematics, Soochow University, Taipei, Taiwan, in
2020, and the M.S. degree from the Institute of Data Science, National Cheng
Kung University, in 2022. He currently serves as an AI engineer at
MobileDrive, Taipei, focusing on the application of learning-based methods for
perception cameras. His research interests primarily focus on computer vision,
machine learning, and deep learning, especially explainable and trustworthy
AI.
---|---