# SelFSR: Self-Conditioned Face Super-Resolution in the Wild via Flow Field
Degradation Network

Xianfang Zeng∗ Jiangning Zhang∗ Liang Liu ∗Equal contribution.  Guangzhong
Tian  and Yong Liu† Member  IEEE X. Zeng, J. Zhang, and L. Liu are with
Institute of Cyber-System and Control, Zhejiang University, Hangzhou 310027,
China (email<EMAIL_ADDRESS><EMAIL_ADDRESS>leonliuz@zju.edu.cn).G. Tian is with Ningbo Research Institute, Zhejiang
University, Ningbo 315000, China (email: gztian@zju.edu.cn).Y. Liu is with
State Key Laboratory of Industrial Control Technology, Zhejiang University,
Hangzhou 310027, China (email: yongliu@iipc.zju.edu.cn).

###### Abstract

In spite of the success on benchmark datasets, most advanced face super-
resolution models perform poorly in real scenarios since the remarkable domain
gap between the real images and the synthesized training pairs. To tackle this
problem, we propose a novel domain-adaptive degradation network for face
super-resolution in the wild. This degradation network predicts a flow field
along with an intermediate low resolution image. Then, the degraded
counterpart is generated by warping the intermediate image. With the
preference of capturing motion blur, such a model performs better at
preserving identity consistency between the original images and the degraded.
We further present the self-conditioned block for super-resolution network.
This block takes the input image as a condition term to effectively utilize
facial structure information, eliminating the reliance on explicit priors,
e.g. facial landmarks or boundary. Our model achieves state-of-the-art
performance on both CelebA and real-world face dataset. The former
demonstrates the powerful generative ability of our proposed architecture
while the latter shows great identity consistency and perceptual quality in
real-world images.

###### Index Terms:

Generative model, adversarial learning, face super-resolution.

##  I Introduction

Face super-resolution (FSR), also known as face hallucination, aims at
restoring high-resolution (HR) face images given low-resolution (LR)
counterparts. This fundamental technology can improve the perceptual quality
of face images in various scenarios like video surveillance or online meeting.
Some recognition tasks such as identity recognition or facial analysis also
benefit from the enhanced details. The performance of super-resolution (SR) on
benchmark datasets has been improved rapidly with numerous proposed approaches
[1, 2, 3, 4, 5]. However, most advanced SR networks perform poorly in the real
scenarios since the noticeable domain gap between the test data and the
artificially synthesized training pairs.

To unbind the limitation of applying SR networks in real-world, various
attempts have been made recently. Those methods can roughly be divided into
three categories: collecting real training pairs with optical zoom [6, 7],
blind super-resolution with kernel estimation [8, 9], as well as constructing
pseudo pairs via deep neural networks [10, 11, 12, 13]. Despite being able to
capture the real degradation, the collection of real pairs is expensive and
laborious. Moreover, SR networks trained on the collected data are still hard
to generalize to the uncovered scenarios. The second scheme tackles super-
resolution in the wild by modeling the unknown downsampling kernel at test
phase. However, those approaches usually assume a parameterized degradation
between LR images and HR. Such an assumption limits the ability to model
complicated distortions in real-world. Hence the third scheme attempts to
estimate the real degradation using neural networks. A degradation network or
domain adaptation module is usually utilized to handle complex sensor noise
and diverse distortions.

Figure 1:  Visual samples of our face super-resolution method. Two images in
the left are results on the CelebA dataset with 8x upsampling scale, while the
right two are from real-world images with 4x upsampling scale.

Falling into the framework of constructing pseudo pairs, our real-world face
super-resolution method, named SelFSR, involves a domain adaptation network in
LR space followed by a SR module. The most distinguished parts of SelFSR come
from the displacement-based adaptation network and the self-conditioned SR
network.

Unlike previous methods [11, 12] that directly utilize CycleGAN estimating the
real degradation, we propose a displacement-based network to generate noisy LR
images. Our motivation is that CycleGAN-based domain adaptation methods hardly
maintain the appearance feature of input images. For example, the skin color
of the input face is usually drifted after the adaptation processing. We argue
that this drawback is caused by estimating noisy images directly in the whole
RGB space, where the huge output space is easy to introduce color bias into
the generated samples. Hence, our adaptation network predicts a flow field
along with an intermediate image, instead of a noisy LR directly. The
intermediate image is forced to approximate the input with an identity loss,
while the estimated flow field warps the intermediate image to generate the
noisy counterpart. Since the flow field indicates the motion of pixels, the
resulting network has a preference on capturing motion noisy in degradation
processing and performs successfully at preserving appearance consistency. As
a comparison, The CycleGAN based models have no inductive bias about motion,
and usually involves the color drifting into the results.

For face super-resolution, we propose the novel self-conditioned block to
construct our SR network. In mainstream FSR methods [4, 14], explicit priors
like facial landmarks are utilized to provide the localization of facial
components as well as global structure. However, this structure information is
naturally contained in face images. For instance, a face image filtered by the
Sobel kernel can provide rough facial boundary information [15]. So we have
the following question: Can we find an effective way to directly utilize the
structure information contained in input images? To this end, we treat the
input image as a condition term containing facial structure information. The
LR image is upsampled to different resolutions and fed to the SR network
multiple times. The proposed self-conditioned block extracts the implicit
structure information from the input and modulates network activations.
Qualitative and quantitative (un)supervised results on CelebA dataset
demonstrate the powerful generative ability of our architecture, which has no
reliance on auxiliary facial priors.

In summary, our contributions are three folds. (i) We propose a novel way to
perform domain adaptation utilizing a displacement-based network. Such a model
performs successfully at identity consistency since its preference for
capturing the motion noise. (ii) We provide a novel backbone for FSR task,
which outperforms previous state-of-the-art methods, with fewer parameters,
and no requirement of explicit facial priors. (iii) Experimental results on
real face dataset indicate that our approach can generate high-quality and
identity-consistent FSR results on real-world images.

##  II Related work

Face generation. Recently, some methods can directly generate realistic faces
from the latent space, which boosted many applications in face editing [16,
17, 18] and face super-resolution [5]. Tero et al. [19] described a
progressive growing methodology for the face generation on 1024$\times$1024
resolution from an underlying code. StyleGAN [20] proposed a style-based
generator that manipulates the activations of image features at different
scales, synthesizing extremely naturalistic face images. Tero et al. [21]
further analyzed the influence of instance normalization and improved the
image quality via path length regularization and modified architecture. As
face super-resolution is a case of conditional generation processing,
researchers can benefit from the advanced methods in face generation. Inspired
by the remarkable success of the style-based architecture in face generation
[20, 21, 22], we proposed the self-conditioned block which takes the input as
a condition term. Unlike some methods [23, 24, 25] which concatenate the input
images onto intermediate feature maps, our self-conditioned block modulates
the activations of each layer given the input.

Image super-resolution. The single image super-resolution performance on
benchmark datasets has been boosted continuously by numerous proposed models
[1, 2, 3, 4, 25], which usually achieve state-of-the-art performance via
advanced network architecture or learning strategy. Some fundamental and
classic network designs in SR models contain recursive learning [26], residual
learning [27], dense connection [28], multi-path learning [29] and attention
mechanism [2, 30, 31]. Kim et al. [26] proposed Deeply-recursive Convolutional
Network (DRCN) to recursively learning high-level representations with a
weight-shared module. In residual learning, the model only predicts the
residual map that transforms the LR image into an HR image. Since the learning
difficulty is significantly reduced with residual learning, it is widely used
in SR models [32, 33, 34]. Further, SRDenseNet [35] introduced dense block
into SR models, utilizing all the previous feature maps to learn the missing
high-frequency details in the LR image. In multi-path learning based SR models
[29, 33, 36], the features are transferred to multiple paths for different
scale or reception field. Multiple representations are then combined to gain
improved performance. As for attention mechanism, Zhang et al. [2] proposed
the channel attention network for image SR task and largely improve
performance. Further, they proposed non-local attention block [37] to extract
local and non-local representations between pixels for a more effective
feature aggregation.

Another important factor is learning strategy like loss function or upsampling
framework. SRGAN [38] achieved photo realistic SR results by introducing
perceptual loss [39] and adversarial loss [40]. ESRGAN [1] utilized a
relativistic version of adversarial loss [41] and got the state-of-the-art
perception-oriented performance at that time, largely boosting the development
of paired image super-resolution. As for upsampling framework, the classical
frameworks are pre-upsampling SR [42] and post-upsampling SR [43, 44].
Recently, the progressive-upsampling framework [45, 46] is proposed to reduce
the learning difficulty and support for multi-scale SR task. A detailed and
comprehensive survey on SR components can be found in recent image super-
resolution reviews [47, 48]. Our SR network also belongs to the progressive-
upsampling framework, where ProSR [45] and LapSRN [46] are the most similar
model to our SR network. The difference is that they predict the residual map
with respect to the LR progressively, while our model take the input as a
condition to modulate feature maps.

As a domain-specific image super-resolution task, face super-resolution models
usually use image priors like facial-structured information to achieve high SR
performance. For instance, FSRNet [14] use facial parsing maps and landmarks
as priors to achieve face SR task. Ma et al. proposed an iterative
collaboration framework between face super-resolution and landmark estimation.
Instead of directly utilizing facial priors, we propose the self-conditioned
block to use the implicit structure information contained in the input.

Real-world image super-resolution. Amount of attempts have been made to tackle
the real image super-resolution task. By adjusting the focal length of a
digital camera, several works collected the training pairs in real scenarios
[6, 49, 7]. Those methods are able to capture the real degradation with an
expensive and laborious collection processing. Another category of approaches
sought for real-world image SR from an algorithmic perspective. Jinjin et al.
[8] proposed an iterative kernel correction at the test phase for blind SR.
Ruofan et al. [9] incorporated the blur-kernel modeling in the training phase
for real LR image super-resolution. However, those methods usually are limited
by the ability of modeling complicated distortions in real-world. Several
researchers, therefore, turned to utilize deep neural networks to estimate the
real degradation [50, 51, 11, 10]. Unlike most existing methods which directly
utilize CycleGAN estimating the real degradation, we propose a displacement-
based architecture to perform the domain adaptation task.

##  III Proposed Method

Given a set of unpaired real-world LR and HR images
$\\{(I_{real,i}^{L})_{i=1}^{M},(I_{real,j}^{H})_{j=1}^{N}\\}$, we aim to learn
a SR network that upscales the LR images and simultaneously ensures the HR
estimation following the real HR distribution. To this end, we propose a two-
stage approach, SelFSR, to perform face super-resolution in the wild. Figure 2
illustrates the training pipeline of our method. The full framework involves
two stages:

  * •

Domain adaptive degradation. We first synthesize clean LR images $I_{syn}^{L}$
by downsampling HR images using a bicubic kernel. A generator
${\mathcal{G}}(\cdot)$ takes $I_{syn}^{L}$ as input to estimate its noisy
counterpart $I_{gen}^{L}$, which is generated by warping the intermediate
image $\tilde{I}_{syn}^{L}$. Meanwhile, $I_{gen}^{L}$ and real LR images
$I_{real}^{L}$ are sent to a domain discriminator ${\mathcal{D}}_{real}^{L}$
for decreasing the distance of two distributions.

  * •

Self-conditioned face super-resolution. We then construct aligned pseudo pairs
$(I_{gen}^{L},I_{real}^{H})$ using estimated noisy LR images and real HR
images. A self-conditioned SR network ${\mathcal{R}}(\cdot)$ is trained with
pseudo pairs to perform real-world face super-resolution. Similarly, the
discriminator ${\mathcal{D}}_{real}^{H}$ is utilized in HR space to enhance
the texture details of generated high-resolution images $I_{gen}^{H}$.

###  III-A Domain Adaptive Degradation

Super-resolution models trained by clean paired images fails in handling real-
world LR images, whose degradation is unknown and more complicated. To tackle
this problem, we introduce the domain adaptation stage to estimate the noisy
LR counterparts of HR images. Unlike previous methods [11, 12] that directly
utilize CycleGAN to transfer the clean LR images to the noisy ones, we propose
a displacement-based architecture to perform this task. Such an architecture
is more likely to capture motion noise and avoid dramatic shifting on identity
features (e.g. skin color) of input images.

(a) Domain adaptation in low-resolution space.
---
(b) Super-resolution with pseudo pairs.
Figure 2: The pipeline of our two-stage unsupervised face super-resolution
framework. (a) The domain adaptation stage is to transfer the synthesized LR
images (by bicubic downsampling) to its noisy counterparts which follow the
real LR distribution. (b) Training super-resolution network
${\mathcal{R}}(\cdot)$ with pseudo pairs $(I_{gen}^{L},I_{real}^{H})$ to adapt
real-world degradation.

As can be seen in Figure 2, the domain adaptation network predicts a pixel
flow field $\mathcal{T}\in\mathbb{R}^{W\times H\times 2}$ as well as an RGB
image $\tilde{I}_{syn}^{L}$. Each element at $\mathcal{T}_{x,y}$ represents
the pixel offset $(\delta{x},\delta{y})$ for the image branch output
$\tilde{I}_{syn}^{L}$. Namely, the pixel ${I}_{gen}^{L}(x,y)$ in the final
output is sampled at the location $(x-\delta{x},y-\delta{y})$ in
$\tilde{I}_{syn}^{L}$, where a generic sampling kernel (e.g. bilinear) is
applied [52]. We denote the sample location $(x-\delta{x},y-\delta{y})$ as
$(x^{s},y^{s})$, the sampling process can be formulated as

$\displaystyle
I_{gen}^{L}(x,y)=\sum_{u}^{W}\sum_{v}^{H}\tilde{I}_{syn}^{L}(u,v)k(x^{s}-u;\Phi_{x})k(y^{s}-v;\Phi_{y})\text{,}$
(1)

where $\Phi_{x}$ and $\Phi_{x}$ are the parameters of sampling kernel $k()$.

We utilize the flow field degradation network ${\mathcal{G}}$ to estimate the
noisy LR images. A domain discriminator ${\mathcal{D}}_{real}^{L}$ is used to
distinguish the generated LR images and the real. During the stage of domain
adaptation, these modules are constrained by following terms.

Adversarial loss in LR. We impose an adversarial constraint [40] for matching
the distribution of the estimated noisy LR images and the real. The
adversarial loss for ${\mathcal{G}}(\cdot)$ is defined as

$\displaystyle\mathcal{L}_{adv}^{L}(\mathcal{G},\mathcal{D}_{real}^{L})$
$\displaystyle=\mathbb{E}\left[\log{\mathcal{D}}_{real}^{L}(I_{real}^{L})\right]$
(2)
$\displaystyle+\mathbb{E}\left[\log(1-{\mathcal{D}}_{real}^{L}({\mathcal{G}}(I_{syn}^{L})))\right]\text{.}$

Identity loss. To keep the identity consistency of estimated LR images and the
input, we impose the identity restriction on the image branch output. It
reduces the L1 distance between $\tilde{I}_{syn}^{L}$ and ${I}_{syn}^{L}$,
denoted as

$\displaystyle\mathcal{L}_{idt}(\tilde{I}_{syn}^{L},{I}_{syn}^{L})=\mathbb{E}[\|\tilde{I}_{syn}^{L}-{I}_{syn}^{L}\|_{1}]\text{.}$
(3)

Smoothness loss. For avoiding quickly-changing and self-crossing in the local
deformation , we enforce the smoothness constraint on the predicted flow field
$\mathcal{T}$. It minimizes the total variation of flow field horizontally and
vertically, denoted as

$\displaystyle\mathcal{L}_{s}(\mathcal{T})=\mathbb{E}[\left\|\nabla_{x}\mathcal{T}\right\|_{1}+\left\|\nabla_{y}\mathcal{T}\right\|_{1}]\text{.}$
(4)

The full objective for domain adaptation processing is the weighted sum of
above terms:

$\displaystyle\mathcal{L}({\mathcal{G}},{\mathcal{D}}_{real}^{L})=$
$\displaystyle\mathcal{L}_{adv}^{L}+\lambda_{idt}\mathcal{L}_{idt}+\lambda_{s}\mathcal{L}_{s}\text{,}$
(5)

where $\lambda_{i}$ is the weight of $i$-th constraint term.

Figure 3: Self-conditioned SR network takes a degraded LR image
${I}_{gen}^{L}$ as input and gradually upscales it to the HR counterpart,
along with corresponding bicubic upsampled image as condition. Green and
orange cubes indicate convolution layer and instance normalization layer,
respectively.

###  III-B Self-Conditioned Face Super-Resolution

Mainstream SR networks usually use the post-upsampling framework, which
involves a heavy trunk component (e.g. 345 convolution layers in ESRGAN [1])
followed by a lightweight upsampling part. This unbalanced design performs
poorly when extended to high scale factor SR task. As illustrated in Figure 3,
we design a balanced backbone for FSR task, assigning similar numbers of
convolution layer for each level. Since the SR operation is segregated into
several small upscaling tasks, the learning difficultly of whole model is
reduced. We further utilize the progressive learning strategy [19] to train
the whole network. Our self-conditioned SR network starts by $\times 2$ scale
factor, gradually upscales LR images by a step of $\times 2$, and finally
achieves a high scale factor. The model capacity is progressively increased
during training process. The resulting network can upscale LR images with
different factors, and easily extend to high scale super-resolution tasks.

To provide facial structure information in each level, LR input is upsampled
to corresponding resolution, and then fed to the self-conditioned block
multiple times. The architecture of self-conditioned block is illustrated in
Figure 4. It takes an image as condition term to estimate the standard
deviation of deep features. The estimated standard deviation $\gamma$ is used
for element-wise modulating the normalized activation $\bar{f}$, formulated as

$\displaystyle\mu_{n,c}^{l}$
$\displaystyle=\frac{1}{HW}\sum_{h=1}^{H}\sum_{w=1}^{W}f_{n,c,h,w}^{l}$ (6)
$\displaystyle\sigma_{n,c}^{l}$
$\displaystyle=\sqrt{\frac{1}{HW}\sum_{h=1}^{H}\sum_{w=1}^{W}\left(f_{n,c,h,w}^{l}-\mu_{n,c}^{l}\right)^{2}+\epsilon}\text{,}$
$\displaystyle\bar{f}_{n,c,h,w}^{l}$
$\displaystyle=(f_{n,c,h,w}^{l}-\mu_{n,c}^{l})/{\sigma_{n,c}^{l}}$
$\displaystyle f^{l+1}_{n,c,h,w}$
$\displaystyle=\gamma_{n,c,h,w}^{l}\cdot\bar{f}_{n,c,h,w}^{l}$

where $f_{n,c,h,w}^{l}$ is the activation before normalization and
$\mu_{n,c}^{l}$ and $\sigma_{n,c}^{l}$ are the mean and standard deviation of
the activation in batch $n$ and channel $c$. This denormalization processing
depends on the condition image and vary with respect to the location, which
make the implicit structure information play an important role in modulating
network activations. In this stage, a SR network ${\mathcal{R}}(\cdot)$ and a
discriminator ${\mathcal{D}}_{real}^{H}$ are trained with the pseudo pairs
$(I_{gen}^{L},I_{real}^{H})$ and these modules are constrained by following
terms.

Adversarial loss in HR. An adversarial training constraint is imposed on the
sets of $(I_{gen}^{H})$ and $(I_{real}^{H})$ to minimize the distance of two
distributions. We choose the hinge version of adversarial loss for
${\mathcal{R}}(\cdot)$, denoted as

$\displaystyle\mathcal{L}_{adv}^{H}(\mathcal{D}_{real}^{H})=$
$\displaystyle-\mathbb{E}\left[\min(0,-1+\mathcal{D}_{real}^{H}(I_{real}^{H}))\right]$
(7)
$\displaystyle-\mathbb{E}\left[\min(0,-1-{\mathcal{D}}_{real}^{H}({\mathcal{R}}(I_{gen}^{L})))\right]$
$\displaystyle\mathcal{L}_{adv}^{H}(\mathcal{R})=$
$\displaystyle-\mathbb{E}\left[{\mathcal{D}}_{real}^{H}({\mathcal{R}}(I_{gen}^{L}))\right]\text{.}$

Reconstruction loss. To maintain the consistency of image contents between
inputs and generated images. We apply reconstruction loss on the input image
$I_{real}^{H}$ and the SR result $I_{gen}^{H}$, that gives

$\displaystyle\mathcal{L}_{rec}(I_{real}^{H},I_{gen}^{H})=\mathbb{E}[\|I_{real}^{H}-I_{gen}^{H}\|_{1}]\text{.}$
(8)

Figure 4: Self-conditioned block takes the input image as condition term,
which is utilized to denormalize deep features. The activation layer is
omitted.

|  |  |  |  |  |  |
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
Bicubic | DIC [4] | FSRGAN [14] | PFSR [53] | DICGAN [4] | Ours-paired | Ours-unpaired | HR

Figure 5: Qualitative comparisons with state-of-the-art FSR methods, where
each row shows the results for one randomly selected person with different
methods. The first column shows results with vanilla bicubic, second to fifth
columns are from different SOTA methods, columns six through seven show our
results with/without paired images, and the last column is the high-resolution
ground truth. Best viewed on screen for more details.

R1 regularization. For improving the convergence robustness of adversarial
training, we adopt the R1 regularization [54]. This regularization penalizes
the discriminator gradients on the true data distribution for better local
stability, denoted as

$\displaystyle\mathcal{L}_{r1}(\mathcal{D}_{real}^{H})=\frac{r}{2}\mathbb{E}_{p(I_{real}^{H})}[\|\nabla
D_{real}^{H}(I_{real}^{H})\|^{2}]\text{,}$ (9)

where $r$ is set to 10 in our experiments. The full objective of super-
resolution stage is

$\displaystyle\mathcal{L}({\mathcal{R}},{\mathcal{D}}_{real}^{H})$
$\displaystyle=\mathcal{L}_{adv}^{H}+\lambda_{rec}\mathcal{L}_{rec}+\lambda_{r1}\mathcal{L}_{r1}\text{,}$
(10)

where $\lambda_{i}$ is the trade-off parameters for $i$-th constraint term.

##  IV Experiments

We first evaluate our model on the synthetic distortion under both paired and
unpaired data settings. We then provide a quantitative and qualitative
comparison against state-of-the-art methods with realistic degradation.
Finally, an ablation study and visualization are performed to evaluate the
effect of each proposed component in SelFSR.

Implementation details. We implement our face super-resolution framework in
PyTorch [55]. All models are trained by Adam optimizer [56] with
$\beta_{1}=0.9$, $\beta_{2}=0.99$, and batch size of 48. We use the trick of
equalized learning rate [19] in our generator and discriminator training. The
learning rate, therefore, is set to 0.003 without adjustment during training.
Training our model on CelebA from scratch takes about 24 hours using 4 Tesla
V100.

TABLE I: Quantitative comparisons with SOTA methods on CelebA and Helen
datasets in several commonly used metrics. Ours-P adopts paired images during
the training phase while Ours-UP uses unpaired images. L represents that extra
facial landmark is involved in the training stage. The bold and underscore
indicate optimal and sub-optimal results, respectively. The upward arrow
indicates that the higher the value, the better, and vice versa.

Method | L | CelebA | Helen | #Param.
---|---|---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | FID $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | FID $\downarrow$
Bicubic | 23.58 | 0.6285 | 0.5219 | 159.06 | 23.89 | 0.6751 | 0.5026 | 171.79 | -
RDN [57] |  | 26.13 | 0.7412 | 0.3135 | 96.14 | 25.34 | 0.7249 | 0.3496 | 104.38 | 24.34 M
FSRNet [14] | ✓ | 26.48 | 0.7718 | 0.2968 | 94.39 | 25.90 | 0.7759 | 0.3539 | 101.13 | 2.15 M
DIC [4] | ✓ | 27.37 | 0.7962 | 0.2565 | 89.24 | 26.69 | 0.7933 | 0.3378 | 94.34 | 21.80 M
FSRGAN [14] | ✓ | 25.06 | 0.7311 | 0.2342 | 85.17 | 24.99 | 0.7424 | 0.2799 | 82.21 | 2.15 M
PFSR [53] | ✓ | 24.43 | 0.6991 | 0.1500 | 69.33 | 24.73 | 0.7323 | 0.2017 | 77.62 | 8.97 M
DICGAN [4] | ✓ | 26.34 | 0.7562 | 0.0840 | 34.49 | 25.96 | 0.7624 | 0.0917 | 50.21 | 21.80 M
Ours-P |  | 26.59 | 0.7696 | 0.0726 | 30.25 | 26.87 | 0.7962 | 0.0911 | 50.03 | 17.89 M
Ours-UP |  | 23.42 | 0.6539 | 0.1141 | 35.61 | 23.59 | 0.6691 | 0.1346 | 67.53 | 17.89 M

Performance Metrics. Under the paired training setting, SR results are
assessed with the standard fidelity oriented metrics, structural similarity
index (SSIM) [58] and Peak Signal to Noise Ratio (PSNR). They are transformed
to YCbCr space first and computed on the Y channel only. Since the SSIM and
PSNR metrics are known to not consistent with the human perception of image
quality, we also use Fréchet Inception Distance (FID) [59] and Learned
Perceptual Image Patch Similarity (LPIPS) [60] to assess the photo-realism of
upscaled images. When the ground truth of HR images is unavailable, we just
report the FID value to measure the distribution distance between the high
resolution images and generated results.

###  IV-A Experiments on Synthetic Distortions

Datasets. We conduct experiments on widely used CelebA [61] and Helen [62]
datasets, following the experimental procedure described by Cheng et al. [4].
They crop facial region in each image based on estimated landmark [63], and
resize it to $128\times 128$ pixels. Then the HR images are downsampled to
$16\times 16$ LR counterparts with the bicubic kernel. For CelebA dataset, we
use 168,854 images for training and 1,000 images for testing. For Helen
dataset, there are 2,005 images for training and 50 images for testing. Only
random horizontal flip is performed as the regular data augmentation in all
experiments.

Hyperparameters. In this subsection, we set the weights in Eq.(10) as
$\lambda_{rec}=150$ and $\lambda_{r1}=3$. For the paired setting, we train our
generator for 160k iterations. The model starts by $\times 2$ scale factor,
and grows up at 20k and 80k steps. The final scale factor is $\times 8$. For
the unsupervised setting, we extend the training scheduler to 280k iterations
and the generator grows up at 40k and 120k steps. Without the groundtruth of
HR images, Eq.(8) is modified to a cycle consistency formulation:

$\displaystyle\mathcal{L}_{rec}(I_{syn}^{L})=\mathbb{E}[\|({\mathcal{R}}(I_{syn}^{L}))_{\downarrow}-I_{syn}^{L}\|_{1}]\text{,}$
(11)

where $(\cdot)_{\downarrow}$ means bicubic degradation.

Comparison with state-of-the-art methods. We show some SR results of different
methods in Figure 5. It can be seen that our model is able to generate photo-
realistic images and synthesize various facial details like: glasses, pupils,
wrinkles, beards and teeth. Among all baselines, DICGAN is most close to our
approach and can output plausible images. However, our model outperforms it at
detail generation. For instance, in the fourth row, DICGAN is failed in
generating correct details of pupils as our model does. Another example is
that the tooth generated by DICGAN looks blur as shown in the second and
fourth row. Compared with other baselines, SelFSR produces much more realistic
textures while other methods yield obvious artifacts and distortions: PFSR is
able to produce realistic SR results but sometimes performs poorly at
preserving facial structures; the SR results created by FSRGAN are over
smooth; DIC achieves bad performance at human perceptual as a PSNR-oriented
method. The qualitative comparison with state-of-the-art face SR methods,
especially DICGAN, demonstrates the powerful generative ability of the
generator constructed by self-conditioned block.

Following previous works [14, 4], we assess the quality of the SR results with
the PSNR and SSIM. Since the above metrics are known to not correlate well
with human perception, we also report the image quality metrics FID as well as
LPIPS. As shown in Table I, our supervised method performs better than DICGAN
and largely outperforms other baselines. Compared with DICGAN, our end-to-end
approach does not need auxiliary face landmark information nor iterative
collaboration during the training phase. Namely, our model is more effective
(no landmark) and efficient (no iteration) in the training phase. It can be
seen that our approach can achieve high performance at all metrics. This
indicates that SelFSR is able to preserve pixel-wise fidelity while increasing
perceptual quality of generated super-resolution images.

Unsupervised results. Furthermore, we evaluate our SR network in an
unsupervised manner where we assuming the LR and HR images are unpaired. The
ground truth is only used to assess the image quality of SR results. As shown
in Figure 5, our model can also generate pleasant SR images under the unpaired
setting. The SR results are comparable to supervised variation as well as
DICGAN though the corresponding PSNR and SSIM values are almost worst. Such an
observation is reasonable since SSIM and PSNR are fidelity oriented metrics
while the ground truth is absent under this setting. For FID and LPIPS, the
unsupervised result only performs slightly worse than the supervised one. This
result indicates a promising potentiality in misaligned scenarios where the LR
and HR images come from the same distribution but have an unavoidable
misalignment, e.g. data captured by a DSLR camera [6].

###  IV-B Experiments on Realistic Distortions

Real-world face image dataset. For real-world face super-resolution, we follow
a similar data setting in previous works [10, 11]. The authors created a face
dataset of 182,866 HR images from CelebA [61], AFLW [64], LS3D-W [65] and
VGGFace2 [66]. LR face images were collected from Widerface [67] for diverse
degradation and noise. In total, there are 53,254 LR images for training and
3,000 images for testing. The HR and LR training images have different
distribution, with $64\times 64$ and $16\times 16$ pixels respectively.

TABLE II: Quantitative results on real-world face super-resolution.  Method |  FID $\downarrow$
---|---
DeepDeblur [68] |  294.96
Wavelet-SRNet [69] |  149.46
SRGAN [38] |  126.75
FSRNet [14] |  148.97
CycleGAN [70] |  37.63
Bulat et al. [10] |  26.41
Shunta et al. [11] |  33.19
Our |  25.07
Figure 6: Qualitative comparisons with state-of-the-art FSR methods on real-
world face super-resolution. Each column shows the results for one randomly
selected person in real world with different methods.

Hyperparameters. During the domain adaptation stage, we set the
hyperparameters in Eq.(5) as $\lambda_{idt}=10$ and $\lambda_{s}=1$. we train
the adaptation network for 180k iterations with a learning rate of 2e-4, which
linearly decays to zero over the last half steps. For the SR network, the
weights in Eq.(10) is set as $\lambda_{rec}=150$ and $\lambda_{r1}=3$. The SR
network is trained for 160k iterations, starting with a factor of $\times 2$
and growing up at 60k steps. The final scale factor is $\times 4$.

Comparison with state-of-the-art methods. Table II shows the quantitative
comparison with the related methods. Our model achieves a slightly better
result than Bulat et al.’s and largely exceeds other baselines. As can be seen
in Figure 6, we further provide some perceptual comparisons to two closest
baselines [10, 11]. All three approaches can generate plausible results.
Compared to the model of Shunta et al., our model is more inclined to generate
smooth results. For instance, we can see intense local noise in their result
at the third column. For the baseline of Bulatet al., our model outperforms it
at maintaining the features of the input, e.g., we can keep the skin color of
the LR images better.

TABLE III: Ablation study for domain adaptation on real-world degradation. Vanilla: train SR network with clean LR-HR pairs and test it on real LR images. Cyc: perform domain adaptation with CycleGAN [70]. FF: flow field degradation network. Idt: identity loss in Eq.( 3). FID is calculated between the final SR results and real HR images. FID‡ denotes the distance between the estimated LR images and the real ones. Vanilla | Cyc | FF | Idt | FID | FID‡
---|---|---|---|---|---
✓ |  |  |  | 61.82 | 24.28
| ✓ |  |  | 33.73 | 14.12
| ✓ |  | ✓ | 29.94 | 12.81
|  | ✓ | ✓ | 25.07 | 12.03
Figure 7: SR results of different variations on real LR faces. The
abbreviation is same as Table III.

###  IV-C Ablation Study

Components in domain adaptation stage. We conduct the ablation study for
domain adaptation on real face dataset. Visual samples of different variations
are displayed in Figure 7. We first assess the performance of our vanilla
variation where the model is trained with clean pairs and directly evaluated
on real LR images. As shown in the second column, there are noticeable
distortions about facial structure in SR results since the domain gap can not
be ignored. We further study the effect of proposed flow field architecture
(FF) by replacing it with CycleGAN. Comparing the results of columns three
through five, we can find that FF outperforms CycleGAN at maintaining the
identity characteristic of the input. The quantitative assessment is
demonstrated in Table III. We can see that the performance of super-resolution
on real face is dramatically improved with adaptive degradation. For two
domain adaptation networks, FF get better FID value than CycleGAN on both the
final SR results and the intermediate translation results. We believe this
improvement is benefited from estimating degradation via flow field. As shown
in Figure 8, we visualize various noisy LR images via disturbing estimated
flow field. It is worth noting that there are no dramatic changes of identity
features in all disturbed images, indicating a pleasurable property of flow
field network at identity consistency.

Figure 8: Visual results of flow field disturbance. The first column is downsampled LR images, the second is noisy LR images estimated by degradation network, and the others are generated by randomly disturbing estimated flow field. TABLE IV: Ablations for components in self-conditioned SR network on CelebA. Top is the results under the supervised setting while the bottom is unsupervised. SCB: Self-conditioned block. Prog.: Progressive training. R1: R1 regularization.  Vanilla | SCB | Prog. | R1 | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | FID $\downarrow$
---|---|---|---|---|---|---|---
✓ |  | ✓ | ✓ | 25.57 | 0.7354 | 0.1105 | 37.12
| ✓ |  | ✓ | 25.83 | 0.7426 | 0.1023 | 34.97
| ✓ | ✓ |  | 25.94 | 0.7483 | 0.0912 | 33.46
| ✓ | ✓ | ✓ | 26.59 | 0.7696 | 0.0726 | 30.25
✓ |  | ✓ | ✓ | 22.72 | 0.6101 | 0.1523 | 56.18
| ✓ |  | ✓ | 22.88 | 0.6135 | 0.1428 | 52.67
| ✓ | ✓ |  | 23.64 | 0.6660 | 0.1238 | 39.09
| ✓ | ✓ | ✓ | 23.42 | 0.6539 | 0.1141 | 35.61

Components in SR network. We further conduct an ablation study to evaluate the
importance of each component in SR network under both supervised and
unsupervised settings. Our vanilla variation concatenates the LR input with
intermediate features instead of using it to modulate network activation. As
shown in the upper half of the Table IV, all metrics are gradually increased
as the different components are added, meaning that each component is useful
for the model. It is worth noting that self-conditioned block improves SR
performance with a large margin, indicating that this block is a more
effective way to fuse structure information contained in the inputs.

The bottom half shows the results under the unpaired setting, and we can come
to a similar conclusion as supervised training for LPIPS and FID metrics.
However, unsupervised training seems not friendly for fidelity oriented
metrics like PSNR and SSIM, as they decrease in a large margin, e.g. 26.59 to
23.43 (-3.16) for PSNR and 0.7696 to 0.6539 (-0.1157) for SSIM. It is a
reasonable result since the pixel-level constraint is not adopted under the
unpaired setting. It can be seen that the _progressive_ strategy plays an
important role for unsupervised training, improving several metrics largely.
We believe the performance improvement is benefited from the training
stability improved by the progressive training, which makes model learn the
overall distribution at beginning and then enhance the details.

###  IV-D Qualitative Results

Visualization of learned structural information. As shown in Figure 9, we
visualized some feature maps of condition branch the self-conditioned block.
Since the normalized activation is element-wise multiplied by the output of
condition branch, the output of the condition branch can be seen as the weight
of each position. Namely, The bright the area is the attention area in the
corresponding channel. In Figure 9, we can see that the shallow feature maps
focus on some the low-level image features like boundary, while the deep
feature maps show some semantic structure information like mouth or face. This
visualization intuitively displays the learned structure information in self-
conditioned block, which we believe is why our model achieved SOTA SR
performance even with less parameters.

Figure 9:  Visualization of some feature maps in condition branch of the self-
conditioned block. The brighter the area, the larger the activation value.
Figure 10: Some visual samples of typical failure case generated by our
method. The first row represents different typical low-resolution samples,
while the second row shows corresponding generated high-resolution images.

Visualization of failure case. We also analyze some failure cases to make our
method more objective. The failure cases can be typically divided into two
categories: the first case is not distinguishable even by the human perceptual
system; the second is some difficult samples like over-blurry or containing
occlusion. The first two columns in the Figure 10 are not distinguishable by
the human eye, let alone by machine. The third column is too fuzzy, while
faces in the fourth column has an extreme angle and the last column have a
non-negligible occlusion. All the above problems will reduce the performance
of the super-resolution network.

##  V Conclusion

In this paper, we propose a novel two-stage framework to perform face super-
resolution in the wild. To avoid the dramatic drifting of the identity
features of the input, we propose a displacement-based network to estimate the
real LR counterparts. Such a model performs better at appearance consistency
with the preference of capturing motion noise. Besides, a backbone built on a
self-conditioned block is presented for face super-resolution. The novel block
has shown a powerful ability at extracting structure information from faces.
As a result, the backbone achieves state-of-the-art performance, with fewer
parameters and no need of facial priors. The experiment results on CelebA and
real face images show high perceptual quality FSR images, demonstrating the
powerful generative ability of our proposed architecture.

##  VI Acknowledgements

This work is supported by the National Natural Science Foundation of China
under Grant 61836015 and Grant 61771193.

## References

  * Wang et al. [2018a] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “Esrgan: Enhanced super-resolution generative adversarial networks,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 0–0.
  * Zhang et al. [2018a] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 286–301.
  * Liu et al. [2020] J. Liu, W. Zhang, Y. Tang, J. Tang, and G. Wu, “Residual feature aggregation network for image super-resolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2359–2368.
  * Ma et al. [2020a] C. Ma, Z. Jiang, Y. Rao, J. Lu, and J. Zhou, “Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 5569–5578.
  * Menon et al. [2020] S. Menon, A. Damian, S. Hu, N. Ravi, and C. Rudin, “Pulse: Self-supervised photo upsampling via latent space exploration of generative models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2437–2445.
  * Zhang et al. [2019a] X. Zhang, Q. Chen, R. Ng, and V. Koltun, “Zoom to learn, learn to zoom,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3762–3770.
  * Cai et al. [2019] J. Cai, H. Zeng, H. Yong, Z. Cao, and L. Zhang, “Toward real-world single image super-resolution: A new benchmark and a new model,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 3086–3095.
  * Gu et al. [2019] J. Gu, H. Lu, W. Zuo, and C. Dong, “Blind super-resolution with iterative kernel correction,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2019, pp. 1604–1613.
  * Zhou and Susstrunk [2019] R. Zhou and S. Susstrunk, “Kernel modeling super-resolution on real low-resolution images,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 2433–2443.
  * Bulat et al. [2018] A. Bulat, J. Yang, and G. Tzimiropoulos, “To learn image super-resolution, use a gan to learn how to do image degradation first,” in _Proceedings of the European Conference on Computer Vision_ , 2018, pp. 185–200.
  * Maeda [2020] S. Maeda, “Unpaired image super-resolution using pseudo-supervision,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 291–300.
  * Gong et al. [2020] D. Gong, W. Sun, Q. Shi, A. v. d. Hengel, and Y. Zhang, “Learning to zoom-in via learning to zoom-out: Real-world super-resolution by generating and adapting degradation,” _arXiv preprint arXiv:2001.02381_ , 2020.
  * Wei et al. [2020] Y. Wei, S. Gu, Y. Li, and L. Jin, “Unsupervised real-world image super resolution via domain-distance aware training,” _arXiv preprint arXiv:2004.01178_ , 2020.
  * Chen et al. [2018] Y. Chen, Y. Tai, X. Liu, C. Shen, and J. Yang, “Fsrnet: End-to-end learning face super-resolution with facial priors,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 2492–2501.
  * Ma et al. [2020b] C. Ma, Y. Rao, Y. Cheng, C. Chen, J. Lu, and J. Zhou, “Structure-preserving super resolution with gradient guidance,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 7769–7778.
  * Zeng et al. [2020] X. Zeng, Y. Pan, M. Wang, J. Zhang, and Y. Liu, “Realistic face reenactment via self-supervised disentangling of identity and pose,” _arXiv preprint arXiv:2003.12957_ , 2020.
  * Shen and Zhou [2020] Y. Shen and B. Zhou, “Closed-form factorization of latent semantics in gans,” _arXiv preprint arXiv:2007.06600_ , 2020.
  * Zhang et al. [2020] J. Zhang, X. Zeng, M. Wang, Y. Pan, L. Liu, Y. Liu, Y. Ding, and C. Fan, “Freenet: Multi-identity face reenactment,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 5326–5335.
  * Karras et al. [2018] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” in _International Conference on Learning Representations (ICLR)_ , 2018.
  * Karras et al. [2019] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2019, pp. 4401–4410.
  * Karras et al. [2020] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of stylegan,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 8110–8119.
  * Park et al. [2019] T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu, “Semantic image synthesis with spatially-adaptive normalization,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Korshunova et al. [2017] I. Korshunova, W. Shi, J. Dambre, and L. Theis, “Fast face-swap using convolutional neural networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 3677–3685.
  * Banerjee et al. [2020] S. Banerjee, W. Scheirer, K. Bowyer, and P. Flynn, “On hallucinating context and background pixels from a face mask using multi-scale gans,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 300–309.
  * Lai et al. [2018] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Fast and accurate image super-resolution with deep laplacian pyramid networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 11, pp. 2599–2613, 2018.
  * Kim et al. [2016a] J. Kim, J. K. Lee, and K. M. Lee, “Deeply-recursive convolutional network for image super-resolution,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1637–1645.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * Huang et al. [2017a] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * Lim et al. [2017] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 136–144.
  * Ren et al. [2016] C. Ren, X. He, Q. Teng, Y. Wu, and T. Q. Nguyen, “Single image super-resolution using local geometric duality and non-local similarity,” _IEEE Transactions on Image Processing_ , vol. 25, no. 5, pp. 2168–2183, 2016\.
  * Chen et al. [2021] C. Chen, D. Gong, H. Wang, Z. Li, and K.-Y. K. Wong, “Learning spatial attention for face super-resolution,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 1219–1231, 2021.
  * Kim et al. [2016b] J. Kim, J. K. Lee, and K. M. Lee, “Accurate image super-resolution using very deep convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1646–1654.
  * Li et al. [2018] J. Li, F. Fang, K. Mei, and G. Zhang, “Multi-scale residual network for image super-resolution,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 517–532.
  * Tai et al. [2017] Y. Tai, J. Yang, X. Liu, and C. Xu, “Memnet: A persistent memory network for image restoration,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 4539–4547.
  * Tong et al. [2017] T. Tong, G. Li, X. Liu, and Q. Gao, “Image super-resolution using dense skip connections,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 4799–4807.
  * Han et al. [2018] W. Han, S. Chang, D. Liu, M. Yu, M. Witbrock, and T. S. Huang, “Image super-resolution via dual-state recurrent networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 1654–1663.
  * Zhang et al. [2019b] Y. Zhang, K. Li, K. Li, B. Zhong, and Y. Fu, “Residual non-local attention networks for image restoration,” _International Conference on Learning Representations (ICLR)_ , 2019.
  * Ledig et al. [2017] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang _et al._ , “Photo-realistic single image super-resolution using a generative adversarial network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 4681–4690.
  * Johnson et al. [2016] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in _European conference on computer vision_. Springer, 2016, pp. 694–711.
  * Goodfellow et al. [2014] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2014.
  * Jolicoeur-Martineau [2018] A. Jolicoeur-Martineau, “The relativistic discriminator: a key element missing from standard gan,” _arXiv preprint arXiv:1807.00734_ , 2018.
  * Dong et al. [2015] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 38, no. 2, pp. 295–307, 2015.
  * Dong et al. [2016] C. Dong, C. C. Loy, and X. Tang, “Accelerating the super-resolution convolutional neural network,” in _Computer Vision – ECCV 2016_ , B. Leibe, J. Matas, N. Sebe, and M. Welling, Eds. Cham: Springer International Publishing, 2016, pp. 391–407.
  * Shi et al. [2016] W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1874–1883.
  * Wang et al. [2018b] Y. Wang, F. Perazzi, B. McWilliams, A. Sorkine-Hornung, O. Sorkine-Hornung, and C. Schroers, “A fully progressive approach to single-image super-resolution,” in _IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 864–873.
  * Lai et al. [2017] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep laplacian pyramid networks for fast and accurate super-resolution,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 5835–5843.
  * Bashir et al. [2021] S. M. A. Bashir, Y. Wang, M. Khan, and Y. Niu, “A comprehensive review of deep learning-based single image super-resolution,” _PeerJ Computer Science_ , vol. 7, p. e621, 2021.
  * Yang et al. [2019] W. Yang, X. Zhang, Y. Tian, W. Wang, J.-H. Xue, and Q. Liao, “Deep learning for single image super-resolution: A brief review,” _IEEE Transactions on Multimedia_ , vol. 21, no. 12, pp. 3106–3121, 2019.
  * Chen et al. [2019] C. Chen, Z. Xiong, X. Tian, Z.-J. Zha, and F. Wu, “Camera lens super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1652–1660.
  * Lugmayr et al. [2019] A. Lugmayr, M. Danelljan, and R. Timofte, “Unsupervised learning for real-world super-resolution,” in _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_. IEEE, 2019, pp. 3408–3416.
  * Yuan et al. [2018] Y. Yuan, S. Liu, J. Zhang, Y. Zhang, C. Dong, and L. Lin, “Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 701–710.
  * Jaderberg et al. [2015] M. Jaderberg, K. Simonyan, A. Zisserman _et al._ , “Spatial transformer networks,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2015.
  * Kim et al. [2019] D. Kim, M. Kim, G. Kwon, and D.-S. Kim, “Progressive face super-resolution via attention to facial landmark,” _arXiv preprint arXiv:1908.08239_ , 2019.
  * Mescheder et al. [2018] L. Mescheder, A. Geiger, and S. Nowozin, “Which training methods for gans do actually converge?” _arXiv preprint arXiv:1801.04406_ , 2018.
  * Van Merriënboer et al. [2018] B. Van Merriënboer, O. Breuleux, A. Bergeron, and P. Lamblin, “Automatic differentiation in ml: Where we are and where we should be going,” in _Advances in neural information processing systems_ , 2018, pp. 8757–8767.
  * Kingma and Ba [2015] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _International Conference on Learning Representations (ICLR)_ , 2015.
  * Zhang et al. [2018b] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 2472–2481.
  * Wang et al. [2004] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * Heusel et al. [2017] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Zhang et al. [2018c] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 586–595.
  * Liu et al. [2015] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 3730–3738.
  * Le et al. [2012] V. Le, J. Brandt, Z. Lin, L. Bourdev, and T. S. Huang, “Interactive facial feature localization,” in _European conference on computer vision_. Springer, 2012, pp. 679–692.
  * Baltrusaitis et al. [2018] T. Baltrusaitis, A. Zadeh, Y. C. Lim, and L.-P. Morency, “Openface 2.0: Facial behavior analysis toolkit,” in _2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)_. IEEE, 2018, pp. 59–66.
  * Koestinger et al. [2011] M. Koestinger, P. Wohlhart, P. M. Roth, and H. Bischof, “Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization,” in _2011 IEEE international conference on computer vision workshops_. IEEE, 2011, pp. 2144–2151.
  * Bulat and Tzimiropoulos [2017] A. Bulat and G. Tzimiropoulos, “How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks),” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1021–1030.
  * Cao et al. [2018] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “Vggface2: A dataset for recognising faces across pose and age,” in _2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)_. IEEE, 2018, pp. 67–74.
  * Yang et al. [2016] S. Yang, P. Luo, C.-C. Loy, and X. Tang, “Wider face: A face detection benchmark,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 5525–5533.
  * Nah et al. [2017] S. Nah, T. Hyun Kim, and K. Mu Lee, “Deep multi-scale convolutional neural network for dynamic scene deblurring,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 3883–3891.
  * Huang et al. [2017b] H. Huang, R. He, Z. Sun, and T. Tan, “Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1689–1697.
  * Zhu et al. [2017] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 2223–2232.


# Discovery of a $584.65$ Hz Burst Oscillation in the Low Mass X-Ray Binary 4U
1730–22

Zhaosheng Li  Key Laboratory of Stars and Interstellar Medium, Xiangtan
University, Xiangtan 411105, Hunan, P.R. China Zhaosheng Li, Yuanyue Pan
<EMAIL_ADDRESS><EMAIL_ADDRESS>Wenhui Yu  Key Laboratory of Stars
and Interstellar Medium, Xiangtan University, Xiangtan 411105, Hunan, P.R.
China  Yongqi Lu  Key Laboratory of Stars and Interstellar Medium, Xiangtan
University, Xiangtan 411105, Hunan, P.R. China  Yuanyue Pan  Key Laboratory of
Stars and Interstellar Medium, Xiangtan University, Xiangtan 411105, Hunan,
P.R. China  Maurizio Falanga  International Space Science Institute (ISSI),
Hallerstrasse 6, 3012 Bern, Switzerland  Physikalisches Institut, University
of Bern, Gesellsschaftstrasse 6, 3012 Bern, Switzerland

###### Abstract

Type-I X-ray burst oscillations are powered by thermonuclear released on the
neutron star (NS) surface in low mass X-ray binaries (LMXBs), where the burst
oscillation frequencies are close to the NS spin rates. In this work, we
report the detection of oscillation at 584.65 Hz during the cooling tail of a
type-I X-ray bursts observed from the accreting NS LMXB 4U 1730–22 in 2022
March 20, by the  Neutron star Interior Composition Explorer (NICER)
telescope. The oscillation signal showed a strong Leahy power, $P_{\rm m}\sim
54.04$, around 584.65 Hz, which has single trial and multiple trials
confidence levels of $7.05\sigma$ and $4.78\sigma$, respectively. The folded
pulse profile of the oscillation in the 0.2–10 keV band showed a sinusoidal
shape with the fraction amplitude rms of $(12.5\pm 1.8)\%$. We found the
oscillation frequency showed insignificant upward drifting, i.e., less than
0.3 Hz, during the cooling tail, similar as the behavior appearing in
accreting millisecond X-ray pulsars (AMXP), and indicate the source could be
an AMXP spinning at 1.71 ms.

X-ray bursts; Low-mass x-ray binary stars; X-ray sources; Neutron stars

††facilities: NICER (Gendreau et al., 2012).††software: Stingray (Huppenkothen
et al., 2019), HEASOFT (NASA High Energy Astrophysics Science Archive Research
Center (2014), Heasarc).

##  1 Introduction

The X-ray source 4U 1730–22 was discovered by Uhuru during its X-ray outburst
in 1972 (Forman et al., 1978; Cominsky et al., 1978). The quiescent state of
the source has been observed by Chandra, which associates with the faint X-ray
source CXOU J173357.5–220156 (Tomsick et al., 2007). Based on the persistent
and quiescent X-ray spectra, 4U 1730–22 was classified as a probable neutron
star low-mass X-ray binary (NS LMXB, Tanaka & Shibazaki, 1996; Chen et al.,
1997). On 2021 June 7, the MAXI/GSC team reported the activities of an X-ray
transient, which is associated with 4U 1730–22 after 50 years quiescent
confirmed by Swift follow-up observations (Kobayashi et al., 2021; Iwakiri et
al., 2021; Kennea et al., 2021a, b).

Type-I X-ray bursts are triggered from unstable thermonuclear burning of
accreting material on the NS surface (Lewin et al., 1993; Galloway et al.,
2008; Galloway & Keek, 2021). The first type-I X-ray burst from the source was
detected by Neutron star Interior Composition Explorer (NICER, Bult et al.,
2021c), confirming definitively, that the source is a transitional LMXB
hosting an accreting NS. Moreover, the optical counterpart of 4U 1730–22 has
been identified, which shows strong hydrogen and weaker helium emission lines
indicating a companion star in main sequence (Russell et al., 2021; Strader et
al., 2021).

Besides type-I X-ray bursts, the detection of coherent pulsation can also
confirm the compact object in a LMXB as a NS. In addition, measuring the NS
spin periods are also important in may other aspects, such as studying the
spin evolution during the accretion process, evaluating the broadening effects
of spectral lines on the NS surface (Chang et al., 2006), measuring the orbit
parameters (see e.g., Strohmayer et al., 2018), determining the NS masses and
radii from photospheric radius expansion bursts (Suleimanov et al., 2020),
constraining on the NS equation of state especially from ultra fast rotational
NS (i.e., sub-millisecond pulsars, Haensel et al., 2009). The spin periods of
a large fraction of NSs have been measured from their coherent emissions
observed in radio or X-ray bands or both (see e.g., Manchester et al., 2005;
Patruno & Watts, 2021; Liu et al., 2006, 2007; Walter et al., 2015).

During type-I X-ray bursts, NS may produce inhomogeneous thermal emissions on
the surface, which manifests as burst oscillations (Strohmayer et al., 1996).
The type-I X-ray burst oscillations provide us an indirect method to measure
the NS spin periods, due to the fact the the oscillation frequencies measured
in accreting millisecond X-ray pulsars (AMXPs) are very close to their
coherent spin frequencies within a few hertz during the persistent emissions
(e.g., Chakrabarty et al. 2003, see Watts 2012 and Bhattacharyya 2022 for
reviews). Totally, five persistent AMXPs have been observed burst
oscillations, and their oscillation frequencies usually showed negligible
upward drifting during the cooling tails (Chakrabarty et al., 2003; Galloway
et al., 2008; Bilous & Watts, 2019).

NICER was launched and installed on the International Space Station on June 3,
2017, which has the capabilities to collect X-ray photons in the energy of
0.2–12 keV with absolute time resolution as high as 100 ns (Gendreau &
Arzoumanian, 2017). After five years operation, NICER has been observed $\sim
56\%$ of all known NS LMXBs. Benefit from its high timing accuracy and
relatively large collecting area, NICER has been detected burst oscillations
from 4U 1728–34 at $\sim$362.5 Hz (Mahmoodifar et al., 2019) and SAX
J1808.4–3658 at 401 Hz (Bult et al., 2019), which were previously found by
RXTE (Strohmayer et al., 1996; Chakrabarty et al., 2003). Recently, the burst
oscillation candidate from XTE J1739–285 has been observed at 386.5 Hz (Bult
et al., 2021a), rather than 1122 Hz reported by Kaaret et al. (2007). However,
most of sources have not been reported the detection of burst oscillations
(see e.g., Aql X–1 and 4U 1636–536, Li et al., 2021; Güver et al., 2022; Zhao
et al., 2022), or the type-I X-ray bursts.

In this letter, we report the detection of the burst oscillation at the
frequency $\sim 584.65$ Hz from 4U 1730–22.

##  2 Observations

Figure 1: Left panel, the 1 s binned light curve of the burst from ObsID
5202200113 (burst #4). We search the time interval between $-5$ s and 75 s.
The contours mark the $Z_{1}^{2}$ power from 25 to 55 with the steps of 5. The
insert zoomed in panel shows the time interval where the oscillation detected.
Right panel, the Leahy and $Z_{1}^{2}$ power marked as vertical dashed lines,
during the burst interval $t\sim 8.5-12.5$ s, in the left panel. The
horizontal dashed lines represent the single trial significance of $4\sigma$
and $6\sigma$, respectively.

In this work, we analyzed all public archived NICER observations of 4U 1730–22
on Modified Julian Date (MJD) 59642.0–59715.3 which have a total unfiltered
exposure of 307.7 ks. The ObsIDs include 42022001(01$\sim$43),
46390101(01$\sim$39) and 52022001(01$\sim$19). We processed the NICER data
analysis using HEASOFT V6.30.1 and the NICER Data Analysis Software
(NICERDAS). We adopted the default selection criteria by using nicerl2 to
filter the cleaned event data. We then applied barycentric corrections using
the tool barycorr to all the event data employing the source coordinates
$\rm{R.A.=263^{\circ}.489792}$, $\rm{D.E.C.=-22^{\circ}.032472}$ (Tomsick et
al., 2007) and the JPL-DE430 ephemeris. The 1 s light curves in the energy of
0.5–10 keV of all observations were extracted. From light curves, we totally
identified nine type-I X-ray bursts, where the observation log and properties
of all bursts are listed in Table. 1.

Figure 2: The folded pulse profile of the burst oscillation in the energy
range of 0.2–10 keV during the interval marked as vertical dashed lines in the
left panel of Fig. 1. The red dashed line represents the best fitted
sinusoidal model, $A+B\sin(2\pi\nu t-\phi_{0})$. In order to clarity, two
cycles are shown.

##  3 Results

Since the spin period or burst oscillation were previously unknown in 4U
1730–22, we carried out blind searches of all type-I X-ray bursts in Table 1
for testing the presence of coherent burst oscillations, where the time
interval of each burst starts $-5$ s prior to the burst trigger. To preserve
as many photons as possible, the cleaned event files in the 0.2–10 keV energy
range were used. We applied the fast Fourier transform (FFT) technique with
Leahy normalization to calculate the power spectra. The Leahy power satisfies
the $\chi^{2}$ distribution with two degree of freedom (Leahy et al., 1983).
We adopt the slide window of $\Delta T=4$ s, each new window moving forward
0.5 s with respect to the previous one. For each window, the statistically
independent Fourier frequencies between 50 and 2000 Hz with steps of 0.25 Hz
and the corresponding power were recorded (see e.g., Bilous & Watts, 2019;
Bult et al., 2021a).

We found the cooling tail in burst #4 shows strong oscillation signal at $\sim
584.65$ Hz with the Leahy power peaked at $P_{\rm m}=54.04$, well exceed
$P_{\rm m}=2$ expected from white noise. It corresponds to a probability of
$1.8\times 10^{-12}$ produced by chance for a single trial, that converts to
the confidence level of $7.05\sigma$. If considering all the searched
frequencies ($N_{\rm Freq.}=1500\times 4$) and segments ($N_{\rm seg.}=160$),
the total trials are $N=N_{\rm Freq.}\times N_{\rm seg.}=9.6\times 10^{5}$. It
should be noticed that the segments are highly overlapped and not independent.
We conservatively estimated that the confidence level of multiple trials is
$1.8\times 10^{-6}$, i.e., $4.78\sigma$.

In order to search for the burst oscillations more accurately, we applied the
$Z_{1}^{2}$–test statistics, which the $Z_{1}^{2}$ power also satisfies the
$\chi^{2}$ distribution with two degree of freedom, based on Stingray
(Buccheri et al., 1983; Huppenkothen et al., 2019) in the frequency range
579.5–589.5 Hz with steps of 0.01 Hz. The cleaned event files in the 0.2–10
keV energy range are also used by applying a moving window method with the
same window size and step as the FFT method above-mentioned. We also
reproduced the oscillation signal for the burst oscillation around 584.61 Hz,
which the $Z_{1}^{2}$ power peaked at 56.37. It converts to a probability of
$5.7\times 10^{-13}$ produced by random process for a single trial, that is
$7.21\sigma$. If considering the total searched frequencies ($N_{\rm
Freq.}=1000$) and segments (same as the FFT method), the probability of
multiple trials is $9.2\times 10^{-8}$. Again, the segments are treated as
independent, the significance of the burst oscillation is modestly estimated
as $5.34\sigma$.

We note that the obtained peak values of the Leahy and $Z_{1}^{2}$ power
dependent a little bit on the offset time. To find the maximum power, we
reduced the slide window offset to 0.01 s, and kept the remains. We obtained
that the maximum of the Leahy and $Z_{1}^{2}$ power are 56.66 and 60.73,
translating to the probabilities of $7.22\sigma$ and $7.50\sigma$ for a single
trail, respectively.

In the left panel of Fig. 1, we produced the light curve of burst #4, and
showed the contours of $Z_{1}^{2}$ with the power changing from 25 to 55. We
found the upward drifting of the oscillation frequency smaller than 0.3 Hz.

The fractional amplitude rms (root-mean-square) is calculated from the
relation,

$A=\left(\frac{P_{\rm s}}{N_{\rm m}}\right)^{1/2}\frac{N}{N_{m}-N_{\rm bkg}},$
(1)

where $P_{\rm s}$ is the power of the signal, $N_{\rm m}$ and $N_{\rm bkg}$
are the total and background X-ray photons collected during the burst
interval. We took $N_{\rm bkg}$ from the averaged count rate before the burst.
The power $P_{\rm s}$ and its standard deviation were estimate from Bilous &
Watts (2019) based on the measured power, $P_{\rm m}$. We obtained the
fractional amplitude rms is $(12.5\pm 1.8)\%$.

Using the oscillation frequency at the highest $Z_{1}^{2}$ power, in Fig. 2,
we folded the pulse profile of burst #4 in the 0.2–10 keV in 16 phase bins
from the 4 s interval marked by vertical dashed lines in the left panel in
Fig. 1. The sinusoidal model, $A+B\sin(2\pi\nu t-\phi_{0})$, has been applied
to fit the pulse profile, resulting in the best fitted values and 1-sigma
uncertainties, $A=544.0\pm 5.8$, $B=61.1\pm 8.2$ and $\phi_{0}=-(0.56\pm
0.02)\pi$, respectively. The goodness of fit has the minimum $\chi^{2}$ of
14.2 for 13 degree of freedom, i.e., 16 phase bins minus three model
parameters, indicating a good fit.

##  4 Discussion

We detected nine type-I X-ray bursts from 4U 1730–22 by analyzing its 2021 and
2022 NICER observations. For the first time, we found that one burst in 4U
1730–22 appears a strong burst oscillation signal during the cooling tail. The
maximum Leahy and $Z_{1}^{2}$ power are 54.04 and 56.37, respectively,
stronger than the burst oscillation power from 4U 1728–34 (Mahmoodifar et al.,
2019) and XTE J1739–285 (Bult et al., 2021b) observed by NICER. The confidence
levels of the signal are $7.05\sigma$ for a single trail and $4.78\sigma$ for
multiple trails, which means a significant detection. The folded pulsation of
the burst oscillation showed a sinusoidal shape, with a fractional rms
amplitude of $(12.5\pm 1.8)\%$. Both features are typical characteristics
shown in many other NS LMXBs (see e.g., Galloway et al., 2008). The burst
decay oscillation and the fractional rms amplitude shown in 4U 1730–22 can be
explained by the surface mode model, where the oscillation waves are excited
at the burst raise in the NS ocean and spread for a large fraction of cooling
tail, or the asymmetric cooling wake model, where different parts of the NS
surface cool at different rates (see e.g., Heyl, 2004; Watts, 2012;
Mahmoodifar & Strohmayer, 2016; Bhattacharyya, 2022).

The oscillation frequency is around 584.65 Hz, which indicates a fast
rotational neutron star in 4U 1730–22 with the spin period of around 1.71 ms.
Moreover, the oscillation frequency showed small upward drifting, less than
0.3 Hz, during the cooling tail, similar as the behaviours appearing in AMXPs
(see e.g., Chakrabarty et al., 2003; Watts, 2012; Bult et al., 2019). Hence,
we suggest that 4U 1730–22 could be also an AMXP spinning at 1.71 ms. During
the process of this work, 4U 1730–22 is still in outburst, and more data will
be collected. Our observed burst oscillation can be searched and verified from
possible new triggered type-I X-ray bursts from this source in near future.
The spectral studies of all type-I X-ray bursts will be published elsewhere.
With almost known spin period, it’s helpful to perform exhaustive searching
the coherent pulsation from the persistent emissions with reduced parameter
spaces when the outburst stops (e.g., Strohmayer et al., 2018). In addition,
radio observations are highly encouraged to carry out, to search for the
coherent radio pulsation and catch the possible transition from accretion-
powered to rotation-powered states, when the source evolves into the X-ray
quiescent state (see Papitto & de Martino, 2022).

Table 1: Summary of burst observations.  Burst No. | NICER | Burst trigger | ObsID start time | Peak count rate |
---|---|---|---|---|---
| ObsID | (TDB MJD) | (YYYY-MM-DD) | ($10^{3}$ ct s-1) |
1 | 4202200125 | 59404.55780 | 2021-07-09 | 3.1 |
2 | 5202200101 | 59639.33493 | 2022-03-01 | 3.7 |
3 | 5202200112 | 59657.91327 | 2022-03-19 | 6.3 |
4 | 5202200113 | 59658.96139 | 2022-03-20 | 6.3 |
5 | 4639010102 | 59664.12261 | 2022-03-26 | 7.6 |
6 | 4639010104 | 59666.95125 | 2022-03-28 | 3.0 |
7 | 4639010113 | 59675.59719 | 2022-04-06 | 2.3 |
8 | 4639010116 | 59678.77329 | 2022-04-09 | 10.3 |
9 | 4639010131 | 59695.09694 | 2022-04-26 | 7.9 |

ZL and YYP were supported by National Natural Science Foundation of China
(12130342, U1938107). This research has made use of data obtained from the
High Energy Astrophysics Science Archive Research Center (HEASARC), provided
by NASA’s Goddard Space Flight Center.

## References

  * Bhattacharyya (2022) Bhattacharyya, S. 2022, in Astrophysics and Space Science Library, Vol. 465, Astrophysics and Space Science Library, ed. S. Bhattacharyya, A. Papitto, & D. Bhattacharya, 125–155, doi: 10.1007/978-3-030-85198-9_5
  * Bilous & Watts (2019) Bilous, A. V., & Watts, A. L. 2019, ApJS, 245, 19, doi: 10.3847/1538-4365/ab2fe1
  * Buccheri et al. (1983) Buccheri, R., Bennett, K., Bignami, G. F., et al. 1983, A&A, 128, 245
  * Bult et al. (2019) Bult, P., Jaisawal, G. K., Güver, T., et al. 2019, ApJ, 885, L1, doi: 10.3847/2041-8213/ab4ae1
  * Bult et al. (2021a) Bult, P., Altamirano, D., Arzoumanian, Z., et al. 2021a, ApJ, 907, 79, doi: 10.3847/1538-4357/abd54b
  * Bult et al. (2021b) —. 2021b, ApJ, 920, 59, doi: 10.3847/1538-4357/ac18c4
  * Bult et al. (2021c) Bult, P. M., Jaisawal, G. K., Jaodand, A., et al. 2021c, The Astronomer’s Telegram, 14769, 1
  * Chakrabarty et al. (2003) Chakrabarty, D., Morgan, E. H., Muno, M. P., et al. 2003, Nature, 424, 42, doi: 10.1038/nature01732
  * Chang et al. (2006) Chang, P., Morsink, S., Bildsten, L., & Wasserman, I. 2006, ApJ, 636, L117, doi: 10.1086/499428
  * Chen et al. (1997) Chen, W., Shrader, C. R., & Livio, M. 1997, ApJ, 491, 312, doi: 10.1086/304921
  * Cominsky et al. (1978) Cominsky, L., Jones, C., Forman, W., & Tananbaum, H. 1978, ApJ, 224, 46, doi: 10.1086/156348
  * Forman et al. (1978) Forman, W., Jones, C., Cominsky, L., et al. 1978, ApJS, 38, 357, doi: 10.1086/190561
  * Galloway & Keek (2021) Galloway, D. K., & Keek, L. 2021, Astrophysics and Space Science Library, 461, 209, doi: 10.1007/978-3-662-62110-3_5
  * Galloway et al. (2008) Galloway, D. K., Muno, M. P., Hartman, J. M., Psaltis, D., & Chakrabarty, D. 2008, ApJS, 179, 360, doi: 10.1086/592044
  * Gendreau & Arzoumanian (2017) Gendreau, K., & Arzoumanian, Z. 2017, Nature Astronomy, 1, 895
  * Gendreau et al. (2012) Gendreau, K. C., Arzoumanian, Z., & Okajima, T. 2012, in Proc. SPIE, Vol. 8443, Space Telescopes and Instrumentation 2012: Ultraviolet to Gamma Ray, 844313, doi: 10.1117/12.926396
  * Güver et al. (2022) Güver, T., Boztepe, T., Ballantyne, D. R., et al. 2022, MNRAS, 510, 1577, doi: 10.1093/mnras/stab3422
  * Haensel et al. (2009) Haensel, P., Zdunik, J. L., Bejger, M., & Lattimer, J. M. 2009, A&A, 502, 605, doi: 10.1051/0004-6361/200811605
  * Heyl (2004) Heyl, J. S. 2004, ApJ, 600, 939, doi: 10.1086/379966
  * Huppenkothen et al. (2019) Huppenkothen, D., Bachetti, M., Stevens, A. L., et al. 2019, ApJ, 881, 39, doi: 10.3847/1538-4357/ab258d
  * Iwakiri et al. (2021) Iwakiri, W., Negoro, H., Nakajima, M., et al. 2021, The Astronomer’s Telegram, 14757, 1
  * Kaaret et al. (2007) Kaaret, P., Prieskorn, Z., in ’t Zand, J. J. M., et al. 2007, ApJ, 657, L97, doi: 10.1086/513270
  * Kennea et al. (2021a) Kennea, J. A., Bahramian, A., Evans, P. A., et al. 2021a, The Astronomer’s Telegram, 14686, 1
  * Kennea et al. (2021b) Kennea, J. A., Evans, P. A., Bahramian, A., et al. 2021b, The Astronomer’s Telegram, 14688, 1
  * Kobayashi et al. (2021) Kobayashi, K., Negoro, H., Serino, M., et al. 2021, The Astronomer’s Telegram, 14683, 1
  * Leahy et al. (1983) Leahy, D. A., Darbro, W., Elsner, R. F., et al. 1983, ApJ, 266, 160, doi: 10.1086/160766
  * Lewin et al. (1993) Lewin, W. H. G., van Paradijs, J., & Taam, R. E. 1993, Space Sci. Rev., 62, 223, doi: 10.1007/BF00196124
  * Li et al. (2021) Li, Z., Pan, Y., & Falanga, M. 2021, ApJ, 920, 35, doi: 10.3847/1538-4357/ac1f15
  * Liu et al. (2006) Liu, Q. Z., van Paradijs, J., & van den Heuvel, E. P. J. 2006, A&A, 455, 1165, doi: 10.1051/0004-6361:20064987
  * Liu et al. (2007) —. 2007, A&A, 469, 807, doi: 10.1051/0004-6361:20077303
  * Mahmoodifar & Strohmayer (2016) Mahmoodifar, S., & Strohmayer, T. 2016, ApJ, 818, 93, doi: 10.3847/0004-637X/818/1/93
  * Mahmoodifar et al. (2019) Mahmoodifar, S., Strohmayer, T. E., Bult, P., et al. 2019, ApJ, 878, 145, doi: 10.3847/1538-4357/ab20c4
  * Manchester et al. (2005) Manchester, R. N., Hobbs, G. B., Teoh, A., & Hobbs, M. 2005, AJ, 129, 1993, doi: 10.1086/428488
  * NASA High Energy Astrophysics Science Archive Research Center (2014) (Heasarc) NASA High Energy Astrophysics Science Archive Research Center (Heasarc). 2014, HEAsoft: Unified Release of FTOOLS and XANADU.  http://ascl.net/1408.004
  * Papitto & de Martino (2022) Papitto, A., & de Martino, D. 2022, in Astrophysics and Space Science Library, Vol. 465, Astrophysics and Space Science Library, ed. S. Bhattacharyya, A. Papitto, & D. Bhattacharya, 157–200, doi: 10.1007/978-3-030-85198-9_6
  * Patruno & Watts (2021) Patruno, A., & Watts, A. L. 2021, in Astrophysics and Space Science Library, Vol. 461, Astrophysics and Space Science Library, ed. T. M. Belloni, M. Méndez, & C. Zhang, 143–208, doi: 10.1007/978-3-662-62110-3_4
  * Russell et al. (2021) Russell, D. M., Baglio, M. C., Saikia, P., Abdelmaguid, M., & Lewis, F. 2021, The Astronomer’s Telegram, 14693, 1
  * Strader et al. (2021) Strader, J., Sokolovsky, K., Aydi, E., et al. 2021, The Astronomer’s Telegram, 14694, 1
  * Strohmayer et al. (1996) Strohmayer, T. E., Zhang, W., Swank, J. H., et al. 1996, ApJ, 469, L9, doi: 10.1086/310261
  * Strohmayer et al. (2018) Strohmayer, T. E., Arzoumanian, Z., Bogdanov, S., et al. 2018, ApJ, 858, L13, doi: 10.3847/2041-8213/aabf44
  * Suleimanov et al. (2020) Suleimanov, V. F., Poutanen, J., & Werner, K. 2020, A&A, 639, A33, doi: 10.1051/0004-6361/202037502
  * Tanaka & Shibazaki (1996) Tanaka, Y., & Shibazaki, N. 1996, ARA&A, 34, 607, doi: 10.1146/annurev.astro.34.1.607
  * Tomsick et al. (2007) Tomsick, J. A., Gelino, D. M., & Kaaret, P. 2007, ApJ, 663, 461, doi: 10.1086/518239
  * Walter et al. (2015) Walter, R., Lutovinov, A. A., Bozzo, E., & Tsygankov, S. S. 2015, A&A Rev., 23, 2, doi: 10.1007/s00159-015-0082-6
  * Watts (2012) Watts, A. L. 2012, ARA&A, 50, 609, doi: 10.1146/annurev-astro-040312-132617
  * Zhao et al. (2022) Zhao, G., Li, Z., Pan, Y., et al. 2022, A&A, 660, A31, doi: 10.1051/0004-6361/202142801