# From Descriptive Richness to Bias: Unveiling the Dark Side of
Generative Image Caption Enrichment

Yusuke Hirota1 Work partially done as an intern at NVIDIA Research.  Ryo
Hachiuma2 Chao-Han Huck Yang2 Yuta Nakashima1
Osaka University1 NVIDIA Research2
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Large language models (LLMs) have enhanced the capacity of vision-language
models to caption visual text. This generative approach to image caption
enrichment further makes textual captions more descriptive, improving
alignment with the visual context. However, while many studies focus on
benefits of generative caption enrichment (GCE), are there any negative side
effects? We compare standard-format captions and recent GCE processes from the
perspectives of “gender bias” and “hallucination”, showing that enriched
captions suffer from increased gender bias and hallucination. Furthermore,
models trained on these enriched captions amplify gender bias by an average of
$30.9$% and increase hallucination by $59.5$%. This study serves as a caution
against the trend of making captions more descriptive.

From Descriptive Richness to Bias: Unveiling the Dark Side of
Generative Image Caption Enrichment

##  1 Introduction

Figure 1: Left: an overview of our analysis. Although the “LLM-enriched”
caption (ShareGPT4V) covers more content than standard COCO (objects described
in captions are bolded), it exhibits hallucination (in yellow) and gender
bias, including describing gender not exist in the image and possible gender-
stereotypical sentence (in purple). Right: a comparison between standard and
enriched captions on caption quality, bias, and hallucination.

Large vision-language models (VLMs), such as BLIP Li et al. (2023a), with
superior performance in multi-modal understanding Lüddecke and Ecker (2022);
Tewel et al. (2022), benefiting from millions of image-caption pairs.
Improving training paradigms Wang et al. (2023); Liu et al. (2024) and data
augmentation strategies Rotstein et al. (2024); Li et al. (2024) are crucial
topics to enhance VLM performance in image captioning.

Among these techniques, Generative language models based Caption Enrichment
(GCE) methods Chen et al. (2023); Chan et al. (2023) have achieved some of the
latest state-of-the-art performances. Unlike standard caption benchmarks,
which concisely describe the salient parts Misra et al. (2016) of an image
(_e.g_., COCO captions Chen et al. (2015)), GCE methods create more
descriptive and semantically enhanced captions. These enhanced textual
captions are aligned to boost downstream performance with large language
models (LLMs).

While many studies emphasize improving caption quality, issues such as
societal bias and hallucination are significant yet often overlooked Zhou et
al. (2023); Zhao et al. (2021); Wang and Russakovsky (2021); Hirota et al.
(2023); Burns et al. (2018) in image captioning. For example, Zhao et al.
(2021) found that the COCO dataset is skewed towards men, and Hirota et al.
(2022) showed that models trained on this biased data generate gender-
stereotypical captions (_e.g_., describing a pink dress for women not wearing
one). These studies have highlighted potential biases in datasets like COCO
and the models trained on them. Addressing this bias is crucial as it can
exacerbate unfairness and risks towards underrepresented groups.

We aim to examine one critical question that has been overlooked in GCE works:
“Although LLM-enriched captions boost VLM performance, do they have negative
effects, regarding societal bias and object hallucination?” To answer this, we
investigate gender bias and hallucination using comprehensive metrics,
examining both datasets and models trained on these datasets for standard
captions (COCO captions) and enriched captions (ShareGPT4V Chen et al. (2023),
FuseCap Rotstein et al. (2024), CapsFusion Yu et al. (2024)).

Our analysis reveals that LLM-enriched captions indeed have negative side
effects, worsening issues of gender bias and hallucination by making captions
more descriptive. Meanwhile, models trained on these enriched captions tend to
amplify these problems further. Finally, we discuss possible causes of these
negative effects and warn against the trend of making captions more
descriptive.

##  2 Evaluation Framework

###  2.1 Generative Caption Enrichment (GCE)

We introduce recent representative approaches to generate enriched captions:
ShareGPT4V Chen et al. (2023), FuseCap Rotstein et al. (2024), and Capsfusion
Yu et al. (2024). All these GCE methods utilize LLMs Brown et al. (2020) or
Large Multi-modal Models (LMMs) OpenAI (2023) to describe images in detail or
summarize different sources of the information.

ShareGPT4V utilizes GPT4-Vision OpenAI (2023) to generate $1.2$M large scale
high-quality captions for incremental training on a strong 7B VLM with strong
generalization and SOTA results.

FuseCap uses several pre-trained off-the-shelf vision models (_e.g_., object
detector) to extract diverse visual information. The outputs from these models
and original captions are fused using ChatGPT Ouyang et al. (2022) to generate
enriched captions.

Capsfusion generates captions using a pre-trained captioner, BLIP Li et al.
(2023a), then fuses them with original captions using ChatGPT.

Table 1: Caption quality, gender bias, and hallucination for upstream and downstream analysis. Red/green indicates the worst/best score for each metric. Recall, gender bias, and hallucination metrics are scaled by 100. | Caption Quality $\uparrow$ |  | Gender bias $\downarrow$ |  | Hallucination $\downarrow$
---|---|---|---|---|---
Captions | Diversity | Length | Recall |  | Gender Err. | LIC | Recall Disp. |  | CHAIRs | CHAIRi
Upstream |  |  |  |  |  |  |  |  |  |
COCO captions | 12,834 | 11.3 | 42.6 |  | 0 | 0 | 7.0 |  | 0 | 0
ShareGPT4V | 25,349 | 166.1 | 61.7 |  | 2.5 | 17.4 | 24.9 |  | 20.7 | 5.7
FuseCap | 25,892 | 39.8 | 59.4 |  | 3.2 | 14.3 | 9.9 |  | 9.2 | 4.0
CapsFusion | 13,158 | 16.9 | 44.6 |  | 1.4 | 1.2 | 7.6 |  | 3.5 | 2.2
Downstream |  |  |  |  |  |  |  |  |  |
COCO captions | 3,312 | 10.9 | 45.7 |  | 3.1 | 5.5 | 7.8 |  | 4.7 | 3.1
ShareGPT4V | 9,573 | 153.8 | 56.3 |  | 3.4 | 14.3 | 30.5 |  | 21.5 | 6.9
FuseCap | 6,341 | 42.0 | 56.9 |  | 4.8 | 17.3 | 16.3 |  | 13.2 | 6.3
CapsFusion | 3,385 | 15.3 | 48.0 |  | 4.2 | 6.3 | 8.8 |  | 7.2 | 4.4

###  2.2 Evaluation metrics

Our analysis focuses on caption quality, societal bias, and hallucination. Let
$\mathcal{D}$ be a dataset of $n$ samples,
$\mathcal{D}=\\{(I_{i},c_{i},a_{i})\mid 1\leq i\leq n\\}$, where each sample
includes an image $I_{i}$, a caption $c_{i}$, and an optional binary gender
label $a_{i}$ (woman or man). We introduce the metrics to evaluate each
aspect.

Caption quality. We evaluate caption quality in three aspects: Vocabulary
diversity is the total number of unique words across all captions in
$\mathcal{D}$. Caption length is the average number of tokens per caption in
$\mathcal{D}$. Recall measures the proportion of objects mentioned in captions
to the total objects in the images. For each caption $c_{i}\in\mathcal{D}$,
recall is calculated as:

$\text{Recall}=\frac{1}{n}\sum_{i=1}^{n}\frac{r_{i}}{o_{i}},$ (1)

where $o_{i}$ is the total number of objects in $I_{i}$, and $r_{i}$ is the
number of relevant objects mentioned in $c_{i}$. Note that conventional
reference-based metrics like CIDEr Vedantam et al. (2015) cannot be applied to
descriptive captions Chan et al. (2023).

Societal bias. We focus on gender bias as gender terms are more frequently
described in captions than other attributes. We adopt three metrics to measure
gender bias: Gender error Burns et al. (2018) measures the rate of incorrect
gender predictions in captions. If a caption $c_{i}\in\mathcal{D}$ with gender
label $a_{i}$ refers to a woman as a man or vice versa, it counts as an error.
The gender error is the proportion of such errors in $\mathcal{D}$. Recall
disparity Hall et al. (2023) evaluates the recall disparity between genders.
Consider two subsets based on $a_{i}$: $\mathcal{D}_{\text{woman}}$ and
$\mathcal{D}_{\text{man}}$. Recall disparity is the average absolute
difference in recall for each object $j$:

$\text{Disparity}=\frac{1}{m}\sum_{j=1}^{m}|\text{Recall}_{\text{man},j}-\text{Recall}_{\text{woman},j}|$

(2)

where $m$ is the total number of COCO objects Lin et al. (2014),
$\text{Recall}_{\text{man},j}$ is the recall of COCO objects $j$ in
$\mathcal{D}_{\text{man}}$, and vice versa. LIC Hirota et al. (2022)
quantifies how gender-stereotypical captions in $\mathcal{D}$ are compared to
human-written captions. It compares the accuracies of two gender classifiers:
one trained on $c_{i}\in\mathcal{D}$ and the other on ground-truth captions.
Higher accuracy for the classifier trained on $c_{i}$ indicates more gender-
stereotypical information in these captions.

Hallucination. We use the CHAIR metric Rohrbach et al. (2018) to evaluate
hallucination in captions. CHAIR has two components: $\text{CHAIR}_{i}$ is the
fraction of mentioned objects in the captions $c_{i}$ that do not appear in
images $I_{i}$:

$\text{CHAIR}_{\text{i}}=\frac{H}{M},$ (3)

where $H$ is the number of hallucinated objects, and $M$ is the total number
of objects mentioned in the captions. $\text{CHAIR}_{s}$ is the fraction of
captions $c_{i}$ with at least one hallucinated object:

$\text{CHAIR}_{\text{s}}=\frac{S_{h}}{n},$ (4)

where $S_{h}$ is the number of captions with hallucinated objects. We focus on
$80$ objects in COCO.

##  3 Evaluation

Figure 2: LIC vs. Recall (left: upstream, right: downstream). The bubble size indicates vocabulary size. LIC tends to increase with higher recall, shown by strong trends (dotted lines) with $R^{2}=0.99$ (left) and $R^{2}=0.97$ (right). Figure 3: $\text{CHAIR}_{\text{s}}$ vs. Recall (left: upstream, right: downstream). The bubble size indicates vocabulary size. $\text{CHAIR}_{\text{s}}$ tends to increase with higher recall, shown by strong trends with $R^{2}=0.80$ (left) and $R^{2}=0.76$ (right). Table 2: Difference in gender bias and hallucination between upstream and downstream (downstream - upstream). Red/green is bias amplification/mitigation. |  $\Delta$Gender bias |  |  $\Delta$Hallucination
---|---|---|---
Captions | Err. | LIC | Disp. |  | Cs | Ci
COCO cap. | 3.1 | 5.5 | 0.8 |  | 4.7 | 3.1
ShareGPT4V | 0.9 | -3.1 | 5.6 |  | 0.8 | 1.2
FuseCap | 1.6 | 3.0 | 6.4 |  | 4.0 | 2.3
CapsFusion | 2.8 | 5.1 | 1.2 |  | 3.7 | 2.2

#### Setup.

We analyze concise (COCO captions) and enriched captions (ShareGPT4V, FuseCap,
CapsFusion) based on the metrics in Section 2.2. Enriched captions are
generated for the COCO training set using these approaches. We first conduct
an upstream analysis of the four datasets and then a downstream analysis of
captions generated by a captioner trained on each dataset. For downstream
analysis, we fine-tune a pre-trained BLIP for 5 epochs with the AdamW
optimizer, generating captions for the COCO validation set. Detailed
experimental settings are in Appendix A.

###  3.1 Upstream & downstream analysis

Figure 4: Recall disparity by visual object.

We present qualitative results in Figure 1 and Appendix B, with key
observations below.

Observation 1. More descriptive, more gender bias. Table 1 (upstream) shows a
clear tendency for gender bias to increase as captions become more
descriptive. For instance, COCO captions have the lowest object coverage
(_i.e_., recall: 42.6) but exhibit the least bias. In contrast, ShareGPT4V and
FuseCap have higher object coverage but higher gender bias than COCO captions
(_e.g_., LIC is 0 for COCO and 17.4 for ShareGPT4V). This observation is
further confirmed by Figure 2 (left), showing a strong correlation between LIC
and recall ($R^{2}=$ 0.99). In other words, making captions more descriptive
increases the risk of gender bias.

Observation 2. Enriched captions exhibit greater recall disparity. In Figure
4, we visualize the difference in recall
($\text{Recall}_{\text{man}}-\text{Recall}_{\text{woman}}$) for the top-$10$
objects that co-occur with images in $\mathcal{D}_{\text{woman}}$ and
$\mathcal{D}_{\text{man}}$. The results show that ShareGPT4V exhibits a more
significant recall disparity for all objects. For example, for the handbag
object, COCO captions show almost no gender difference, while ShareGPT4V
exhibits a strong bias towards men. This further validates the risk of gender
bias in enriched captions.

Observation 3. More descriptive, more hallucination. A similar trend between
descriptiveness and hallucination is also evident in Table 1 (upstream). COCO
captions, which has the lowest object coverage, exhibits the lowest
hallucination rates. Conversely, ShareGPT4V, with the highest object coverage,
shows significantly increased hallucination rates compared to COCO captions
(_e.g_., $\text{CHAIR}_{s}$ is 0 for COCO and 20.7 for ShareGPT4V). This trend
is corroborated by Figure 3 (left), highlighting a strong correlation between
hallucination rates and recall ($R^{2}=$ 0.80). Thus, making captions more
descriptive increases hallucination risks.

Observation 4. Models trained on the datasets inherit/amplify bias and
hallucination. Table 1 (downstream) shows that models inherit the dataset’s
bias tendencies. Specifically, the model trained on the least descriptive
captions (_i.e_., COCO captions) exhibits the smallest bias and hallucination,
while models trained on the most descriptive captions, ShareGPT4V and FuseCap,
show significant bias and hallucination. Figures 2 and 3 (right) further
demonstrate that the models inherit the datasets’ bias and hallucination.
Furthermore, Table 2 shows that in most cases, the models amplify the
dataset’s biases. For example, ShareGPT4V’s recall disparity worsens from
$24.9$ to $30.5$ ($\Delta=5.6$), and $\text{CHAIR}_{s}$ increases from $20.7$
to $21.5$ ($\Delta=0.8$). These results highlight the severe issue of dataset
bias, as it directly affects the outcomes of the trained models.

##  4 Discussion on Possible Sources of Bias

To enhance descriptiveness, GEC methods heavily rely on LLMs to improve
textual alignment. However, issues with gender bias and hallucination Gunjal
et al. (2024) have been explored in these LLMs. The enrichment process, which
depends on text representations, risks incorporating these inherent biases
into the final captions. Furthermore, the lack of human oversight in the
caption generation process can exacerbate these issues. Without iterative
human intervention to correct biases, the inaccuracies of LLMs remain
unaddressed, leading to increased bias and hallucination. Introducing human-
in-the-loop Yang et al. (2019) could mitigate these problems by ensuring that
captions are free from gender-stereotypical descriptions.

##  5 Conclusion

We examined standard and LLM-enriched captions for gender bias and
hallucination, deriving key insights: GCE-based image captioning exacerbates
these bias, which are further amplified in downstream models. We argue that
further efforts must be invested to the problems to strike a balance between
descriptive richness and incremental bias.

## Limitations

#### Attributes other than gender.

We focused our analysis on gender bias for societal bias. This is because
gender-related terms are more frequently described in captions compared to
other attributes, making gender bias particularly prominent in captions Hirota
et al. (2022). However, previous works have shown that racial bias, though not
as pronounced as gender bias, is also present in captioning models Zhao et al.
(2021). Analyzing racial bias and bias of other attributes requires future
studies and efforts.

#### Evaluation metrics.

While our analysis demonstrated various critical problems in enriched captions
(_e.g_., they exacerbate bias and hallucination), there may be aspects that we
can further investigate. For example, we can consider other attributes for
societal bias analysis and utilize hallucination metrics that account for
elements beyond objects. However, our analysis is robust and highlights
critical issues in enriched captions, serving as a counterpoint to the trend
of making captions more descriptive and benefiting the community.

#### Source datasets other than COCO.

In our analysis, we used COCO as the source for images for two reasons: (1)
COCO images come with high-quality, human-annotated concise captions,
providing a solid basis for evaluating concise captions; and (2) COCO has been
extensively analyzed in existing research for societal bias and hallucination
Li et al. (2023b). We did not use other image-caption datasets (_e.g_., Google
Conceptual Captions Sharma et al. (2018), LAION Schuhmann et al. (2022))
because the quality of the accompanying captions is lower, making the analysis
results less reliable.

## Acknowledgments

We thank David Chan from UC Berkeley and Yen-Ting Lin from National Taiwan
University for their invaluable contributions during the initial discussions
that enhanced this project.

## References

  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  In _NeurIPS_.
  * Burns et al. (2018) Kaylee Burns, Lisa Anne Hendricks, Kate Saenko, Trevor Darrell, and Anna Rohrbach. 2018.  Women also snowboard: Overcoming bias in captioning models.  In _ECCV_.
  * Chan et al. (2023) David Chan, Austin Myers, Sudheendra Vijayanarasimhan, David A Ross, and John Canny. 2023.  Ic3: Image captioning by committee consensus.  In _EMNLP_.
  * Chen et al. (2023) Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023.  Sharegpt4v: Improving large multi-modal models with better captions.  _arXiv preprint arXiv:2311.12793_.
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_.
  * Devlin et al. (2015) Jacob Devlin, Hao Cheng, Hao Fang, Saurabh Gupta, Li Deng, Xiaodong He, Geoffrey Zweig, and Margaret Mitchell. 2015.  Language models for image captioning: The quirks and what works.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_. Association for Computational Linguistics.
  * Gunjal et al. (2024) Anisha Gunjal, Jihan Yin, and Erhan Bas. 2024.  Detecting and preventing hallucinations in large vision language models.  In _AAAI_.
  * Hall et al. (2023) Melissa Hall, Laura Gustafson, Aaron Adcock, Ishan Misra, and Candace Ross. 2023.  Vision-language models performing zero-shot tasks exhibit gender-based disparities.  In _ICCV Workshops_.
  * Hirota et al. (2022) Yusuke Hirota, Yuta Nakashima, and Noa Garcia. 2022.  Quantifying societal bias amplification in image captioning.  In _CVPR_.
  * Hirota et al. (2023) Yusuke Hirota, Yuta Nakashima, and Noa Garcia. 2023.  Model-agnostic gender debiased image captioning.  In _CVPR_.
  * Li et al. (2024) Jiaxuan Li, Duc Minh Vo, Akihiro Sugimoto, and Hideki Nakayama. 2024.  Evcap: Retrieval-augmented image captioning with external visual-name memory for open-world comprehension.  In _CVPR_.
  * Li et al. (2023a) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_.
  * Li et al. (2023b) Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023b.  Halueval: A large-scale hallucination evaluation benchmark for large language models.  In _EMNLP_.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft COCO: Common objects in context.  In _ECCV_.
  * Liu et al. (2024) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024.  Improved baselines with visual instruction tuning.  In _CVPR_.
  * Louizos et al. (2017) Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. 2017.  Causal effect inference with deep latent-variable models.  _Advances in neural information processing systems_ , 30.
  * Lüddecke and Ecker (2022) Timo Lüddecke and Alexander Ecker. 2022.  Image segmentation using text and image prompts.  In _CVPR_.
  * Misra et al. (2016) Ishan Misra, C Lawrence Zitnick, Margaret Mitchell, and Ross Girshick. 2016.  Seeing through the human reporting bias: Visual classifiers from noisy human-centric labels.  In _CVPR_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4v(ision) system card.  In _OpenAI Blog_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  In _NeurIPS_.
  * Rohrbach et al. (2018) Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018.  Object hallucination in image captioning.  In _EMNLP_.
  * Rotstein et al. (2024) Noam Rotstein, David Bensaïd, Shaked Brody, Roy Ganz, and Ron Kimmel. 2024.  Fusecap: Leveraging large language models for enriched fused image captions.  In _WACV_.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022.  Laion-5b: An open large-scale dataset for training next generation image-text models.  In _NeurIPS_.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _ACL_.
  * Tewel et al. (2022) Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. 2022.  Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic.  In _CVPR_.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation.  In _CVPR_.
  * Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015.  Show and tell: A neural image caption generator.  In _CVPR_.
  * Wang and Russakovsky (2021) Angelina Wang and Olga Russakovsky. 2021.  Directional bias amplification.  In _ICML_.
  * Wang et al. (2023) Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. 2023.  Visionllm: Large language model is also an open-ended decoder for vision-centric tasks.  In _NeurIPS_.
  * Yang et al. (2022) Chao-Han Huck Yang, I-Te Danny Hung, Yi Ouyang, and Pin-Yu Chen. 2022.  Training a resilient q-network against observational interference.  In _AAAI_.
  * Yang et al. (2019) Yiwei Yang, Eser Kandogan, Yunyao Li, Prithviraj Sen, and Walter S Lasecki. 2019.  A study on interaction in human-in-the-loop machine learning for text analytics.  In _IUI Workshops_.
  * Yu et al. (2024) Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Xinlong Wang, and Jingjing Liu. 2024.  Capsfusion: Rethinking image-text data at scale.  In _CVPR_.
  * Zhao et al. (2021) Dora Zhao, Angelina Wang, and Olga Russakovsky. 2021.  Understanding and evaluating racial biases in image captioning.  In _ICCV_.
  * Zhou et al. (2023) Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2023.  Analyzing and mitigating object hallucination in large vision-language models.  In _ICLR_.

##  Appendix A Appendix Settings of the Experiments

###  A.1 Upstream analysis

For the upstream analysis, we use images of the COCO training set as an image
source. For every image, we apply GCE approaches introduced in Section 2.1,
obtaining enriched captions for each approach. We compare these enriched
captions with concise captions (COCO captions) in terms of the metrics
introduced in Section 2.2.

###  A.2 Downstream analysis

For the downstream analysis, we fine-tune pre-trained BLIP on one of the four
datasets (_i.e_., COCO captions, ShareGPT4V, FuseCap, or CapsFusion). We fine-
tune the models for 5 epochs with the AdamW optimizer. Then, the fine-tuned
models are used to generate captions for the COCO validation set. The
generated captions for the validation set are used for the downstream
analysis.

Figure 5: Qualitative examples of the comparison between COCO captions and
ShareGPT4V. Objects described in captions are bolded. Gender bias and
hallucination are highlighted in purple and yellow, respectively. Figure 6:
Qualitative examples of the comparison between COCO captions and FuseCap.
Objects described in captions are bolded. Gender bias and hallucination are
highlighted in purple and yellow, respectively. Figure 7: Qualitative examples
of the comparison between COCO captions and CapsFusion. Objects described in
captions are bolded. Gender bias and hallucination are highlighted in purple
and yellow, respectively.

##  Appendix B Additional Qualitative Examples

We show some qualitative examples where enriched captions generated by GCE
methods exhibit gender bias and hallucination. The examples are shown in
Figures 5, 6 and 7 (ShareGPT4V, FuseCap, and CapsFusion, respectively).

The enriched captions suffer from gender misclassification (_e.g_., bottom of
Figure 5), incorrectly describing two people in the image as a couple (_e.g_.,
top of Figure 7), describing nonexistent individuals with different genders
(_e.g_., bottom of Figure 6), and object hallucination (in all the figures).
These results further confirm the negative impacts of GCE.

###  B.1 Non-LLM based Caption Enrichment

We also would like to credit previous works Devlin et al. (2015); Vinyals et
al. (2015) of non pre-training based language modeling to enhance image
captioning by providing structured linguistic patterns and vocabulary.
However, without the depth of large language models, such systems may exhibit
bias and limited expressiveness, struggling to generate diverse and
contextually nuanced captions. These models often rely on statistical
techniques, which can constrain their descriptive capabilities compared to
their more advanced counterparts. In other words, how to incorporate structure
knowledge refinement Louizos et al. (2017); Yang et al. (2022) or graphical
structure would also be important for LLM-based caption enrichment in future
studies.


# The Cauchy problem for the energy-critical inhomogeneous nonlinear
Schrödinger equation with inverse-square potential

RoeSong Jang, JinMyong An, JinMyong Kim∗
Faculty of Mathematics, Kim Il Sung University, Pyongyang, Democratic People’s
Republic of Korea
∗ Corresponding Author
Email address<EMAIL_ADDRESS>
###### Abstract

In this paper, we study the Cauchy problem for the energy-critical
inhomogeneous nonlinear Schrödinger equation with inverse-square potential

$iu_{t}+\Delta u-c|x|^{-2}u=\lambda|x|^{-b}|u|^{\sigma}u,\;u(0)=u_{0}\in
H^{1},\;(t,x)\in\mathbb{R}\times\mathbb{R}^{d},$

where $d\geq 3$, $\lambda=\pm 1$, $0<b<2$, $\sigma=\frac{4-2b}{d-2}$ and
$c>-c(d):=-\left(\frac{d-2}{2}\right)^{2}$. We first prove the local well-
posedness as well as small data global well-posedness and scattering in
$H^{1}$ for $c>-\frac{(d+2-2b)^{2}-4}{(d+2-2b)^{2}}c(d)$ and
$0<b<\frac{4}{d}$, by using the contraction mapping principle based on the
Strichartz estimates. Based on the local well-posedness result, we then
establish the blowup criteria for solutions to the equation in the focusing
case $\lambda=-1$. To this end, we derive the sharp Hardy-Sobolev inequality
and virial estimates related to this equation.

Keywords: Inhomogeneous nonlinear Schrödinger equation; Inverse-square
potential; Energy-critical; Well-posedness; Blowup; Hardy-Sobolev inequality;
Virial estimates

MR(2020) Subject Classification: 35Q55, 35A01, 35B44

##  1 Introduction

In this paper, we consider the Cauchy problem for the inhomogeneous nonlinear
Schrödinger equation with inverse-square potential, denoted by INLSc equation,

$\left\\{\begin{array}[]{l}{iu_{t}-P_{c}u=\lambda|x|^{-b}|u|^{\sigma}u,\;(t,x)\in\mathbb{R}\times\mathbb{R}^{d},}\\\
{u\left(0,\;x\right)=u_{0}\left(x\right),}\end{array}\right.$ (1.1)

where $d\geq 3$, $u:\;\mathbb{R}\times\mathbb{R}^{d}\to\mathbb{C}$,
$u_{0}:\;\mathbb{R}^{d}\to\mathbb{C}$, $b,\;\sigma>0$, $\lambda=\pm 1$ and
$P_{c}=-\Delta u+c|x|^{-2}$ with $c>-c(d):=-\left(\frac{d-2}{2}\right)^{2}$.
$\lambda=-1$ corresponds to the focusing case and $\lambda=1$ corresponds to
the defocusing case. The restriction on $c$ comes from the sharp Hardy
inequality:

$\frac{(d-2)^{2}}{4}\int_{\mathbb{R}^{d}}{|x|^{-2}\left|u(x)\right|^{2}dx}\leq\int_{\mathbb{R}^{d}}{\left|\nabla
u(x)\right|dx},\leavevmode\nobreak\ \forall u\in H^{1}(\mathbb{R}^{d}),$ (1.2)

which ensures that $P_{c}$ is a positive operator. The INLSc equation appears
in a variety of physical settings, for example, in nonlinear optical systems
with spatially dependent interactions (see e.g. [3] and the references
therein). In particular, when $c=0$, it can be thought of as modeling
inhomogeneities in the medium in which the wave propagates (see e.g. [17]).
When $b=0$, the equation (1.1) also appears in various areas of physics, for
instance in quantum field equations, or in the study of certain black hole
solutions of the Einstein equations (see e.g. [5, 15]).

The case $b=c=0$ is the classic nonlinear Schrödinger (NLS) equation which has
been been widely studied over the last three decades (see e.g. [8, 21, 25] and
the references therein). The case $b=0$ and $c\neq 0$ is known as the NLS
equation with inverse-square potential, denoted by NLSc equation, which has
also been extensively studied in recent years (see e.g. [10, 18, 20, 23, 26]
and the references therein). Moreover, when $c=0$ and $b\neq 0$, we have the
inhomogeneous nonlinear Schrödinger (INLS) equation, which has also attracted
a lot of interest in recent years (see e.g. [1, 2, 6, 9, 13] and the
references therein).

On the other hand, the inhomogeneous nonlinear Schrödinger with potential in
the following form:

$\left\\{\begin{array}[]{l}{iu_{t}+\Delta
u-Vu=\lambda|x|^{-b}|u|^{\sigma}u,\;(t,x)\in\mathbb{R}\times\mathbb{R}^{d},}\\\
{u\left(0,\;x\right)=u_{0}\left(x\right).}\end{array}\right.$ (1.3)

has also been studied by several authors in recent years. For example, Dinh
[11] studied the well-posedness, scattering and blowup for (1.3) when $d=3$,
$b>0$, $\lambda=\pm 1$ and $V$ is a real-valued potential satisfying $V\in
K_{0}\cap L^{\frac{3}{2}}$ and $\left\|V_{-}\right\|<4\pi$, where
$V_{-}:=\min\left\\{V,\;0\right\\}$ and $K_{0}$ is defined as the closure of
bounded and compactly supported functions with respect to the Kato norm

$\left\|V\right\|_{K}:=\sup_{x\in\mathbb{R}^{3}}\int_{\mathbb{R}^{3}}{\frac{|V(y)|}{|x-y|}dy.}$

Luo [22] also studied the stability and multiplicity of standing waves for
(1.3) with $V=|x|^{2}$ (harmonic potential), $\lambda=-1$ and $b<0$. The case
$V(x)=c|x|^{-2}$ with $c>-c(d)$ (inverse-square potential) and $b>0$ was
considered by [7, 24].

In this paper, we are interested in (1.3) with $V(x)=c|x|^{-2}$ with $c>-c(d)$
and $b>0$, i.e. we study the INLSc equation (1.1).

Before recalling the known results for the INLSc equation (1.1) and stating
our main results, let us give some information about this equation. The INLSc
equation (1.1) is invariant under the scaling,

$u_{\lambda}(t,x):=\lambda^{\frac{2-b}{\sigma}}u\left(\lambda^{2}t,\lambda
x\right),\leavevmode\nobreak\ \lambda>0.$

An easy computation shows that

$\left\|u_{\lambda}(0)\right\|_{\dot{H}^{s}}=\lambda^{s-\frac{d}{2}+\frac{2-b}{\sigma}}\left\|u_{0}\right\|_{\dot{H}^{s}},$

which implies that the critical Sobolev index is given by

$s_{c}=\frac{d}{2}-\frac{2-b}{\sigma}.$ (1.4)

Note that, if $s_{c}=0$ (alternatively
$\sigma=\sigma_{\star}:=\frac{4-2b}{d}$) the problem is known as the mass-
critical or $L^{2}$-critical; if $s_{c}=1$ (alternatively
$\sigma=\sigma^{\star}:=\frac{4-2b}{d-2}$) it is called energy-critical or
$\dot{H}^{1}$-critical. The problem is known as intercritical (mass-
supercritical and energy-subcritical) if $0<s_{c}<1$ (alternatively
$\sigma_{\star}<\sigma<\sigma^{\star}$). On the other hand, solutions to the
INLSc equation (1.1) conserve the mass and energy, defined respectively by

$M\left(u(t)\right):=\int_{\mathbb{R}^{d}}{\left|u(t,x)\right|^{2}dx},$ (1.5)
$E_{b,c}\left(u(t)\right):=\int_{\mathbb{R}^{d}}{\frac{1}{2}\left|\nabla
u(t,x)\right|^{2}+\frac{c}{2}|x|^{-2}\left|u(t,x)\right|^{2}+\frac{\lambda}{\sigma+2}|x|^{-b}\left|u(t,x)\right|^{\sigma+2}dx}.$
(1.6)

Let us recall the known results for the INLSc equation (1.1). Using the energy
method, Suzuki [24] showed that if 111 Note that the author in [24] considered
(1.1) with $c=-c(d)$. The authors in [7] pointed out that the proof for the
case $c>-c(d)$ is an immediate consequence of the previous one. $d\geq 3$,
$0<\sigma<\sigma^{\star}$, $c>-c(d)$ and $0<b<2$, then the INLSc equation
(1.1) is locally well-posed in $H^{1}_{c}(\mathbb{R}^{d})$ (which is
equivalent to $H^{1}(\mathbb{R}^{d}))$. It was also proved that any local
solution of (1.1) with $u_{0}\in H^{1}_{c}(\mathbb{R}^{d})$ extends globally
in time if either $\lambda=1$ (defocusing case) or $0<\sigma<\sigma_{\star}$
for $\lambda=-1$ (focusing, mass-subcritical case). Recently, Campos-Guzmán
[7] established the sufficient conditions for global existence and blowup in
$H^{1}_{c}(\mathbb{R}^{d})$ for $d\geq 3$, $\lambda=-1$ and
$\sigma_{\star}\leq\sigma<\sigma^{\star}$, using a Gagliardo-Nirenberg-type
estimate. They also studied the local well-posedness and small data global
well-posedness under some assumption on $b$ and $c$ in the energy-subcritical
case $\sigma<\sigma^{\star}$ with $d\geq 3$ by using the standard Strichartz
estimates combined with the fixed point argument. Furthermore, they showed a
scattering criterion and construct a wave operator in
$H^{1}_{c}(\mathbb{R}^{d})$, for the intercritical case. As mentioned above,
the authors in [7, 24] studied the local and global well-posedness as well as
blowup and scattering in $H^{1}_{c}(\mathbb{R}^{d})$ with $d\geq 3$ for the
INLSc equation (1.1) in the energy–subcritical case
$\sigma<\sigma^{\star}\left(=\frac{4-2b}{d-2}\right)$.

In this paper, we study the well-posedness and blowup in
$H^{1}(\mathbb{R}^{d})$ with $d\geq 3$ for the INLSc equation (1.1) in the
energy-critical case $\sigma=\sigma^{\star}=\frac{4-2b}{d-2}$.

First, we prove the local well-posedness as well as small data global well-
posedness and scattering by using the contraction mapping principle based on
Strichartz estimates.

######  Theorem 1.1.

Let $d\geq 3$, $0<b<\frac{4}{d}$, $\sigma=\frac{4-2b}{d-2}$ and
$c>-\frac{(d+2-2b)^{2}-4}{(d+2-2b)^{2}}c(d)$. If $u_{0}\in
H^{1}(\mathbb{R}^{d})$, then there exists $T=T(u_{0})>0$ such that (1.1) has a
unique solution

$u\in
L^{\gamma(r)}\left(\left[-T,\;T\right],\;H^{1,r}(\mathbb{R}^{d})\right),$
(1.7)

where $\left(\gamma(r),\;r\right)$ is an admissible pair satisfying

$r=\frac{2d(d+2-2b)}{d^{2}-2db+4}.$ (1.8)

Moreover, for any admissible pair $\left(\gamma\left(p\right),\;p\right)$, we
have

$u\in
L^{\gamma\left(p\right)}\left(\left[-T,\;T\right],\;H^{1,p}(\mathbb{R}^{d})\right).$
(1.9)

If $\left\|u_{0}\right\|_{\dot{H}^{1}(\mathbb{R}^{d})}$ is sufficiently small,
then the above solution is global and scatters.

######  Remark 1.2.

Theorem 1.1 can be seen as the extension of the well-posedness result of NLSc
equation (see Proposition 3.3 of [10]) to the INLSc equation.

######  Remark 1.3.

In Theorem 1.1, the restriction $b<\frac{4}{d}$ comes from the fractional
Hardy inequality (Lemma 3.1). And the restriction
$c>-\frac{(d+2-2b)^{2}-4}{(d+2-2b)^{2}}c(d)$ comes from the equivalence of
Sobolev spaces $\dot{H}^{1,r}_{c}\sim\dot{H}^{1,r}$.

Based on the local well-posedness result above, we study the blowup phenomena
for the focusing, energy–critical INLSc equation.

Let $0<b<2$, $c>-c(d)$, and let $C_{HS}(b,c)$ be the sharp constant in the
Hardy-Sobolev inequality related to the focusing, energy–critical INLSc
equation (1.1), namely,

$C_{HS}(b,c)=\inf_{f\in\dot{H}^{1}_{c}\setminus\left\\{0\right\\}}{\frac{\left\|f\right\|_{\dot{H}^{1}_{c}}}{\left\||x|^{-b}|f|^{\sigma^{\star}+2}\right\|^{\frac{1}{\sigma^{\star}+2}}_{L^{1}}}}.$

We will see in Lemma 4.1 that:

  1. 1.

When $-c(d)<c\leq 0$, the sharp constant $C_{HS}(b,c)$ is attained by the
function

$W_{b,c}(x):=\frac{\left[\varepsilon(d-b)(d-2)\beta^{2}\right]^{\frac{d-2}{4-2b}}}{\left[\varepsilon+|x|^{(2-b)\beta}\right]^{\frac{d-2}{2-b}}|x|^{\rho}},$
(1.10)

with $\beta=1-\frac{2\rho}{d-2}$, for all $\varepsilon>0$ (see (2.2) for the
definition of $\rho$).

  2. 2.

If $c>0$, $C_{HS}(b,c)\leq C_{HS}(b,0)$.

We have the following blowup result for the focusing, energy-critical INLSc
equation.

######  Theorem 1.4.

Let $d\geq 3$, $0<b<\frac{4}{d}$, $\lambda=-1$,
$c>-\frac{(d+2-2b)^{2}-4}{(d+2-2b)^{2}}c(d)$ and $\sigma=\frac{4-2b}{d-2}$.
Let $u_{0}\in H^{1}(\mathbb{R}^{d})$ and $u$ be the corresponding solution to
(1.1). Suppose that either $E_{b,c}(u_{0})<0$, or if $E_{b,c}(u_{0})\geq 0$,
we assume that $E_{b,c}(u_{0})<E_{b,\bar{c}}(W_{b,\bar{c}})$ and
$\left\|u_{0}\right\|_{\dot{H}^{1}_{c}}>\left\|W_{b,\bar{c}}\right\|_{\dot{H}^{1}_{\bar{c}}}$,
where $\bar{c}=\min\left\\{c,\;0\right\\}$. If $xu_{0}\in L^{2}$ or $u_{0}$ is
radial, then the solution $u$ blows up in finite time.

######  Remark 1.5.

  1. 1.

In Theorem 1.4, the restrictions on $b$ and $c$ only come from the local well-
posedness result (Theorem 1.1). If we can prove the local existence of
solution for the wider range of $b$ and $c$, the result of Theorem 1.4 still
holds.

  2. 2.

Theorem 1.4 can be seen as the extension of the blowup result of NLSc equation
(see Theorem 1.12 of [10]) to the INLSc equation.

This paper is organized as follows. In Section 2, we recall some useful facts
which are used in this paper. In Section 3, we prove Theorem 1.1. In Section
4, we derive the sharp Hardy-Sobolev inequality and virial estimates related
to INLSc equation to prove Theorem 1.4.

##  2 Preliminaries

Let us introduce the notation used throughout the paper. As usual, we use
$\mathbb{C}$, $\mathbb{R}$ and $\mathbb{N}$ to stand for the sets of complex,
real and natural numbers, respectively. $C>0$ will denote positive universal
constant, which can be different at different places. $a\lesssim b$ means
$a\leq Cb$ for some constant $C>0$. We also write $a\sim b$ if $a\lesssim
b\lesssim a$. We denote by $p^{\prime}$ the dual number of
$p\in\left[1,\;\infty\right]$, i.e. $1/p+1/p^{\prime}=1$. As in [25], for
$s\in\mathbb{R}$ and $1<p<\infty$, we denote by $H^{s,p}(\mathbb{R}^{d})$ and
$\dot{H}^{s,p}(\mathbb{R}^{d})$ the usual nonhomogeneous and homogeneous
Sobolev spaces associated to the Laplacian $-\Delta$. As usual, we abbreviate
$H^{s,2}(\mathbb{R}^{d})$ and $\dot{H}^{s,2}(\mathbb{R}^{d})$ as
$H^{s}(\mathbb{R}^{d})$ and $\dot{H}^{s}(\mathbb{R}^{d})$, respectively.
Similarly, we define Sobolev spaces in terms of $P_{c}$ via

$\left\|f\right\|_{\dot{H}_{c}^{s,p}(\mathbb{R}^{d})}=\left\|(P_{c})^{\frac{s}{2}}f\right\|_{L^{r}(\mathbb{R}^{d})},\leavevmode\nobreak\
\left\|f\right\|_{H_{c}^{s,p}(\mathbb{R}^{d})}=\left\|(1+P_{c})^{\frac{s}{2}}f\right\|_{L^{r}(\mathbb{R}^{d})}.$

We also abbreviate
$\dot{H}_{c}^{s}(\mathbb{R}^{d})=\dot{H}_{c}^{s,2}(\mathbb{R}^{d})$ and
$H_{c}^{s}(\mathbb{R}^{d})=H_{c}^{s,2}(\mathbb{R}^{d})$. Note that by sharp
Hardy inequality (1.2), we see that

$\left\|f\right\|_{\dot{H}_{c}^{1}(\mathbb{R}^{d})}\sim\left\|f\right\|_{\dot{H}^{1}(\mathbb{R}^{d})}\leavevmode\nobreak\
\textrm{for}\leavevmode\nobreak\ c>-c(d).$ (2.1)

For $I\subset\mathbb{R}$ and $\gamma\in\left[1,\;\infty\right]$, we will use
the space-time mixed space
$L^{\gamma}\left(I,\;X\left(\mathbb{R},^{d}\right)\right)$ whose norm is
defined by

$\left\|f\right\|_{L^{\gamma}\left(I,\;X(\mathbb{R}^{d})\right)}=\left(\int_{I}\left\|f\right\|_{X(\mathbb{R}^{d})}^{\gamma}dt\right)^{\frac{1}{\gamma}},$

with a usual modification when $\gamma=\infty$, where $X(\mathbb{R}^{d})$ is a
normed space on $\mathbb{R}^{d}$. Given normed spaces $X$ and $Y$, $X\subset
Y$ means that $X$ is continuously embedded in $Y$, i.e. there exists a
constant $C\left(>0\right)$ such that $\left\|f\right\|_{Y}\leq
C\left\|f\right\|_{X}$ for all $f\in X$. If there is no confusion,
$\mathbb{R}^{d}$ will be omitted in various function spaces.

Next, we recall the equivalence between the usual Sobolev space defined by
$-\Delta$ and the one defined by $P_{c}$. For convenience, we define the
following number:

$\rho:=\frac{d-2}{2}-\sqrt{\left(\frac{d-2}{2}\right)^{2}+c}.$ (2.2)

######  Lemma 2.1 (Equivalence of Sobolev spaces, [19]).

Let $d\geq 3$, $c>-c(d)$ and $0<s<2$.

  1. 1.

If $1<p<\infty$ satisfies
$\frac{s+\rho}{d}<\frac{1}{p}<\min\left\\{1,\frac{d-\rho}{d}\right\\}$, then
$\left\|f\right\|_{\dot{H}^{s,p}}\lesssim\left\|f\right\|_{\dot{H}_{c}^{s,p}}$
for all $f\in C^{\infty}_{0}(\mathbb{R}^{d}\setminus\left\\{0\right\\})$.

  2. 2.

If $1<p<\infty$ satisfies
$\max\left\\{\frac{s}{d},\;\frac{\rho}{d}\right\\}<\frac{1}{p}<\min\left\\{1,\frac{d-\rho}{d}\right\\}$,
then
$\left\|f\right\|_{\dot{H}^{s,p}_{c}}\lesssim\left\|f\right\|_{\dot{H}^{s,p}}$
for all $f\in C^{\infty}_{0}(\mathbb{R}^{d}\setminus\left\\{0\right\\})$.

######  Remark 2.2.

Let $0<s<2$.

  1. 1.

When $c>0$, $\left\|f\right\|_{\dot{H}^{s,p}_{c}}$ is equivalent to
$\left\|f\right\|_{\dot{H}^{s,p}}$, provided that $1<p<\frac{d}{s}$.

  2. 2.

When $-c(d)\leq c<0$, $\left\|f\right\|_{\dot{H}^{s,p}_{c}}$ is equivalent to
$\left\|f\right\|_{\dot{H}^{s,p}}$, provided that
$\frac{d}{d-\rho}<p<\frac{d}{s+\rho}$.

We end this section by recalling the Strichartz estimates for the INLSc
equation (1.1).

######  Definition 2.3.

Let $d\geq 3$. We say that a pair $(\gamma(p),p)$ is admissible, if

$2\leq p\leq\frac{2d}{d-2},\leavevmode\nobreak\
\frac{2}{\gamma(p)}=\frac{d}{2}-\frac{d}{p}.$ (2.3)

######  Lemma 2.4 (Strichartz estimates, [4, 5]).

Let $d\geq 3$ and $c>-c(d)$. Then for any $s\in\mathbb{R}$ and any admissible
pairs $(\gamma(p),p)$, $(\gamma(r),r)$, we have

$\left\|e^{-itP_{c}}f\right\|_{L^{\gamma(p)}(\mathbb{R},\;\dot{H}_{c}^{s,p})}\lesssim\left\|f\right\|_{\dot{H}_{c}^{s}},$
(2.4)
$\left\|\int_{0}^{t}e^{-i(t-\tau)P_{c}}f(\tau)d\tau\right\|_{L^{\gamma(p)}(\mathbb{R},\;\dot{H}_{c}^{s,p})}\lesssim\left\|f\right\|_{L^{\gamma(r)^{\prime}}(\mathbb{R},\;\dot{H}_{c}^{s,r^{\prime}})}.$
(2.5)

##  3 Local and global well-posedness

In this section, we prove Theorem 1.1. To establish the nonlinear estimates,
we recall the following fractional Hardy inequality which is a direct
consequence of Theorem 3.1 of [14].

######  Lemma 3.1 (Fractional Hardy Inequality).

Let $1<p<\infty$ and $0<s<\frac{d}{p}$. Then we have

$\left\||x|^{-s}f\right\|_{L^{p}(\mathbb{R}^{d})}\lesssim\left\|f\right\|_{\dot{H}^{s,p}(\mathbb{R}^{d})}.$

Using Lemma 3.1, we have the following nonlinear estimates.

######  Lemma 3.2.

Let $\bar{r}=\frac{2d}{d-2}$, $r=\frac{2d(d+2-2b)}{d^{2}-2db+4}$,
$0<b<\frac{4}{d}$ and $\sigma=\frac{4-2b}{d-2}$. Then we have

$\left\||x|^{-b}|u|^{\sigma}u\right\|_{\dot{H}^{1,\bar{r}^{\prime}}}\lesssim\left\|u\right\|_{\dot{H}^{1,r}}^{\sigma+1},$
(3.1)
$\left\||x|^{-b}|u|^{\sigma}v\right\|_{\dot{H}^{1,\bar{r}^{\prime}}}\lesssim\left\|u\right\|_{\dot{H}^{1,r}}^{\sigma}\left\|v\right\|_{L^{r}}.$
(3.2)

###### Proof.

Noticing that

$\left|\nabla\left(|x|^{-b}|u|^{\sigma}u\right)\right|\lesssim|x|^{-b-1}|u|^{\sigma+1}+|x|^{-b}|u|^{\sigma}|\nabla
u|,$ (3.3)

we have

$\displaystyle\begin{split}\left\||x|^{-b}|u|^{\sigma}u\right\|_{\dot{H}_{\bar{r}^{\prime}}^{1}}=\left\|\nabla\left(|x|^{-b}|u|^{\sigma}u\right)\right\|_{\bar{r}^{\prime}}\lesssim\left\||x|^{-b-1}|u|^{\sigma+1}\right\|_{\bar{r}^{\prime}}+\left\||x|^{-b}|u|^{\sigma}\nabla
u\right\|_{\bar{r}^{\prime}}.\end{split}$ (3.4)

First we estimate
$\left\||x|^{-b-1}|u|^{\sigma+1}\right\|_{\bar{r}^{\prime}}$. We can see that

$\frac{1}{\bar{r}^{\prime}}=\left(\sigma+1\right)\left(\frac{1}{r}-\frac{1}{d}\left(1-\frac{b+1}{\sigma+1}\right)\right).$

Putting

$\frac{1}{\rho}:=\frac{1}{r}-\frac{1}{d}\left(1-\frac{b+1}{\sigma+1}\right),$

we have $\dot{H}^{1-\frac{b+1}{\sigma+1},r}\subset L^{\rho}$. Here, we use the
fact $1-\frac{b+1}{\sigma+1}>0\Leftrightarrow b<\frac{4}{d}$. Using Lemma 3.1,
we have

$\left\||x|^{-b-1}|u|^{\sigma+1}\right\|_{\bar{r}^{\prime}}=\left\||x|^{-\frac{b+1}{\sigma+1}}u\right\|_{\rho}^{\sigma+1}\lesssim\left\||x|^{-\frac{b+1}{\sigma+1}}u\right\|_{\dot{H}^{1-\frac{b+1}{\sigma+1},r}}^{\sigma+1}\lesssim\left\|u\right\|_{\dot{H}^{1,r}}^{\sigma+1}.$
(3.5)

Next we estimate $\left\||x|^{-b}|u|^{\sigma}\nabla
u\right\|_{\bar{r}^{\prime}}$. We get

$\sigma\left(\frac{1}{r}-\frac{1}{d}\left(1-\frac{b}{\sigma}\right)\right)+\frac{1}{r}=\frac{1}{r^{\prime}}.$

Putting

$\frac{1}{\gamma}:=\frac{1}{r}-\frac{1}{d}\left(1-\frac{b}{\sigma}\right),$

and noticing $1-\frac{b}{\sigma}>0$, we have
$\dot{H}^{1-\frac{b}{\sigma},r}\subset L^{\gamma}$. Hence it follows from
Hölder inequality and Lemma 3.1 that

$\left\||x|^{-b}|u|^{\sigma}\nabla
u\right\|_{\bar{r}^{\prime}}\leq\left\||x|^{-\frac{b}{\sigma}}u\right\|_{\gamma}^{\sigma}\left\|\nabla
u\right\|_{r}\lesssim\left\||x|^{-\frac{b}{\sigma}}u\right\|_{\dot{H}^{1-\frac{b}{\sigma},r}}^{\sigma}\left\|u\right\|_{\dot{H}_{r}^{1}}\lesssim\left\|u\right\|_{\dot{H}^{1,r}}^{\sigma+1}.$
(3.6)

In view of (3.4)–(3.6), we immediately have (3.1). Similarly we also have

$\left\||x|^{-b}|u|^{\sigma}v\right\|_{L^{r^{\prime}}}\lesssim\left\||x|^{-\frac{b}{\sigma}}u\right\|_{\gamma}^{\sigma}\left\|v\right\|_{r}\leq\left\||x|^{-\frac{b}{\sigma}}u\right\|_{\dot{H}^{1-\frac{b}{\sigma},r}}^{\sigma}\left\|v\right\|_{r}\lesssim\left\|u\right\|_{\dot{H}^{1,r}}^{\sigma}\left\|v\right\|_{r},$

this concludes the proof. ∎

###### Proof of Theorem 1.1.

We can easily see that $(\gamma(r),r)$ is admissible, where $r$ is given in
(1.8). Furthermore, using Remark 2.2, we can easily verify that
$\dot{H}^{1,r}_{c}$ is equivalent to $\dot{H}^{1,r}$ provided that
$c>-\frac{(d+2-2b)^{2}-4}{(d+2-2b)^{2}}c(d)$. Putting
$\bar{r}=\frac{2d}{d-2}$, we can also see that
$\dot{H}^{1,\bar{r}^{\prime}}_{c}\sim\dot{H}^{1,\bar{r}^{\prime}}$. Noticing

$\frac{1}{\gamma\left(\bar{r}\right)^{\prime}}=\frac{\sigma+1}{\gamma(r)},$
(3.7)

and using Lemma 3.2, Hölder inequality, we immediately have

$\
\left\||x|^{-b}|u|^{\sigma}u\right\|_{L^{\gamma\left(\bar{r}\right)^{{}^{\prime}}}(I,\;\dot{H}^{1,\bar{r}^{\prime}})}\lesssim\left\|u\right\|_{L^{\gamma(r)}(I,\;\dot{H}^{1,r})}^{\sigma+1},$
(3.8)

where $I\subset\mathbb{R}$ is an interval. Using (3.7), Lemma 3.2 and Hölder
inequality, we also have

$\left\||x|^{-b}|u|^{\sigma}u\right\|_{L^{\gamma\left(\bar{r}\right)^{{}^{\prime}}}(I,\;L^{\bar{r}^{\prime}})}\lesssim\left\|u\right\|_{L^{\gamma(r)}(I,\;\dot{H}^{1,r})}^{\sigma}\left\|u\right\|_{L^{\gamma(r)}(I,\;L^{r})}.$
(3.9)

In view of (3.8) and (3.9), we have

$\left\||x|^{-b}|u|^{\sigma}u\right\|_{L^{\gamma\left(\bar{r}\right)^{{}^{\prime}}}(I,\;H^{1,\bar{r}^{\prime}})}\lesssim\left\|u\right\|_{L^{\gamma(r)}(I,\;\dot{H}^{1,r})}^{\sigma}\left\|u\right\|_{L^{\gamma(r)}\left(I,\;H^{1,r}\right)}.$
(3.10)

On the other hand, noticing that

$\left||x|^{-b}|u|^{\sigma}u-|x|^{-b}\left|v\right|^{\sigma}v\right|\lesssim|x|^{-b}(|u|^{\sigma}+|v|^{\sigma})|u-v|,$

using Lemma 3.2, we have

$\displaystyle\begin{split}\left\||x|^{-b}|u|^{\sigma}u-|x|^{-b}\left|v\right|^{\sigma}v\right\|_{L^{\bar{r}^{\prime}}}&\lesssim\left\||x|^{-b}\left(|u|^{\sigma}+\left|v\right|^{\sigma}\right)\left(u-v\right)\right\|_{L^{\bar{r}^{\prime}}}\\\
&\lesssim\left(\left\|u\right\|_{\dot{H}^{1,r}}^{\sigma}+\left\|v\right\|_{\dot{H}^{1,r}}^{\sigma}\right)\left\|u-v\right\|_{L^{r}}.\end{split}$
(3.11)

Using (3.7), (3.11) and Hölder inequality, we immediately have

$\displaystyle\begin{split}&\left\||x|^{-b}|u|^{\sigma}u-|x|^{-b}\left|v\right|^{\sigma}v\right\|_{L^{\gamma\left(\bar{r}\right)^{{}^{\prime}}}(I,\;L^{\bar{r}^{\prime}})}\\\
&\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\
\lesssim\left(\left\|u\right\|_{L^{\gamma(r)}(I,\;\dot{H}^{1,r})}^{\sigma}+\left\|v\right\|_{L^{\gamma(r)}(I,\;\dot{H}^{1,r})}^{\sigma}\right)\left\|u-v\right\|_{L^{\gamma(r)}(I,\;L^{r})}.\end{split}$
(3.12)

First, we prove the local well-posedness. Let $T>0$ and $M>0$ which will be
chosen later. We define the following complete metric space

$D=\left\\{u\in
L^{\gamma(r)}\left(I,\;H^{1,r}\right):\;\left\|u\right\|_{L^{\gamma(r)}(I,\;H^{1,r})}\leq
M\right\\},$

which is equipped with the metric

$d\left(u,\;v\right)=\left\|u-v\right\|_{L^{\gamma\left(r\right)}(I,\;L^{r})},$

where $I=[-T,\;T]$. We consider the mapping

$T:\;u(t)\to
e^{-itP_{c}}u_{0}+\int_{0}^{t}e^{-i(t-\tau)P_{c}}|x|^{-b}\left|u(\tau)\right|^{\sigma}u(\tau)d\tau=:u_{L}+u_{NL}.$

Lemma 2.4 (Strichartz estimates) yields that

$\left\|u_{L}\right\|_{L^{\gamma(r)}\left(I,\;H^{1,r}\right)}\sim\left\|u_{L}\right\|_{L^{\gamma(r)}(I,\;H^{1,r}_{c})}\lesssim\left\|u_{0}\right\|_{H^{1}_{c}}\sim\left\|u_{0}\right\|_{H^{1}},$
(3.13)
$\left\|u_{NL}\right\|_{L^{\gamma(r)}\left(I,\;H^{1,r}\right)}\sim\left\|u_{NL}\right\|_{L^{\gamma(r)}(I,\;H^{1,r}_{c})}\lesssim\left\||x|^{-b}|u|^{\sigma}u\right\|_{L^{\gamma\left(\bar{r}\right)^{{}^{\prime}}}(I,\;H^{1,\bar{r}^{\prime}}_{c})},$
(3.14) $\left\|Tu-
Tv\right\|_{L^{\gamma(r)}(I,\;L^{r})}\lesssim\left\||x|^{-b}|u|^{\sigma}u-|x|^{-b}\left|v\right|^{\sigma}v\right\|_{L^{\gamma\left(\bar{r}\right)^{{}^{\prime}}}\left(I,\;L^{\bar{r}^{\prime},2}\right)}.$
(3.15)

In view of (3.13), we can see that
$\left\|u_{L}\right\|_{L^{\gamma(r)}\left(\left[-T,\;T\right],\;H^{1,r}\right)}\to
0$, as $T\to 0$. Take $M>0$ such that $CM^{\sigma}\leq\frac{1}{4}$ and $T>0$
such that

$\left\|u_{L}\right\|_{L^{\gamma(r)}\left(\left[-T,\;T\right],\;H^{1,r}\right)}\leq\frac{M}{2}.$
(3.16)

Using (3.10), (3.14), (3.16), and the fact
$\dot{H}^{1,\bar{r}^{\prime}}_{c}\sim\dot{H}^{1,\bar{r}^{\prime}}$, we have

$\left\|Tu\right\|_{L^{\gamma(r)}\left(I,\;H^{1,r}\right)}\leq\frac{M}{2}+C\left\|u\right\|_{L^{\gamma(r)}\left(I,\;H^{1,r}\right)}^{\sigma+1}\leq
M.$ (3.17)

In view of (LABEL:GrindEQ__3_12_) and (3.15), we have

$\left\|Tu-Tv\right\|_{L^{\gamma(r)}\left(I,\;L^{r}\right)}\leq
2CM^{\sigma}\left\|u-v\right\|_{L^{\gamma\left(r\right)}(I,\;L^{r})}\leq\frac{1}{2}\left\|u-v\right\|_{L^{\gamma(r)}(I,\;L^{r})}.$
(3.18)

(3.17) and (3.18) imply that $T:(D,d)\to(D,d)$ is a contraction mapping. From
Banach fixed point theorem, there exists a unique solution $u$ of (1.1) in
$(D,d)$. Furthermore for any admissible pair
$\left(\gamma\left(p\right),\;p\right)$, it follows from Lemma 2.4 (Strichartz
estimates) and (3.10) that

$\left\|u\right\|_{L^{\gamma\left(p\right)}\left(I,\;H^{1,p}\right)}\lesssim\left\|u_{0}\right\|_{H^{1}}+\left\|u\right\|_{L^{\gamma\left(r\right)}\left(I,\;H^{1,r}\right)}^{\sigma+1},$

which implies $u\in L^{\gamma\left(p\right)}\left(I,\;H^{1,p}\right)$. This
completes the proof of the local well-posedness.
Next we prove the global well-posedness with small initial data. We define the
following complete metric space

$E=\left\\{u\in
L^{\gamma(r)}\left(\mathbb{R},,\;H^{1,r}\right):\;\left\|u\right\|_{L^{\gamma(r)}\left(\mathbb{R},,\;\dot{H}^{1,r}\right)}\leq
m,\;\left\|u\right\|_{L^{\gamma(r)}(\mathbb{R},\;H^{1,r})}\leq M\right\\},$

which is equipped with the metric

$d\left(u,\;v\right)=\left\|u-v\right\|_{L^{\gamma\left(r\right)}(\mathbb{R},\;L^{r})}.$

Using Lemma 2.4 (Strichartz estimates) and (3.8), it follows from the facts
$\dot{H}^{1,r}_{c}\sim\dot{H}^{1,r}$ and
$\dot{H}^{1,\bar{r}^{\prime}}_{c}\sim\dot{H}^{1,\bar{r}^{\prime}}$ that

$\left\|Tu\right\|_{L^{\gamma(r)}\left(\mathbb{R},,\;\dot{H}^{1,r}\right)}\leq
C\left\|u_{0}\right\|_{\dot{H}^{1}}+C\left\|u\right\|_{L^{\gamma(r)}(\mathbb{R},\;\dot{H}^{1,r})}^{\sigma+1}.$
(3.19)

Similarly, using Lemma 2.4 (Strichartz estimates), (3.10) and
(LABEL:GrindEQ__3_12_), we also have

$\left\|Tu\right\|_{L^{\gamma(r)}\left(\mathbb{R},,\;H^{1,r}\right)}\leq
C\left\|u_{0}\right\|_{H^{1}}+C\left\|u\right\|_{L^{\gamma(r)}(\mathbb{R},\;\dot{H}^{1,r})}^{\sigma}\left\|u\right\|_{L^{\gamma(r)}\left(\mathbb{R},,\;H^{1,r}\right)},$
(3.20) $\left\|Tu-
Tv\right\|_{L^{\gamma(r)}\left(\mathbb{R},,\;L^{r}\right)}\leq
C\left(\left\|u\right\|_{L^{\gamma(r)}(\mathbb{R},\;\dot{H}^{1,r})}^{\sigma}+\left\|v\right\|_{L^{\gamma\left(r\right)}(\mathbb{R},\;\dot{H}^{1,r})}^{\sigma}\right)\left\|u-v\right\|_{L^{\gamma(r)}\left(\mathbb{R},,\;L^{r}\right)}.$
(3.21)

Put $m=2C\left\|u_{0}\right\|_{\dot{H}^{1}}$,
$M=2C\left\|u_{0}\right\|_{H^{1}}$ and
$\delta=2\left(4C\right)^{-\frac{\sigma+1}{\sigma}}$. If
$\left\|u_{0}\right\|_{\dot{H}^{1}}\leq\delta$, i.e.
$Cm^{\sigma}<\frac{1}{4}$, then it follows from (3.19)–(3.21) that

$\left\|Tu\right\|_{L^{\gamma(r)}\left(\mathbb{R},,\;\dot{H}^{1,r}\right)}\leq
m,$ $\;\left\|Tu\right\|_{L^{\gamma(r)}\left(\mathbb{R},,\;H^{1,r}\right)}\leq
M,$ $\left\|Tu-
Tv\right\|_{L^{\gamma(r)}\left(\mathbb{R},,\;L^{r}\right)}\leq\frac{1}{2}\left\|u-v\right\|_{L^{\gamma(r)}\left(\mathbb{R},,\;L^{r}\right)}.$

So $T:(E,d)\to(E,d)$ is a contraction mapping and there exists a unique
solution $u$ in $E$. The scattering result with small initial data can be
proved using the standard argument and we omit the details. This concludes the
proof. ∎

##  4 Blowup

In this section, we prove Theorem 1.4. To arrive at this goal, we derive the
sharp Hardy-Sobolev inequality as well as the standard virial identity and
localized virial estimate related to the focusing, energy-critical INLSc
equation.

###  4.1 Sharp Hardy-Sobolev inequality

In this subsection, we consider the sharp Hardy-Sobolev inequality related to
the focusing, energy-critical INLSc equation:

$\left(\int_{\mathbb{R}^{d}}{|x|^{-b}\left|f\right|^{\sigma^{\star}+2}dx}\right)^{\frac{1}{\sigma^{\star}+2}}\leq
C_{HS}(b,c)\left\|f\right\|_{\dot{H}^{1}_{c}},$ (4.1)

where the sharp constant $C_{HS}(b,c)$ is defined by

$C_{HS}(b,c)=\inf_{f\in\dot{H}^{1}_{c}\setminus\left\\{0\right\\}}{\frac{\left\|f\right\|_{\dot{H}^{1}_{c}}}{\left(\int{|x|^{-b}\left|f\right|^{\sigma^{\star}+2}dx}\right)^{\frac{1}{\sigma^{\star}+2}}}}.$
(4.2)

######  Lemma 4.1 (Sharp Hardy-Sobolev inequality).

Let $d\geq 3$, $0<b<2$ and $c>-c(d)$.

  1. 1.

If $-c(d)<c\leq 0$, then the equality in (4.1) is attained by function
$W_{b,c}(x)$ given in (1.10).

  2. 2.

If $c>0$, then $C_{HS}(b,c)\leq C_{HS}(b,0)$.

###### Proof.

The proof of Item 1 can be found in [16]. Using the fact $c>0$, we immediately
have that $\left\|f\right\|_{\dot{H}^{1}}<\left\|f\right\|_{\dot{H}^{1}_{c}}$
for any $f\in\dot{H}^{1}\setminus\left\\{0\right\\}$. Hence it follows from
Item 1 that

$\left(\int{|x|^{-b}\left|f\right|^{\sigma^{\star}+2}dx}\right)^{\frac{1}{\sigma^{\star}+2}}\leq
C_{HS}(b,0)\left\|f\right\|_{\dot{H}^{1}}<C_{HS}(b,0)\left\|f\right\|_{\dot{H}^{1}_{c}},$

which implies that $C_{HS}(b,c)\leq C_{HS}(b,0)$. This completes the proof. ∎

######  Remark 4.2.

When $b=0$ and $c>0$, it is known that $C_{HS}(b,c)=C_{HS}(b,0)$ and the
equality in (4.1) is never attained. In fact, $C_{HS}(0,c)\geq C_{HS}(0,0)$
can be proved by considering $f_{n}(x)=W_{0,c}(x-x_{n})$ for any sequence
$x_{n}\rightarrow\infty$. See [10, 18] for details. But when $b>0$, we could
not apply this argument and we don’t know whether $C_{HS}(b,c)=C_{HS}(b,0)$.

Next, we recall some properties related to $W_{b,c}$. Lemma 2.2 of [16] also
shows that $W_{b,c}$ with $-c(d)<c\leq 0$ solves the equation

$P_{c}W_{b,c}=|x|^{-b}\left|W_{b,c}\right|^{\sigma^{\star}}W_{b,c},$

and satisfies

$\left\|W_{b,c}\right\|_{\dot{H}^{1}_{c}}^{2}=\int{|x|^{-b}W_{b,c}^{\sigma^{\star}+2}dx}.$
(4.3)

Hence, we have for $-c(d)<c\leq 0$,

$\left\|W_{b,c}\right\|_{\dot{H}^{1}_{c}}^{2}=\int{|x|^{-b}W_{b,c}^{\sigma^{\star}+2}dx}=C_{HS}(b,c)^{-\frac{2(d-b)}{2-b}},\leavevmode\nobreak\
\left\|W_{b,c}\right\|_{\dot{H}^{1}_{c}}^{\sigma^{\star}}=C_{HS}(b,c)^{-(\sigma^{\star}+2)},$
(4.4)
$E_{b,c}\left(W_{b,c}\right)=\frac{1}{2}\left\|W_{b,c}\right\|_{\dot{H}^{1}_{c}}^{2}-\frac{1}{\sigma^{\star}+2}\int{|x|^{-b}W_{b,c}^{\sigma^{\star}+2}dx}=\frac{2-b}{2(d-b)}C_{HS}(b,c)^{-\frac{2(d-b)}{2-b}}.$
(4.5)

Moreover, for any $c>-c(d)$, we have

$C_{HS}(b,c)\leq
C_{HS}(b,\bar{c})=\left\|W_{b,\bar{c}}\right\|_{\dot{H}^{1}_{\bar{c}}}^{-\frac{2-b}{d-b}}=\left\||x|^{-b}W_{b,\bar{c}}^{\sigma^{\star}+2}\right\|_{L^{1}}^{-\frac{2-b}{2(d-b)}}=\left[\frac{2(d-b)}{2-b}E_{b,\bar{c}}\left(W_{b,\bar{c}}\right)\right]^{-\frac{2-b}{2(d-b)}}.$
(4.6)

###  4.2 Virial estimates

In this subsection, we derive the standard virial identity and localized
virial estimate related to the focusing INLSc equation. Given a real valued
function $a$, we define the virial potential by

$V_{a}(t):=\int{a(x)\left|u(t,x)\right|^{2}dx.}$

A simple computation shows that the following result holds.

######  Lemma 4.3 ([10]).

Let $d\geq 3$ and $c>-c(d)$. If $u:I\times\mathbb{R}^{d}\rightarrow\mathbb{C}$
is a smooth-in-time and Schwartz-in-space solution to $iu_{t}-P_{c}u=N(u)$,
with $N(u)$ satisfying $\textnormal{Im}(N(u)\bar{u})=0$, then we have for any
$t\in I$,

$\frac{d}{dt}V_{a}(t)=2\int{\nabla a(x)\cdot\textnormal{Im}(\bar{u}(t,x)\nabla
u(t,x))dx},$

and

$\displaystyle\begin{split}\frac{d^{2}}{dt^{2}}V_{a}(t)&=-\int{\Delta^{2}a(x)|u(t,x)|^{2}dx+4\sum_{j,k=1}^{d}{\int{\partial_{jk}^{2}a(x)\textnormal{Re}(\partial_{k}u(t,x)\partial_{j}\bar{u}(t,x))dx}}}\\\
&+4c\int{\nabla a(x)\cdot\frac{x}{|x|^{4}}|u(t,x)|^{2}dx}+2\int{\nabla
a(x)\cdot\left\\{N(u),u\right\\}_{p}(t,x)dx},\end{split}$

where
$\left\\{f,g\right\\}_{p}:=\textnormal{Re}(f\nabla\bar{g}-g\nabla\bar{f})$ is
the momentum bracket.

Note that if $N(u)=-|x|^{-b}|u|^{\sigma}u$, then

$\left\\{N(u),u\right\\}_{p}=\frac{\sigma}{\sigma+2}\nabla(|x|^{-b}|u|^{\sigma+2})+\frac{2}{\sigma+2}\nabla(|x|^{-b})|u|^{\sigma+2}.$

Hence, we immediately have the following result.

######  Corollary 4.4.

If $u$ is a smooth-in-time and Schwartz-in-space solution to the focusing
INLSc equation, then we have for any $t\in I$,

$\displaystyle\begin{split}\frac{d^{2}}{dt^{2}}V_{a}(t)&=-\int{\Delta^{2}a(x)|u(t,x)|^{2}dx+4\sum_{j,k=1}^{d}{\int{\partial_{jk}^{2}a(x)\textnormal{Re}(\partial_{k}u(t,x)\partial_{j}\bar{u}(t,x))dx}}}\\\
&+4c\int{\nabla
a(x)\cdot\frac{x}{|x|^{4}}|u(t,x)|^{2}dx}-\frac{2\sigma}{\sigma+2}\int{\Delta
a(x)|x|^{-b}|u(t,x)|^{\sigma+2}dx}\\\ &+\frac{4}{\sigma+2}\int{\nabla
a(x)\cdot\nabla(|x|^{-b})|u(t,x)|^{\sigma+2}dx}.\end{split}$

We have the following standard virial identity for the focusing INLSc
equation.

######  Lemma 4.5 (Standard Virial Identity).

Let $d\geq 3$, $0<b<2$ and $c>-c(d)$. Let $u_{0}\in H^{1}$ be such that
$|x|u_{0}\in L^{2}$ and $u:I\times\mathbb{R}^{d}\to\mathbb{C}$ be the
corresponding solution to the focusing INLSc equation. Then, $|x|u\in
C\left(I,\;L^{2}\right)$. Moreover, for any $t\in I$,

$\frac{d^{2}}{dt^{2}}\left\|xu(t)\right\|_{L^{2}}^{2}=8\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}-\frac{4(d\sigma+2b)}{\sigma+2}\int{|x|^{-b}\left|u(t,x)\right|^{\sigma+2}dx.}$
(4.7)

###### Proof.

The first claim follows from the standard approximation argument and we omit
the details (see e.g. Proposition 6.5.1 of [8] for details). It remains to
prove (4.7). Applying Corollary 4.4 with $a(x)=|x|^{2}$, we have

$\displaystyle\begin{split}\frac{d^{2}}{dt^{2}}\left\|xu(t)\right\|_{L^{2}}^{2}&=\frac{d^{2}}{dt^{2}}V_{|x|^{2}}(t)\\\
&=8\int{\left|\nabla
u(t,x)\right|^{2}+c|x|^{-2}\left|u(t,x)\right|^{2}dx}-\frac{4(d\sigma+2b)}{\sigma+2}\int{|x|^{-b}\left|u(t,x)\right|^{\sigma+2}dx}\\\
&=8\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}-\frac{4(d\sigma+2b)}{\sigma+2}\int{|x|^{-b}\left|u(t,x)\right|^{\sigma+2}dx,}\end{split}$

this completes the proof. ∎

Next we derive the localized virial estimate which is used to prove the blowup
for the focusing INLSc equation with radial data. To do so, we introduce a
function $\theta:\left[0,\;\infty\right)\to\left[0,\;\infty\right)$ satisfying

$\theta(r)=\left\\{\begin{array}[]{l}{r^{2},\leavevmode\nobreak\
\textrm{if}\;0\leq r\leq 1,}\\\ {\textrm{const},\leavevmode\nobreak\
\textrm{if}\;r\geq 2,}\end{array}\right.\textrm{and}\leavevmode\nobreak\
\leavevmode\nobreak\ \theta^{\prime\prime}(r)\leq 2,\leavevmode\nobreak\
\leavevmode\nobreak\ \textrm{for}\leavevmode\nobreak\ \leavevmode\nobreak\
r\geq 0.$ (4.8)

For $R>1$, we define the radial function

$\varphi_{R}(x)=\varphi_{R}(r):=R^{2}\theta(r/R),\;r=|x|.$ (4.9)

One can easily see that

$2-\varphi^{\prime\prime}_{R}(r)\geq
0,\;2-\frac{\varphi^{\prime}_{R}(r)}{r}\geq 0,\;2d-\Delta\varphi_{R}(x)\geq
0.$ (4.10)

######  Lemma 4.6 (Localized Virial Estimate).

Let $d\geq 3$, $0<b<2$, $c>-c(d)$, $R>1$ and $\varphi_{R}$ be as in (4.9). Let
$u:I\times\mathbb{R}^{d}\to\mathbb{C}$ be a radial solution to the focusing
INLSc equation. Then for any $\varepsilon>0$ and any $t\in I$,

$\displaystyle\begin{split}\frac{d^{2}}{dt^{2}}V_{\varphi_{R}}(t)&\leq
8\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}-\frac{4(d\sigma+2b)}{\sigma+2}\int{|x|^{-b}\left|u(t,x)\right|^{\sigma+2}dx}\\\
&+O\left(R^{-2}+\varepsilon^{-\frac{\sigma}{4-\sigma}}R^{-{\frac{2\left[(d-1)\sigma+2b\right]}{4-\sigma}}}+\varepsilon\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}\right).\end{split}$
(4.11)

###### Proof.

We use the argument similar to that used to prove the localized virial
estimates for INLS equation and NLSc equation (see [9, 10]). Applying
Corollary 4.3 with $a(x)=\varphi_{R}(x)$, we have

$\displaystyle\begin{split}\frac{d^{2}}{dt^{2}}V_{\varphi_{R}}(t)&=-\int{\Delta^{2}\varphi_{R}(x)|u(t,x)|^{2}dx}+4\sum_{j,k=1}^{d}{\int{\partial_{jk}^{2}\varphi_{R}(x)\textnormal{Re}(\partial_{k}u(t,x)\partial_{j}\bar{u}(t,x))dx}}\\\
&+4c\int{\nabla\varphi_{R}(x)\cdot\frac{x}{|x|^{4}}|u(t,x)|^{2}dx}-\frac{2\sigma}{\sigma+2}\int{\Delta\varphi_{R}(x)|x|^{-b}|u(t,x)|^{\sigma+2}dx}\\\
&+\frac{4}{\sigma+2}\int{\nabla\varphi_{R}(x)\cdot\nabla(|x|^{-b})|u(t,x)|^{\sigma+2}dx}.\end{split}$

Since $\varphi_{R}(x)=|x|^{2}$ for $|x|<R$, it follows from Lemma 4.5 that

$\displaystyle\begin{split}\frac{d^{2}}{dt^{2}}V_{\varphi_{R}}(t)&=8\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}-\frac{4(d\sigma+2b)}{\sigma+2}\int{|x|^{-b}\left|u(t,x)\right|^{\sigma+2}dx}-8\left\|u(t)\right\|_{\dot{H}^{1}_{c}(|x|>R)}^{2}\\\
&+\frac{4(d\sigma+2b)}{\sigma+2}\int_{|x|>R}{|x|^{-b}\left|u(t,x)\right|^{\sigma+2}dx}-\int_{|x|>R}{\Delta^{2}\varphi_{R}(x)|u(t,x)|^{2}dx}\\\
&+4\sum_{j,k=1}^{d}{\int_{|x|>R}{\partial_{jk}^{2}\varphi_{R}(x)\textnormal{Re}(\partial_{k}u(t,x)\partial_{j}\bar{u}(t,x))dx}}\\\
&+4c\int_{|x|>R}{\nabla\varphi_{R}(x)\cdot\frac{x}{|x|^{4}}|u(t,x)|^{2}dx}-\frac{2\sigma}{\sigma+2}\int_{|x|>R}{\Delta\varphi_{R}(x)|x|^{-b}|u(t,x)|^{\sigma+2}dx}\\\
&+\frac{4}{\sigma+2}\int_{|x|>R}{\nabla\varphi_{R}(x)\cdot\nabla(|x|^{-b})|u(t,x)|^{\sigma+2}dx}.\end{split}$

Since $\Delta\varphi_{R}\lesssim 1$, $\Delta^{2}\varphi_{R}\lesssim R^{-2}$
and $\nabla\varphi_{R}(x)\cdot\nabla(|x|^{-b})\lesssim|x|^{-b}$, we have

$\displaystyle\begin{split}\frac{d^{2}}{dt^{2}}V_{\varphi_{R}}(t)&=8\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}-\frac{4(d\sigma+2b)}{\sigma+2}\int{|x|^{-b}\left|u(t,x)\right|^{\sigma+2}dx}\\\
&-8\left\|u(t)\right\|_{\dot{H}^{1}_{c}(|x|>R)}^{2}+4\sum_{j,k=1}^{d}{\int_{|x|>R}{\partial_{jk}^{2}\varphi_{R}(x)\textnormal{Re}(\partial_{k}u(t)\partial_{j}\bar{u}(t))dx}}\\\
&+4c\int_{|x|>R}{\nabla\varphi_{R}(x)\cdot\frac{x}{|x|^{4}}|u(t,x)|^{2}dx}\\\
&+O\left(\int_{|x|>R}{R^{-2}|u(t)|^{2}+|x|^{-b}|u(t)|^{\sigma+2}dx}\right).\end{split}$

Using (4.10) and the fact

$\partial^{2}_{jk}=\left(\frac{\delta_{jk}}{r}-\frac{x_{j}x_{k}}{r^{3}}\right)\partial_{r}+\frac{x_{j}x_{k}}{r^{2}}\partial^{2}_{r},$

we can see that

$4\sum_{j,k=1}^{d}{\partial_{jk}^{2}\varphi_{R}(\partial_{k}u\partial_{j}\bar{u})dx}\leq
2|\nabla u|^{2},\leavevmode\nobreak\ \nabla\varphi_{R}\cdot x\leq 2|x|^{2}.$

Hence, we have

$\displaystyle\begin{split}&-8\left\|u(t)\right\|_{\dot{H}^{1}_{c}(|x|>R)}^{2}+4\sum_{j,k=1}^{d}{\int_{|x|>R}{\partial_{jk}^{2}\varphi_{R}\textnormal{Re}(\partial_{k}u\partial_{j}\bar{u})dx}}+4c\int_{|x|>R}{\nabla\varphi_{R}(x)\cdot\frac{x}{|x|^{4}}|u|^{2}dx}\\\
&\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leq 4c\int_{|x|>R}{\left(\nabla\varphi_{R}(x)\cdot
x-2|x|^{2}\right)\frac{|u|^{2}}{|x|^{4}}}=-4c\int_{|x|>R}{\left(2-\frac{\varphi^{\prime}_{R}(r)}{r}\right)\frac{|u(t)|}{|x|^{2}}dx}\\\
&\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\
\leq\max\left\\{-4cS,0\right\\}\int_{|x|>R}{R^{-2}|u(t)|^{2}dx}\leq\max\left\\{-4cSM(u(t)),0\right\\}R^{-2},\end{split}$

where $S=\max_{r\geq 1}{2-\frac{\theta^{\prime}(r)}{r}}$. The conservation of
mass implies

$\displaystyle\begin{split}\frac{d^{2}}{dt^{2}}V_{\varphi_{R}}(t)&\leq
8\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}-\frac{4(d\sigma+2b)}{\sigma+2}\int{|x|^{-b}\left|u(t,x)\right|^{\sigma+2}dx}\\\
&+O\left(R^{-2}+\int_{|x|>R}{|x|^{-b}|u(t)|^{\sigma+2}dx}\right).\end{split}$

Using the same argument as in the proof of Lemma 3.4 of [9], it follows from
the fact $\dot{H}^{1}\sim\dot{H}^{1}_{c}$, we have

$\int_{|x|>R}{|x|^{-b}|u(t)|^{\sigma+2}dx}\lesssim
R^{-\frac{(d-1)\sigma+2b}{2}}\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{\frac{\sigma}{2}},$

whose proof will be omitted. Next we use the Young inequality 222 Let $a$, $b$
be non-negative real numbers and $p$, $q$ be positive real numbers satisfying
$\frac{1}{p}+\frac{1}{q}=1$. Then for any $\varepsilon$, we have
$ab\lesssim\varepsilon a^{p}+\varepsilon^{-\frac{q}{p}}b^{q}$. to get for any
$\varepsilon>0$,

$R^{-\frac{(d-1)\sigma+2b}{2}}\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{\frac{\sigma}{2}}\lesssim\varepsilon^{-\frac{\sigma}{4-\sigma}}R^{-{\frac{2\left[(d-1)\sigma+2b\right]}{4-\sigma}}}+\varepsilon\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2},$

this completes the proof. ∎

###  4.3 Proof of Theorem 1.4

In this subsection, we prove Theorem 1.4. We divide the study in two cases:
$E_{b,c}(u_{0})<0$ and $E_{b,c}(u_{0})\geq 0$.

  * •

The case $E_{b,c}(u_{0})<0$.

First, we consider the case $xu_{0}\in L^{2}$. Applying the standard virial
identity (4.7) and the conservation of energy, we have

$\displaystyle\begin{split}\frac{d^{2}}{dt^{2}}\left\|xu(t)\right\|_{L^{2}}^{2}&=8\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}-\frac{4(d\sigma^{\star}+2b)}{\sigma^{\star}+2}\int{|x|^{-b}\left|u(t,x)\right|^{\sigma^{\star}+2}dx}\\\
&=4(d\sigma^{\star}+2b)E_{b,c}\left(u(t)\right)-2(d\sigma^{\star}-4+2b)\left\|u\right\|_{\dot{H}^{1}_{c}}^{2}<0,\end{split}$

where we used the fact $d\sigma^{\star}-4+2b>0$. By the classical argument of
Glassey [12], it follows that the solution $u$ blows up in finite time.

Next, we consider the case $u_{0}$ is radial. Using the localized virial
estimate (4.11) and the conservation of energy, we have

$\displaystyle\begin{split}\frac{d^{2}}{dt^{2}}V_{\varphi_{R}}(t)&\leq
8\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}-\frac{4(d\sigma^{\star}+2b)}{\sigma^{\star}+2}\int{|x|^{-b}\left|u(t,x)\right|^{\sigma^{\star}+2}dx}\\\
&+O\left(R^{-2}+\varepsilon^{-\frac{\sigma^{\star}}{4-\sigma^{\star}}}R^{-{\frac{2\left[(d-1)\sigma^{\star}+2b\right]}{4-\sigma^{\star}}}}+\varepsilon\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}\right)\\\
&=4(d\sigma^{\star}+2b)E_{b,c}\left(u(t)\right)-2(d\sigma^{\star}-4+2b)\left\|u\right\|_{\dot{H}^{1}_{c}}^{2}\\\
&+O\left(R^{-2}+\varepsilon^{-\frac{\sigma^{\star}}{4-\sigma^{\star}}}R^{-{\frac{2\left[(d-1)\sigma^{\star}+2b\right]}{4-\sigma^{\star}}}}+\varepsilon\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}\right),\end{split}$

for any $t$ in the existence time and for any $\varepsilon>0$. Since
$d\sigma^{\star}-4+2b>0$, we take $\varepsilon>0$ small enough and $R>1$ large
enough depending on $\varepsilon$ to have that

$\frac{d^{2}}{dt^{2}}V_{\varphi_{R}}(t)\leq
2(d\sigma^{\star}+2b)E_{b,c}\left(u_{0}\right)<0,$

for any $t$ in the existence time. This shows that the solution $u$ must blow
up in finite time.

  * •

The case $E_{b,c}(u_{0})\geq 0$.

By the definition of the energy (1.6) and Lemma 4.1, we have

$\displaystyle\begin{split}E_{b,c}\left(u(t)\right)&=\frac{1}{2}\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}-\frac{1}{\sigma^{\star}+2}\left\||x|^{-b}|u|^{\sigma^{\star}+2}\right\|_{L^{1}}\\\
&\geq\frac{1}{2}\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}-\frac{C_{HS}(b,\bar{c})^{\sigma^{\star}+2}}{\sigma^{\star}+2}\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{\sigma^{\star}+2}=:g\left(\left\|u(t)\right\|_{\dot{H}^{1}_{c}}\right),\end{split}$

where

$g(y)=\frac{1}{2}y^{2}-\frac{C_{HS}(b,\bar{c})^{\sigma^{\star}+2}}{\sigma^{\star}+2}y^{\sigma^{\star}+2}.$
(4.12)

It also follows from (4.6) that

$g(\left\|W_{b,\bar{c}}\right\|_{\dot{H}^{1}_{\bar{c}}})=E_{b,\bar{c}}(W_{b,\bar{c}}).$

By the conservation of energy and the assumption
$E_{b,c}(u_{0})<E_{b,\bar{c}}(W_{b,\bar{c}})$, we can see that

$g(\left\|u(t)\right\|_{\dot{H}^{1}_{c}})\leq
E_{b,c}\left(u(t)\right)=E_{b,c}\left(u_{0}\right)<E_{b,\bar{c}}(W_{b,\bar{c}}).$

By the assumption
$\left\|u_{0}\right\|_{\dot{H}^{1}_{c}}>\left\|W_{b,\bar{c}}\right\|_{\dot{H}^{1}_{\bar{c}}}$
and the continuity argument, we have

$\left\|u(t)\right\|_{\dot{H}^{1}_{c}}>\left\|W_{b,\bar{c}}\right\|_{\dot{H}^{1}_{\bar{c}}},$
(4.13)

for any $t$ as long as the solution exists. (4.13) is improved as follows.
Pick $\delta>0$ small enough such that

$E_{b,c}(u_{0})\leq(1-\delta)E_{b,\bar{c}}(W_{b,\bar{c}}),$ (4.14)

which implies that

$g(\left\|u(t)\right\|_{\dot{H}^{1}_{c}})\leq(1-\delta)E_{b,\bar{c}}(W_{b,\bar{c}}).$
(4.15)

Using (4.6), (4.12) and (4.15), we have

$\frac{d-b}{2-b}\left(\frac{\left\|u(t)\right\|_{\dot{H}^{1}_{c}}}{\left\|W_{b,\bar{c}}\right\|_{\dot{H}^{1}_{\bar{c}}}}\right)^{2}-\frac{d-2}{2-b}\left(\frac{\left\|u(t)\right\|_{\dot{H}^{1}_{c}}}{\left\|W_{b,\bar{c}}\right\|_{\dot{H}^{1}_{\bar{c}}}}\right)^{\sigma^{\star}+2}\leq
1-\delta.$

The continuity argument shows that there exits $\delta^{\prime}>0$ depending
on $\delta$ such that

$\frac{\left\|u(t)\right\|_{\dot{H}^{1}_{c}}}{\left\|W_{b,\bar{c}}\right\|_{\dot{H}^{1}_{c}}}\geq
1+\delta^{\prime}.$ (4.16)

Then we can take $\varepsilon>0$ small enough such that

$8\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}-\frac{4(d\sigma^{\star}+2b)}{\sigma^{\star}+2}\left\||x|^{-b}|u|^{\sigma^{\star}+2}\right\|_{L^{1}}+\varepsilon\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}\leq-c<0,$
(4.17)

for any $t$ in the existence time. In fact, using the conservation of energy,
(4.6), (4.14) and (4.16), we have

$\displaystyle\begin{split}\textrm{LHS(4.17)}&=4(d\sigma^{\star}+2b)E_{b,c}\left(u(t)\right)+(8+\varepsilon-2d\sigma^{\star}-4b)\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}\\\
&\leq
4(1-\delta)(d\sigma^{\star}+2b)E_{b,\bar{c}}(W_{b,\bar{c}})+(8+\varepsilon-2d\sigma^{\star}-4b)(1+\delta^{\prime})^{2}\left\|W_{b,\bar{c}}\right\|_{\dot{H}^{1}_{\bar{c}}}^{2}\\\
&=\left\|W_{b,\bar{c}}\right\|_{\dot{H}^{1}_{\bar{c}}}^{2}\left[\frac{8(2-b)}{d-2}\left(1-\delta-(1+\delta^{\prime})^{2}\right)+\varepsilon(1+\delta^{\prime})^{2}\right].\end{split}$

Hence, by taking $\varepsilon>0$ small enough, we can get (4.17).

First, we consider the case $xu_{0}\in L^{2}$ satisfying
$E_{b,c}(u_{0})<E_{b,\bar{c}}(W_{b,\bar{c}})$ and
$\left\|u_{0}\right\|_{\dot{H}^{1}_{c}}>\left\|W_{b,\bar{c}}\right\|_{\dot{H}^{1}_{\bar{c}}}$.
Using the standard virial identity (4.7) and (4.17), we have

$\frac{d^{2}}{dt^{2}}\left\|xu(t)\right\|_{L^{2}}^{2}=8\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}-\frac{4(d\sigma^{\star}+2b)}{\sigma^{\star}+2}\left\||x|^{-b}|u|^{\sigma^{\star}+2}\right\|_{L^{1}}\leq-c<0,$

which implies that the solution blows up in finite time.

Next, we consider the case $u_{0}$ is radial, and satisfies
$E_{b,c}(u_{0})<E_{b,\bar{c}}(W_{b,\bar{c}})$ and
$\left\|u_{0}\right\|_{\dot{H}^{1}_{c}}>\left\|W_{b,\bar{c}}\right\|_{\dot{H}^{1}_{\bar{c}}}$.
Using the localized virial estimates (4.11), we have

$\displaystyle\begin{split}\frac{d^{2}}{dt^{2}}V_{\varphi_{R}}(t)\leq&8\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}-\frac{4(d\sigma^{\star}+2b)}{\sigma^{\star}+2}\left\||x|^{-b}|u|^{\sigma^{\star}+2}\right\|_{L^{1}}\\\
&+O\left(R^{-2}+\varepsilon^{-\frac{\sigma^{\star}}{4-\sigma^{\star}}}R^{-{\frac{2\left[(d-1)\sigma^{\star}+2b\right]}{4-\sigma^{\star}}}}+\varepsilon\left\|u(t)\right\|_{\dot{H}^{1}_{c}}^{2}\right),\end{split}$

for any $\varepsilon>0$ and any $t$ in the existence time. Taking
$\varepsilon>0$ small enough and $R>1$ large enough depending on
$\varepsilon$, it follows from (4.17) that

$\frac{d^{2}}{dt^{2}}V_{\varphi_{R}}(t)\leq-c/2<0,$

which implies that the solution must blow up in finite time. This completes
the proof.

## References

  * [1] An, J., Kim, J.: Local well-posedness for the inhomogeneous nonlinear Schrödinger equation in $H^{s}(\mathbb{R}^{n})$. Nonlinear Anal. Real World Appl., 59, 103268 (2021)
  * [2] Ardila, A. H., Cardoso, M.: Blow-up solutions and strong instability of ground states for the inhomogeneous nonlinear Schrödinger equation. Commun. Pure Appl. Anal., 20(1), 101–119 (2021)
  * [3] Belmonte-Beitia, J., Pérez-García, V. M., Vekslerchik, V., Torres, P. J.: Lie symmetries and solitons in nonlinear systems with spatially inhomogeneous nonlinearities. Phys. Rev. Lett., 98(6), 064102 (2007)
  * [4] Bouclet, J. M., Mizutani, H.: Uniform resolvent and Strichartz estimates for Schrödinger equations with critical singularities. Trans. Amer. Math. Soc., 370, 7293–7333 (2018)
  * [5] Burq, N., Planchon, F., Stalker, J., Tahvildar-Zadeh, A.S.: Strichartz estimates for the wave and Schrödinger equations with the inverse-square potential. J. Funct. Anal., 203, 519–549 (2003)
  * [6] Campos, L.: Scattering of radial solutions to the inhomogeneous nonlinear Schrödinger equation. Nonlinear Anal., 202, 112118 (2021)
  * [7] Campos, L., Guzmán, C. M.: On the inhomogeneous NLS with inverse-square potential. Z. Angew. Math. Phys., 72:143, (2021) Avaliable at https://doi.org/10.1007/s0033-021-01560-4.
  * [8] Cazenave, T.: Semilinear Schrödinger Equations, Amer. Math. Soc., Providence, (2003)
  * [9] Dinh, V. D.: Blowup of $H^{1}$ solutions for a class of the focusing inhomogeneous nonlinear Schrödinger equation. Nonlinear Anal., 174, 169–188 (2018)
  * [10] Dinh, V. D.: Global exsitence and blowup for a class of focusing nonlinear Schrödinger equation with inverse-square potential. J. Math. Anal. Appl., 468, 270–303 (2018)
  * [11] Dinh, V. D.: Global dynamics for a class of inhomogeneous nonlinear Schrödinger equations with potential. Math. Nachr., 294(4), 672–716 (2021)
  * [12] Glassey, R. T.: On the blowing up of solutions to the Cauchy problem for nonlinear Schrödinger equations. J. Math. Phys., 18, 1794–1797 (1977)
  * [13] Guzmán, C. M.: On well posedness for the inhomogeneous nonlinear Schrödinger equation. Nonlinear Anal. Real World Appl. 37, 249–286 (2017)
  * [14] Hajaiej, H., Yu, X., Zhai, Z.: Fractional Gagliardo-Nirenberg and Hardy inequalities under Lorentz norms. J. Math. Anal. Appl., 396, 569–577 (2012)
  * [15] Kalf, H., Schmincke, U. W., Walter, J., Wust, R.: On the spectral theory of Schrödinger and Dirac operators with strongly singular potentials, in: Spectral Theory and Differential Equations, in: Lect. Notes in Math., vol. 448, Springer, Berlin, 1975
  * [16] Kang, D. S., Peng, S. J.: Positive solutions for elliptic equations with critical Sobolev-Hardy exponents. Appl. Math. Lett., 17, 411–416 (2004)
  * [17] Kartashov, Y. V., Malomed, B. A., Vysloukh, V. A., Belic, M. R., Torner, L.: Rotating vortex clusters in media with inhomogeneous defocusing nonlinearity. Optic Letters., 42(3), 446–449 (2017)
  * [18] Killip, R., Miao, C., Visan, M., Zhang, J., Zheng, J.: The energy-critical NLS with inverse-square potential. Discrete Contin. Dyn. Syst. 37, 3831–3866 (2017)
  * [19] Killip, R., Miao, C., Visan, M., Zhang, J., Zheng, J.: Sobolev spaces adapted to the Schrödinger operator with inverse-square potential. Math. Z., 288(3-4), 1273–1298 (2018)
  * [20] Killip, R., Murphy, J., Visan, M., Zheng, J.: The focusing cubic NLS with inverse-square potential in three space dimensions. Differential Integral Equations, 30(3-4), 161–206 (2017)
  * [21] Linares, F., Ponce, G.: Introduction to Nonlinear Dispersive Equations, second ed., Universitext. Springer, New York, 2015
  * [22] Luo, X.: Stability and multiplicity of standing waves for the inhomogeneous NLS equation with a harmonic potential. Nonlinear Anal. Real World Appl., 45, 688–703 (2019)
  * [23] Lu, J., Miao, C., Murphy, J.: Scattering in $H^{1}$ for the intercritical NLS with an inverse-square potential. J. Differential Equations, 264(5), 3174–3211 (2018)
  * [24] Suzuki, T.: Solvability of nonlinear Schrödinger equations with some critical singular potential via generalized Hardy-Rellich inequalities. Funkcial. Ekvac., 59(1), 1–34 (2016)
  * [25] Wang, B. X., Huo, Z., Hao, C., Guo, Z.: Harmonic Analysis Method for Nonlinear Evolution Equations, I, World Scientific, Singapore, 2011
  * [26] Yang, K.: Scattering of the focusing energy-critical NLS with inverse-square potential in the radial case. Comm. Pure Appl. Anal., 20(1), 77–99 (2021)


# A 2-$\mu$J, 12-class, $91\%$ Accuracy Spiking Neural Network Approach For
Radar Gesture Recognition

Ali Safa, , André Bourdoux, , Ilja Ocket, , Francky Catthoor, , Georges G.E.
Gielen Ali Safa, Ilja Ocket, Francky Catthoor and Georges G.E Gielen are with
imec and the Department of Electrical Engineering, KU Leuven, 3001 Leuven,
Belgium (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Georges.Gielen@kuleuven.be).André Bourdoux is with
imec, 3001 Leuven, Belgium (e-mail: Andre.Bourdoux@imec.be).

###### Abstract

Radar processing via spiking neural networks (SNNs) has recently emerged as a
solution in the field of ultra-low-power wireless human-computer interaction.
Compared to traditional energy- and area-hungry deep learning methods, SNNs
are significantly more energy efficient and can be deployed in the growing
number of compact SNN accelerator chips, making them a better solution for
ubiquitous IoT applications. We propose a novel SNN strategy for radar gesture
recognition, achieving more than 91% of accuracy on two different radar
datasets. Our work significantly differs from previous approaches as 1) we use
a novel radar-SNN training strategy, 2) we use quantized weights, enabling
power-efficient implementation in real-world SNN hardware, and 3) we report
the SNN energy consumption per classification, clearly demonstrating the real-
world feasibility and power savings induced by SNN-based radar processing. We
release evaluation code to help future research.

###### Index Terms:

Radar gesture recognition, spiking networks

## Supplementary Material

Evaluation code is available at: https://tinyurl.com/yu598c7e

##  I Introduction

Wireless human-computer interaction using radar-based gesture recognition
systems has attracted large interest during the past decade, enabling
applications such as smart domotics, AR/VR headsets and many other touchless
interfacing solutions that are key for a more hygienic, post-COVID-19 world
[1]. In order to embed radar sensing into ubiquitous, ultra-low-power IoT
devices, research at the hardware side has mainly been devoted to high-level
integration of radar transceivers [2] with a focus on energy and area
efficiency [3]. In contrast, research at the signal processing side has mainly
been devoted to the use of high-accuracy deep neural networks (DNNs) known to
be rather energy- and area-hungry [4, 5]. State-of-the-art DNN-based
techniques either rely on the use of an expensive desktop-grade GPU [4] or
either on the use of a lower-power and lower-area embedded GPU (e.g., 10-W
Nvidia Jetson Nano) [5], still ill-suited for ultra-low-power applications
like ubiquitous IoT.

Figure 1: $\boldsymbol{\mu}$Doppler-based radar-SNN architecture proposed to
solve the 5-class 8-GHz dataset of [6] with $93\%$ of accuracy Figure 2:
Range-Doppler radar-SNN architecture proposed to solve the 12-class dataset of
[4] with $91\%$ of accuracy.

Very recently, the use of energy-efficient spiking neural networks (SNNs) for
radar processing has grown to become an emerging topic in radar sensing and is
currently being investigated by many teams [6, 7, 8, 9]. Algorithm-wise, SNNs
differ from DNNs as they communicate inter-neural information asynchronously,
using binary spikes that are only emitted when the neuron membrane potential
reaches a specific threshold. In contrast to DNNs, SNNs do not require
expensive multiply-accumulate operations at the input of each neuron, but make
use of inexpensive add operations only. Hardware-wise, SNNs can be integrated
near the radar sensor (see Fig. 1) as sub-threshold analog circuits, reaching
more than 5 orders of magnitude lower power consumption compared to embedded
GPUs [5, 11, 12].

Still, the development of SNN-based radar processing is at an early stage. In
this letter, our aim is to propose a novel SNN architecture for radar gesture
recognition using a different approach than the ones used in previously
presented radar-SNN systems. Compared to previous works [6, 8, 9], which
either use the $\mu$Doppler pre-processing [13] or the range-Doppler pre-
processing [4], we demonstrate that our novel radar-SNN approach is compatible
with both pre-processing techniques. In contrast to the work in [7], our
approach is purely SNN-based, while the system of [7] uses an SNN followed by
classical machine learning techniques such as Random Forest, which cannot be
deployed in sub-threshold analog SNN circuits. Compared to [7, 8, 9], our
system uses implementation-ready, quantized weights (typical bit width in SNN
hardware is $<8$ bits [10, 6]), while none of the aforementioned works
quantize their weights, making their reported performances (85%-98%) unclear
when deployed in real-world hardware (12-class $91\%$ with 6-bit weights and
5-class $93\%$ with 4-bit weights in our work). Finally, in contrast to most
previously mentioned works [7, 8, 9], we report an estimate of our SNN energy
consumption when deployed in dedicated SNN hardware [6]. We assess the
performance of our system on two different radar gesture datasets: the
12-class Google Soli dataset of [4] and the 5-class 8-GHz dataset of [6]. We
report a $\times 3$ increase in the number of gesture classes compared to
state-of-the-art quantized-weight SNNs.

Figure 3: SNN architecture used for radar processing. Each spiking map slice
corresponding to each time step is fed one by one to the network and the IF
neurons change state according to their self recurrence (as denoted by the
black recurrence arrows).

##  II Radar-SNN Processing Pipeline

###  II-A 5-class 8-GHz dataset and pre-processing

The dataset of [6] contains radar ADC data with $N_{chirps}=192$ chirps per
frame and with a variable number of frames per gesture acquisition
$N_{frames}$ (step 1 in Fig. 1). $\mu$Doppler signatures [13] are acquired for
each gesture acquisition in the dataset by first computing the range profiles
$R_{n}[k]$ for each chirp $n=1,...,N_{tot}$ (where $N_{tot}$ is the total
number of chirps). $R_{n}[k]$ is acquired by DFT using a Blackman window [14]
(step 2 in Fig. 1). Then, we apply the Short-Time Fourier Transform (STFT) to
the sequence $\tilde{R}_{n}[k^{*}]=R_{n}[k^{*}]-R_{n-1}[k^{*}]$ (step 3 in
Fig. 1), which removes the strong DC component during each analysis window
[15], as follows:

$\Theta[m,f]=\sum_{n=-\infty}^{\infty}\tilde{R}_{n}[k^{*}]g_{s}[n-mR]e^{-j2\pi
fn}$ (1)

where $k^{*}$ denotes the range bin where the gestures are executed, $g_{s}$
denotes a Hanning window of length $s$, and $R$ is the hop size ($s=192$ and
$R=8$ throughout this paper). $k^{*}$ is known a priori as the gestures are
executed at 2 meters from the radar. We define the $\mu$Doppler signature as
$|\Theta[m,f]|$ which is a matrix of size $(N_{T}\times s)$ with $N_{T}$ given
by [9]:

$N_{T}=\left\lfloor{\frac{N_{frames}N_{chirps}-N_{overlap}}{R}}\right\rfloor$
(2)

where $N_{overlap}=s-R$ is the number of overlapping bins between successive
windows. Radar maps to be fed to the SNN are obtained by cutting
$|\Theta[m,f]|$ along dimension $m$ into images of $48$ time samples.
$\left\lfloor{\frac{N_{T}}{48}}\right\rfloor$ examples are thus obtained for
each acquisition. By balancing the dataset and by removing the first and the
last $6$ example maps to remove start-up (when the human simply sits in front
of the radar before performing gestures) and ending artefacts (when the human
reaches out to the radar to stop it), we obtain a balanced dataset with a
total of $1695$ $\mu$Doppler examples.

Each example map is then normalized between $[0,1]$. Out-of-band noise is
removed through the band-limiting of the Doppler frequency axis by keeping the
normalized frequency range between $[-0.26,0.26]$ only. This frequency band
was identified visually by evaluating the maximal significant extent of the
Doppler spectra in the dataset. Then, we use soft thresholding [16] to remove
in-band noise in each Doppler spectrum (step 4 in Fig. 1). The soft
thresholding is performed by keeping the $k$ largest values and pad the
remaining ones to $0$. We choose $k$ heuristically by considering that more
than half of the Doppler samples within the normalized frequencies
$[-0.26,0.26]$ are significant (i.e. not noise), which leads to the choice of
$k=\left\lfloor{\frac{192\times(0.26-(-0.26))}{2}}\right\rfloor-1=48$ (we
tried other $k$ values around $48$, but did not observe any significant boost
in SNN accuracy). After step 4, Fig. 1 shows an example radar map, resulting
from the $\mu$Doppler pre-processing described above.

The pre-processed radar $\mu$Doppler maps must then be converted into event
streams to be compatible with the spiking nature of our SNN. Each pixel of the
map is coded as a spike train of length $T_{inf}$ (number of time steps per
inference). We encode each pixel using Time-To-First-Spike (TTFS) encoding
(step 5 in Fig. 1), where a pixel of value $v\in[0,1]$ is quantized into an
event train containing one spike located at index
$T_{inf}-\left\lfloor{vT_{inf}}\right\rfloor$ [17]. If the pixel is equal to
0, then no spikes are emitted. As we are aiming at low-latency inference, we
choose $T_{inf}=4$ time steps.

###  II-B 12-class Soli dataset and pre-processing

The Soli dataset [4] has been acquired using a 60-GHz FMCW radar and is
composed of 12 classes with a total of 5500 CFAR-processed range-Doppler
magnitude acquisitions. Each gesture acquisition is a collection of maps
$RD[t,l,m]$ where $t$ is the frame index, $l$ is the range index and $m$ is
the Doppler index (see step 1 in Fig. 2), with a varying number of time steps
$t\in[1,T_{fr}]$ per acquisition. First, we average and sub-sample each
gesture acquisition $RD[t,l,m]$ (with varying $T_{fr}$) along $t$ (step 2 in
Fig. 2) to a fixed number $T_{inf}<T_{fr}$ $\forall T_{fr}$ of frames per
acquisition, as follows:

$RD[n,l,m]=\frac{T_{inf}}{T_{fr}}\sum_{t=n}^{n+\frac{T_{fr}}{T_{inf}}}RD[t,l,m]$
(3)

where $n$ is the sub-sampled time index. Then, the resulting frames are
converted to binary images $RD_{b}[n,l,m]$ by thresholding against $0$ (step 3
in Fig. 2). Therefore, for any pixel coordinate $(l^{*},m^{*})$,
$RD_{b}[n,l^{*},m^{*}]$ represents a spike train of length $T_{inf}$, set to
$28$ (minimum $T_{fr}$ in the dataset).

###  II-C Spiking neural network for classification

To classify the spiking radar tensors, we use the SNN architecture shown in
Fig. 3 with Integrate and Fire (IF) neurons:

$\begin{cases}V^{k+1}=V^{k}+J_{in}\hskip 5.0pt\mbox{and}\hskip
5.0ptS=0&\mbox{if }V^{k}<1\\\ V^{k+1}=0\hskip 5.0pt\mbox{and}\hskip
5.0ptS=1&\mbox{if }V^{k}\geq 1\end{cases}$ (4)

where $V^{k}\geq 0$ is the neural membrane potential at time step $k$,
$J_{in}$ is the neuron input and $S$ is the spiking output. As the derivative
of spikes as a function of the membrane potential is ill-defined, we create a
custom neuron model using the pyTorch framework [18] which behaves as (4) in
forward pass. For the backward pass, we approximate the derivative using a
Gaussian function (5) as the surrogate derivative [19]. This enables the use
of back-propagation in the spiking domain.

$\sigma^{\prime}(V)\approx\frac{1}{\sqrt{2\pi}}e^{-2V^{2}}$ (5)

The layer-by-layer description of our SNN architecture (Fig. 3) is the
following. After the spike train encoding of the radar maps, we use a
$(5,5,12)$ convolutional layer. At each time step, the convolution result is
fed to the IF neuron layer $\sigma_{1}$. Then, the spiking tensor at the
output of $\sigma_{1}$ is down-sampled via MaxPooling and the resulting tensor
is flattened to a 1-dimensional spiking vector. Then, two fully-connected
spiking layers are used and the 12- or 5-dimensional output of $\sigma_{3}$
(corresponding to the 12 or 5 gesture classes) is accumulated over time in a
vector $A$. Finally, $A$ is transformed via SoftMax into class probabilities.
Our network architecture search was conducted with the objective of achieving
a $>90\%$ accuracy with heavily quantized weights (at most 6-bit) and a small
network size.

For training, we use the Adam optimizer [20] with learning rate $10^{-3}$. The
batch size is $128$ and the SNN is first trained for 14 epochs with full-bit
weights and 1 epoch with quantized weights in the forward pass and full-bit
weights in the backward pass. The accuracy of our SNN is assessed using 6-fold
cross validation.

##  III Experimental Results

Table I reports the performance of our proposed system (entry 6 for the 8-GHz
dataset and entry 7 for the Soli dataset) against the state of the art. We
evaluate the energy per classification $E_{c}$ of our SNN using the hardware
metrics of the $\mu$Brain SNN chip, described in [6]:

$E_{c}=N_{spikes}\times E_{dyn}+\delta T\times P_{stat}$ (6)

where $N_{spikes}$ is the maximum number of spikes during classification,
$E_{dyn}=2.1$ pJ is the energy per spike, $P_{stat}=73$ $\mu$W is the static
leakage power and $\delta T$ is the inference time. Even though a smaller
$\delta T$ can be reached by adjusting the bias voltages that control the
delay cells in [6], we assume $\delta T=4$ ms for the 8-GHz dataset
($T_{inf}=4$) and $\delta T=28$ ms for the Soli dataset ($T_{inf}=28$) to
provide an upper bound estimate on $E_{c}$.

Architecture | $\boldsymbol{N_{c}}$ | Accuracy | $\boldsymbol{E_{c}}$ | $\boldsymbol{N_{bits}}$
---|---|---|---|---
1) DNN [5] | 12 | 94% | 330 mJ | 32-f
2) SNN-STDP [9] | 8 | 85% | - | 32-f
3) SNN-conv [8] | 4 | 98.5% | - | 32-f
4) SNN-RF [7] | 11 | 98% | - | 4-i & 32-f
5) SNN-conv [6] | 4 | 93.4% | 340 nJ | 4-i
6) This work (8-GHz) | 5 |  $93\pm 2$% | 351 nJ | 4-i
7) This work (Soli) | 12 |  $91\pm 1$% | 2 $\mu$J | 6-i
TABLE I: Our proposed system compared to the state of the art. $N_{c}$ is the
number of classes, $E_{c}$ is the energy consumption per classification (not
reported for entries 2-4) and $N_{bits}$ is the number of bits for the network
weights (f and i stand for float and integer respectively).

Out of the implementation-ready SNNs using quantized weights only (entries 5,
6 and 7 in Table I), our work significantly outperforms entry 5 by up to
$\times 3$ higher number of gesture classes $N_{c}$, while having a similar
accuracy and $N_{bits}$, with $E_{c}$ of the same order of magnitude. All
other entries in Table I either rely on DNNs (entry 1) and RF (entry 4), being
ill-suited for ultra-low-power IoT, or do not quantize their weights (entries
2 and 3, giving unclear performance in real-world SNN hardware). In addition,
our work achieves a recognition accuracy close to the accuracy of the DNN in
entry 1 [5], while consuming more than two orders of magnitude less energy per
inference. Finally, entries 6 and 7 clearly show how our system trades off
$N_{c}$, $E_{c}$ and $N_{bits}$ for a target accuracy of $>90\%$.

##  IV Conclusion

This letter has presented a novel radar-SNN architecture for ultra-low-power
radar gesture recognition, significantly outperforming existing
implementation-ready SNNs in terms of classification performance. The
presented approach has reported several key innovations compared to previous
radar-SNN systems such as a novel radar-SNN training strategy and radar to
spike encoding approaches. Radar-SNN evaluation code has also been provided,
which helps lighting the way for the emerging area of SNN-based radar
processing.

## Acknowledgment

The authors thank Dr. Federico Corradi and Dr. Lars Keuninckx for the
discussions and guidance, and the Flanders AI research program for partially
supporting this work.

## References

  * [1] Chin, A., Chu, J., Perera, M., Hui, K., Yen, H. L., Chan, M., Peiris, M., Poon, L. (2020). ”Stability of SARS-CoV-2 in different environmental conditions.” The Lancet. Microbe, 1(1), e10. https://doi.org/10.1016/S2666-5247(20)30003-3
  * [2] J. Rimmelspacher, R. Ciocoveanu, G. Steffan, M. Bassi and V. Issakov, ”Low Power Low Phase Noise 60 GHz Multichannel Transceiver in 28 nm CMOS for Radar Applications,” 2020 IEEE Radio Frequency Integrated Circuits Symposium (RFIC), 2020, pp. 19-22, doi: 10.1109/RFIC49505.2020.9218297.
  * [3] Y. Liu et al., ”9.3 A680 $\mu$W Burst-Chirp UWB Radar Transceiver for Vital Signs and Occupancy Sensing up to 15m Distance,” 2019 IEEE International Solid- State Circuits Conference - (ISSCC), San Francisco, CA, USA, 2019, pp. 166-168, doi: 10.1109/ISSCC.2019.8662536.
  * [4] Wang, S., Song, J., Lien, J., Poupyrev, I., Hilliges, O. (2016). ”Interacting with Soli: Exploring Fine-Grained Dynamic Gesture Recognition in the Radio-Frequency Spectrum,” (pp. 851-860).
  * [5] Y. Sun, T. Fei, X. Li, A. Warnecke, E. Warsitz and N. Pohl, ”Real-Time Radar-Based Gesture Detection and Recognition Built in an Edge-Computing Platform,” in IEEE Sensors Journal, vol. 20, no. 18, pp. 10706-10716, 15 Sept.15, 2020, doi: 10.1109/JSEN.2020.2994292.
  * [6] Stuijt, J., Sifalakis, M., Yousefzadeh, A., Corradi, F. (2021). ”$\mu$Brain: An Event-Driven and Fully Synthesizable Architecture for Spiking Neural Networks.” Frontiers in Neuroscience, 15, 538.
  * [7] Tsang, I. et al., (2021). ”Radar-Based Hand Gesture Recognition Using Spiking Neural Networks.” Electronics, 10(12).
  * [8] Arsalan M. et al., (2021). ”Resource Efficient Gesture Sensing Based on FMCW Radar using Spiking Neural Networks”
  * [9] D. Banerjee et al., ”Application of Spiking Neural Networks for Action Recognition from Radar Data,” 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, United Kingdom, 2020, pp. 1-10, doi: 10.1109/IJCNN48605.2020.9206853.
  * [10] C. Frenkel, M. Lefebvre, J. Legat and D. Bol, ”A 0.086-mm2 12.7-pJ/SOP 64k-Synapse 256-Neuron Online-Learning Digital Spiking Neuromorphic Processor in 28-nm CMOS,” in IEEE Transactions on Biomedical Circuits and Systems, vol. 13, no. 1, pp. 145-158, Feb. 2019, doi: 10.1109/TBCAS.2018.2880425.
  * [11] S. Moradi, N. Qiao, F. Stefanini and G. Indiveri, ”A Scalable Multicore Architecture With Heterogeneous Memory Structures for Dynamic Neuromorphic Asynchronous Processors (DYNAPs),” in IEEE Transactions on Biomedical Circuits and Systems, vol. 12, no. 1, pp. 106-122, Feb. 2018, doi: 10.1109/TBCAS.2017.2759700.
  * [12] M. Davies et al., ”Loihi: A Neuromorphic Manycore Processor with On-Chip Learning,” in IEEE Micro, vol. 38, no. 1, pp. 82-99, January/February 2018, doi: 10.1109/MM.2018.112130359.
  * [13] Victor C. Chen (2014). ”Radar Micro-Doppler Signatures: Processing and Applications,” Institution of Engineering and Technology.
  * [14] F. J. Harris, ”On the use of windows for harmonic analysis with the discrete Fourier transform,” in Proceedings of the IEEE, vol. 66, no. 1, pp. 51-83, Jan. 1978, doi: 10.1109/PROC.1978.10837.
  * [15] B. Vandersmissen, N. Knudde, A. Jalalvand, I. Couckuyt, A. Bourdoux, W. De Neve, and T. Dhaene, “Indoor person identification using a low-power fmcw radar,” IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 7, pp. 3941–3952, July 2018
  * [16] H. Xu, Z. Wang, H. Yang, D. Liu and J. Liu, ”Learning Simple Thresholded Features With Sparse Support Recovery,” in IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 4, pp. 970-982, April 2020, doi: 10.1109/TCSVT.2019.2901713.
  * [17] B. Rueckauer and S. Liu, ”Conversion of analog to spiking neural networks using sparse temporal coding,” 2018 IEEE International Symposium on Circuits and Systems (ISCAS), Florence, 2018, pp. 1-5, doi: 10.1109/ISCAS.2018.8351295.
  * [18] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A. (2017). Automatic differentiation in PyTorch.
  * [19] E. O. Neftci, H. Mostafa and F. Zenke, ”Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks,” in IEEE Signal Processing Magazine, vol. 36, no. 6, pp. 51-63, Nov. 2019, doi: 10.1109/MSP.2019.2931595.
  * [20] Kingma, D., Ba, J. (2014). ”Adam: A Method for Stochastic Optimization,” International Conference on Learning Representations.