Alkis Koudounas♢, Moreno La Quatra♣, Lorenzo Vaiani♢, Luca Colomba♢, Giuseppe
Attanasio♡, Eliana Pastor♢, Luca Cagliero♢, Elena Baralis♢

# ITALIC: An ITALian Intent Classification Dataset

# ITALIC: An Italian Intent Classification Dataset

###### Abstract

Recent large-scale Spoken Language Understanding datasets focus predominantly
on English and do not account for language-specific phenomena such as
particular phonemes or words in different lects. We introduce ITALIC, the
first large-scale speech dataset designed for intent classification in
Italian. The dataset comprises 16,521 crowdsourced audio samples recorded by
70 speakers from various Italian regions and annotated with intent labels and
additional metadata. We explore the versatility of ITALIC by evaluating
current state-of-the-art speech and text models. Results on intent
classification suggest that increasing scale and running language adaptation
yield better speech models, monolingual text models outscore multilingual
ones, and that speech recognition on ITALIC is more challenging than on
existing Italian benchmarks. We release both the dataset and the annotation
scheme to streamline the development of new Italian SLU models and language-
specific datasets.

Index Terms: spoken language understanding, speech recognition, human-computer
interaction

##  1 Introduction

Spoken Language Understanding (SLU) is pivotal in enabling human-machine
interaction through natural language. However, language distribution in high-
quality SLU resources is skewed toward a small set of languages, particularly
English [1, 2]. Prior work proposed training resources for languages other
than English to let models learn language-specific phonemes, words, or
phrases; however, such resources are either not specifically designed for
human-machine interaction [3] or lack audio recordings in the target language,
preventing end-to-end (E2E) learning from speech [4, 5].

In this paper, we introduce ITALIC, the first large-scale ITAlian Language
Intent Classification audio dataset, including 16,521 audio samples spanning
18 domains, 60 intents, and recorded by 70 speakers from a variety of Italian
regions. To build the collection, we extracted and annotated all Italian
interactions in the MASSIVE dataset [5] and enriched them by annotating every
recording with speaker- and channel-related attributes. We provide the
speaker's self-declared region of origin, gender, age, and instruction level,
the level of background noise, and the type of recording device used.

This metadata allows a variety of additional analyses beyond intent
classification, such as speaker recognition, text-to-speech, age estimation,
and linguistic variety identification.

To highlight the versatility of ITALIC and the richness of the provided
metadata, we benchmark current state-of-the-art speech and text models on the
intent classification and automatic speech recognition tasks. The experimental
results show that 1) model scale and ASR adaptation improve the performance of
speech models in terms of generalization to unseen speakers and robustness to
noise, 2) monolingual text models outperform multilingual ones, and 3) zero-
shot speech recognition performs worse than existing Italian benchmarks.

Our contributions are as follows:

  * •

We introduce the first Italian large-scale intent classification dataset with
recordings and transcripts of virtual assistant utterances. We enrich every
instance with self-declared information about the speaker and the recording
channel.

  * •

We provide several baselines with current state-of-the-art speech and text
models. Through experimental results, we show the strengths and weaknesses of
such models and highlight the most promising direction to improve SLU models.

  * •

We release the dataset, annotation scheme, and code for the baselines to
encourage further research on the collection and, more broadly, on SLU in
Italian.

##  2 Related Work

Most well-known SLU benchmark datasets are mainly in English. For example,
Fluent Speech Commands [1] is an open-source SLU dataset including 31 intents
and 30,043 English utterances. SLURP [2] is also an English-only dataset with
single turn user interactions with a home assistant. MASSIVE [5] extends the
latter by including more than one million utterances across 51 languages with
annotations for Natural Language Understanding (NLU) tasks, but no non-English
audio recordings. AUDIO SNIPS [6] is the audio version of the SNIPS NLU
dataset. It contains both audio samples in English and French annotated with
the corresponding intents.

Limited efforts have been made to develop speech and language understanding
systems specifically for the Italian language. One such attempt is the
AlmaWave-SLU [4], which involves the generation of an Italian data collection
derived from English AUDIO SNIPS utterances through speech transcription and
machine translation. However, this corpus lacks Italian audio recordings,
rendering it unsuitable for Italian SLU tasks. Mozilla Common Voice [3] and
Google Fleurs [7] include Italian audio recordings with transcriptions.
However, both datasets do not provide intent annotations nor metadata on the
recording conditions or the speakers’ regional origins. EMOVO [8] collects
emotional speech recordings from six native Italian speakers. IDEA [9] is a
dataset for modeling dysarthric speech and includes isolated words, recorded
under controlled conditions, covering a wide range of phonemes. Differently
from ITALIC, these Italian datasets are not explicitly created for SLU tasks.

Table 1: Gender and age distribution in ITALIC. Gender | Age
---|---
Female | Male | [18-25] | [26-40] | [41-55] |  $\geq$56
42.96% | 57.04% | 10.63% | 63.86% | 10.78% | 14.73%

##  3 The ITALIC Dataset

###  3.1 Data collection

The ITALIC dataset was crowdsourced through a custom web platform. Both native
and non-native Italian speakers participated. We required participants to
record themselves while reading a short instruction randomly sampled from the
MASSIVE [5] dataset. The latter consists of utterance transcripts and
associated intents. We use the transcripts as prompts for crowd workers to
read out and record. We do not annotate intents and instead use those supplied
by MASSIVE. After giving the participants a list of annotation guidelines, we
did not intervene in the process or supervise the recording sessions. They
used their own devices and chose freely when and how to contribute recordings.

Optionally, participants could declare additional information about themselves
and recording conditions through an anonymous registration form. Specifically,
we asked for age (a numerical integer), gender (male, female, non-binary,
undeclared), region of origin in Italy, country of origin (if not Italian),
instruction level, and presence of any speech impairment (e.g., lisp or
stuttering). Recording conditions include the input device adopted (laptop,
smartphone, or headphones) and environmental noise level (no noise to very
noisy).Such rich additional metadata will enable future per-group analyses,
e.g., studying service quality of IC models over Italian regions.

To ensure high-quality annotations, at least two individuals reviewed each
sample. We consider a sample valid if 1) the utterance is intelligible from
the recording and 2) it is coherent with the provided prompt. We validated the
entire set of recordings as follows. Once all samples were annotated, we ran a
first validation round considering the entire collection. Then, we extracted
all non-valid samples and ran a second annotation and validation round
considering only those samples. We repeated the process until no non-valid
samples were left.

###  3.2 Data Characterization

We extracted and annotated every sample in the Italian split of MASSIVE [5],
for a total of 15.46 hours of recording and 60 different intents. The final
dataset consists of 16,521 audio recordings by 70 distinct volunteers. All but
one self-identified as native Italian speakers. Only four speakers declared
some speech impairment. Native speakers are distributed across 13 Italian
regions, which concentrate a large number of linguistic and diatopic
variations [10]. Table 1 reports the distribution of gender and age in the
dataset, while Figure 1 shows the distribution over the region of origin.

The audio length goes from 1.14 to a maximum of 38.34 seconds and an average
of 3.37 seconds. We WAV-encoded recordings with a sampling frequency of 16
kHz.

Figure 1: Distribution of utterances across Italian regions. Darker colors
represent higher absolute counts.

Dataset splits. For experimental and reproducibility purposes, we release
three splitting configurations of increasing difficulty111The dataset, splits,
and annotation scheme are publicly available at https://github.com/RiTA-
nlp/ITALIC.:

  * •

massive: It uses the official training and test splits of MASSIVE [5] and
includes all our speakers. Furthermore, the distribution of noisy utterances
is randomly apportioned between the two partitions.

  * •

speaker: We stratify on speakers, i.e., all recordings of a speaker belong to
either the training, validation, or test split. This division will help to
assess whether the models generalize to unseen speakers;

  * •

noisy: The test set consists exclusively of data annotated with the highest
background noise level, while the train and validation sets contain either
noiseless or data with a low noise level, randomly split.

Table 2 reports statistics on the number of utterances, speakers, and hours of
recordings for each splitting configuration.

Table 2: Dataset statistics. Configuration | # Utterances | # Hours | # Speakers
---|---|---|---
massive | train | 11514 | 10.80 | 69
validation | 2033 | 1.90 | 68
test | 2974 | 2.76 | 69
speaker | train | 13123 | 12.33 | 56
validation | 1957 | 1.67 | 7
test | 1441 | 1.46 | 7
noisy | train | 13742 | 12.93 | 69
validation | 1526 | 1.44 | 66
test | 1253 | 1.09 | 9

###  3.3 Tasks

ITALIC enables a range of SLU and Natural Language Understanding (NLU) tasks.
In this paper, following the original setup in MASSIVE [5], we provide
baselines for the intent classification and automatic speech recognition
tasks.

Given the rich set of annotations we release, practitioners may use ITALIC in
other tasks, such as speaker identification, text-to-speech, age estimation,
or region of origin identification. We leave the analysis in such contexts to
future work.

##  4 Experiments

In this section, we present the experimental setup and results for evaluating
the performance of various models on the ITALIC dataset. We focus on intent
classification and automatic speech recognition. Our goal is to leverage the
rich annotations in ITALIC to evaluate state-of-the-art models in terms of
accuracy, robustness to noise, and generalization to unseen speakers.
Additionally, this investigation seeks to explore the influence of two key
factors, namely the knowledge of the Italian language and the variability of
recording conditions, including variations in noise levels and speaker
characteristics.

###  4.1 Experimental Setting

Models. We use established E2E transformer-based models for both IC and ASR
setups. We address IC using either raw audio signals or text transcripts. As
speech SLU models, we test several XLSR variants, a speech representation
network pretrained on 53 [11] or 128 [12] languages. We conduct experiments on
two model sizes, i.e., 300M and 1B parameters, testing XLSR-53 300M and
XLSR-128 300M and 1B. We build two additional baselines by testing language
adaptation on out-of-domain data. In practice, we employ two models, i.e.,
XLSR-53 300M and XLSR-128 1B, that have been fine-tuned via ASR on the Italian
split of the Mozilla Common Voice dataset [3]. As text NLU models, we include
multilingual BERT [13], multilingual BART [14], and two variants of these
models pre-trained on Italian data only [15, 16]. Note that both models
include Italian as a pretraining language.

For the ASR task, we use Whisper [17] in three variants: small (244M
parameters), medium (769M), and large (1.5B).

We fine-tune each IC model attaching to the encoder architecture a final
classification layer.222In our experiments on intent classification, we use a
classification layer on top of the BART encoder. We use model implementation
and weights from the transformers library [18] in all experiments.

Data. We fine-tune models on the training split of one of the configurations
of ITALIC, i.e., massive, speaker, or noisy, and report the result on the test
set.

Evaluation Metrics. We measure accuracy and F1 Macro scores for IC and the
Word Error Rate (WER) and Character Error Rate (CER) for ASR.

Hyperparameter Setup. We ran a manual hyperparameter search and followed fine-
tuning procedures and guidelines from relevant literature.

We provide detailed information about the models used for the evaluation, the
hyperparameter setup, and the fine-tuning procedure in the official project
repository.1

###  4.2 Results on Intent Classification

Table 3: Accuracy and F1 Macro results of E2E-SLU models and adapted variants (FT: ✓). Best result per splitting configuration in bold. Split | Model | # params | FT | Accuracy | F1
---|---|---|---|---|---
massive | XLSR-128 | 300M |  | 76.16 | 76.11
XLSR-128 | 1B |  | 77.07 | 77.08
XLSR-53 | 300M | ✓ | 81.34 | 81.31
XLSR-128 | 1B | ✓ | 83.39 | 83.25
speaker | XLSR-128 | 300M |  | 73.42 | 73.04
XLSR-128 | 1B |  | 79.11 | 79.08
XLSR-53 | 300M | ✓ | 83.69 | 83.62
XLSR-128 | 1B | ✓ | 84.18 | 84.05
noisy | XLSR-128 | 300M |  | 78.29 | 78.21
XLSR-128 | 1B |  | 76.48 | 76.06
XLSR-53 | 300M | ✓ | 81.01 | 80.94
XLSR-128 | 1B | ✓ | 82.20 | 82.43

E2E SLU. Table 3 reports the result of speech SLU models on the ITALIC
dataset. As expected, adaptation to Italian via ASR fine-tuning yields better
models, with the adapted XLSR-128 1B model being the best across all splitting
configurations. Relative to the non-adapted version, XLSR-128 1B achieved
+6.17 and +6.37 F1 points on the massive and noisy configurations,
respectively. Except for one case (XLSR-128, noisy), the results also suggest
that larger models yield better results, although not as much as adapting
models to Italian.

Notably, all models, except XLSR-128 300M, achieved higher performance on the
challenging speaker configuration compared to massive, indicating their
generalizability to different speakers and their ability to handle variations
in speaking styles and accents. As expected, all models showed lower
performance on the noisy subset, with the only exception of the XLSR-128 300M
model. This finding highlights the impact of recording conditions on model
performance, motivating the need for more resources and training procedures
that closely match real-world scenarios.

Table 4: Accuracy and F1 Macro results of text NLU models with multilingual (PT: M) and monolingual (PT: I) pretraining for the massive configuration. Best result in bold. Model | # params | PT | Accuracy | F1
---|---|---|---|---
BART | 611M | M | 87.16 | 83.53
BERT | 167M | M | 86.21 | 82.93
BART | 141M | I | 86.65 | 83.82
BERT | 110M | I | 88.43 | 85.57

NLU. Table 4 reports the results of addressing IC with text NLU models on the
massive splitting configuration. We do not test text models on speaker and
noisy as these configurations are specifically customized for speech models
and tasks.

Of particular interest is the performance of the Italian pre-trained BERT
model, which, despite having fewer parameters, exhibits superior performance
compared to the BART and BERT models pre-trained on multilingual data.
Utilizing Italian data can yield valuable improvements in the performance of
NLU models, even though the performance gap between the models is less
pronounced. Our study demonstrates that, particularly for encoder-based models
such as BERT, smaller models pre-trained on Italian data can achieve
comparable or even superior performance compared to their larger multilingual
counterparts.

Since two human inspectors have validated all ITALIC samples, we can safely
assume that voice recordings closely match the text transcripts. We can then
impute any difference in performance across modalities, i.e., speech and text,
to the inherent difficulty of learning from the two kinds of raw data.
Interestingly, we note that the best text model, monolingual BERT 110M,
achieves +2.32 F1 points, marking a gap despite having x9 fewer parameters. We
can draw similar conclusions comparing other pairs of speech and text models.
These findings underscore the importance of a dedicated dataset for SLU tasks
for better interpretation of the spoken language.

###  4.3 Results on Automatic Speech Recognition

Table 5: WER and CER results of Whisper models in a zero-shot setup (S: ZS) and adapted variants (S: FT). Best result per splitting configuration in bold. Split | Model | # params | S | WER | CER
---|---|---|---|---|---
massive | large | 1.5B | ZS | 11.46 | 5.01
small | 244M | FT | 4.82 | 1.49
medium | 769M | FT | 3.41 | 0.92
large | 1.5B | FT | 3.06 | 0.82
speaker | large | 1.5B | ZS | 8.65 | 3.93
small | 244M | FT | 3.81 | 0.99
medium | 769M | FT | 2.92 | 0.70
large | 1.5B | FT | 2.74 | 0.61
noisy | large | 1.5B | ZS | 15.41 | 7.67
small | 244M | FT | 8.46 | 2.95
medium | 769M | FT | 5.83 | 1.92
large | 1.5B | FT | 5.29 | 1.70

Although the ITALIC dataset was not specifically designed for ASR, its various
speaker and recording conditions make it a valuable resource for analyzing the
performance of Italian ASR models. For this task, we use the large Whisper
model [17] in zero-shot settings and its fine-tuned version for Italian ASR in
three different sizes: small, medium, and large. The results are presented in
Table 5.

Our investigation reveals that all the evaluated models performed well, with
low WER and CER scores (the estimated human WER is approximately 4% [19]).
Whisper large is best across the board, further highlighting the importance of
model size. As expected, there is a clear performance degradation on the noisy
configuration, more marked for smaller models.

Applying Whisper to zero-shot speech recognition yields the worst performance
on all splits, with a gap from the fine-tuned variant (Whisper large, S: FT)
largest in noisy and smallest in speaker. In absolute terms, it achieves a WER
of 8.65 on the speaker configuration and a much worse 15.41 on noisy. These
results are sensibly worse than standard Italian benchmarks such as Mozilla CV
[3] (WER: 7.1) or Google Fleur [7] (WER: 4.0).

These findings prove ITALIC challenging for current state-of-the-art SLU and
NLU models and underscore its importance as a novel Italian resource. With
accurate recordings of real-world human-to-voice assistant interactions and
rich annotations, ITALIC paves the way for new research and development of
Italian models.

##  5 Conclusions

We presented ITALIC, the first large-scale Italian audio dataset specifically
designed for intent classification. The collection is comprehensive of text
transcripts, recordings, and additional metadata about the speaker and the
recording channel. We evaluated the performance of current state-of-the-art
speech and text models on the intent classification and automatic speech
recognition tasks, demonstrating the impact of model selection and pretraining
on performance. We release the dataset, annotation schema, and code to foster
future research in this area.

Our future work includes expanding the ITALIC dataset to enhance its diversity
and representativeness of the Italian language and exploring new tasks that
can be tackled with this dataset. Future enhancements will involve adding more
speakers with diverse backgrounds, including non-native speakers, and further
extending the dataset to address any potential gap in coverage. We also aim to
develop a large-scale multilingual data collection platform to facilitate the
creation of similar datasets in other languages.

##  6 Limitations

The ITALIC dataset is valuable for evaluating models for the Italian SLU and
ASR tasks. However, some limitations must be taken into account when
interpreting the results. While the dataset includes recordings from a wide
range of Italian regions, it only partially represents all dialects and
linguistic varieties. Additionally, the dataset is mainly composed of
recordings from native Italian speakers, which may not be representative of
scenarios where the user of a voice assistant has a non-native accent. We
envisioned the dataset to represent a broad spectrum of individuals, from non-
binary to speakers with speech impairments—however, only a limited number of
volunteers identified as such. We will promote future dataset releases
capturing more speech nuances. Finally, the dataset only includes one
recording per sentence. Including multiple recordings of the same sentence by
different speakers would allow a more comprehensive evaluation of model
performance, which is of key importance for SLU domain [20].

Overall, the ITALIC dataset provides a strong foundation for evaluating
Italian SLU and ASR models; addressing these limitations will enable more
comprehensive evaluations and further advances in these fields.

##  7 Acknowledgments

This project is a joint effort of members of the ``Risorse per la Lingua
Italiana'' open community. We would like to thank all the crowd workers who
participated to our campaign and the reviewers for their helpful comments.
This work is partially supported by the FAIR - Future Artificial Intelligence
Research (PE00000013) and the spoke ``FutureHPC & BigData'' of the ICSC -
``National Centre for HPC, Big Data and Quantum Computing'' funded by the
European Union Next-GenerationEU, and SmartData@PoliTO center on Big Data and
Data Science. This manuscript reflects only the authors views and opinions,
neither the European Union nor the European Commission can be considered
responsible for them.

## References

  * [1] L. Lugosch, M. Ravanelli, P. Ignoto, V. S. Tomar, and Y. Bengio, ``Speech model pre-training for end-to-end spoken language understanding,'' in _Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019_ , G. Kubin and Z. Kacic, Eds. ISCA, 2019, pp. 814–818. [Online]. Available: https://doi.org/10.21437/Interspeech.2019-2396
  * [2] E. Bastianelli, A. Vanzo, P. Swietojanski, and V. Rieser, ``SLURP: A spoken language understanding resource package,'' in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , B. Webber, T. Cohn, Y. He, and Y. Liu, Eds. Association for Computational Linguistics, 2020, pp. 7252–7262. [Online]. Available: https://doi.org/10.18653/v1/2020.emnlp-main.588
  * [3] R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, ``Common voice: A massively-multilingual speech corpus,'' in _Proceedings of the Twelfth Language Resources and Evaluation Conference_. Marseille, France: European Language Resources Association, May 2020, pp. 4218–4222. [Online]. Available: https://aclanthology.org/2020.lrec-1.520
  * [4] V. Bellomaria, G. Castellucci, A. Favalli, and R. Romagnoli, ``Almawave-slu: A new dataset for SLU in italian,'' _CoRR_ , vol. abs/1907.07526, 2019\. [Online]. Available: http://arxiv.org/abs/1907.07526
  * [5] J. FitzGerald, C. Hench, C. Peris, S. Mackie, K. Rottmann, A. Sanchez, A. Nash, L. Urbach, V. Kakarala, R. Singh, S. Ranganath, L. Crist, M. Britan, W. Leeuwis, G. Tur, and P. Natarajan, ``Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages,'' 2022. [Online]. Available: https://arxiv.org/abs/2204.08582
  * [6] A. Coucke, A. Saade, A. Ball, T. Bluche, A. Caulier, D. Leroy, C. Doumouro, T. Gisselbrecht, F. Caltagirone, T. Lavril, M. Primet, and J. Dureau, ``Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces,'' _CoRR_ , vol. abs/1805.10190, 2018. [Online]. Available: http://arxiv.org/abs/1805.10190
  * [7] A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna, ``Fleurs: Few-shot learning evaluation of universal representations of speech,'' _2022 IEEE Spoken Language Technology Workshop (SLT)_ , pp. 798–805, 2022.
  * [8] G. Costantini, I. Iaderola, A. Paoloni, and M. Todisco, ``EMOVO corpus: an Italian emotional speech database,'' in _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)_. Reykjavik, Iceland: European Language Resources Association (ELRA), May 2014, pp. 3501–3504. [Online]. Available: http://www.lrec-conf.org/proceedings/lrec2014/pdf/591_Paper.pdf
  * [9] M. Marini, M. Viganò, M. Corbo, M. Zettin, G. Simoncini, B. Fattori, C. D’Anna, M. Donati, and L. Fanucci, ``Idea: An italian dysarthric speech database,'' in _2021 IEEE Spoken Language Technology Workshop (SLT)_ , 2021, pp. 1086–1093.
  * [10] M. Maiden and M. Parry, _The dialects of Italy_. Routledge, 2006.
  * [11] A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, ``Unsupervised Cross-Lingual Representation Learning for Speech Recognition,'' in _Proc. Interspeech 2021_ , 2021, pp. 2426–2430.
  * [12] A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, ``XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale,'' in _Proc. Interspeech 2022_ , 2022, pp. 2278–2282.
  * [13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ``BERT: Pre-training of deep bidirectional transformers for language understanding,'' in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423
  * [14] Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer, ``Multilingual denoising pre-training for neural machine translation,'' _Transactions of the Association for Computational Linguistics_ , vol. 8, pp. 726–742, 2020.
  * [15] S. Schweter, ``Italian bert and electra models,'' Nov. 2020. [Online]. Available: https://doi.org/10.5281/zenodo.4263142
  * [16] M. La Quatra and L. Cagliero, ``Bart-it: An efficient sequence-to-sequence model for italian text summarization,'' _Future Internet_ , vol. 15, no. 1, 2023. [Online]. Available: https://www.mdpi.com/1999-5903/15/1/15
  * [17] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, ``Robust speech recognition via large-scale weak supervision,'' _arXiv preprint arXiv:2212.04356_ , 2022.
  * [18] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush, ``Transformers: State-of-the-art natural language processing,'' in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. Online: Association for Computational Linguistics, Oct. 2020, pp. 38–45. [Online]. Available: https://aclanthology.org/2020.emnlp-demos.6
  * [19] R. P. Lippmann, ``Speech recognition by machines and humans,'' _Speech Communication_ , vol. 22, no. 1, pp. 1–15, 1997. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167639397000216
  * [20] A. Koudounas, E. Pastor, G. Attanasio, V. Mazzia, M. Giollo, T. Gueudre, L. Cagliero, L. de Alfaro, E. Baralis, and D. Amberti, ``Exploring subgroup performance in end-to-end speech models,'' in _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023, pp. 1–5.


# Learning to Detect with Constant False Alarm Rate

Tzvi Diskin, Uri Okun and Ami Wiesel

(February 2022)

###### Abstract

We consider the use of machine learning for hypothesis testing with an
emphasis on target detection. Classical model-based solutions rely on
comparing likelihoods. These are sensitive to imperfect models and are often
computationally expensive. In contrast, data-driven machine learning is often
more robust and yields classifiers with fixed computational complexity.
Learned detectors usually provide high accuracy with low complexity but do not
have a constant false alarm rate (CFAR) as required in many applications. To
close this gap, we propose to add a term to the loss function that promotes
similar distributions of the detector under any null hypothesis scenario.
Experiments show that our approach leads to near CFAR detectors with similar
accuracy as their competitors.

###### Index Terms:

hypothesis testing, deep learning

##  I Introduction

The deep learning revolution has led many to apply machine learning methods to
classical problems in all fields including statistics. Examples range from
estimation [1, 2, 3, 4, 5, 6] to detection [7, 8, 9, 10, 11, 12]. Deep
learning is a promising approach for deriving high accuracy and low complexity
alternatives when classical solutions are intractable. To facilitate this
switch, we must ensure that the learned solutions are accurate but also
satisfy other classical requirements. In this paper, we focus on learning
detectors for composite hypothesis testing. We claim that current solutions
deliver on their promises but lack the Constant False Alarm Rate (CFAR)
requirement which is critical in many applications. To close this gap, we
provide a framework for learning accurate CFAR detectors.

Detection theory begins with simple hypothesis testing where a detector must
decide between two fully specified distributions. The classical solution is
the Likelihood Ratio Test (LRT) which is optimal in terms of maximizing the
detection probability subject to a false alarm constraint. Composite
hypothesis testing is a more challenging setting where the hypotheses involve
unknown deterministic parameters. A CFAR detector is invariant to these
parameters and has identical false alarm probabilities as long as no target is
present. This allows the user to set the thresholds a priori. A popular
approach is the Generalized Likelihood Ratio Test (GLRT) which can be
interpreted as estimating the unknown parameters and then plugging them into a
standard LRT. GLRT performs well when the number of unknowns is relatively
small and is asymptotically CFAR. On the other hand, it is generally sub-
optimal in finite sample settings and may be computationally expensive.

Data-driven classifiers are the machine learning counterpart to model-based
detectors. In simple settings with no unknown deterministic parameters, it is
well known that the optimal Bayes classifier converges to the LRT with a
specific false alarm rate. More advanced classifiers can also maximize the
cumulative detection rate over a wide range of false alarms, also known as the
area under the curve (AUC) [13, 14].

There is a growing body of works on using machine learning for target
detection. In the context of hyperspectral imagery, [11] introduced the use of
support vector machines (SVM). Deep neural networks (DNN) were proposed in
[8]. An important ingredient of these works is the use of an artificial
training set where real noise data is augmented with synthetically planted
targets. In the context of radar detection, SVMs were considered in [10].
Specific CFAR radar detectors were developed by relying on CFAR features [15,
16, 17].

The main contribution of this paper is a framework for learning CFAR detectors
denoted by CFARnet. The framework is general purpose and can be applied to
arbitrary composite hypothesis testing problems. CFARnet is based on adding a
penalty function to the optimization which decreases the distances between the
distributions under different parameter values. To optimize CFARnet we rely on
empirical and differentiable distances that have recently become popular in
unsupervised deep learning. Our numerical experiments, show that the resulting
networks are approximately CFAR while paying a negligible price in terms of
accuracy.

The CFAR notion is closely related to the topics of “fairness” and “out of
distribution (OOD)” generalization which have recently attracted considerable
attention in the machine learning literature, e.g., [18, 19]. CFAR can be
interpreted as a fairness property with respect to the unknown deterministic
parameters. The closest work is [20] which also enforces “equalized odds”
using a distance between distributions. A main difference is that CFAR is a
one-sided fairness property and requires equal rates only in the null
hypothesis. Algorithmically, [20] compares the high dimensional joint
distribution of the predictions and the unknown parameters, whereas we only
consider the scalar distribution of the predictions. This makes our method
significantly cheaper in terms of computational complexity. Indeed, we rely on
a simple kernel based distance and do not require sophisticated adversarial
networks.

##  II Background on statistical distances

We begin with a brief background on statistical distances.

######  Definition 1.

Let $X\sim p(X)$ and $Y\sim p(Y)$ be two random variables. A statistical
distance $d(X;Y)$ is a function that satisfies $d(X;Y)\geq 0$ with equality if
and only if $p(X)=p(Y)$.

A distance which has recently become popular is the Maximum Mean Discrepancy
(MMD) [21]:

$d_{\rm{MMD}}(X;Y)=\mathbf{E}[k(X,X^{\prime})]\\\
+\mathbf{E}[k(Y,Y^{\prime})]-2\mathbf{E}[k(X,Y)]$ (1)

where $X$ and $X^{\prime}$ are independent and identically distributed
(i.i.d.), and so are $Y$ and $Y^{\prime}$. The function $k(\cdot,\cdot)$ is a
characteristic kernel over a reproducing kernel Hilbert space, e.g., the
Gaussian Radial Basis Function (RBF).

Recent advances in deep generative models allow us to optimize distances as
MMD in an empirical and differentiable manner. For this purpose, we need to
represent each distribution using a small dataset. Let $\\{X_{i}\\}_{i=1}^{N}$
and $\\{Y_{i}\\}_{i=1}^{N}$ be i.i.d. realizations of $X$ and $Y$,
respectively. Then, an empirical version of the MMD can be used where the
expetations in (1) are replaced by their empirical estimates. More advanced
metrics can be obtained using the tools of generative adversarial networks
(GANs). In this paper, we only deal with distances between scalar random
variables and simple MMD distances suffice.

##  III Problem formulation

We consider a binary hypothesis test. Let $\boldsymbol{x}$ be an observed
random vector whose distribution $p(\boldsymbol{x};\mathbf{z})$ depends on an
unknown deterministic parameter $\mathbf{z}$. The value of $\mathbf{z}$
defines two possible hypotheses

$\displaystyle y=0:\quad\mathbf{z}\in{\mathcal{Z}}_{0}$ $\displaystyle
y=1:\quad\mathbf{z}\in{\mathcal{Z}}_{1}.$ (2)

The goal is to design a detector $\hat{y}(\boldsymbol{x})\in\\{0,1\\}$ as a
function of $\boldsymbol{x}$ that will identify the true hypothesis
$y\in\\{0,1\\}$. Performance is measured in terms of probability of correct
detection, also known as True Positive Rate (TPR):

$\displaystyle P_{\rm TPR}(\mathbf{z})=\;$ $\displaystyle
P(\hat{y}(\boldsymbol{x})=1;y=1)$ (3)

and probability of false alarm, also known as False Positive Rate (FPR):

$\displaystyle P_{\rm FPR}(\mathbf{z})=P(\hat{y}(\boldsymbol{x})=1;y=0)$ (4)

In practice, the user typically provides a false alarm constraint $P_{\rm
FPR}\leq\alpha$ that must be satisfied and the goal is to maximize $P_{\rm
TPR}$.

It is standard to consider detectors of the form

$\hat{y}\left(\boldsymbol{x}\right)=\begin{cases}0&T\left(\boldsymbol{x}\right)<\gamma\\\
1&T\left(\boldsymbol{x}\right)\geq\gamma\end{cases}$ (5)

where $T(\boldsymbol{x})$ is a function of the measurements and $\gamma$ is a
threshold. This structure allows users to tune $P_{\rm FPR}$ by adjusting the
threshold. Performance is usually visualized using the Receiver Operating
Characteristic (ROC) which plots the TPR as a function of the FPR. In signal
processing applications, users are often interested in a region of very low
FPRs, e.g., $10^{-1}-10^{-3}$ and the goal is to maximize the TPR
probabilities in this area.

A main challenge in detection theory are the unknown parameters under the null
hypothesis $y=0$. The false alarm probability (FPR) is generally a function of
these parameters and cannot be controlled without their knowledge. Therefore,
it is often preferable to restrict the attention to CFAR detectors.

######  Definition 2.

A detector $T(\boldsymbol{x})$ is CFAR if its distribution is invariant to the
value of $\mathbf{z}\in{\mathcal{Z}}_{0}$.

As we will review below, many classical detectors are CFAR or asymptotically
CFAR. With the growing trend of switching to machine learning, the goal of
this paper is to introduce a framework for learning CFAR detectors.

##  IV Classical Likelihood based Detectors

In this section, we provide a short background on classical detectors based on
likelihood ratios. In the simple case where
${\mathcal{Z}}_{0}=\\{\mathbf{z}_{0}\\}$ and
${\mathcal{Z}}_{1}=\\{\mathbf{z}_{1}\\}$ are singletons, hypothesis testing
has an optimal solution known as the Likelihood Ratio Test (LRT) due to Neyman
Pearson [22, p. 65]. LRT theory states that the optimal detector for
maximizing detection subject to a given false alarm probability is:

$\displaystyle T_{\rm LRT}(\boldsymbol{x})$
$\displaystyle=2{\rm{log}}\frac{p(\boldsymbol{x};\mathbf{z}=\mathbf{z}_{1})}{p(\boldsymbol{x};\mathbf{z}=\mathbf{z}_{0})}$
(6)

and the threshold $\gamma$ is chosen to satisfy the false alarm (FPR)
constraint.

The more realistic scenario is composite hypotheses testing where one or both
of the hypotheses involve unknown parameters and there is no simple and
optimal solution. A popular heuristic is the Generalized Likelihood Ratio Test
(GLRT) that estimates the unknown parameters using the Maximum Likelihood
technique and plugs them into the LRT detector:

$\displaystyle T_{\rm GLRT}(\boldsymbol{x})$
$\displaystyle=2{\rm{log}}\frac{\max_{\mathbf{z}\in{\mathcal{Z}}_{1}}p(\boldsymbol{x};\mathbf{z})}{\max_{\mathbf{z}\in{\mathcal{Z}}_{0}}p(\boldsymbol{x};\mathbf{z})}$
(7)

Setting the threshold to ensure a fixed $P_{\rm{FPR}}$ is non trivial.
Fortunately, under regular conditions, GLRT is asymptotically CFAR and thus
its threshold can be set for all values of the unknown parameters
simultaneously.

GLRT is probably the most popular solution to composite hypothesis testing. It
gives a simple recipe that performs well under asymptotic conditions. Its main
downsides are that it is sensitive to deviations from its theoretical model,
it is generally sub-optimal under finite sample settings and that it may be
computationally expensive. Both the nominator and denominator of the GLRT
involve optimization problems that may be large scale, non-linear and non-
convex. Therefore, there is an ongoing search for robust and low cost
alternatives.

##  V Machine Learning for Detection

In this section, we explain the use of machine learning for hypothesis
testing. The starting point to any data-driven learning is a training set.
Hypothesis testing relies on a probabilistic model
$p(\boldsymbol{x};\mathbf{z})$ and we need to use this model in order to
generate a synthetic dataset. Hybrid settings involving a mixture of real and
artificial samples are also common. For example, it is standard to plant
synthetic targets on real noise samples [11]. Due to space limitations, we
leave these hybrid extensions for the journal version of this paper.

The main challenge in generating data is that $y\in\\{0,1\\}$ and $\mathbf{z}$
are not random variables but deterministic parameters without any prior
distribution. A natural heuristic is to assume uniform fake priors, e.g.,
choose half of labels as $y=0$ and half as $y=1$ and assume that $\mathbf{z}$
is uniformly distributed on its corresponding domains. For each $y_{i}$ and
$\mathbf{z}_{i}$, we then generate a measurement $\boldsymbol{x}_{i}$
according to the true $p\left(\boldsymbol{x};{\mathbf{z}_{i}}\right)$ and
obtain a synthetic dataset

${\mathcal{D}}_{N}=\\{\boldsymbol{x}_{i},\mathbf{z}_{i},y_{i}\\}_{i=1}^{N}.$
(8)

Next, a class of possible detectors ${\mathcal{T}}$ is chosen in order to
tradeoff expressive power with computational complexity in test time. The
class is usually a fixed differentiable neural network architecture. In our
context, it also makes sense to reuse existing ingredients from classical
detector as non-linear features or internal sub-blocks.

Finally, the learned detector is defined as the minimizer of an empirical loss
function

$\min_{\hat{{T}}\in{\mathcal{T}}}\frac{1}{N}\sum_{i=1}^{N}L(\hat{T}(\boldsymbol{x}_{i}),y_{i}).$
(9)

where $L(\cdot;\cdot)$ is a classification loss function. Ideally, we would
like to minimize the zero-one loss which corresponds to the average
probability of error. Practically, for efficient optimization, a smooth and
convex surrogate loss, as the hinge or cross entropy functions, is preferable.
The overall procedure for learning a detector is summarized in Algorithm 1.

Algorithm 1 Fitting detectors for $p(\boldsymbol{x};\mathbf{z})$

  * •

Choose $p^{\rm{fake}}(y)$.

  * •

Choose $p^{\rm{fake}}(\mathbf{z};y)$.

  * •

For each $i=1,\cdots,N$:

Generate $y_{i}$.

Generate $\mathbf{z}_{i}$ given $y_{i}$.

Generate $\boldsymbol{x}_{i}$ given $\mathbf{z}_{i}$.

  * •

Solve

$\min_{\hat{{T}}\in{\mathcal{T}}}\frac{1}{N}\sum_{i=1}^{N}L(\hat{T}(\boldsymbol{x}_{i}),y_{i})$.

Learned detectors have strong theoretical guarantees in simple testing
problems where $y$ uniquely defines $\mathbf{z}$. With a sufficiently
expressive class of detectors and a large enough training set, minimizing the
zero-one loss leads to the Bayes optimal detector. It is identical to the LRT
with the threshold

$\gamma=\frac{p^{\rm fake}(y=0)}{p^{\rm fake}(y=1)}.$ (10)

Minimizers of Bayes risk consistent surrogates, as the hinge loss, also
asymptotically converge to this LRT [23]. Thus, any simple LRT can be
approximated by tuning $p^{\rm fake}(y)$ to achieve a desired $P_{\rm FPR}$.

To approximate LRT for a wide range of $P_{\rm FPR}$ it is preferable to
minimize the AUC. There are many works in the machine learning literature on
fitting classifiers with this goal through an increased computational
complexity [14, 13]. There are also variants that focus on partial regimes in
the ROC [24].

To evaluate the machine learning approach to detection we turned to numerical
experiments. We completed a wide range of simulations in different target
detection scenarios comparing the classical solutions with different learned
detectors based on various loss functions. We examined both simple and
composite settings, with and without hidden variables. The conclusions were
that learned detectors usually perform similarly to the their corresponding
(G)LRTs. In some experiments, some detectors had a small advantage but the
differences were not uniform nor significant. Our conclusion is therefore that
simple classifiers with wide priors on the unknown parameters are practically
sufficient for achieving an optimal ROC in most problems. On the negative
side, most of the experiments showed that the learned detectors were not CFAR
and resulted in significantly different false alarm rates for different values
of $\mathbf{z}\in{\mathcal{Z}}_{0}$. To close this gap, in the next section we
propose a framework for learning CFAR detectors.

##  VI Learning CFAR detectors

Figure 1: Performance graphs in terms of FPR, TPR and thresholds for
$\sigma\in\\{0,1\\}$. Top row is with Gaussian noise and indeed the Gaussian
GLRT is best both in terms of ROC and CFAR. NET succeeds in achieving its ROC
but is not CFAR. CFAR-NET is both accurate and CFAR. Bottom row is with non-
Gaussian noise and the learned detectors beat the Gaussian GLRT in accuracy.
CFAR-NET is near CFAR with a slight decrease in accuracy.

In this section, we introduce CFARnet, a framework for designing CFAR
detectors using machine learning. CFARnet introduces two modifications to
Algorithm 1. First, we augment the classification loss with a penalty function
that ensures similar distributions for all values of
$\mathbf{z}\in{\mathcal{H}}_{0}$. Second, in order to optimize this penalty,
we generate multiple $\\{\boldsymbol{x}_{ij}\\}_{j=1}^{M}$ for each
$\mathbf{z}_{i}$ and use them to empirically approximate the CFAR penalty.

The main idea is adding a penalty to the objective function that promotes a
CFAR. The penalty is defined as a sum of distance functions between the
distributions of $\hat{T}$ under different values of $\mathbf{z}$:

$\displaystyle
R(\hat{T})=\sum_{\mathbf{z},\tilde{\mathbf{z}}\in{\mathcal{Z}}_{0}}d\left(\hat{T}(\boldsymbol{x});\hat{T}(\tilde{\boldsymbol{x}})\right)$
(11)

where $d(\cdot;\cdot)$ is any statistical distance as detailed in Section II.
The distributions of $\hat{T}(\boldsymbol{x})$ and
$\hat{T}(\tilde{\boldsymbol{x}})$ are implicitly defined through

$\displaystyle\boldsymbol{x}\sim p(\boldsymbol{x};\mathbf{z})$
$\displaystyle\tilde{\boldsymbol{x}}\sim
p(\boldsymbol{x};\tilde{\mathbf{z}}).$ (12)

Clearly, any CFAR test must satisfy $R(\hat{T})=0$.

To minimize a loss with a CFAR penalty, we need to represent each distribution
using a small dataset. For each $\mathbf{z}_{i}$, we synthetically generate
multiple observations $\boldsymbol{x}_{ij}$ for $j=1,\cdots,M$. Similarly, for
each $\tilde{\mathbf{z}}_{i}$ we compute multiple
$\tilde{\boldsymbol{x}}_{ij}$. We then plug these into the empirical distances
as detailed in Sec. II:

$\displaystyle\hat{R}(\hat{T})=\sum_{i,\tilde{i}}\hat{d}\left(\\{\hat{T}(\boldsymbol{x}_{ij}\\}_{j=1}^{M});\\{\hat{T}(\tilde{\boldsymbol{x}}_{\tilde{i}j})\\}_{j=1}^{M}\right)$
(13)

Altogether, we recommend to use a hyper-parameter $\alpha>0$ that trades off
the importance of the classification accuracy versus the CFAR penalty. The
overall procedure for learning a CFAR detector is summarized in Algorithm 2.

Algorithm 2 Fitting CFAR detectors for $p(\boldsymbol{x};\mathbf{z})$

  * •

Choose $p^{\rm{fake}}(y)$.

  * •

Choose $p^{\rm{fake}}(\mathbf{z};y)$.

  * •

For each $i=1,\cdots,N$:

Generate $y_{i}$.

Generate $\mathbf{z}_{i}$ given $y_{i}$.

For $j=1,\cdots,M$:

Generate $\boldsymbol{x}_{ij}$ given $\mathbf{z}_{i}$.

  * •

Solve

$\min_{\hat{{T}}\in{\mathcal{T}}}\frac{1}{N}\sum_{i=1}^{N}L(\hat{T}(\boldsymbol{x}_{i}),y_{i})+\alpha\hat{R}(\hat{T})$.

##  VII Numerical experiments

In this section, we demonstrate the advantages of CFARnet via numerical
experiments. We consider a basic yet realistic target detection scenario in
which both the target amplitude and the noise scaling are unknown:

$\displaystyle\boldsymbol{x}=A\mathbf{1}+\sigma\boldsymbol{n}$ (14)

where $\mathbf{1}$ is a target vector of ones, $\boldsymbol{n}$ is a random
vector with i.i.d. noise variables, and $\mathbf{z}=[A,\sigma]$ are
deterministic unknown parameters

$\displaystyle-1\leq A\leq 1,\qquad 0.5\leq\sigma\leq 1.$ (15)

The goal is to decide between

$\displaystyle y=0:\quad A=0$ $\displaystyle y=1:\quad A\neq 0.$ (16)

We compare three detectors:

  * •

GLRT: assuming Gaussian noise, the classical GLRT has a simple closed form
solution
$T_{\rm{GLRT}}=(\boldsymbol{x}^{T}\mathbf{1})^{2}/(\boldsymbol{x}^{T}\boldsymbol{x})$
and is known to be CFAR.

  * •

NET: a learned neural network as in Algorithm 1. We choose a uniform fake
prior for the unknown parameters in (15). The architecture is based on four
non-linear features: the sample mean of $\boldsymbol{x}$, its sample variance
and robust versions of the two based on the median. These features are passed
through a fully connected neural network, and are optimized to minimize a
cross entropy loss using PyTorch.

  * •

CFAR-NET: a learned neural network as in Algorithm 2. Architecture and
implementation are all identical to NET. Loss is cross entropy with an MMD
CFAR penalty with parameter $\alpha=1$.

The first experiment considers Gaussian noise. In the first row of Fig. 1, we
plot the two ROCs for different values of $\sigma$. To examine the CFAR
property, we also plot the FPRs under different parameters. As expected, it is
easy to see that the Gaussian GLRT performs well and is CFAR. NET provides
similar accuracy as illustrated in its ROC but is non-CFAR and results in
significantly different FPR when we change $\sigma$. On the other hand, CFAR-
NET is both accurate and near CFAR.

The second experiment is more challenging and considers non-Gaussian noise.
The setting is identical as before except for the noise distribution

$\displaystyle p(n_{k})=(1-\epsilon)N(0,1)+\epsilon N(0,100)$ (17)

where $\epsilon=0.1$. There is no simple GLRT for this setting. The results
are provided in the second row of Fig. 1. In this case, the Gaussian GLRT is
no longer optimal and the two learned detectors provide a significantly better
ROC. In terms of CFAR, GLRT is still invariant to the nuisance parameter, but
the FPR of NET is dependent on its value. As promised, CFAR-NET is both
accurate and CFAR.

## Acknowledgment

The authors would like to thank Yoav Wald and Yiftach Beer for fruitful
discussions and helpful insights in the initial stages of this project.

## References

  * [1] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE transactions on pattern analysis and machine intelligence, vol. 38, no. 2, pp. 295–307, 2015.
  * [2] G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis, and R. Willett, “Deep learning techniques for inverse problems in imaging,” IEEE Journal on Selected Areas in Information Theory, vol. 1, no. 1, pp. 39–56, 2020.
  * [3] L. Gabrielli, S. Tomassetti, S. Squartini, and C. Zinato, “Introducing deep machine learning for parameter estimation in physical modelling,” in Proceedings of the 20th International Conference on Digital Audio Effects, 2017\.
  * [4] V. Dua, “An artificial neural network approximation based decomposition approach for parameter estimation of system of ordinary differential equations,” Computers & chemical engineering, vol. 35, no. 3, 2011.
  * [5] R. Dreifuerst and R. W. Heath Jr, “SignalNet: A low resolution sinusoid decomposition and estimation network,” arXiv preprint arXiv:2106.05490, 2021.
  * [6] T. Diskin, Y. C. Eldar, and A. Wiesel, “Learning to estimate without bias,” Preprint arXiv:2110.12403, 2021.
  * [7] N. Samuel, T. Diskin, and A. Wiesel, “Learning to detect,” IEEE Transactions on Signal Processing, vol. 67, no. 10, pp. 2554–2564, 2019.
  * [8] L. Girard, V. Roy, P. Giguère, and T. Eude, “Deep neural network training using synthetic signatures for rare target detection in SWIR hyperspectral imagery,” in 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, pp. 4420–4423, IEEE, 2021.
  * [9] A. Brighente, F. Formaggio, G. M. Di Nunzio, and S. Tomasin, “Machine learning for in-region location verification in wireless networks,” IEEE Journal on Selected Areas in Communications, vol. 37, no. 11, pp. 2490–2502, 2019.
  * [10] D. de la Mata-Moya, M. P. Jarabo-Amores, J. M. de Nicolás, and M. Rosa-Zurera, “Approximating the Neyman–Pearson detector with 2C-SVMs. application to radar detection,” Signal Processing, vol. 131, pp. 364–375, 2017.
  * [11] A. Ziemann, M. Kucer, and J. Theiler, “A machine learning approach to hyperspectral detection of solid targets,” in Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXIV, vol. 10644, p. 1064404, International Society for Optics and Photonics, 2018.
  * [12] J. Theiler, S. Matteoli, and A. Ziemann, “Bayesian detection of solid subpixel targets,” in 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, pp. 3213–3216, IEEE, 2021.
  * [13] A. Herschtal and B. Raskutti, “Optimising area under the ROC curve using gradient descent,” in Proceedings of the twenty-first international conference on Machine learning, p. 49, 2004.
  * [14] U. Brefeld, T. Scheffer, et al., “AUC maximizing support vector learning,” in Proceedings of the ICML 2005 workshop on ROC Analysis in Machine Learning, 2005.
  * [15] C.-H. Lin, Y.-C. Lin, Y. Bai, W.-H. Chung, T.-S. Lee, and H. Huttunen, “DL-CFAR: A novel cfar target detection method based on deep learning,” in 2019 IEEE 90th Vehicular Technology Conference (VTC2019-Fall), pp. 1–6, IEEE, 2019.
  * [16] J. Akhtar and K. E. Olsen, “A neural network target detector with partial CA-CFAR supervised training,” in 2018 International Conference on Radar (RADAR), pp. 1–6, IEEE, 2018.
  * [17] J. Akhtar, “Training of neural network target detectors mentored by SO-CFAR,” in 2020 28th European Signal Processing Conference (EUSIPCO), pp. 1522–1526, IEEE, 2021.
  * [18] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz, “Invariant risk minimization,” arXiv preprint arXiv:1907.02893, 2019.
  * [19] Y. Wald, A. Feder, D. Greenfeld, and U. Shalit, “On calibration and out-of-domain generalization,” arXiv preprint arXiv:2102.10395, 2021.
  * [20] Y. Romano, S. Bates, and E. Candes, “Achieving equalized odds by resampling sensitive attributes,” Advances in Neural Information Processing Systems, vol. 33, pp. 361–371, 2020.
  * [21] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola, “A kernel two-sample test,” The Journal of Machine Learning Research, vol. 13, no. 1, pp. 723–773, 2012.
  * [22] S. Kay, Fundamentals of Statistical Signal Processing: Detection theory.  Fundamentals of Statistical Si, Prentice-Hall PTR, 1998.
  * [23] P. L. Bartlett, M. I. Jordan, and J. D. McAuliffe, “Convexity, classification, and risk bounds,” Journal of the American Statistical Association, vol. 101, no. 473, pp. 138–156, 2006.
  * [24] H. Narasimhan and S. Agarwal, “A structural SVM based approach for optimizing partial AUC,” in International Conference on Machine Learning, pp. 516–524, PMLR, 2013.