classifiers in speech spoofing detection. Detailed results of the SOTA
countermeasures are presented in Table XVII.

###  Experimental Analysis of the SOTA countermeasures with a CNN-GRU based
classifier

In this experiment, a CNN-GRU [34] classifier was used to classify the SOTA
countermeasures in order to test their effectiveness against the SS, VC, and
replay attacks, and the results of this experiment are presented in Table
XVIII. The results demonstrate that in the case of the VSDC dataset, the
linear and spectral magnitude-based LFCC and SCMC features achieved a
comparatively similar performance, with an EER of 0.02 and a min T-DCF of
0.05. Due to the versatility of the VSDC dataset, the SCFC features scored an
EER of 0.18 and a min T-DCF of 0.35. In particular, the results from this
experiment proved to be optimal for the VSDC dataset, in comparison with the
GMM, SVM, and CNN-GRU classifiers. Moreover, the SOTA countermeasures achieved
the best result on the CNN-GRU classifiers. In contrast, in the case of
ASVspoof19, LPCC was the best performing countermeasure, with an EER of 0.02
and a min T-DCF of 0.47, while the MFCC and MSRCC countermeasures have the
lowest EERs of 0.22 and 0.18, respectively. Therefore, it was concluded from
this experiment that the performance of the existing SOTA countermeasure
differs in the presence of distinct classifiers and may not function
adequately in the presence of advanced spoofing attempts. In addition, we can
infer that the CNN-GRU classifier proved to be the optimal classifier for both
of the datasets. However, the effectiveness of the SOTA countermeasure across
corpora must still be demonstrated. Therefore, in the next section we provide
a cross-corpus evaluation of the SOTA countermeasures.

TABLE XVIII: Experimental performance analysis of the SOTA countermeasures with a CNN-GRU based classifier. Feature | Train Dataset | Test Dataset | EER | min T-DCF
---|---|---|---|---
LFCC [113] | VSDC | VSDC | 0.02 | 0.05
ASVspoof19 | ASVspoof19 | 0.15 | 0.41
SSFC[80] | VSDC | VSDC | 0.04 | 0.07
ASVspoof19 | ASVspoof19 | 0.17 | 0.46
MFCC [58] | VSDC | VSDC | 0.03 | 0.07
ASVspoof19 | ASVspoof19 | 0.22 | 0.548
SCFC[44] | VSDC | VSDC | 0.18 | 0.35
ASVspoof19 | ASVspoof19 | 0.09 | 0.25
LPCC [91] | VSDC | VSDC | 0.18 | 0.03
ASVspoof19 | ASVspoof19 | 0.02 | 0.47
SCMC[44] | VSDC | VSDC | 0.02 | 0.05
ASVspoof19 | ASVspoof19 | 0.17 | 0.46
MSRCC [80] | VSDC | VSDC | 0.21 | 0.03
ASVspoof19 | ASVspoof19 | 0.22 | 0.50
PSRCC [80] | VSDC | VSDC | 0.23 | 0.03
ASVspoof19 | ASVspoof19 | 0.19 | 0.53
RFCC [73] | VSDC | VSDC | 0.11 | 0.22
ASVspoof19 | ASVspoof19 | 0.04 | 0.11
IMFCC[9] | VSDC | VSDC | 0.03 | 0.09
ASVspoof19 | ASVspoof19 | 0.18 | 0.50
APGDF [67] | VSDC | VSDC | 0.08 | 0.26
ASVspoof19 | ASVspoof19 | 0.14 | 0.37
CQCC [83] | VSDC | VSDC | 0.09 | 0.23
ASVspoof19 | ASVspoof19 | 0.18 | 0.49
GTCC [85] | VSDC | VSDC | 0.04 | 0.09
ASVspoof19 | ASVspoof19 | 0.19 | 0.53
RPS [73] | VSDC | VSDC | 0.04 | 0.09
ASVspoof19 | ASVspoof19 | 0.19 | 0.53

###  Cross-corpus evaluations of SOTA countermeasures

The SOTA countermeasures are compared in this section using cross-corpus
evaluation. Cross-corpus evaluation and generalisation was one of the key
needs mentioned by the in-field researcher [79], which we resolved in this
experiment. A key shortcoming in previous research is the lack of cross-corpus
experiments [69]. Cross-corpus examination demonstrates the countermeasure’s
effectiveness in dealing with unknown aspects of spoofing attacks by removing
prior bias from the model in the form of characteristics shared by a
particular dataset, and allows it to perform a classification based solely on
how the model recognises the unique aspects of spoofing attacks. In addition,
this experiment was performed to determine the comprehensiveness of the
countermeasures to speech samples, including unknown and diverse spoofing
artifacts such as distinct VC algorithms, SSD techniques, and others. During
the experiment, the countermeasures were trained using one dataset and the
testing was done using the speech samples from another. There are some
similarities between the two datasets. Replayed speech samples, for instance,
can be found in the VSDC and ASVspoof19 databases. Initially, the
countermeasures were trained using VSDC data samples and then tested with the
replayed spoof speech samples from the ASVspoof corpus. The SOTA
countermeasure was then trained using ASVspoof and evaluated using VSDC data
samples. This evaluation was performed against three distinct classifiers, and
the results are shown below in Tables XIX, XX, and XXI.

####  IX-E1 Cross-corpus evaluation of the SOTA countermeasures using a GMM
classifier

In this subsection, performance of the SOTA countermeasure was against a GMM
classifier, and the results are reported in Table XIX. The results demonstrate
that when the classifier was trained on the VSDC dataset and tested on the
ASVspoof19 dataset, the countermeasure PSRCC [80] outperformed the SOTA
countermeasures, with an EER of 0.25 and a min TDCF of 0.39. When trained on
the ASVspoof19 dataset and evaluated on the VSDC dataset, RFCC [73] was the
top performing countermeasure, with the lowest EER of 0.39 and the lowest min
T-DCF of 0.27. In contrast, when training on VSDC and testing with the
ASVspoof dataset, the countermeasure SSFC [44] obtained the highest EER of
0.80 and min-tdcf of 0.99. Similarly, the SSFC [80] countermeasure obtained
the highest EER of 0.67. It is observed from the results that the EER of the
SOTA countermeasures varies from 0.25 to 0.80, but min-TDCF ranges from 0.81
to 1.0. Only the PSRCC [80], MSRCC[80], RFCC [73], GTCC [85], and APGDF [67]
countermeasures are able to achieve lower min-TDCF, at 0.39, 0.47, 0.27, 0.38,
and 0.59, respectively. The EER performance metric is used to evaluate the
effectiveness of spoofing countermeasures, whereas the min-TDCF evaluation
parameter can be used to evaluate the efficacy of ASVs. Consequently, based on
these results, we may conclude that the effectiveness of the SOTA
countermeasures will be called into question when employed for ASV
evaluations.

Surprisingly, no specific SOTA countermeasure performed considerably better in
both cross-corpus evaluations. For instance, when trained using ASVspoof19 and
evaluated against the VSDC dataset, the countermeasure MSRCC worked
effectively. In the ASVspoof dataset test, EER increased by up to 11%. Despite
being the worst performer in the standalone tests, PSRCC performed well when
trained on VSDC and evaluated with ASVspoof19. While phase-based spectral
features outperformed the state-of-the-art in cross-corpus evaluation, the
current investigation revealed that when utilizing the GMM classifier, the
performance of the existing countermeasures differed greatly in cross-corpus
evaluation. Moreover, the results show that when the system was unfamiliar
with the testing speech samples, the countermeasures failed to accurately
recognize the spoofed speech samples. However, determining the
countermeasure’s cross-corpus performance using a single classifier is not
practical for fairness and may be biased. Therefore, along with the GMM
classifier, we demonstrate the efficacy of the SOTA countermeasures against
the DL-based CNN and CNN-GRU classifiers, and the results are reported below.

TABLE XIX: Cross corpus evaluation of the SOTA countermeasures with GMM. Features | Training | Testing | EER | min-tdcf
---|---|---|---|---
CQCC [83] | VSDC | ASVspoof19 | 0.40 | 0.81
ASVspoof19 | VSDC | 0.66 | 0.99
LFCC [113] | VSDC | ASV’19 PA | 0.60 | 0.99
ASV’19 PA | VSDC | 0.45 | 0.99
SSFC [80] | VSDC | ASVspoof19 | 0.80 | 0.99
ASVspoof19 | VSDC | 0.67 | 0.99
RPS [73] | VSDC | ASVspoof19 | 0.50 | 0.93
ASVspoof19 | VSDC | 0.40 | 0.94
LPCC [91] | VSDC | ASVspoof19 | 0.41 | 0.89
ASVspoof19 | VSDC | 0.62 | 1.0
PSRCC [80] | VSDC | ASVspoof19 | 0.25 | 0.39
ASVspoof19 | VSDC | 0.67 | 0.99
MSRCC [80] | VSDC | ASVspoof19 | 0.29 | 0.47
ASVspoof19 | VSDC | 0.40 | 0.94
SCFC [44] | VSDC | ASVspoof19 | 0.58 | 0.99
ASVspoof19 | VSDC | 0.48 | 0.99
SCMC[44] | VSDC | ASVspoof19 | 0.60 | 0.97
ASVspoof19 | VSDC | 0.46 | 0.87
MFCC [58] | VSDC | ASVspoof19 | 0.42 | 0.97
ASVspoof19 | VSDC | 0.45 | 0.93
IMFCC [9] | VSDC | ASVspoof19 | 0.45 | 0.99
ASVspoof19 | VSDC | 0.45 | 0.96
RFCC [73] | VSDC | ASVspoof19 | 0.34 | 0.84
ASVspoof19 | VSDC | 0.39 | 0.27
GTCC [85] | VSDC | ASVspoof19 | 0.36 | 0.38
ASVspoof19 | VSDC | 0.56 | 1.0
APGDF [67] | VSDC | ASVspoof19 | 0.30 | 0.59
ASVspoof19 | VSDC | 0.61 | 0.99

####  IX-E2 Cross corpus evaluation of SOTA countermeasures with a CNN based
classifier

In this section, cross-corpus evaluations of the SOTA countermeasures with a
CNN-based classifier are discussed, and the results are presented in Table XX.
The results demonstrate that in VSDC training and ASVspoof19 testing, the
phase-oriented PSRCC [80] countermeasure performed the best and obtained an
EER of 0.37 and a min T-DCF of 0.94. In contrast, in the case of ASVspoof19
training and VSDC testing, the SCMC [44] features achieved the minimum values
for EER and min T-DCF, at 0.35 and 0.96, respectively. Although the
countermeasures and the CNN-based classifier performed marginally better when
trained with VSDC and tested with the ASVspoof19 dataset, several of the
front-end features performed significantly worse. This uncertainty shows that
the performance of these countermeasures remains ambiguous against unseen
attacks without having prior knowledge of the speech sample.

TABLE XX: Cross corpus evaluation of the SOTA countermeasures with a CNN based classifier. Feature | Train Dataset | Test Dataset | EER | min T-DCF
---|---|---|---|---
IMFCC [9] | VSDC | ASVspoof19 | 0.43 | 0.99
ASVspoof19 | VSDC | 0.57 | 1.0
SCMC [44] | VSDC | ASVspoof19 | 0.51 | 0.99
ASVspoof19 | VSDC | 0.35 | 0.96
RFCC [73] | VSDC | ASVspoof19 | 0.40 | 0.96
ASVspoof19 | VSDC | 0.50 | 0.91
RPS [73] | VSDC | ASVspoof19 | 0.68 | 0.99
ASVspoof19 | VSDC | 0.55 | 0.98
SSFC [80] | VSDC | ASVspoof19 | 0.72 | 0.99
ASVspoof19 | VSDC | 0.39 | 0.91
PSRCC [80] | VSDC | ASVspoof19 | 0.37 | 0.94
ASVspoof19 | VSDC | 0.51 | 0.97
MSRCC [80] | VSDC | ASVspoof19 | 0.46 | 0.52
ASVspoof19 | VSDC | 0.47 | 0.93
CQCC [83] | VSDC | ASVspoof19 | 0.48 | 0.98
ASVspoof19 | VSDC | 0.66 | 1.0
LFCC [113] | VSDC | ASVspoof19 | 0.51 | 0.99
ASVspoof19 | VSDC | 0.37 | 0.99
MFCC [58] | VSDC | ASVspoof19 | 0.47 | 0.93
ASVspoof19 | VSDC | 0.52 | 0.94
SCFC[44] | VSDC | ASVspoof19 | 0.53 | 0.99
ASVspoof19 | VSDC | 0.54 | 0.97
LPCC [91] | VSDC | ASVspoof19 | 0.46 | 0.98
ASVspoof19 | VSDC | 0.63 | 1.0
GTCC [85] | VSDC | ASVspoof19 | 0.50 | 0.98
ASVspoof19 | VSDC | 0.41 | 0.91
APGDF [67] | VSDC | ASVspoof19 | 0.44 | 0.99
ASVspoof19 | VSDC | 0.56 | 1.0

####  IX-E3 Cross corpus measurements of the SOTA countermeasures with a CNN-
GRU based classifier

A CNN-GRU classifier was used to evaluate the countermeasures across corpora
in this subsection, and the results are shown in Table XX. In the instance of
VSDC training and ASVspoof19 testing, the APGDF-based countermeasure [67]
obtained an optimal EER of 0.34 and a minimum T-DCF score of 0.82. With an EER
of 0.44, the RFCC [73] and IMFCC [9] were ranked second and third,
respectively. In the case of ASVspoof19 training and VSDC testing, the MFCC-
based countermeasure [58] outperformed the SOTA countermeasures with the
lowest EER of 0.34 and the lowest min T-DCF score of 0.97. The outcomes of
this experiment were similar to the results of the CNN-based cross-corpus
evaluations. The min-TDCF of the SOTA countermeasures was greater than the
countermeasures’ standalone testing. In the instance of ASVspoof19 dataset
training and VSDC testing, the results demonstrated that the majority of the
countermeasures’ minimum T-DCF was at or around 1. These findings demonstrate
the ineffectiveness of existing countermeasures and classifiers against
unknown spoofing attacks. Furthermore, these findings have raised concerns
regarding the effectiveness of existing protections against complicated
contemporary spoofing attacks.

TABLE XXI: Cross corpus evaluation of the countermeasures with a CNN-GRU based classifier. Feature | Train Dataset | Test Dataset | EER | min T-DCF
---|---|---|---|---
LFCC [113] | VSDC | ASVspoof19 | 0.47 | 0.99
ASVspoof19 | VSDC | 0.53 | 0.98
MFCC [58] | VSDC | ASVspoof19 | 0.45 | 0.94
ASVspoof19 | VSDC | 0.34 | 0.97
SCFC[44] | VSDC | ASVspoof19 | 0.49 | 0.99
ASVspoof19 | VSDC | 0.59 | 0.99
LPCC [91] | VSDC | ASVspoof19 | 0.65 | 0.99
ASVspoof19 | VSDC | 0.45 | 1.0
SCMC[44] | VSDC | ASVspoof19 | 0.45 | 0.99
ASVspoof19 | VSDC | 0.65 | 1.0
PSRCC [80] | VSDC | ASVspoof19 | 0.521 | 0.987
ASVspoof19 | VSDC | 0.57 | 0.99
MSRCC [80] | VSDC | ASVspoof19 | 0.49 | 0.99
ASVspoof19 | VSDC | 0.54 | 1.0
RFCC [73] | VSDC | ASVspoof19 | 0.44 | 0.99
ASVspoof19 | VSDC | 0.67 | 1.0
IMFCC [9] | VSDC | ASVspoof19 | 0.44 | 0.99
ASVspoof19 | VSDC | 0.61 | 1.0
APGDF [67] | VSDC | ASVspoof19 | 0.34 | 0.82
ASVspoof19 | VSDC | 0.51 | 0.96
CQCC [83] | VSDC | ASVspoof19 | 0.50 | 0.99
ASVspoof19 | VSDC | 0.43 | 1.0
GTCC [85] | VSDC | ASVspoof19 | 0.49 | 0.99
ASVspoof19 | VSDC | 0.54 | 1.0
SSFC [80] | VSDC | ASVspoof19 | 0.51 | 0.99
ASVspoof19 | VSDC | 0.56 | 1.0
RPS [73] | VSDC | ASVspoof19 | 0.50 | 0.96
ASVspoof19 | VSDC | 0.55 | 0.98

###  Overall performance of the SOTA countermeasures and a way forward

One of the shortfalls in the current research into audio spoof detection is
that cross-corpus evaluations are rarely done, limiting knowledge of the
generalizability of the features and full countermeasure frameworks. To
overcome this knowledge gap, this comparative analysis performs a cross-corpus
evaluation, where one dataset is used for training and another for evaluation,
to show the effectiveness of these features at generalized spoof detection
across different conditions present in the respective datasets.

Based on the results of this study’s experiments, we can infer that the no-
countermeasures from the selected ones were consistent over a wide range of
datasets and classifiers. As shown in the preceding section, several
countermeasures, such as CQCC [83] and RFCC [73], performed better overall in
standalone testing of the countermeasures. However, when the dataset and
classifier were changed, the identical countermeasure failed to perform well.
While PSRCC [80] and MSRCC [80] did not perform much better in standalone
testing, they did significantly better in cross-corpus examination. After
extensive testing, it is extremely difficult to select the best countermeasure
from among those tested because some countermeasures performed well with one
data set but failed with the other, while others performed in reverse with
other classifiers. However, we may infer that classifiers and datasets have a
significant influence on the present countermeasures. In addition, in
standalone corpus testing with four alternative classifiers, the constant Q
based CQCC [83] performed consistently with only a minor decline in
performance across classifiers. The countermeasures PSRCC [80] and APGDF [67]
scored marginally better than each other when the classifier varied and in
cross-corpus evaluation. As a consequence of the extensive testing with
various datasets and classifiers, the effectiveness of the countermeasures was
called into question. It also demonstrates the significance of cross-corpus
and variable classifier evaluation of proposed solutions.

###  Performance Evaluation of recent deep learned and End-to-End solutions

Following the observation of a performance drop in SOTA countermeasures, we
conducted a cross-corpus sub-experiment utilizing contemporary deep learning,
along with an end-to-end solution. In this subsection, the performance of
recent deep learning and end-to-end solutions is evaluated. The results are in
Table XXII.

Several deep learning and end-to-end solutions have recently demonstrated
outstanding performance in the recent ASV challenges [33, 78, 110, 2, 49]. In
the latest ASVspoof2021 competition, for instance, the end-to-end solution
ASSIST [33] achieved the best EER and min-TDCF on the ASVspoof2021-LA dataset.
Similarly, [110] and [78] also performed well in this task. Many participants,
however, presented deep learning and end-to-end systems that had been trained
and tested to mitigate only a single attack. To demonstrate the general
effectiveness of the countermeasures, we performed an experiment in which we
evaluated the higher performing solutions to the ASV challenges across
corpora. Each solution trained and tested on ASVspoof2019-LA for the
challenge, evaluated against the LA speech samples of the ASVspoof2021 corpus.
Similarly, if the model’s stated results were based on ASVspoof2021, we
examined that model’s performance against ASVspoof2019. The key reasoning
behind this experiment was to evaluate and report the generalizability of the
countermeasures.

To measure generalizability, we selected the top-performing models with
publicly available code. It was observed from the results that the top
performer, ASSIST [33], and its companion model ASSIST-L, achieved EER values
of 0.83% and 0.99%, respectively, against the ASVspoof2019 dataset. However,
the performance of these models deteriorated when tested against speech
samples from the ASVspoof2021 dataset. Specifically, the EER scores increased
to 15.84% and 15.69%, and the minimum t-DCFs rose from 0.027% to 0.537%, and
from 0.030% to 0.520%, respectively.

When tested against ASV spoof 2021, the EER for the solution proposed in [110]
increased from 2.17% to 23.1%, and the minimum TDCF increases from 0.055% to
0.759%, however the worst performance was observed in the case of the RawBoost
solution proposed in [78]. Although the proposed solution performed optimally
with the augmented data approach with EER score of 2.17%, the EER and minimum
t-DCF increased dramatically when tested on other datasets and without
augmentation. In the case of ASVspoof2021 and without data augmentation, EER
increased by 43%, and the minimum t-DCF increased from 0.301% to 0.903%.

TABLE XXII: Computational details of the state-of-the-art countermeasures. Model | ASVspoof2019 | ASVspoof2021 | VSDC
---|---|---|---
EER | min t-DCF | EER | min t-DCF | EER | min t-DCF
ASSIST [33] | 0.83 | 0.027 | 15.84 | 0.537 | 51.21 | –
ASSIST-L [33] | 0.99 | 0.030 | 15.69 | 0.520 | 59.32 | –
RawBoost [78] | 2.17 | 0.055 | 23.1 | 0.759 | 64.90 | –
One class Learning [110] | 5.38 | 0.309 | 48.7 | 0.901 | 37.95 | –
Res2Net34 [49] | 1.67 | 0.058 | 48.00 | 0.994 | 29.00 | –
Res2Net50 [49] | 1.63 | 0.049 | 39.00 | 0.999 | 24.00 | –
ResNet34-LA [2] | 10.53 | 0.19 | 26.55 | 0.34 | 40.23 | –
ResNet34-PA [2] | 6.61 | 0.15 | 49.51 | 0.41 | 34.12 | –

Although the adaptation of the Resnet-34 architecture published in [2]
reported an EER of 5.32% against the ASVSpoof2019-LA dataset and 5.74% against
the ASVSpoof2019-PA dataset, we observed EERs of 10.53% and 6.61%,
respectively, in our configured environment. When tested on ASVSpoof2021,
performance dropped even further, with EER results of 26.55% on LA and 49.51%
on PA.

To further test the extensibility of this and other methods, we performed
cross-attack, cross-corpus testing, training on ASVspoof2019-PA, and testing
against ASVSpoof2021-DF, which achieved an EER 42.21%. Similar performance
deterioration was observed for [2] on the distinct dataset; the EER rose to
48.00% and 39.00% for res2net34 and res2net50, respectively. The performance
decreased when tested on the ASVspoof2021 dataset, and the worst performance
was observed against the chained replay VSDC datasets. These scores imply that
while these methods perform well on a specific dataset, the results do not
transfer well to others. This test demonstrates the impact of formulating a
solution based on a particular corpus, as the performance of deep learning and
end-to-end solutions degrades significantly when presented with unknowns. Each
static dataset differs and has distinct discrepancies due to varied factors,
such as the ambient setting, gadgets, microphonic influence, and the spoofing
technique used to produce spoof samples. Therefore, developing a spoofing
countermeasure without taking into consideration the aforementioned factors
may lead to performance degradation when tested across corpora. Thus, based on
these results, we conclude that there is a dire need to concentrate on a broad
approach that mitigates the aforementioned variances while developing robust
voice spoofing defenses.

###  Limitation and Future work

Aside from the effectiveness of the presented work, there are certain
limitations to this study. Due to contradictory technical issues and lack of
code availability, only fourteen countermeasures were chosen to do the cross-
corpus and multi-classifier evaluation. Moreover, in accordance with the
advancement of the voice spoofing domain and deep learning methodologies such
as CNNs, transformers, and so on, we compared four recent top-rated (by EER
and min-TDCF scores on ASVspoof datasets) deep learning and end-to-end
solutions with cross corpus evaluation. Despite the fact that just four models
were examined, the findings motivate us to continue working in the deep
learning and end-to-end arenas. Thus, we want to expand this work in the
future to include the most recent deep learning/end-to-end solutions in order
to provide an apples-to-apples comparison. Lastly, we intend to create a
benchmark for comparing CNN and DNN-based architectures for the automatic
speaker verification system.

##  Existing Issues and Future Needs

This section describes the challenges and issues observed during this review
as well as the future requirements for developing dependable and secure anti-
spoofing solutions.

###  Need for Explainable AI in Audio Forensics

Explainable AI (XAI) is an emerging field which emphasises the need for human
understanding through what has traditionally been a black box process. An
important gain over the interaction between humans and XAI is a level of trust
in the process, which studies have shown does not currently exist [20].
Existing deepfake detection approaches are typically designed to perform batch
analysis over a large dataset, however, when these techniques are employed in
the field by journalists or law enforcement, there may only be a small set of
videos available for analysis. A numerical score parallel to the probability
of an audio or video being real or fake is not valuable to these users if it
cannot be confirmed with an appropriate proof of the score. In those
situations, it is very common to demand an explanation for the numerical score
in order for the analysis to be believed before publication or utilization in
a court of law, however, most deepfake detection methods lack such an
explanation. If results for single interactions between humans and computers
are to be accepted it is vitally important to involve XAI early in the
process.

###  Robust PAD for ASV in Smart Homes

Because the Smart Home concept encourages hands-free automation, bio-metric
technologies, and particularly automatic speech recognition, are the ideal way
to regulate and personalize access. However, while contemporary spoofing
systems are extremely descriptive and specific when detecting presentation
attacks, current speaker identification systems may be vulnerable. Therefore,
most of these PAD systems are ineffective in the current environment where the
types of presentation attacks are unknown. When deployed to smart homes, the
risk of attack increases significantly. Hence, there is an immediate need to
build a strong PAD system to protect ASV systems in order to accelerate the
safe use of ASVs in Smart Home applications.

###  What type of model is best for given scenario?

Because countermeasures operate differently in varying conditions there is an
urgent need to shift the research community’s focus to environmental or
scenario-based spoofing countermeasures. The environment is an essential part
of spoofing detection countermeasures; for instance, several countermeasures
may fail to provide adequate performance when surrounded by a loud environment
or background noise. As a result, a thorough comparison study is required to
find the best countermeasure for specific scenarios.

###  Need for a new Audio Forensics datasets

Voice-controlled devices (VCDs) have transformed the Internet of Things (IoT),
enabling the enhanced development, personalization, and automation, of smart
homes through voice activation. Unfortunately, these VCDs are currently
largely unprotected, and can be exploited via spoofing attacks. Existing
datasets, such as the Voice spoofing detection corpus (VSDC), ASVspoof 2017,
ASVspoof 2019, and ReMASC, contain large-scale replay (i.e., single and multi-
order replays) and basic cloning audio files, but newer, deep learning-based
attacks may easily defeat current solutions, introducing a new level of
complexity for anti-spoofing algorithms. Moreover, large-scale datasets like
ASVspoof 2017 and ASVspoof 2019 lack cloning design parameters, which are an
essential component of current VCDs. Therefore, an enhanced cloning dataset is
needed in order to stay ahead of the curve of advanced voice cloning spoof
attacks. For creation of such dataset, new generative algorithms such as
diffusion models could be employed. Also, there is need to develop datasets
that contains multiple forgeries at frame level in a single file. The new
datasets also need to be unbiased with respect to gender and ethnicity.
subsectionThe role of Adversarial Machine Learning on existing Audio spoofing
countermeasures High-performance spoofing countermeasure systems for automated
speaker verification (ASV) have been presented during the ASVspoof sessions.
However, the robustness of these systems in the face of adversarial attacks
has yet to be thoroughly examined, while adversarial attacks pose a
significant threat to ASV systems and their countermeasures. Currently, there
are only a limited number of countermeasures in this domain. It is therefore
necessary to develop a more adversarial-aware ASV system.

###  Fairness-enabled voice spoofing countermeasures and ASV

Fairness is defined as the lack of bias or preference toward a person or a
group based on intrinsic or acquired attributes. Prior to their widespread
commercialization, it is necessary to study the bias and impartiality of
spoofing detection and speaker verification systems across populations. No
quantitative evaluation has been reported to study the fairness (w.r.t gender,
ethnicity etc) in the spoofing detectors across protected parameters, and even
integrated voice countermeasure. To avoid the real-world consequences of a
biased and flawed system that favors a single sub-group or class, it is vital
to ensure the impartiality of data sets used for training and testing spoofing
systems.

###  Attack-aware Audio Forensics Countermeasures

ASV (automated speaker verification) systems should always discard both non-
target (voice produced by unrelated speakers) and spoofed (reprocessed or
transformed) signals. However, little consideration has been given to how ASV
systems should be changed when they encounter spoofing attacks, or even when
they pair up with spoofing countermeasures, much less to how both systems
might be jointly optimized.

###  Need for Cross-dataset Evaluation

The need for cross dataset evaluation is important to consider when evaluating
the performance of any aspect of an audio spoof detection system as shown in
this paper. Cross dataset evaluation gives important insight into the
robustness of the tested features and can provide additional information and
direction toward a more general and complete ASV system.

###  Physics of AI

Current deepfake detectors face challenges, particularly due to incomplete,
sparse, and noisy data in training phases. There is a need to explore
innovative AI architectures, algorithms, and approaches that “bake in” the
physics, mathematics, and prior knowledge relevant to deep fakes and other
forgeries. Embedding physics and prior knowledge, using knowledge-infused
learning, into AI will help to overcome the challenges of sparse data and will
facilitate the development of generative models that are causal and
explanatory.

##  Conclusion

This research provides a comprehensive review of voice spoofing attacks,
countermeasures proposed to counter certain attacks, publicly accessible
datasets, and the criteria used to evaluate countermeasure performance in
voice spoofing detection. This paper also includes an experimental comparison
of the state-of-the-art (SOTA) voice spoofing countermeasures against multi-
and cross-datasets and classifiers. Based on the results of the experiments,
this study concludes that characteristics that better capture microphone
signatures and harmonic distortions give improved detection performance for
the identification of PA attacks. Furthermore, we show that the efficacy of
the SOTA countermeasure varies significantly with changes in datasets and
classifiers. Extensive testing reveals that these countermeasures, along with
CNN and CNN-GRU classifiers, perform well. Detailed experiments also
demonstrate the significance of cross-corpus evaluation for future voice
spoofing technologies.

##  Reproducibility

We have provided a GitHub
repository222https://github.com/smileslab/Comparative-Analysis-Voice-Spoofing
that contains all the code and explanation needed to reproduce results of
models evaluated to test their generalizability. The README.pdf file contains
explanation of how to run the experiments.

##  Acknowledgement

This material is based upon work supported by the National Science Foundation
(NSF) under award number 1815724 and Michigan Transnationals Research and
Commercialization (MTRAC), Advanced Computing Technologies (ACT) award number
292883. Any opinions, findings, and conclusions or recommendations expressed
in this material are those of the author(s) and do not necessarily reflect the
views of the NSF and MTRAC ACT.

## References

  * [1] Federico Alegre, Asmaa Amehraye, and Nicholas Evans.  Spoofing countermeasures to protect automatic speaker verification from voice conversion.  In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 3068–3072. IEEE, 2013.
  * [2] PR Aravind, Usamath Nechiyil, Nandakumar Paramparambath, et al.  Audio spoofing verification using deep convolutional neural networks by transfer learning.  arXiv preprint arXiv:2008.03464, 2020.
  * [3] Tuba Arif, Ali Javed, Mohammed Alhameed, Fathe Jeribi, and Ali Tahir.  Voice spoofing countermeasure for logical access attacks detection.  IEEE Access, 9:162857–162868, 2021.
  * [4] B. T. Balamurali, Kinwah Edward Lin, Simon Lui, Jer-Ming Chen, and Dorien Herremans.  Toward robust audio spoofing detection: A detailed comparison of traditional and learned features.  IEEE Access, 7:84229–84241, 2019.
  * [5] BT Balamurali, Kinwah Edward Lin, Simon Lui, Jer-Ming Chen, and Dorien Herremans.  Toward robust audio spoofing detection: A detailed comparison of traditional and learned features.  IEEE Access, 7:84229–84241, 2019.
  * [6] Roland Baumann, Khalid Mahmood Malik, Ali Javed, Andersen Ball, Brandon Kujawa, and Hafiz Malik.  Voice spoofing detection corpus for single and multi-order audio replays.  Computer Speech & Language, 65:101132, 2021.
  * [7] Stephen R Bond, Anke Hoeffler, and Jonathan RW Temple.  Gmm estimation of empirical growth models.  Available at SSRN 290522, 2001.
  * [8] Weicheng Cai, Danwei Cai, Wenbo Liu, Gang Li, and Ming Li.  Countermeasures for automatic speaker verification replay spoofing attack: On data augmentation, feature representation, classification and fusion.  In INTERSPEECH, pages 17–21, 2017.
  * [9] Sandipan Chakroborty and Goutam Saha.  Improved text-independent speaker identification using fused mfcc & imfcc feature sets based on gaussian filter.  International Journal of Signal Processing, 5(1):11–19, 2009.
  * [10] Nanxin Chen, Yanmin Qian, Heinrich Dinkel, Bo Chen, and Kai Yu.  Robust deep feature for spoofing detection—the sjtu system for asvspoof 2015 challenge.  In Sixteenth Annual Conference of the International Speech Communication Association, 2015.
  * [11] Xinhui Chen, You Zhang, Ge Zhu, and Zhiyao Duan.  UR Channel-Robust Synthetic Speech Detection System for ASVspoof 2021\.  In Proc. 2021 Edition of the Automatic Speaker Verification and Spoofing Countermeasures Challenge, pages 75–82, 2021.
  * [12] Zhuxin Chen, Zhifeng Xie, Weibin Zhang, and Xiangmin Xu.  ResNet and Model Fusion for Automatic Spoofing Detection.  In Proc. Interspeech 2017, pages 102–106, 2017.
  * [13] Bhusan Chettri, Tomi Kinnunen, and Emmanouil Benetos.  Deep generative variational autoencoding for replay spoof detection in automatic speaker verification, 2020.
  * [14] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman.  Voxceleb2: Deep speaker recognition.  arXiv preprint arXiv:1806.05622, 2018.
  * [15] Héctor Delgado, Nicholas Evans, Tomi Kinnunen, Kong Aik Lee, Xuechen Liu, Andreas Nautsch, Jose Patino, Md Sahidullah, Massimiliano Todisco, Xin Wang, et al.  Asvspoof 2021: Automatic speaker verification spoofing and countermeasures challenge evaluation plan.  arXiv preprint arXiv:2109.00535, 2021.
  * [16] Héctor Delgado, Massimiliano Todisco, Md Sahidullah, Nicholas Evans, Tomi Kinnunen, Kong Aik Lee, and Junichi Yamagishi.  Asvspoof 2017 version 2.0: meta-data analysis and baseline enhancements.  In Odyssey 2018-The Speaker and Language Recognition Workshop, 2018\.
  * [17] H. Dinkel, N. Chen, Y. Qian, and K. Yu.  End-to-end spoofing detection with raw waveform cldnns.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4860–4864, 2017.
  * [18] Serife Kucur Ergünay, Elie Khoury, Alexandros Lazaridis, and Sebastien Marcel.  On the vulnerability of speaker verification to realistic voice spoofing.  In 2015 IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS), pages 1–6. IEEE, 2015.
  * [19] Roberto Font, Juan M Espín, and María José Cano.  Experimental analysis of features for replay attack detection-results on the asvspoof 2017 challenge.  In Interspeech, pages 7–11, 2017.
  * [20] Julie Gerlings, Arisa Shollo, and Ioanna Constantiou.  Reviewing the need for explainable artificial intelligence (xai), 2020\.
  * [21] Yuan Gong, Jian Yang, Jacob Huber, Mitchell MacKnight, and Christian Poellabauer.  Remasc: Realistic replay attack corpus for voice controlled systems.  arXiv preprint arXiv:1904.03365, 2019.
  * [22] Tharshini Gunendradasan, Saad Irtza, Eliathamby Ambikairajah, and Julien Epps.  Transmission line cochlear model based am-fm features for replay attack detection.  In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6136–6140, 2019.
  * [23] Tharshini Gunendradasan, Buddhi Wickramasinghe, Phu Le, Eliathamby Ambikairajah, and Julien Epps.  Detection of replay-spoofing attacks using frequency modulation features.  In INTERSPEECH, pages 636–640, 09 2018.
  * [24] Farman Hassan and Ali Javed.  Voice spoofing countermeasure for synthetic speech detection.  In 2021 International Conference on Artificial Intelligence (ICAI), pages 209–212, 2021.
  * [25] Lian Huang and Chi-Man Pun.  Audio replay spoof attack detection by joint segment-based linear filter bank feature extraction and attention-enhanced densenet-bilstm network.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:1813–1825, 2020.
  * [26] Ting Huang, Hongxia Wang, Yi Chen, and Peisong He.  Gru-svm model for synthetic speech detection.  In Hongxia Wang, Xianfeng Zhao, Yunqing Shi, Hyoung Joong Kim, and Alessandro Piva, editors, Digital Forensics and Watermarking, pages 115–125, Cham, 2020. Springer International Publishing.
  * [27] Arindam Jati, Chin-Cheng Hsu, Monisankha Pal, Raghuveer Peri, Wael AbdAlmageed, and Shrikanth Narayanan.  Adversarial attack and defense strategies for deep speaker recognition systems.  Computer Speech & Language, 68:101199, Jul 2021.
  * [28] Ali Javed, Khalid Mahmood Malik, Aun Irtaza, and Hafiz Malik.  Towards protecting cyber-physical and iot systems from single-and multi-order voice spoofing attacks.  Applied Acoustics, 183:108283, 2021.
  * [29] Ali Javed, Khalid Mahmood Malik, Hafiz Malik, and Aun Irtaza.  Voice spoofing detector: A unified anti-spoofing framework.  Expert Systems with Applications, 198:116770, 2022.
  * [30] Zhe Ji, Zhi-Yi Li, Peng Li, Maobo An, Shengxiang Gao, Dan Wu, and Faru Zhao.  Ensemble learning for countermeasure of audio replay spoofing attack in asvspoof2017.  In Interspeech, pages 87–91, 2017.
  * [31] Akhil Jose, Justin Joseph, Glan Devadhas, and M. M. Shinu.  Influence of filter bank structure on the statistical significance of coefficients in cepstral analysis for acoustic signals.  In Sabu M. Thampi, S ri Krishnan, Juan Manuel Corchado Rodriguez, Swagatam Das, Michal Wozniak, and Dhiya Al-Jumeily, editors, Advances in Signal Processing and Intelligent Recognition Systems, pages 91–104, Cham, 2018\. Springer International Publishing.
  * [32] Jee-weon Jung, Hee-Soo Heo, Hemlata Tak, Hye-jin Shim, Joon Son Chung, Bong-Jin Lee, Ha-Jin Yu, and Nicholas Evans.  Aasist: Audio anti-spoofing using integrated spectro-temporal graph attention networks, 2021.
  * [33] Jee-weon Jung, Hee-Soo Heo, Hemlata Tak, Hye-jin Shim, Joon Son Chung, Bong-Jin Lee, Ha-Jin Yu, and Nicholas Evans.  Aasist: Audio anti-spoofing using integrated spectro-temporal graph attention networks.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6367–6371. IEEE, 2022.
  * [34] Jee-weon Jung, Hye-jin Shim, Hee-Soo Heo, and Ha-Jin Yu.  Replay attack detection with complementary high-resolution information using end-to-end dnn for the asvspoof 2019 challenge.  arXiv preprint arXiv:1904.10134, 2019.
  * [35] Jee-weon Jung, Hemlata Tak, Hye-jin Shim, Hee-Soo Heo, Bong-Jin Lee, Soo-Whan Chung, Ha-Jin Yu, Nicholas Evans, and Tomi Kinnunen.  Sasv 2022: The first spoofing-aware speaker verification challenge.  arXiv preprint arXiv:2203.14732, 2022.
  * [36] Madhu R Kamble, Hardik B Sailor, Hemant A Patil, and Haizhou Li.  Advances in anti-spoofing: from the perspective of asvspoof challenges.  APSIPA Transactions on Signal and Information Processing, 9, 2020\.
  * [37] Tomi Kinnunen, Kong Aik Lee, Héctor Delgado, Nicholas Evans, Massimiliano Todisco, Md Sahidullah, Junichi Yamagishi, and Douglas A Reynolds.  t-dcf: a detection cost function for the tandem assessment of spoofing countermeasures and automatic speaker verification.  arXiv preprint arXiv:1804.09618, 2018.
  * [38] Tomi Kinnunen, Md Sahidullah, Héctor Delgado, Massimiliano Todisco, Nicholas Evans, Junichi Yamagishi, and Kong Aik Lee.  The asvspoof 2017 challenge: Assessing the limits of replay spoofing attack detection.  2017\.
  * [39] Tomi Kinnunen, Md Sahidullah, Héctor Delgado, Massimiliano Todisco, Nicholas Evans, Junichi Yamagishi, and Kong Aik Lee.  The asvspoof 2017 challenge: Assessing the limits of replay spoofing attack detection.  In INTERSPEECH, 08 2017.
  * [40] Tomi H. Kinnunen, Md. Sahidullah, Mauro Falcone, Luca Costantini, Rosa González Hautamäki, Dennis Alexander Lehmann Thomsen, Achintya Kumar Sarkar, Z. Tan, Héctor Delgado, Massimiliano Todisco, Nicholas W. D. Evans, Ville Hautamäki, and Kong-Aik Lee.  Reddots replayed: A new replay spoofing attack corpus for text-dependent speaker verification research.  2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5395–5399, 2017.
  * [41] Pavel Korshunov, André R Gonçalves, Ricardo PV Violato, Flávio O Simões, and Sébastien Marcel.  On the use of convolutional neural networks for speech presentation attack detection.  In 2018 IEEE 4th international conference on identity, security, and behavior analysis (ISBA), pages 1–8. IEEE, 2018.
  * [42] Pavel Korshunov, Sébastien Marcel, Hannah Muckenhirn, André R Gonçalves, AG Souza Mello, RP Velloso Violato, Flávio O Simoes, M Uliani Neto, Marcus de Assis Angeloni, José Augusto Stuchi, et al.  Overview of btas 2016 speaker anti-spoofing competition.  In 2016 IEEE 8th international conference on biometrics theory, applications and systems (BTAS), pages 1–6. IEEE, 2016.
  * [43] Felix Kreuk, Yossi Adi, Moustapha Cisse, and Joseph Keshet.  Fooling end-to-end speaker verification with adversarial examples.  In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 1962–1966. IEEE, 2018.
  * [44] Jia Min Karen Kua, Tharmarajah Thiruvaran, Mohaddeseh Nosratighods, Eliathamby Ambikairajah, and Julien Epps.  Investigation of spectral centroid magnitude and frequency for speaker recognition.  In Odyssey, page 7, 2010.
  * [45] Cheng-I Lai, Alberto Abad, Korin Richmond, Junichi Yamagishi, Najim Dehak, and Simon King.  Attentive filtering networks for audio replay attack detection.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6316–6320. IEEE, 2019.
  * [46] Cheng-I Lai, Nanxin Chen, Jesús Villalba, and Najim Dehak.  Assert: Anti-spoofing with squeeze-excitation and residual networks.  arXiv preprint arXiv:1904.01120, 2019.
  * [47] Galina Lavrentyeva, Sergey Novoselov, Andzhukaev Tseren, Marina Volkova, Artem Gorlanov, and Alexandr Kozlov.  Stc antispoofing systems for the asvspoof2019 challenge.  arXiv preprint arXiv:1904.05576, 2019.
  * [48] Kong-Aik Lee, Anthony Larcher, Guangsen Wang, Patrick Kenny, Niko Brümmer, David A. van Leeuwen, Hagai Aronowitz, Marcel Kockmann, Carlos Vaquero, Bin Ma, Haizhou Li, Themos Stafylakis, Md. Jahangir Alam, Albert Swart, and Javier Pérez.  The reddots data collection for speaker recognition.  In INTERSPEECH, 2015.
  * [49] Xu Li, Na Li, Chao Weng, Xunying Liu, Dan Su, Dong Yu, and Helen Meng.  Replay and synthetic speech detection with res2net architecture.  In ICASSP 2021-2021 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 6354–6358. IEEE, 2021.
  * [50] Leian Liu and Jichen Yang.  Study on feature complementarity of statistics, energy, and principal information for spoofing detection.  IEEE Access, 8:141170–141181, 2020.
  * [51] Xuechen Liu, Md Sahidullah, and Tomi Kinnunen.  Spoofing-aware speaker verification with unsupervised domain adaptation.  arXiv preprint arXiv:2203.10992, 2022.
  * [52] Youxuan Ma, Zongze Ren, and Shugong Xu.  Rw-resnet: A novel speech anti-spoofing model using raw waveform, 2021\.
  * [53] K. M. Malik, A. Javed, H. Malik, and A. Irtaza.  A light-weight replay detection framework for voice controlled iot devices.  IEEE Journal of Selected Topics in Signal Processing, 14(5):982–996, 2020.
  * [54] Khalid Mahmood Malik, Ali Javed, Hafiz Malik, and Aun Irtaza.  A light-weight replay detection framework for voice controlled iot devices.  IEEE Journal of Selected Topics in Signal Processing, 14(5):982–996, 2020.
  * [55] Khalid Mahmood Malik, Ali Javed, Hafiz Malik, and Aun Irtaza.  A light-weight replay detection framework for voice controlled iot devices.  IEEE Journal of Selected Topics in Signal Processing, 14(5):982–996, 2020.
  * [56] Aakshi Mittal and Mohit Dua.  Automatic speaker verification systems and spoof detection techniques: review and analysis.  International Journal of Speech Technology, pages 1–30, 2021.
  * [57] João Monteiro, Jahangir Alam, and Tiago H Falk.  An ensemble based approach for generalized detection of spoofing attacks to automatic speaker recognizers.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6599–6603. IEEE, 2020.
  * [58] Lindasalwa Muda, Mumtaj Begam, and Irraivan Elamvazuthi.  Voice recognition algorithms using mel frequency cepstral coefficient (mfcc) and dynamic time warping (dtw) techniques.  arXiv preprint arXiv:1003.4083, 2010.
  * [59] Hema A Murthy and Venkata Gadde.  The modified group delay function and its application to phoneme recognition.  In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)., volume 1, pages I–68. IEEE, 2003.
  * [60] Parav Nagarsheth, Elie el Khoury, Kailash Patil, and Matt Garland.  Replay attack detection using dnn for channel discrimination.  In INTERSPEECH, 2017.
  * [61] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman.  Voxceleb: a large-scale speaker identification dataset.  arXiv preprint arXiv:1706.08612, 2017.
  * [62] Ravika Naika.  An overview of automatic speaker verification system.  In Intelligent Computing and Information and Communication, pages 603–610. Springer, 2018.
  * [63] Sergey Novoselov, Alexandr Kozlov, Galina Lavrentyeva, Konstantin Simonchik, and Vadim Shchemelinin.  Stc anti-spoofing systems for the asvspoof 2015 challenge.  In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5475–5479. IEEE, 2016.
  * [64] Hemant A Patil and Madhu R Kamble.  A survey on replay attack detection for automatic speaker verification (asv) system.  In 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1047–1053. IEEE, 2018.
  * [65] Dipjyoti Paul, Monisankha Pal, and Goutam Saha.  Novel speech features for improved detection of spoofing attacks.  In 2015 Annual IEEE India Conference (INDICON), pages 1–6. IEEE, 2015.
  * [66] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [67] Padmanabhan Rajan, Tomi Kinnunen, Cemal Hanilci, Jouni Pohjalainen, and Paavo Alku.  Using group delay functions from all-pole models for speaker recognition.  In INTERSPEECH, pages 2489–2493. Citeseer, 2013.
  * [68] Amir Mohammad Rostami, Mohammad Mehdi Homayounpour, and Ahmad Nickabadi.  Efficient attention branch network with combined loss function for automatic speaker verification spoof detection.  arXiv preprint arXiv:2109.02051, 2021.
  * [69] Md Sahidullah, Héctor Delgado, Massimiliano Todisco, Tomi Kinnunen, Nicholas Evans, Junichi Yamagishi, and Kong-Aik Lee.  Introduction to voice presentation attack detection and recent advances.  In Handbook of biometric anti-spoofing, pages 321–361. Springer, 2019.
  * [70] Md Sahidullah, Héctor Delgado, Massimiliano Todisco, Hong Yu, Tomi Kinnunen, Nicholas Evans, and Zheng-Hua Tan.  Integrated spoofing countermeasures and automatic speaker verification: An evaluation on asvspoof 2015.  2016\.
  * [71] Md Sahidullah, Tomi Kinnunen, and Cemal Hanilçi.  A comparison of features for synthetic speech detection, 2015.
  * [72] M. S. Saranya, R. Padmanabhan, and Hema A. Murthy.  Replay attack detection in speaker verification using non-voiced segments and decision level feature switching.  In 2018 International Conference on Signal Processing and Communications (SPCOM), pages 332–336, 2018.
  * [73] Ibon Saratxaga, Inmaculada Hernáez, Michael Pucher, and Inaki Sainz.  Perceptual importance of the phase related information in speech.  In INTERSPEECH, volume 2, 09 2012.
  * [74] Gajan Suthokumar, Kaavya Sriskandaraja, Vidhyasaharan Sethu, Chamith Wijenayake, and Eliathamby Ambikairajah.  Phoneme specific modelling and scoring techniques for anti spoofing system.  In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6106–6110, 2019.
  * [75] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks, 2014.
  * [76] Hemlata Tak, Jee-weon Jung, Jose Patino, Madhu Kamble, Massimiliano Todisco, and Nicholas Evans.  End-to-end spectro-temporal graph attention networks for speaker verification anti-spoofing and speech deepfake detection, 2021.
  * [77] Hemlata Tak, Jee-weon Jung, Jose Patino, Massimiliano Todisco, and Nicholas Evans.  Graph attention networks for anti-spoofing, 2021.
  * [78] Hemlata Tak, Madhu Kamble, Jose Patino, Massimiliano Todisco, and Nicholas Evans.  Rawboost: A raw data boosting and augmentation method applied to automatic speaker verification anti-spoofing, 2021.
  * [79] Choon Beng Tan, Mohd Hanafi Ahmad Hijazi, Norazlina Khamis, Zuraini Zainol, Frans Coenen, Abdullah Gani, et al.  A survey on presentation attack detection for automatic speaker verification systems: State-of-the-art, taxonomy, issues and future direction.  Multimedia Tools and Applications, 80(21):32725–32762, 2021.
  * [80] Prasad A Tapkir, Ankur T Patil, Neil Shah, and Hemant A Patil.  Novel spectral root cepstral features for replay spoof detection.  In 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1945–1950. IEEE, 2018.
  * [81] Prasad A Tapkir and Hemant A Patil.  Significance of teager energy operator phase for replay spoof detection.  In 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1951–1956. IEEE, 2018.
  * [82] Zhongwei Teng, Quchen Fu, Jules White, Maria E Powell, and Douglas C Schmidt.  Sa-sasv: An end-to-end spoof-aggregated spoofing-aware speaker verification system.  arXiv preprint arXiv:2203.06517, 2022.
  * [83] Massimiliano Todisco, Héctor Delgado, and Nicholas W. D. Evans.  Constant q cepstral coefficients: A spoofing countermeasure for automatic speaker verification.  Comput. Speech Lang., 45:516–535, 2017.
  * [84] Massimiliano Todisco, Xin Wang, Ville Vestman, Md Sahidullah, Héctor Delgado, Andreas Nautsch, Junichi Yamagishi, Nicholas Evans, Tomi Kinnunen, and Kong Aik Lee.  Asvspoof 2019: Future horizons in spoofed and fake audio detection.  arXiv preprint arXiv:1904.05441, 2019.
  * [85] Xavier Valero and Francesc Alias.  Gammatone cepstral coefficients: Biologically inspired features for non-speech audio classification.  IEEE Transactions on Multimedia, 14(6):1684–1689, 2012.
  * [86] Ville Vestman, Tomi Kinnunen, Rosa González Hautamäki, and Md Sahidullah.  Voice mimicry attacks assisted by automatic speaker verification.  Computer Speech & Language, 59:36–54, 2020.
  * [87] SVM Vishwanathan and M Narasimha Murty.  Ssvm: a simple svm algorithm.  In Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN’02 (Cat. No. 02CH37290), volume 3, pages 2393–2398. IEEE, 2002.
  * [88] Qian Wang, Xiu Lin, Man Zhou, Yanjiao Chen, Cong Wang, Qi Li, and Xiangyang Luo.  Voicepop: A pop noise based anti-spoofing system for voice authentication on smartphones.  In IEEE INFOCOM 2019 - IEEE Conference on Computer Communications, pages 2062–2070, 2019.
  * [89] Xin Wang, Junichi Yamagishi, Massimiliano Todisco, Héctor Delgado, Andreas Nautsch, Nicholas Evans, Md Sahidullah, Ville Vestman, Tomi Kinnunen, Kong Aik Lee, et al.  Asvspoof 2019: A large-scale public database of synthesized, converted and replayed speech.  Computer Speech & Language, 64:101114, 2020.
  * [90] Marcin Witkowski, Stanislaw Kacprzak, Piotr Zelasko, Konrad Kowalczyk, and Jakub Galka.  Audio replay attack detection using high-frequency features.  In Interspeech, pages 27–31, 2017.
  * [91] Eddie Wong and Sridha Sridharan.  Comparison of linear prediction cepstrum coefficients and mel-frequency cepstrum coefficients for language identification.  In Proceedings of 2001 International Symposium on Intelligent Multimedia, Video and Speech Processing. ISIMP 2001 (IEEE Cat. No. 01EX489), pages 95–98. IEEE, 2001.
  * [92] Haibin Wu, Po chun Hsu, Ji Gao, Shanshan Zhang, Shen Huang, Jian Kang, Zhiyong Wu, Helen Meng, and Hung yi Lee.  Spotting adversarial samples for speaker verification by neural vocoders, 2021.
  * [93] Haibin Wu, Xu Li, Andy T. Liu, Zhiyong Wu, Helen Meng, and Hung yi Lee.  Adversarial defense for automatic speaker verification by cascaded self-supervised learning models, 2021.
  * [94] Haibin Wu, Xu Li, Andy T. Liu, Zhiyong Wu, Helen Meng, and Hung yi Lee.  Improving the adversarial robustness for speaker verification by self-supervised learning, 2021.
  * [95] Haibin Wu, Andy T. Liu, and Hung yi Lee.  Defense for black-box attacks on anti-spoofing models by self-supervised learning, 2020.
  * [96] Haibin Wu, Songxiang Liu, Helen Meng, and Hung yi Lee.  Defense against adversarial attacks on spoofing countermeasures of asv, 2020.
  * [97] Xiang Wu, Ran He, Zhenan Sun, and Tieniu Tan.  A light cnn for deep face representation with noisy labels.  IEEE Transactions on Information Forensics and Security, 13(11):2884–2896, 2018.
  * [98] Zhenzong Wu, Rohan Kumar Das, Jichen Yang, and Haizhou Li.  Light convolutional neural network with feature genuinization for detection of synthetic speech attacks, 2020.
  * [99] Zhizheng Wu, Nicholas Evans, Tomi Kinnunen, Junichi Yamagishi, Federico Alegre, and Haizhou Li.  Spoofing and countermeasures for speaker verification: A survey.  speech communication, 66:130–153, 2015.
  * [100] Zhizheng Wu, Sheng Gao, Eng Siong Cling, and Haizhou Li.  A study on replay attack and anti-spoofing for text-dependent speaker verification.  In Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2014 Asia-Pacific, pages 1–5, 2014.
  * [101] Zhizheng Wu, Ali Khodabakhsh, Cenk Demiroglu, Junichi Yamagishi, Daisuke Saito, Tomoki Toda, and Simon King.  Sas: A speaker verification spoofing database containing diverse attacks.  In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4440–4444. IEEE, 2015.
  * [102] Zhizheng Wu, Tomi Kinnunen, Nicholas Evans, Junichi Yamagishi, Cemal Hanilçi, Md Sahidullah, and Aleksandr Sizov.  Asvspoof 2015: the first automatic speaker verification spoofing and countermeasures challenge.  In Sixteenth annual conference of the international speech communication association, 2015.
  * [103] Zhizheng Wu, Junichi Yamagishi, Tomi Kinnunen, Cemal Hanilçi, Mohammed Sahidullah, Aleksandr Sizov, Nicholas Evans, Massimiliano Todisco, and Héctor Delgado.  Asvspoof: the automatic speaker verification spoofing and countermeasures challenge.  IEEE Journal of Selected Topics in Signal Processing, 11(4):588–604, 2017.
  * [104] Zhizheng Wu, Junichi Yamagishi, Tomi Kinnunen, Cemal Hanilçi, Mohammed Sahidullah, Aleksandr Sizov, Nicholas Evans, Massimiliano Todisco, and Héctor Delgado.  Asvspoof: The automatic speaker verification spoofing and countermeasures challenge.  IEEE Journal of Selected Topics in Signal Processing, 11(4):588–604, 2017.
  * [105] Jichen Yang and Rohan Das.  Long-term high frequency features for synthetic speech detection.  Digital Signal Processing, 97:102622, 11 2019.
  * [106] Jichen Yang and Rohan Kumar Das.  Low frequency frame-wise normalization over constant-q transform for playback speech detection.  Digital Signal Processing, 2019.
  * [107] Jichen Yang and Rohan Kumar Das.  Improving anti-spoofing with octave spectrum and short-term spectral statistics information.  Applied Acoustics, 157:107017, 2020.
  * [108] Jichen Yang, Rohan Kumar Das, and Nina Zhou.  Extraction of octave spectra information for spoofing attack detection.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(12):2373–2384, 2019.
  * [109] Jichen Yang, Longting Xu, Bo Ren, and Yunyun Ji.  Discriminative features based on modified log magnitude spectrum for playback speech detection, 2020.
  * [110] You Zhang, Fei Jiang, and Zhiyao Duan.  One-class learning towards synthetic voice spoofing detection.  IEEE Signal Processing Letters, 28:937–941, 2021.
  * [111] You Zhang, Ge Zhu, and Zhiyao Duan.  A probabilistic fusion framework for spoofing aware speaker verification.  arXiv preprint arXiv:2202.05253, 2022.
  * [112] Fang Zheng, Guoliang Zhang, and Zhanjiang Song.  Comparison of different implementations of mfcc.  Journal of Computer science and Technology, 16(6):582–589, 2001\.
  * [113] Xinhui Zhou, Daniel Garcia-Romero, Ramani Duraiswami, Carol Espy-Wilson, and Shihab Shamma.  Linear versus mel frequency cepstral coefficients for speaker recognition.  In 2011 IEEE Workshop on Automatic Speech Recognition & Understanding, pages 559–564. IEEE, 2011.


# Characterization of atmospheric properties at the future sites of the
Cherenkov Telescope Array

a, Dušan Mandáta, Miroslav Pecha, Ladislav Chytkab, Jakub Juryšeka,b, Michael
Prouzaa, Petr Janečeka, Petr Trávníčeka, Jiří Blažeka, Tomasz Bulikc, Marek
Cieslarc, Mariusz Suchenekd, Vincenzo Rizie, Ermanno Pietropaoloe, Marco
Iarlorie, Carla Aramof, Laura Valoreg, Federico Di Pierrog, Piero Vallaniag,
Davide Depaolig, Martin Willh, Markus Gaugj and Lluís Fontj for the CTA
consortium and Martin Mašeka, Jiří Eliášeka, Martin Jelíneki and Sergey
Karpova
aFZU – Institute of Physics of the Czech Academy of Sciences, Czech Republic
bPalacký University, Olomouc, Czech Republic
cAstronomical Observatory, Department of Physics, University of Warsaw, Poland
dFaculty of Physics, University of Warsaw, Poland
eINFN-GSGC-L’Aquila and CETEMPS/DSFC, Università degli Studi dell’Aquila,
Italy
fINFN Napoli and Dipartimento di Fisica, Università di Napoli “Federico II”,
Italy
gINFN Torino, Italy
hMax Planck Institute for Physics, Munich, Germany
iAstronomical Institute of the Czech Academy of Sciences, Czech Republic
jDepartament de Fisica, and CERES-IEEC, Universitat Autonoma de Barcelona,
Spain
E-mail

###### Abstract:

Advanced knowledge of the detailed atmospheric properties of both the future
sites of the Cherenkov Telescope Array is essential in preparation of the
arrival of the first scientific data. Meteorological variables are studied
using a dedicated characterization station installed at the southern site in
Chile and a wealth of data from existing observatories around the northern
site on the La Palma island. Campaigns using radiosondes launched on balloons
are foreseen to complement these data in the near future. Cloudiness during
the night has been continuously monitored at both sites for several years
using All-sky Cameras which assess the presence of clouds based on detection
of stars. The integrated aerosol optical depth over the southern site has been
measured using a Sun/Moon Photometer since 2016 and the small robotic FRAM
telescope since 2017; identical instruments have been deployed at the northern
site in autumn 2018. Also in October 2018, the ARCADE Raman lidar (RL) has
started to take measurements on routine basis at the northern site, providing
data on the vertical profile of the aerosol optical properties (i.e.,
extinction and scattering) and of the water vapour mixing ratio. We present
the data currently available from these instruments from both sites with
emphasis on characteristics important for the (future) operation of Imaging
Atmospheric Cherenkov Telescopes.

Figure 1:  Distribution of the star fraction recorded by the All-sky Cameras
at the future CTA sites between 11/2015–12/2018 for the CTA South site and
bewteen 10/2015–06/2019 for the CTA North site. The number of individual
measurements used (count) is provided as well as the _clear fraction_ – the
amount of measurements with less than 20 % of the sky covered by clouds.

##  1 Introduction

The future sites of the Cherenkov Telescope Array (CTA) – the CTA South site
near Cerro Armazones, Chile and the CTA North site on La Palma, Canary
Islands, Spain – are equipped with various devices used for atmospheric
characterization. Their brief technical description is mostly provided in [1]
(and references cited therein) and the experience from operating some of them
as prototypes of atmospheric monitoring devices to be used during CTA
operations is presented in [2] together with information on the covered time
periods and instrument availability. In this paper, we focus on the results
obtained at the sites and provide additional information necessary for proper
interpretation of the data.

##  2 Cloud coverage

The cloud coverage above the sites is monitored during nighttime using the
All-sky Cameras (ASC) [3], which observe stars over the whole sky
simultaneously. Each 30-second exposure is automatically processed and stars
detected on the image are compared with the expectation from a catalog. The
distribution of the resulting _star fraction_ is shown in Fig. 1; this
quantity is, for a full-sky analysis, a very good approximation of the
fraction of sky not covered by clouds. For this analysis, only images taken
with Moon under the horizon were used, to avoid systematic effects in the
distributions – for operational assessment of cloudiness, all other images can
be used as well. For both sites, the _clear fraction_ – the amount of time
when less than 20 % of the sky covered is by clouds and thus the conditions
are favorable for observations – is very large (83 % and 92 %).

Figure 2:  Distribution of the VAOD measured by the different instruments at
both sites. In the CTA North site, the ARCADE data span the period
10/2018–04/2019 while FRAM data were taken between 11/2018–05/2019 and
Photometer data between 10/2018–06/2019. In the CTA South site, the FRAM data
cover 09/2017–12/2018 while the Photometer data come mostly from
08/2017–12/2018 with a small contribution from three months in 2016. Figure 3:
Time sequence of VAOD measurements from the CTA North site in 12/2018.

##  3 Aerosols

###  3.1 Instruments and uncertainties

The transparency of the atmosphere above the sites is monitored using the FRAM
telescopes [4], the Sun/Moon Photometers [5], and, at CTA North, also with the
ARCADE Raman Lidar system [6] (which also provides other quantities, such as
the vertical profiles of back-scatter and extinction coefficients and water
vapor profiles). The former two instruments provide only the integral value of
the vertical aerosol optical depth (VAOD) through the whole atmosphere, while
the Lidar measures the whole vertical profile. For the sake of comparison of
the data, we use the VAOD value at 3 km above the ground, which is above the
Atmospheric Boundary Layer, where most of the aerosols usually reside, but
still close enough to the instrument for a very precise measurement. Clearly,
this value is expected to be less or equal to the integral measurement. Due to
constraints from astronomical observatories located around the CTA North
sites, the measurements are taken only twice a day during twilight. After
accumulating data for 15 minutes, the estimated error is less than 0.015 in
VAOD at 3 km. The ARCADE system is fixed and thus the measurements are always
conducted in the vertical direction and at a single wavelength of 355 nm.

The Sun/Moon Photometer operates in two rather distinct modes. During the day,
using the light from the Sun, which is well characterized, stable and
abundant, the accuracy of the VAOD measurements is estimated to be better than
0.01 (note that the VAOD is a dimensionless quantity). During the night, if
the Moon is at least 40 % illuminated and at least 10 degrees above the
horizon, it can be used as a source as well, but the error is usually
estimated to be roughly 0.04. Our analysis shows that this can likely be
improved, using stringent quality cuts on the data and removing the residual
dependence of the calibration constant of the instrument on lunar phase (which
points to improper description of Moon brigtness in the model used) using a
fit over many lunar cycles. Considering that the corrected data show, for CTA
North, a correlation with the FRAM measurements with a mean shift of 0.0045
and RMS spread of 0.011 (see Fig. 2 in [2]), it is unlikely for both the
devices to possess a much larger uncertainty, barring a correlated systematic
shift. However the CTA South data must be interpreted with caution, possibly
because of an issue related to rapid fluctuations of the sensitivity of the
device. The Photometer measurement is a directed one, in a direction always
determined by the position of the Sun or the Moon. It has a large amount of
filters available, from which the 440-nm one is most suitable for the
comparison with other instruments.

The FRAM telescopes determine the VAOD using photometry of a large number of
stars in a large span of altitudes above the horizon and thus seen through
varying column depths of air. Assuming horizontal uniformity of aerosols at
least in the close vicinity of the instrument (several kilometers), the VAOD
can be extracted from a fit of observed light extinction as a function of the
altitude above the horizon. This method has been recently significantly
improved using laboratory measurements of various properties of the system
[7]; furthermore a thorough study of uncertainties has been done. The
statistical uncertainty of a single measurement is 0.002–0.004 and most of the
systematic uncertainties (from sources such as the subtraction of molecular
extinction, knowledge of the non-linearity of the camera and spectral response
of the system etc.) combined are found to be 0.006. A dominant uncertainty of
0.015 is related to the choice of photometric software and requires further
examination, but the outlook for the method of choosing the correct model
using internal consistency of the data is promising, which would bring the
over systematic uncertainty (correlated plus uncorrelated) to around 0.01. The
FRAM measurements are performed as a scan between the horizon and the zenith
in a chosen azimuth, giving them some directionality, but the extraction of
the VAOD assumes a degree of horizontal homogeneity anyway. Using Johnson B
filter, the bandpass spans roughly 100 nm, with an effective mean wavelength
of 440 nm for typical aerosols.

###  3.2 Results

Fig. 2 shows the comparison of the VAOD distribution measured by the different
instruments. The problematic behavior of the CTA South Photometer is obvious
from the presence of spurious negative VAOD values. The difference between the
ARCADE data and other measurements is significant compared to the stated
uncertainties and shall be investigated further, given that the Raman Lidar is
traditionally considered to be the reference instrument for aerosol
measurements – however, it is important to note, that data from all
instruments should be considered preliminary, pending improvements in the
analyses.

Another point to note is that the periods covered slightly differ and even if
the devices were covering the same periods, they did not take data at the same
time and in the same direction. In particular, the FRAM measurement requires a
full undisturbed swath of clear sky for the VAOD fit, which means that in
periods of scattered clouds, it does not produce VAOD data, while for the
Photometer, only the direction of the Moon needs to be cloud-free and for the
ARCADE measurement, any clouds higher than 3 km above observation level are
irrelevant. This dependence on weather may reflect in a bias in the
distribution, as aerosol concentration can easily be correlated with weather.
To alleviate this issue, an algorithm that selects the direction for the FRAM
observations based on real-time ASC data in order to avoid clouds as much as
possible is being implemented. Moreover, the FRAM has to specifically avoid
the direction of the Moon on the sky (as direct moonlight saturates the
camera), where the Photometer is taking data, so the effect any inhomogeneity
is exacerbated.

Figure 4:  The fraction of wind speed above a certain velocity measured at
three different heights at the CTA South site between 02/2016–07/2018. The
left panels shows data with 1-second granularity, while in the right panel,
averages over 120 seconds are used to smooth out wind gusts. Over 75 million
1-second measurements have been recorded in this time period.  Figure 5:
Cumulative distributions of temperature (top panels) and relative humidity
(bottom panels) observed at the future CTA sites. For CTA South, data from the
weather station installed on a 10-metier mast as a part of the site
characterization station are used, for CTA North, the data come from a weather
station operated by the MAGIC Collaboration on the roof of the MAGIC Counting
house 111An article describing this weather station and its results is
currently in preparation – M. Gaug, O. Blanch, L. Font, M. Doro and J.
Zapatero, _15 years of MAGIC weather station data_.. Additionally to
distributions of all measured values, the values for nighttime periods
(defined by nautical twilight) are shown separately. The data from the CTA
South site cover the period between 05/2015–12/2018 and the data from CTA
North the period between 11/2012–02/2019.

Fig. 3 shows a time sequence of VAOD observations in a selected time period in
12/2018. The typical trend of increase during the day and then decrease during
the night is obvious as is a slow change of conditions over several days. The
daytime Photometer measurements are the most reliable and thus seeing the
nighttime measurements from Photometer and FRAM interpolate well between days
is encouraging. However the data also show that the nighttime and daytime
behavior of aerosols can be significantly different and thus the improvements
in data analysis for the nighttime data are worth pursuing for the purpose of
site characterization.

From the point of view of the future CTA Observatory, both Sun/Moon Photometer
and FRAM data suggest that both sites are exceptionally clean, with very
consistent aerosol contents over time, while the ARCADE measurements could be
interpreted as slightly less favorable for the northern site.

##  4 Meteorological observations

A large amount of meteorological observations on both sites are available
thanks to the dedicated site characterization station deployed at the CTA
South site and to several major astrophysical observatories close to the CTA
North site, among all the MAGIC telescopes located directly within the future
CTA area. From the monitored variables, possibly the most impactful on the
construction and operation of the CTA Observatory is the measurement of wind
in three heights above ground at the CTA South site as shown in Fig. 4. No
wind gust over 27 m/s has been recorded at the site and the differences
between the different heights are small. Further values of interest for the
safe and efficient operations of telescopes are the distributions of
temperature and humidity (Fig. 5) – note that the CTA South site is located in
one of the driest deserts of the planet and thus the humidity is consistently
low, while at the CTA North site there are periods of high humidity when the
upper reaches of Roque de los Muchachos are engulfed by clouds. The
temperature differences between day and night are also significantly more
pronounced in the desert of the CTA South site.

## Acknowledgements

We gratefully acknowledge financial support from the agencies and
organizations listed here: www.cta-observatory.org/consortium_acknowledgments,
in particular by Ministry of Education, Youth and Sports of the Czech Republic
(MEYS) under the projects MEYS LM2015046, LTT17006 and EU/MEYS
CZ.02.1.01/0.0/0.0/16_013/0001403 and European Structural and Investment Fund
and MEYS (Project CoGraDS – CZ.02.1.01/0.0/0.0/15_003/0000437), CETEMPS-
UNIVAQ-Italy and NCN, Poland. The installation of the devices on La Palma
would not be possible without the support from the MAGIC Collaboration. This
work was conducted in the context of the CTA Consortium.

## References

  * [1] J. Ebr et al., _Atmospheric calibration of the Cherenkov Telescope Array_ , _PoS ICRC2017_ 833, 2018.
  * [2] J. Ebr et al., _Prototype operations of atmospheric calibration devices for the Cherenkov Telescope Array_ , these proceedings.
  * [3] D. Mandát and M. Pech, _All Sky Camera for CTA Site characterization_ , _EPJ Conf. Proc._ 144 01005, 2017.
  * [4] P. Janeček et al., _FRAM telescopes and their measurements of aerosol content at the Pierre Auger Observatory and at future sites of the Cherenkov Telescope Array_ , _EPJ Conf. Proc._ 197 02008, 2019.
  * [5] J. Juryšek et al., _Sun/Moon photometer for Cherenkov Telescope Array – first results_ , _PoS ICRC2017_ 853, 2018.
  * [6] M. Iarlori et al., _The Raman LIDAR for the pre-production phase of Cherenkov Telescope Array_ , _EPJ Conf. Proc._ 197 02004, 2019.
  * [7] J. Ebr et al., _New developments in aerosol measurements using stellar photometry_ , _EPJ Conf. Proc._ 197 02007, 2019.