Programmable Functional Materials Lab, Brain and Consciousness Research
Center, Moscow, 121205, Russia Center for Photonics and 2D Materials, Moscow
Institute of Physics and Technology, Dolgoprudny, 141700, Russia Programmable
Functional Materials Lab, Brain and Consciousness Research Center, Moscow,
121205, Russia Programmable Functional Materials Lab, Brain and Consciousness
Research Center, Moscow, 121205, Russia

# Terahertz photoconductivity in bilayer graphene transistors: evidence for
tunneling at gate-induced junctions

Dmitry A. Mylnikov<EMAIL_ADDRESS>Center for Photonics and 2D
Materials, Moscow Institute of Physics and Technology, Dolgoprudny, 141700,
Russia  Elena I. Titova  Center for Photonics and 2D Materials, Moscow
Institute of Physics and Technology, Dolgoprudny, 141700, Russia  Mikhail A.
Kashchenko  Programmable Functional Materials Lab, Brain and Consciousness
Research Center, Moscow, 121205, Russia  Ilya V. Safonov  Center for Photonics
and 2D Materials, Moscow Institute of Physics and Technology, Dolgoprudny,
141700, Russia  Sergey S. Zhukov  Center for Photonics and 2D Materials,
Moscow Institute of Physics and Technology, Dolgoprudny, 141700, Russia
Valentin A. Semkin  Center for Photonics and 2D Materials, Moscow Institute of
Physics and Technology, Dolgoprudny, 141700, Russia  Kostya S. Novoselov
Institute for Functional Intelligent Materials, National University of
Singapore, Singapore, 117575, Singapore  Denis A. Bandurin  Department of
Materials Science and Engineering, National University of Singapore, 117575,
Singapore  Dmitry A. Svintsov  Center for Photonics and 2D Materials, Moscow
Institute of Physics and Technology, Dolgoprudny, 141700, Russia

###### Abstract

Photoconductivity of novel materials is the key property of interest for
design of photodetectors, optical modulators, and switches. Despite the
photoconductivity of most novel 2d materials has been studied both
theoretically and experimentally, the same is not true for 2d $p$-$n$
junctions that are necessary blocks of most electronic devices. Here, we study
the sub-terahertz photocoductivity of gapped bilayer graphene with
electrically-induced $p$-$n$ junctions. We find a strong positive contribution
from junctions to resistance, temperature resistance coefficient and photo-
resistivity at cryogenic temperatures $T\sim 20$ K. The contribution to these
quantities from junctions exceeds strongly the bulk values at uniform channel
doping even at small band gaps $\sim 10$ meV. We further show that positive
junction photoresistance is a hallmark of interband tunneling, and not of
intra-band thermionic conduction. Our results point to the possibility of
creating various interband tunneling devices based on bilayer graphene,
including steep-switching transistors and selective sensors.

Understanding the interplay between optical and electrical properties of novel
materials is important for design of various optoelectronics devices,
including optical modulators Moduators_conducting_oxides, Modulators_review
and switches EO_switch_graphene_oxide, EO_switch_liquid_crystal, photo-
transistors Phototransistor_2d_double_heterojunction,
Phototransistor_negative_cap, Phototransistor_organic and photo-detectors
Photodetectors_2d_review, Photodetectors_2d_review_2. The phenomenon of
photocondcutivty (PC), the change in conduction upon illumination, enables to
extract the information about fundamental materials’ properties, such as band
gaps and spectra Teppe_PC_ZeroFieldSplitting, Teppe_PC_TopologicalTransition,
and collective excitations CR_overtones,
Transport_detection_of_collective_excitations. On the other hand, PC has
enabled the practical implementation of detectors in most electromagnetic
ranges HEB_superconductors, Bolometers_review, market. The search for
photoconductive materials is nevertheless continued, mainly to ensure fast
photoresponse and sensitivity in terahertz (THz) frequency range where
absorption of most semiconductors is blocked by the band gap
THz_detectors_review.

Among numerous novel 2d materials, graphene and its bilayer (BLG) attract a
special attention in THz detection applications bandurin_resonant_2018,
castilla_2019, gayduchenko_2021, Efetov_BLG_calorimeter. The selective light-
induced heating of electronic sub-system in carbon layers ensures fast,
potentially sub-picosecond, photoresponse tielrooij_2015. The gate-induced
doping and electric tunability of band structure enable the induction of
light-sensitive $p$-$n$ junctions without any chemical modifications
Ryzhii_graphene_pin-detectors. Though the highly-sensitive light detection by
electrically-induced junctions in graphene castilla_2019 and its bilayer
gayduchenko_2021 have been reported, the origins of photoresponse and even
transport through such junctions have yet to be understood. The complexity of
transport through junctions in BLG stems from comparable values of band gap,
Fermi energy, temperature, and disorder amplitude. Depending on sample quality
and temperature, the transport through such junctions may be dominated by
thermionic emission, variable-range hopping, or tunneling
Oostinga_bandgap_opening, alymov_2016, Levitov_oscillatory_tunneling,
VRH_conduction_BLG_1, VRH_conduction_BLG_2, Shklovsky_VRH_2d.

Figure 1: (a) Schematic representation of split-gate BLG device. Illuminated
antenna connected to source and drain acts as alternating voltage source. Top
gate voltages $V_{TG1}$ and $V_{TG2}$ define an in-plane $p$-$n$ junction,
while the back gate voltage $V_{BG}$ induces the gap (b,c) Micro-photographs
of the fabricated device: channel with split gate (b) and bow-tie antenna (c).

Here, we clarify the physics of transport in electrically-induced BLG $p$-$n$
junctions via combined experimental and theoretical studies of terahertz
photoconductivity. The structure under study represents a bilayer channel with
global back gate and split top gate. The gating architecture enables to switch
between gapless and gapped states in the channel, and between uniform and
ambipolar doping profiles simultaneously. Previously, the photoconductivity
induced by electron heating in graphene bilayer has been studied only in
’junctionless’ state, and high bolometric responsivity was demonstrated
therein yan_2012. Our study shows that PC signal is strongly enhanced once the
$p$-$n$ junctions in the channel are induced. Moreover, the observed negative
PC is opposite to that in conventional junctions, where light enhances the
conduction. We show theoretically that positive photoresistance induced by
electron heating appears if the transport at the junction is dominated by
interband tunneling alymov_2016, gayduchenko_2021. Indeed, light-induced
heating increases the electron kinetic energy and pushes them out of the
’tunneling window’ where conduction and valence bands overlap.

Our devices were fabricated by encapsulating the bilayer graphene (BLG)
between two slabs of hexagonal boron nitride (hBN) of thickness 60 nm and 40
nm for top and bottom slabs using a standard dry transfer technique
kretinin_2014 on top of silicon wafer with 280 nm SiO2. Etching of BLG/hBN
slab was used for channel patterning to rectangular shape of width $W=3$
$\mu$m and length $L=6$ $\mu$m. Subsequently, e-beam sputtering was used to
deposit top gate and quasi one-dimensional Au/Cr source and drain contacts
1d_contact. Finally, plasma etching was used to induce a narrow ($\sim 200$
nm) trench between the gates (Fig. 1). Further details of sample fabrication
detail are given in the Supplementary material, Sec. I. We have studied two
devices with similar geometry and obtained identical trends in gate-controlled
resistivity and photoconductivity.

It is important that a part of graphene channel between in-plane contact and
top gate of $300-400$ nm length is controlled by the back gate only.
Therefore, $p$-$n$ junctions in the graphene channel are induced either when
voltages of opposite polarity are applied to top gates #1 and #2 (mid-channel
junction), or when opposite polarities are applied to bottom gate and a pair
of top gates (near-contact junction).

To prove the strong effect of gate-induced $p$-$n$ junctions on charge
transport, we start with electrical measurements. BLG resistivity was measured
with a conventional lock-in technique by passing alternating current of
amplitide $I_{\rm bias}=280$ nA. The device was mounted in an optical cryostat
and cooled down to $T=25$ K temperature measured directly on silicon chip.
This temperature, if not stated otherwise, is assumed in all further results.

The measured map of two-point resistance vs voltages at the two gates, $V_{\rm
BG}$ and $V_{TG2}$, is shown in Fig. 2. In agreement with previous
measurements Oostinga_bandgap_opening, Stampfer_AEM_transport_spectroscopy,
the maximum resistance is achieved at the map diagonal. Its slope, $\Delta
V_{\rm BG}/\Delta V_{\rm TG}\approx 0.2$, determines the ratio of gates’
efficiency, and coincides well with the ratio of the effective thickness of
top and bottom dielectrics. Expectantly, the resistance maximum is achieved at
largest absolute values of back gate voltages, which is consistent with gap
opening in the channel induced by transverse field. A secondary resistance
maximum seen at $V_{\rm BG}\approx 0$ can be attributed to the neutrality
condition of regions not covered by the top gate.

A prominent feature of the measured resistance maps is the considerable
enhancement of the two-point channel resistance $R$ in case when $p$-$n$
junctions are induced. To prove the junction-related origin, we classify the
four characteristic regions in Fig. 2(a) in accordance with polarity of four
characteristic sections in the channel. For example, the notation $nnpn$ (top
left part of the map) corresponds to $n$-doping of near-contact areas,
$n$-doping under the first top gate and $p$-doping under the second one. Only
the uniformly doped region have relatively small resistance guaranteed by the
absence of the junctions. A similar asymmetry of transfer curves was seen in
several Bandurin_EdgeCurrentsShunt, Koppens_Science_VSHE (not all
Stampfer_AEM_transport_spectroscopy) previous measurements of BLG, but
scarcely attained proper attention gayduchenko_2021. Slightly higher
resistance at the $p$-side, $R_{pppp}>R_{nnnn}$ may be attributed to electron
doping of BLG by Au/Cr contacts.

Figure 2: (a,b) Resistance maps of the BLG device at $T=25$ K obtained by
sweeping (a) back gate voltage and top gate voltage $V_{TG2}$ at fixed
$V_{TG1}=0$ (b) two top gate voltages at fixed $V_{BG}=10$ V. Insets in (b)
show the band diagrams at biasing conditions marked by points 1-4 (c)
Arrhenius plot of the device resistance recorded at CNP of one of the gated
regions (while other regions are heavily doped) for two different $V_{BG}$,
blue line for $V_{BG}=10$ V ($V_{TG2}=-2.1$ V, $V_{TG1}=0.4$ V), orange line
for $V_{BG}=20$ V ($V_{TG2}=-4.2$ V, $V_{TG1}=-1.7$ V). Dashed lines represent
Arrhenius fit $R\sim\exp(\Delta/2kT)$. (d) $T$-dependent resistance of BLG
channel at different doping: $n$-doped channel with contacts $R_{e}=R_{1}$,
resistance due to induced $p$-$n$ junctions $R_{2}-R_{1}$ and $R_{3}-R_{1}$,
additional resistance of hole-doped channel $R_{4}-R_{2}$ and $R_{4}-R_{3}$,
resistance of $p$-$n$ junctions $R_{pn}$.

Further insights into junction effects on net resistance are obtained by
sweeping two top gate voltages at fixed value of $V_{BG}$. An example obtained
at $V_{BG}=+10$ V is shown in Fig. 2(b). Clearly, only $n$-doping of both
gated regions, identical to doping of near-contact areas, promotes low
resistance. Inverting the back gate voltage polarity, one obtains the
resistance map with minimal resistance in the left bottom corner (see
Supplementary material, Sec. IV). It proves that junctions of interest are
formed at the boundary between single- and double-gated regions, but not at
the metal-graphene contact.

We further perform temperature-dependent transport measurements providing
insights into mechanisms of resistivity. The $T$-dependence of resistance at
the charge neutrality point (CNP) follows the Arrhenius law,
$R=R_{0}\exp\\{\Delta/2kT\\}$, where $\Delta$ is the bandgap, see Fig. 2(c).
The extracted values of $\Delta$ at $V_{\rm BG}=10$ V and $V_{\rm BG}=20$ V
are 11 meV and 23 meV, respectively. These values are in fair agreement with
theoretical prediction $\Delta=\alpha|C_{\rm TG}V_{\rm TG}-C_{\rm BG}V_{\rm
BG}|/\varepsilon_{0}$, where $C$ is the gate-channel capacitance per unit
area, and $\alpha\approx 0.05$ eV$\cdot$nm/V Falko_bandgap_theory, yielding
$\Delta=13$ and 26 meV, respectively. At the smallest temperatures,
$T^{*}\approx 21$ K, the resistance tends to saturation indicating the
leakages through edge states and/or hopping via defect levels
Bandurin_EdgeCurrentsShunt, Oostinga_bandgap_opening.

To single out the effect of $p$-$n$ junctions on $T$-dependent resistivity, we
further show the $R(T)$-curves for four characteristic configurations with
identical absolute values of carrier density in the bulk. These
configurations, $nn$, $pn$, $np$, and $pp$, are marked by points 1-4 in Fig.
2(c). Their temperature-dependent resistances are shown in Fig. 2(d). The
resistance of uniformly-doped $n$-channel $R_{1}$ is order of 600 $\Omega$ and
increases with temperature, which is common for phonon-limited transport.
Importantly, the added resistances of non-uniform channel $R_{2}-R_{1}$ and
$R_{3}-R_{1}$ also grow with temperature. We further extract the $T$-dependent
resistivity of a single $p$-$n$ junction $R_{pn}$, which are assumed identical
in all regions. The analysis accounts for different bulk conductivities of
electrons $R_{e}$ and holes $R_{h}$ which can emerge due to different
effective masses Band_asymmetry. Under these assumptions,
$R_{2}-R_{1}=2R_{pn}+R_{h}-R_{e}$ and $R_{4}-R_{2}=R_{h}-R_{e}$, which is
sufficient for determination of $R_{pn}$. The result of extraction, shown in
Fig. 2(d) with dashed line, again grows with $T$. This lets us conclude that
$p$-$n$ junctions in BLG have a positive temperature coefficient of resistance
(TCR). This contrasts to conventional junctions in Si, Ge, and III-V
semiconductors.

Figure 3: (a-c) Maps of BLG device photo-resistance at different back gate
voltages indicated in plots. White dashed lines correspond to CNP. (d,e)
Slices of photoresistance maps at $V_{TG1}$ 2.5 V above (s1) and below CNP
(s2). (f,g) Measured temperature coefficient of resistance at slices (s1) and
(s2) recorded at $V_{BG}=10$ V. In panels (d-g), $V^{0}_{TG2}$ is the charge
neutrality voltage different for each $V_{BG}$

Now it is tempting to see whether the measured $T$-dependent resistivity of
split-gate BLG device makes any fingerprints on the electro-optical
properties. To this end, we measure the device photoconductivity upon
illumination with sub-terahertz radiation. During electro-optical
measurements, radiation from the sub-THz source (0.13 THz IMPATT diode) was
focused to the sample through Si lens glued to the back side of the substrate.
Moderate resistivity of substrate (12 $\Omega\cdot$cm, B-doped Si) ensured its
transparency. A specially designed wideband THz bow-tie antenna mylnikov_2022
was connected to source and drain contacts. The polarization of the light was
set along the axis of the bow-tie antenna. The radiation power was chosen so
that the device operated in a linear mode, and was 0.5 $\mu$W (Supplementary
material, Sec. V). The photoconductivity was measured using a double-
modulation technique: the radiation from sub-THz source was modulated at
$f_{\mathrm{light}}=34$ Hz, the while bias current through the sample
oscillated at $f_{c}=111$ Hz. The signal measured at difference frequency
$f_{\mathrm{light}}-f_{c}$ carries the information about PC.

The measured maps of relative photoresistace (PR) are presented in Figs.
3(a)-(c). Its pattern is highly non-trivial, depending on the carrier density
in the channel. First, the photoresistance is negative in the vicinity of CNPs
under the first and the second gates (marked by white dashed lines). The width
of cross-shaped negative photoresistance area grows with increasing the band
gap. Second, the photoresistance changes sign abruptly as we shift either of
top gate voltages from its CNP. The only exception of this rule occurs if the
channel is uniformly doped. In the latest case, the photoresistance signal is
very small, so that even its sign cannot be resolved. Nearly-zero signal for
uniformly doped channel persists if we invert the back gate voltage (see
Supplementary material, Sec. IV).

From the first glance, it may be tempting to explain the observed features of
THz PR by the variations of bulk temperature resistance coefficient $dR_{\rm
bulk}/dT$. Indeed, in the linear regime $\delta R_{\mathrm{THz}}=(dR/dT)\delta
T_{\mathrm{THz}}$. As radiation always heats the electrons, $\delta
T_{\mathrm{THz}}>0$, and the sign of photoresistance coincides with that of
$dR/dT$. The negative PR near the CNP is then readily explained by thermal
generation of electrons and holes as the THz radiation heats the charge
carriers. The negativity of PR should persist as far as the Fermi level stays
within the gap. This fact is reflected in a wider negative PR cross with
increasing the back gate voltage (compare Figs. 3(a) and (c)). The positive PR
away from CNP is typically explained by phonon-limited transport in the
’metallic state’, where $dR/dT$ is positive Pump-probe_PC_BLG, Pump-probe-PC-
BLG2. However, in our case, this explanation fails. Indeed, this scenario does
not explain the asymmetrically vanishing photoresistance for uniformly doped
channel.

We now argue that strong positive PR of non-uniformly doped BLG is a
consequence of peculiar $T$-dependent resistance of $p$-$n$ junctions. To this
end, we compare the measured values of $\delta R_{\mathrm{THz}}/R$ and $dR/dT$
along specific slices $s_{1}$ and $s_{2}$ of the full map. The data at slices
are shown in panels (d)-(g) of Fig. 3 and cover all four combinations of
doping ($nn$, $np$, $pn$, and $pp$). One observes that positive PR correlates
well with positive $dR/dT$ in the $np$, $pn$, and $pp$ regions away from CNP.
The absolute value of $dR/dT$ in the $nn$-region, where junctions are absent,
is smaller, and the sign of $dR/dT$ cannot be unambiguously resolved.

Figure 4: Calculated dependences of tunnel $p$-$n$ junction resistance on
temperature (dashed lines) along with experimentally extracted resistances of
two junctions (solid lines). The data is normalized to resistance at the
lowest temperature of measurements, $T_{0}=22.5$ K. Different colors
correspond to different Fermi energies in the $p$\- and $n$-regions (marked in
the inset)

The simultaneous measurements of $\delta R_{\mathrm{THz}}$ and $dR/dT$ at
various doping profiles in the channel hint on junction-induced asymmetric
photoresistance. Still, the positive sign of PR (and of $dR/dT$) of $p$-$n$
junctions in graphene bilayer raises extra questions. Indeed, if the barrier
between $p$ and $n$ sides is overcome by thermally activated carriers, the
value of $dR/dT$ should be negative, as it occurs for conventional
semiconductor junctions. Even if the conduction occurs via variable-range
hopping through defect levels Oostinga_bandgap_opening or electron-hole
puddles Shklovsky_VRH_2d (specific to narrow-gap 2d systems), this should also
result in negative TCR.

We proceed to show that positive TCR of $p$-$n$ BLG junctions can be explained
by a simple interband tunneling theory accounting for finite-temperature
smearing of the Fermi surface. The relevant band diagram is shown in the inset
of Fig. 4. Intuitively, the electrons capable of tunneling are constrained in
energy by the edges of conduction and valence bands, $E_{C}$ in the $n$-region
and $E_{V}$ in the $p$-region. Raising the temperature increases the average
kinetic energy of electron and ’pushes’ them away from the tunneling window
$E_{V}-E_{C}$. Most of thermally activated electrons are now blocked by the
gap and do not contribute to transport, and only a small fraction of thermally
activated electrons may overcome the barrier.

Direct calculations of tunneling junction conductance based on Esaki theory
confirm this idea. Assuming energy-independent tunneling probability $D_{0}$
and constant densities of states $\rho_{c}$ and $\rho_{v}$ in the conduction
and valence band, we arrive at a remarkably simple formula for $T$-dependent
tunneling conductance:

$G_{\rm tun}(T)\propto D_{0}\rho_{c}\rho_{v}[f(E_{C},T)-f(E_{V},T)],$ (1)

where $f(E,T)=[1+e^{(E-F)/kT}]^{-1}$ is the Fermi function, and $F$ is the
Fermi level (details are in Supplementary material, Sec. VI). The result of
theoretical calculations shown in Fig. 4 for various Fermi energies (not known
exactly) produces a fair agreement with experiment. Further extension of the
model including the energy-dependent density of states in BLG may improve the
agreement.

Though the observed dependences of photoresistance on doping can be explained
by variations of TCR for the overall FET structure, the variations of THz-
induced temperature increase $\delta T_{\mathrm{THz}}$ may also play role.
Particularly, the presence of extra junction resistances in non-uniformly
doped channel may block the cooling of hot electrons via leakage to the
contacts Fonf_PRX_Cooling. Simple estimates show that an average increase in
electron temperature in the presence of junction resistances $R_{pn}$ is
larger than that in uniform channel by a factor of
$1+6R_{pn}/R_{\mathrm{bulk}}$. This fact can also contribute to the observed
larger PR in the non-uniform channel, compared to the uniformly doped case.

The measured PC lets us to evaluate the performance of our device acting as a
sub-THz bolometer. The power incident on the antenna of 0.8 mm2 area of device
#2 is estimated as $P\approx 40$ nW, which accounts for angular divergence of
the beam and losses in lenses and intentionally introduced filters
(Supplementary material, Sec. V). Under such conditions, the maximum
photoresistance reaches $\delta R_{\mathrm{THz}}/R\approx 0.06$. This allows
us to estimate the voltage responsivity $r_{V}$ and noise equivalent power

${\rm NEP}^{*}=\frac{S_{\mathrm{V}}}{\delta R_{\rm THz}I_{\rm bias}/P},$ (2)

where $S_{\mathrm{V}}=S_{1/f}+S_{\mathrm{JN}}$ is the total voltage noise
comprised of Flicker ($S_{1/f}$) and Johnson-Nyquist ($S_{\mathrm{JN}}$)
components. At 34 Hz measurement frequency the measured noise $S_{\rm V}=63$
nV/Hz1/2, wich results in ${\rm NEP}^{*}=47$ pW/Hz1/2. NEP can be further
reduced at higher modulation frequencies $f^{*}\gtrsim 15$ kHz, where the
$1/f$-noise will surrender to thermal noise. This allows us to estimate the
minimum achievable ${\rm NEP}_{\rm min}^{*}\sim 3$ pW/Hz1/2. This is a good
value for the bolometric class of photodetectors, it is comparable to low-
temperature germanium bolometers low_Ge_1961. The obtained values of $r_{V}$
and ${\rm NEP^{*}}$ at 25 K are of the same order as in yan_2012 for BLG under
IR illumination in terms of external responsivity, but will become superior to
reported previously if our device is cooled to the same temperature of 5 K.

In conclusion, our measurements have revealed a strong impact of electrically-
induced $p$-$n$ junctions in bilayer graphene on its electrical conduction at
cryogenic temperatures. Further combined studies of temperature-dependent
resistivity and THz photoresitivity have revealed an unexpected increase in
junction resistance with increasing $T$. We have shown that this anomaly can
be explained with interband tunneling theory taking into account the thermal
smearing of the Fermi surface. Instructively, the measured photoresistivity of
BLG $p$-$n$ junctions is rather high, and leads to quite a large estimate of
external responsivity $r_{V}\approx 1200$ V/W and low ${\rm NEP}^{*}$ of
$3...47$ pW/Hz1/2 at $T=25$ K.

The Supporting Information contains following subsections: (I) Device
fabrication, (II) Carrier concentration and bandgap in BLG, (III) Carrier
mobility extraction, (IV) Device behaviour under inverted back gate, (V) NEP
Calculation, (VI) Calculation of the junction tunneling resistance

The work of D.M., E.T., M.K., V.S. and D.S. (sample fabrication, electrical
and THz measurements) was supported by the Russian Science Foundation, grant #
21-79-20225. The devices were fabricated using the equipment of the Center of
Shared Research Facilities (MIPT). S.Z. acknowledges the support of the the
Ministry of Science and Higher Education of the Russian Federation (grant #
FSMG-2021-0005). E.T., M.K., I.S., and K.N. thank Vladimir Potanin via Brain
and Conscionsness Research Center.

## Author declarations

### Conflict of interest

The authors have no conflicts of interest to disclose.

### Author contributions

D.S. and D.B. conceived the idea of experiment; K.N., D.B., E.T. and M.K.
established the technology of sample fabrication at MIPT; M.K. fabricated the
samples; D.M. and S.Z. designed the experiment; D.M., E.T., and V.S. performed
experimental measurements; D.M. treated the experimental data; I.S. and D.S.
developed the theoretical model; D.M., I.S. and D.S. wrote the text. All
authors contributed to the discussions of data.

### Data availability

Original data are available from the corresponding author upon a reasonable
request.


# From Interpretable Filters to Predictions of Convolutional Neural Networks
with Explainable Artificial Intelligence

Shagufta Henna, Juan Miguel Lopez Alcaraz Shagufta Henna and Juan Miguel Lopez
Alcaraz are with the Department of Computing, Atlantic Technological
University, Donegal, IrelandManuscript received April 19, 2021; revised August
16, 2021.

###### Abstract

Convolutional neural networks (CNN) are known for their excellent feature
extraction capabilities to enable the learning of models from data, yet are
used as black boxes. An interpretation of the convolutional filtres and
associated features can help to establish an understanding of CNN to
distinguish various classes. In this work, we focus on the explainability of a
CNN model called as $cnn_{explain}$ that is used for Covid-19 and non-Covid-19
classification with a focus on the interpretability of features by the
convolutional filters, and how these features contribute to classification.
Specifically, we have used various explainable artificial intelligence (XAI)
methods, such as visualizations, SmoothGrad, Grad-CAM, and LIME to provide
interpretation of convolutional filtres, and relevant features, and their role
in classification. We have analyzed the explanation of these methods for
Covid-19 detection using dry cough spectrograms. Explanation results obtained
from the LIME, SmoothGrad, and Grad-CAM highlight important features of
different spectrograms and their relevance to classification.

###### Index Terms:

Deep Learning Model Explanation, XAI, Explainable AI, Transparency in AI,
Grad-CAM, LIME, SmoothGrad.

††publicationid: pubid: 0000–0000/00$00.00 © 2021 IEEE

##  I Introduction

Artificial intelligence (AI) has several applications in various industries to
improve goods, methods, and research. Recent years have demonstrated
significant success in the AI subfield known as deep learning (DL)/neural
networks due to its good performance for multidimensional data. The deep
learning model consists of multiple processing layers, to model complex
nonlinear input-output relationships, and the ability to perform pattern
recognition and feature extraction from low-level input data. Although, these
deep learning approaches, can capture non-linear relationships in input data,
however, often comes at the cost of limited interpretability. DL models are
notorious to elude immediate interpretability by humans, thereby limiting
their applications in certain situations, such as medical diagnosis.
Explainable artificial intelligence (XAI) aims to explain and interpret the
internals of machine learning or deep learning models [1]. The role of XAI in
medical diagnosis plays a critical role, due to clinicians who want to
consider AI-driven decisions as a second opinion. The explainability of “how”
the deep learning model makes a prediction may seem a global explanation of
the network, however, the explainability of the “why” is more inclined toward
the contribution of features towards classification.

Undoubtedly, XAI provides a better environment for deep learning applications
in many ways, and it comes with limitations and challenges. As an example,
when it comes to the raw explainable results based on visualization, there are
two main concerns as reviewed by [1]. Firstly, there is human attention
incapacity to deduce XAI decision-making explanation maps. Secondly,
appropriate explainable AI techniques are unavailable to quantitatively
provide better interpretability and explanation to augment visualizations.

The challenges that XAI present in healthcare have recently been investigated
by various researchers, specifically for the clinical decision support system
(CDSS) [2]. A work by [3] found that pathologists who had local model
interpretations before the model adoption require more insights on model
characteristics for complete trustworthiness. Further, authors in [4] show
that identifying the rationale for explainability aids in the selection of XAI
approaches that can fill gaps when creating user experiences. Another work by
[5] suggest that to obtain user acceptability, a CDSS should have high
tractability, which necessitates human reasoning, stepwise inference,
explanation capability, and user-familiar terminology. Overall, the
explainability for CDSS demands easily understandable terminologies and an
explanation of the neural network architecture.

The trend of XAI for medical applications is relatively new, however, there
are some efforts in this domain to explain the role of deep neural networks.
[6] implemented a gradient-weighted class activation map (Grad-CAM) algorithm
for Covid-19 features detection using X-ray images. Another similar work by
[7] implemented a layer-wise relevance propagation (LRP) algorithm to
highlight breast cancer features in mammogram images. [8] proposed the Grad-
CAM algorithm and its derivatives for the explainability of traffic accident
anticipation in automated driving systems. Another work in [9] retrieved
important job skills from a job classification task with local interpretable
model-agnostic explanations (LIME) algorithm. Authors in [10] proposed an
explainable algorithm that analyzes the performance of various explainable
algorithms by removing noise for disease detection. [11] implemented an
explainable algorithm based on dimensionality reduction for traffic
classification. [12] presented an explainable method based on a hybrid neural
network for acoustic scene classification.

To our best knowledge, all the recent works on the explainability of neural
networks focus on model internals. There is no work, specifically
convolutional neural networks (CNN) to explain the CNN filters to interpret
features and their role in Covid-19 classification.

This research focuses on the explainability of CNN, a popular neural network
for image classfication with a particular focus on its filtres. Further, it
also investigates various explainability approaches for the local explanation
of features interpreted by CNN filtres and the decisions made by the
classification layer. The major contributions are listed as follows:

  * •

We evaluate the accuracy of CNN for Covid-19 detection using dry cough to show
its suitability for explainability. From an application point of view, this is
different from [6] and [13] because their approaches detect Covid-19 based on
X-ray and CT images respectively. On the other hand, our work focuses on cough
sounds that can support a faster and cheaper diagnosis.

  * •

We presented features visualization of CNN to show a positive correlation
between the complexity of patterns learned by the network. The proposed
convolutional filters visualization is the most relevant pattern in a deep
neural network. On the contrary, this work puts an effort to analyze the
patterns learned by the filters of each hidden layer of CNN.

  * •

We showed that the visualization of CNN layers helps to understand a clear and
distinct characteristic between a positive and a negative class to help to
interpret the model predictions. Our work focuses on features that the CNN
related to each class using the process of activation maximization. This is
different from the work of [14] which uses learned patterns to distinguish the
classes.

  * •

We evaluated various XAI methods such as Smooth-Grad, Grad-CAM, and LIME to
explain the CNN filtres and classification when applied for Covid-19 detection
using dry cough. This differs from the existing works that focus on the
application of saliency maps to explain model internals, and segmentation by
occlusion to evaluate the feature importance using inputs and outputs.

##  II Convolutional Neural Network for Covid-19 Detection

This section presents the CNN model used for Covid-19 classification using dry
cough spectrograms. The model internals as shown in Figure 1 that are later
explained using various XAI approaches in Section III.

Figure 1 shows that the CNN takes a spectrogram as input and consists of three
convolutional layers with the primary function to extract features from the
pixeled images. This is followed by a flattening process converting the two-
dimensional image to a single-dimensional matrix suitable for the fully
connected layer. The fully connected layer enables learning based on non-
linear combinations of extracted features and its output is passed to another
fully connected layer that outputs the probabilities of Covid-19 and non-
Covid-19 classes.

Algorithm 1 shows the steps of CNN used for Covid-19 detection. The algorithm
performs classification as Covid-19 or non-Covid-19 using the spectrogram
input. It computes the loss using the predicted label $\hat{y}$ and true label
$y$ that is later backpropagated to adjust the weights and bias using an
optimizer. The basic steps given in Algorithm 1 are explained in the sections
below.

Figure 1: Convolutional neural network for Covid-19 detection.

1

Data: $Spectrogram$ $X$, $w$,$b$

Result: Model $cnn_{explain}$

2 for  _each epoch_ do

3 for  _each batch x $\in X$ and c_ do

4 $z_{t}^{(1)}=\mathbb{R}\left(w^{(1)}x+b^{(1)}\right)$

5 $z_{t}^{(1)}=MaxPool\left(z_{t}^{(1)}\right)$

6 $z_{t}^{(1)}=DropOut\left(z_{t}^{(1)}\right)$

7 $z_{t}^{(2)}=\mathbb{R}\left(w^{(2)}z_{t}^{(2-1)}+b^{(2)}\right)$

8 $z_{t}^{(2)}=MaxPool\left(z_{t}^{(2)}\right)$

9 $z_{t}^{(2)}=DropOut\left(z_{t}^{(2)}\right)$

10 $z_{t}^{(3)}=\mathbb{R}\left(w^{(3)}z_{t}^{(3-1)}+b^{(3)}\right)$

11 $z_{t}^{(3)}=MaxPool\left(z_{t}^{(3)}\right)$

12 $z_{t}^{(3)}=DropOut\left(z_{t}^{(3)}\right)$

13 $z_{t}^{(f)}=Flatten\left(z_{t}^{(3)}\right)$

14 $z_{t}^{(4)}=\mathbb{R}\left(w^{(4)}z_{t}^{(f)}{+b}^{(4)}\right)$

15 $z_{t}^{(4)}=DropOut\left(z_{t}^{(4)}\right)$

16 $z_{t}^{(5)}=(\hat{y})=\sigma(w^{(4)}z_{t}^{(4)}+b^{(4)})$

17 $cnn_{explain}\leftarrow z_{t}^{(5)}$

18 $\hat{y}\leftarrow cnn_{explain}.predict()$

19 $cnn_{explain}loss=y-(\hat{y})$ $\triangleright$ calculates loss

20 backpropagation($cnn_{explain}$) $\triangleright$ backpropagate loss

21 Optimize($cnn_{explain}$) $\triangleright$ minimize loss

22

23 end for

24

25 end for

26

Algorithm 1 CNN model for Covid-19 detection

###  II-A Dry Cough Audio to Spectrograms

The neural network in Algorithm 1 takes spectrograms as an input. These
spectrograms are the result of conversion from audio to images, representing
the meaningful audio information to train a convolutional neural network.
Figure 2 shows the wave plot of the dry cough audio. The y- axis represents
the amplitude of the cough sound, whereas, the x-axis shows the time of the
audio using a sampling rate of 44,100 hertz. The wave plot in the Figure shows
approximately 308,700 data points for 7 seconds that reflects the information
contained in the audio.

Figure 2: Wave plot of dry cough audio.

Figure 3 shows the scaled Mel spectrogram generated using two steps: magnitude
spectrogram generation, and mapping from spectrogram to Mel scale. The
magnitude spectrogram is computed by applying the short-time Fourier transform
(STFT) [18].

Figure 3: Scaled Mel spectrogram.

$Y\left(m,k\right)=\ \left|S(m,k)\right|^{2}$ (1)

SFTF generates a matrix of magnitudes that are later squared to obtain an
image representation of spectrograms as given in Equation 1. In the Equation,
$m$ denotes the signal and $k$ as the window of the spectrogram computation.
Later, the frequencies of magnitude spectrogram are mapped to the Mel scale as
given in Equation 2 [15]. The scaled Mel-spectrogram in Figure 3 shows low
frequencies after amplification and high frequencies using compression with
the help of the Minmax scaling process to fit the values into the 8-bit range,
thereby retaining significant information in a smaller size image.

$m=2595{log}_{10}\left(1+\frac{f}{700}\ \right)$ (2)

Figure 4: Mel spectrogram colour exchange and image flipping.

Figure 5 shows the scaled Mel spectrogram using inversion of color intensity,
where black areas represent high and white as low energies, respectively. This
results in a final spectrogram, where the flipping of the image moves the low
frequencies at the bottom, making it ready as the input to the CNN.

###  II-B CNN Initialization

Algorithm 1 takes the weights $w$ and biases $b$ to train all the layers of
the CNN that are later adjusted to minimize the training cost or loss. The
weights in convolutional layers are interpreted as kernels, i.e., a weight
matrix multiplied with input to extract relevant features. For 2D
convolutions, kernels, as used in the Algorithm 1, are also called filtres.
The bias shifts the activation function by adding a constant value to it. The
weights of the neural network are initialized using the Glorot uniform based
on the size of the neural network as given in Equation 3 [16]. The Equation
randomly initializes the weights of the neural network using the number of
input and hidden units in the neural network. Using the $4$ input and $5$
hidden units, the Equation calculates the random weights in a range of -0.82
to +0.82. The bias is initialized with a value of $0$.

${GU}_{limits}=\ \sqrt{\frac{6}{N_{in}+N_{out}}}$ (3)

###  II-C Feature Extraction using Convolutional Filtres

Figure 5 illustrates the basic process of applying a kernel or filter to the
normalized spectrogram. It shows the complete feature map after passing the
filter through all the possible positions on the Mel-Spectrogram. The filter
represents the weight with the values initialized using the Glorot uniform
function given in Equation 3. Wee have considered a filter size of $3\times
3$. However, the size can be adjusted based on the feature retrieval specific
to the area of a Mel-Spectrogram. This filter is later multiplied with the dry
cough spectrogram, followed by a dot product with added bias later to generate
a value [17]. Finally, the resultant values are added to a feature map. The
same filter convolves from left to right and up to down with the Mel-
Spectrogram adding more values to the feature map. The values in the feature
map correspond to Mel-Spectrogram pixels while maintaining lass-specific pixel
correlation.

Figure 5: Complete feature map generated by the convolutional filters.

###  II-D Activation and Max Pooling

Equation 4 shows the activation function for the first three convolutional and
the first fully connected layer of the CNN. The activation function is based
on the Rectified Linear Unit (ReLu) activation with values between 0 and the
input value (Nair and Hinton, 2010).

$\mathbb{R}\left(z\right)=max(0,z)$ (4)

After the activation function, Max pooling is applied to the feature map. Max
pooling is considered another new filter, however, it takes only the maximum
value from each feature map region, constituting a reduced size feature map.
In Algorithm 1, we have considered a pool size of $2\times 2$, although its
size can be adjusted based on the input and convolutional filters.

###  II-E Dropout and Flattening

Once Max pooling is complete, dropout is applied for the removal of input-
output connections to regularize the network. In the neural network for
Covid-19 classification, the dropout process drops approximately 25% of the
features and is applied $4$ times in Algorithm 1. Once the dropout process is
complete, we consider the flattening process for the CNN that converts the
2-dimensional feature map to a 1-dimensional feature space to feed to the
fully connected layer as given in Algorithm 1 [19].

The fully connected layer also known as the dense layer connects each unit of
the last layer with all the units of the same layer. Our model used for
explainability as given in Algorithm 1 uses two dense layers, each with its
weights and bias.

###  II-F Covid-19 Classification and Training

This section illustrates the Covid-19 and non-Covid-19 classification using
the dry cough spectrogram using the CNN Algorithm 1. This classification is
explained later using various explainable AI approaches. Algorithm 1 shows the
use of the SoftMax activation $\sigma(z)$ function as given in Equation 5
based on the features extracted by the convolutional filters. SoftMax function
assigns a confidence score to each class by applying an exponential function.

${\sigma(z)}_{i}=\ \frac{e^{zi}}{\sum_{j=1}^{k}e^{zj}}$ (5)

Equation 6 shows how SoftMax function calculates the probability $\mathbb{P}$
of each class $y$ at given timestep $t$ from the $k$ classes, i.e., Covid-19
and non-Covid-19 from the input features $f$. The function results in a
predicted class called $\hat{y}$.

$\hat{y}_{t,k}\equiv\mathbb{P}\left(y_{t}=k|x\right)=\frac{\exp(w_{k}^{(5)}h_{t}^{(4)}+b_{k}^{(5)})}{\sum_{j}{exp(w_{j}^{(5)}h_{t}^{(4)}+b_{j}^{(5)})}}$
(6)

Line 16-18 illustrates the loss computation, i.e., the difference between true
class label $y$ and predicted class label $\hat{y}$, that is backpropagated
from the output layer to the input layer to adjust the weights of each layer
using the Adam optimizer. Adam optimizer is based on adaptive moment
estimation. It incorporates anticipated values $E$ of previous gradients as
given in Equation 7. Adam optimizer function calculates momentum using various
steps consisting of distinct parameters, thereby leading to faster
convergence.

$\theta_{t+1,i}=\theta_{t,i}-\frac{\eta}{\sqrt{E[G}_{t,ii}+\epsilon]}\cdot{E[g}_{t,i}]$
(7)

##  III Explainability of CNN for Covid-19 Detection

This section discusses the explainable AI approaches to help interpret the
deep learning model given in Algorithm 1 and explained in Section II. This
section starts with the visualization of features to help interpret the role
of these features in the Covid-19 classification. After feature
visualizations, we have used SmoothGrad Algorithm to highlight the importance
of features of a spectrogram to Covid-19 predictions. This is followed by the
GradCAM to provide visualization of the spectrogram to indicate the important
regions in a feature map in the last convolutional layer as given in Algorithm
1. Finally, explainable AI approach LIME provides the feature importance of a
spectrogram using segments.

###  III-A Feature Visualization by Optimization

We have used two approaches of visualization to explain the features and the
learning of convolutional neural network given in Algorithm 1. The first
approach as given in Algorithm 2 helps to interpret convolutional filters
using visualization, whereas the second one given in Algorithm 3 provides an
interpretation of dense layers in Algorithm 1. These algorithms involve the
use of a loss function to maximize the activation of desired neurons within
the selected layer.

Algorithm 2 helps to understand the first convolutional layer filter for
simple textures, and the second from Algorithm 3 for complicated textures and
patterns from the dry cough spectrogram. The last convolutional filter in
Algorithm 1 learns objects or fragments of objects from the spectrogram. In
the case of dry cough spectrograms, these objects are various frequencies
activated according to the dry cough audio, thereby making its interpretation
significantly challenging.

1

Data: $cnn_{explain}$ from Algorithm 1

Result: Filter $A_{desired}$

2 $model_{filters}\leftarrow\emptyset$

3 for  _each $layer_{i}$ in $cnn_{explain}$_ do

4 if  _$layer_{i}==covolutional$_ then

5 $layer_{i}^{conv}\leftarrow layer_{i}$

6 end if

7 $layer_{i}^{conv}.set\\_activation(linear)$ $\triangleright$ set the model
activation as linear $model_{filters}.inputs\leftarrow cnn_{explain}.inputs$
$model_{filters}.outputs\leftarrow layer_{i}^{conv}.outputs$ $\triangleright$
set the model activation as linear

8 end for

9for  _each $filter_{i}$ in $model_{filters}$_ do

10 if  _$filter_{i}$ is desired_ then

11 $A_{desired}\leftarrow filter_{i}$

12 return $A_{desired}$ $\triangleright$ return the desired filter for
visualization

13 end if

14

15 end for

16

Algorithm 2 Feature visualization of convolutional filters.

Data: $cnn_{explain}$ from Algorithm 1

Result: $model_{dense}$

1 for  _each $layer_{i}$ in $cnn_{explain}$_ do

2 if  _$layer_{i}==dense$_ then

3 $layer_{i}^{dense}\leftarrow layer_{i}$

4 end if

5 $layer_{i}^{dense}.set\\_activation(linear)$ $\triangleright$ set the model
activation as linear

6 $model_{dense}.param\leftarrow cnn_{explain}.param$

7 $model_{dense}.outputs\leftarrow layer_{i}^{dense}.outputs$ $\triangleright$
set the dense model output as outputs of the dense layer

8 end for

9for  _each $unit_{k}$ in $model_{dense}$_ do

10 $img_{k}\leftarrow creatImage(model_{dense}.inputSample)$

11 $model_{dense}\leftarrow maxActivation(img_{k},unit_{k}$) $\triangleright$
maximize activation of Image in each unit of dense layer

12 end for

13

Algorithm 3 Feature visualization of dense layer.

Algorithm 2 presents the steps required for the visualization of convolutional
filter. The first step extracts the required convolutional layer from the CNN
model. Later, it enables the linear activation function to take the raw input.
This is followed by the creation of a new model with the same inputs as the
first model and the output of the last layer. Slicing the model into the
desired number of convolutional layer transforms it to a new model that does
not change the first model by collecting its outputs. Lines 9 to 12 iterates
over desired filters for visualization returning an image to distinguish the
Covid-19 or non-Covid-19 classes based on specific neural network level
patterns. These patterns along with the filters and parameters are learned
during the first model training.

Algorithm 3 lists the steps used for the visualization of the last dense layer
using the SoftMax classification. The first step in the algorithm extracts the
desired layer and changes the SoftMax as an activation function for the
analysis. This change brings desirable linearity to the model outputting the
probabilities of the Covid-19/non-Covid-19 classes with a sum of 1. The second
sets the layer’s activation function to linear. Similar to Algorithm 3, the
third creates a new model based on the specification of the first model. This
is to prevent the modification to the existing model while explicitly
revealing the CNN model internals at this stage. This is followed by an
iterative process to apply the activation maximization function that generates
a random and optimized image for Covid-19/non-Covid-19 as predicted by the
Algorithm 1.

###  III-B Smooth Gradient (SmoothGrad) for cnnexplain Model Interpretation

This section presents SmoothGrad to interpret cnnexplain in Algorithm 1. The
method focuses on the explanation of the feature importance of input dry cough
Mel-spectrogram image as given in Figure 3 and outputs a sensitive map.
Specifically, the SmoothGrad aims to maximize the class score as given in
Equations 8 and 9.

$class\left(x\right)=argmax_{c\in C}S_{c}(x)\\\ $ (8)

$M_{c}\left(x\right)=\frac{\partial S_{c}(x)}{\partial x}$ (9)

Data: Image $x$, noise spread level $N(0,\sigma^{2})$, number of samples
$n$,$cnn_{explain}$ model

Result: SmoothGrad sensitive map ${M}^{{}^{\prime}}_{c}$

1 $totalImages\leftarrow x.{n}$

2for  _each image $x$ in totalImages_ do

3 $x\leftarrow x+N(0,\sigma^{2})$

4 $S_{c}(x)=cnn_{explain}.predict(x)$

5 $M_{c}(x)=\ \frac{\partial S_{c}(x)}{\partial x}$

6 ${M}^{{}^{\prime}}_{c}\leftarrow\frac{\sum_{i=1}^{n}M_{ci}(x)}{n}$
$\triangleright$ sensitive map smoothed with noise base

7 end for

8

Algorithm 4 SmoothGrad.

For the given input features $x$ extracted from the dry cough Mel-spectrogram,
Activation function in Equation 8 selects a class $c$, i.e, covid-19/non-
covid-19 from the set of classes $C\in\\{$Covid-19, non-Covid-19$\\}$ to
maximize the class score. Equation 9 computes a sensitive map
$M_{c}\left(x\right)$ for the input Mel-spectrogram where $\partial S_{c}$
represents the derivative gradient of the class score function for input
$\partial x$. This measures any change in classification score concerning the
change in each feature, i.e., pixels of the Mel-spectrogram. This map,
however, also includes noise or random pixels in Mel-spectrogram. This is
attributed to the local variations in score function $S_{c}$ during the
backpropagation process

Equation 10 selects the saliency map ${M^{\prime}}_{C}\left(x\right)$ of a
Mel-spectrogram estimating the local average of the gradient values with a
stochastic approximation. ${M^{\prime}}_{C}\left(x\right)$ is computed by
sampling similar images by adding random noise with a gaussian distribution
with a mean of 0 and standard deviation to set $N\left(0,\ \sigma^{2}\right)$,
then averaging the resultant sensitive maps for each Mel-spectrograme for $n$
samples.

${M^{\prime}}_{C}\left(x\right)=\ \frac{1}{n}\sum_{1}^{n}{M_{c}(x+N\left(0,\
\sigma^{2}\right))}$ (10)

Algorithm 4 shows steps for sensitive map calculation. Lines 2-6 illustrate
the iterative steps to apply noise using a gaussian distribution with a mean
of 0 and a standard deviation$N\left(0,\ \sigma^{2}\right)$ to each Mel-
spectrogram. This augmented Mel-spectrogram $x$ is then classified by the
cnnexplain model returning a class score. The application of noise during this
process improves sensitive map called as stochastic resonance process. This
process replicates the Mel-spectrogram with random noise that strengthens the
pixel patterns between the Mel-spectrogram samples without amplifying the
random noise. Line 5 calculates the sensitive map with the partial derivatives
of the class score ${M^{\prime}}_{C}\left(x\right)$ for the augmented Mel-
spectrogram $x$. Finally, line 6 returns the sensitive map smoothed with noise
base on the average of sensitive maps $\frac{\sum_{i=1}^{n}M_{ci}(x)}{n}$ for
each augmented Mel-spectrogram. Later, during the performance evaluation,
Algorithm 4 provides meaningful insights for Covid-19/ non-Covid-19 class for
the input Mel-spectrogram.

###  III-C Gradient-weighted Class Activation Mapping (Grad-CAM)

Similar to SmoothGrad, Grad-CAM outputs a sensitivity map for a Mel-
spectrogram image. However, in contrast to SmoothGrad , it emphasizes the
areas of importance by the feature maps in the last convolutional layer for
the Covid-19/non-Covid-19 classification. Extracting the importance of pixels
in a Mel-spectrogram by the last layer/feature map leads to the collection of
both the Covid-19 and non-Covid-19 classes that can be predicted. In our work,
we focus on a single class to conduct an analysis of class differentiation
using the activation of each sample area.

To collect the importance of each pixel in a spectrogram for image
classification, derivatives by backpropagation are required. This process can
lead to confusion in the feature’s importance collection during training.
However, in this work, we have considered the backpropagation at an inference
phase. This is given in Algorithm 5. It computes the global and averaged
pooled gradients $\alpha_{k}^{c}of$ of each feature map $A^{k}$ passed through
the ReLU.

1

Data: Image $x$, class $c$,$cnn_{explain}$ model

Result: Sensitive map $GC^{c}_{+}$

2 forward $x$ in $cnn_{explain}$ up to $h_{t}^{-1}$

3 $y^{c}\leftarrow collect(c)$ $\triangleright$ Collect raw score of desired
class $c$.

4 $y^{!c}=0$ $\triangleright$ set other classes to 0.

5

6for  _each $A\in h_{t}^{-1}$_ do

7

8 $\frac{dy^{c}}{dA^{k}}$ $\triangleright$ Backpropagate $c$ score in feature
maps of the last layer $A^{k}$.

9 end for

10

11for  _each $A\in h_{t}^{-1}$_ do

12 $\alpha_{k}^{c}=\ \frac{1}{Z}\sum i_{width}\sum j_{height}$
$\triangleright$ Global average pooling of gradient for each feature map.

13 end for

14

15$GC^{C}=\alpha_{1}A^{1}+\ \alpha_{2}A^{2}\ldots\
\sum_{K}\alpha_{k}^{c}A^{k}$ $\triangleright$ Weighted sum of feature maps.

16 $GC^{c}_{+}=ReLu(max,{GC}^{c})$ $\triangleright$ ReLu function.

Algorithm 5 Grad-CAM

Algorithm 5 presents all the steps by the GradCAM to compute the sensitive
map. Line 1 passes the Mel-spectrogram using the forward pass in the last
convolutional layer of the $cnn_{explain}$ model. Line 2-3 collects the score
of the desired class $c$, i.e., Covid-19 or non-Covid-19, and sets the score
of the other class to $0$. Lines 4-6 refer to the backpropagation of the class
score for the Mel-spectrogram in the convolutional filters to calculate the
gradients, representing the feature importance of the Mel-spectrogram as
$A^{k}$. Lines 7-9 iteratively perform global average pooling of gradients of
each feature map, returning the weights of each feature map. Line 10 computes
the weighted sum of the feature maps with their respective weights, called as
the sensitive map.

The first step is to send an image input in a forward pass in the model up to
the last convolutional layer. The second step is to collect the score of the
desired class and set the other classes’ scores to zero in step 3. Step number
four refers to the backpropagation of the class score obtained concerning the
input image but only in the convolutional filters of the convolutional last
layer to obtain the gradients, representing the feature importance of the
image. Lines 7-9 iteratively perform global average pooling of gradients of
each feature map, returning the weights of each feature map. Line 10 computes
the weighted sum of the feature maps with their respective weights, called the
sensitive map. In the later step, negative values in the sensitive map are
eliminated using the ReLu function, thereby highlighting only the positively
correlated features in the Mel-spectrogram.

W have resized the sensitive map to facilitate its visualization and
comparison for the corresponding Mel-spectrogram. This is needed as the
convolutional filters of the last layer are smaller than the original Mel-
spectrogram input because of the image transformation. Algorithm 5results in
map $GC^{c}_{+}$ highlighting the most important areas of a Mel-spectrogram by
each feature map $A$ corresponding to the last convolutional layer
$h_{t}^{-1}$ contributing significantly to the prediction covid-19 or non-
covid-19 class.

###  III-D Local Interpretable Model-Agnostic Explanations (LIME)

The fourth method, we have considered for the explainability of cnnexpain is
called the LIME algorithm. LIME considers image classification with a focus on
specific areas of an image contributing to the Covid-19/non-Covid-19 class. In
contrast to SmoothGrad and GradCAM that average sensitive maps and compute a
sensitive map from the last filter, the LIME algorithm aims to identify the
areas of importance by occlusion. LIME uses an interpretable model to
interpret a model by approximating the black-box cnnexplain. This
interpretable model supports straightforward computations to understand than
the algorithm trying to explain. LIME is also called the model-agnostic
approach due to its functional independence,i.e., zero-intervention with the
functionality of model internals. This feature makes LIME the only method to
explain the cnnexplain without seeing its internal functions.

1

Data: Image $x$, number of features $n$,$cnn_{explain}$ model

Result: Essential segments of class $y$, segmentsy

2

3segments $s$ = Cluster($x$)

4 combinations $c$ = $\frac{S!}{r!\left(S-r\right)!}$, where r = S-1, S-2

5 if  _$!(s\in c)$_ then

6 $s\leftarrow$ average as $s\prime$ for $s\in c$

7 end if

8$Px=150(c)\cdot s$ $\triangleright$ perturbed images.

9

10$P$ = cnnexplain.predict($Px$) $\triangleright$ predictions.

11for  _$p_{x}\in Px$ _ do

12 $d=\frac{p_{x}\cdot{x}}{|p_{x}||x|}$

13 $D\leftarrow D\cup d$

14 return $D$

15 $\triangleright$ return distances $D$.

16 end for

17$d_{m}in\leftarrow 0$

18 $d_{m}ax\leftarrow 1$

19

20for  _$d\in D$ _ do

21 $W\leftarrow d$

22 $LM\leftarrow createLinear()$ $\triangleright$ create interpretable linear
model

23 $y\leftarrow LM$.predict $(PX.W)$ $\triangleright$ predictions of desired
class $c$ using $LM$.

24 segments${}_{y}=$ coefficients$(n,LM)$ $\triangleright$ $n$ most
significant coefficients of LM

25 end for

26

Algorithm 6 LIME.

Algorithm 6 presents the steps required for the explainability by the LIME.
The first step segments the input image to be explained with the help of a
quick-shift segmentation algorithm. The algorithm computes the segments, also
known as super-pixels or features in the image. Line 2 in Algorithm 6 creates
the possible combinations of the segments. This involves including or
excluding these segments by the factorial product of the total segments S! and
segments r! at each combination. The inclusion of segments in each combination
provides better differentiation later in the model. Line 3-6 in the algorithm
creates 150 perturbed images based on the segment’s activations in 150 random
combinations. If a segment is excluded in the combination, an average of the
other segments is added to keep the image size excluded from meaningful
information.

Line 6 shows the Covid-19/non-Covid-19 class predictions with the
$cnn_{explain}$ on the 150 perturbed images. This returns a class core for
each perturbed image for each class. Line 8-13 starts a loo over the set of
perturbed images which computes a cosine distance towards the original image.
This distance measures the similarity or differences of the permuted image
with the original image containing all the feature segments in it. Lines 14
and 15 assign these distances in a range between 0 and 1 to interpret the
values as weights of importance for each perturbation image. Step 16 to 19
creates an easily interpretable linear model in which outputs are the
predictions towards the covid-19/non-Covid-19 class. These predictions are
based on the perturbed images with their respective importance of weights.
Lines 20-21 assign the most significant features for a class in an image,
i.e., the number of segments with more significant coefficients in the linear
model. The larger the coefficient, the larger the impact on the class
prediction. For a visual representation, we will plot only the essential
segments regarding the Covid-19 or non-Covid-19 class, and mask the rest of
the image.

##  IV Explainability of cnnexplain for Covid-19 Detection using Dry Cough

This section presents the performance evaluation and analysis of XAI methods
for Covid-19 detection using dry cough audio. It describes hardware/software,
dataset, preprocessing, model, and hyperparameters.

###  IV-A Experiment Setup

We have used Linux OS, intel Xeon CPU processor at 2.00GHz with 2 CPU cores,
13GB of RAM, and 24 gigabytes of VRAM from a Tesla K80 graphic card. To
implement the CNN model and XAI methods, we have considered libraries
including Keras, a deep learning library built on top of TensorFlow, an open-
source library for machine learning tasks;tf-Keras-vis, a deep learning
visualization toolkit for Keras and LIME for interpretable model-agnostic
explanations. Further, we have also considered Librosa [20] which is a package
for music and audio analysis.

We have used a public dataset [21] that consists of 160 Covid-19 positive
audios. The dataset includes 19 positive classes and 21 random negative
classes. The reduction of negative classes at this stage is to address the
issue of imbalanced classes, later used for training and predictions.

###  IV-B Pre-processing/Augmentation

The 40 original audios are augmented using time-stretching the audio from 30%
to 190% of the original audio. It is further combined with other techniques
including noise aggregation which adds random samples of values dispersed at
regular intervals with a mean of 0 and a standard deviation of 1. Moreover, a
time-shifting method that moves the sound to the beginning/end of the audio
along the time axis with an interval of 15% of the sample rate of 44,100 Hz is
also used. These augmentation techniques add 2720 new audios from 40 samples,
resulting in a total of 2760 audios.

Further to augmentation, we have also converted the audios to a spectrogram
using a sample rate of 44100 hertz and computing the magnitude spectrogram,
and mapping it to a Mel scale. The conversion uses 1024 samples per fast
Fourier transform window, 512 overlapping samples between successive frames to
avoid information loss, and 128 samples of Mels. Mels represent bins of the
spectrogram to its height. Finally, the data in the spectrogram is distributed
by exchanging black and white colors in the image and flipping them from top
to bottom of the spectrogram.

###  IV-C Model Parameters and Hyperparameters

The $cnn_{explain}$ model consists of 3 convolutional layers and two fully
connected layers. The input layer takes the spectrogram as $128\times 820$
pixels along with three channels representing red, green, and blue color
space. The convolutional layer is 2D and contains 16 units of filters. The
output layer uses the max-pooling process for complexity reduction resulting
in an image shape of $64\times 410$ pixels. We have considered a dropout of
20% for the output layer for regularizing the $cnn_{explain}$ to prevent
overfitting. The second convolutional layer consists of 32 filters. This layer
also has a max-pooling process that reduces the image size to $23\times 205$
pixels, and a dropout of 20% of connections. Finally, the third convolutional
layer consists of 64 filters followed by a max-pooling with $16\times 102$
with a dropout of 20%.

The $cnn_{explain}$ model uses a total of 6,708,450 trainable parameters. We
have conducted experiments with 20 training epochs per iteration with a batch
size of 128, and have trained the model using the Adam optimizer. Adam
optimizer is known for its best performance for CNN. The learning rate to
train the $cnn_{explain}$ is kept at 0.001. The activation function considered
for all the layers is ReLu with the Softmax at the output layer. An early
stopping mechanism is used to avoid overfitting and accelerate training, if
accuracy does not increases in 5 epochs.

###  IV-D Performance Analysis

As it is important to keep a balance between the positive and negative classes
to avoid bias during the model training, this section presents an exploratory
analysis of data. Further, we have evaluated the performance of the
$cnn_{explain}$ in terms of sparse cross-entropy loss and accuracy as
discussed below.

Figure 6 presents a visualization of the image’s shapes in the dataset,
precisely the width of images that vary after the augmentation of the audios.
The $cnn_{explain}$ only accepts input images with the same shapes, therefore,
this visual analysis provides insights for the best width for the resizing of
the images. The localization of the group of images with the same width size
is between 500 and 1000 pixels followed by the group between 0 and 500.
Although, these groups are close enough for resizing, there are some outliers
in categories of 3,000 pixels up to 7,000 pixels. Statistically, the mean of
the group is 1,014 pixels and the median 718. Thus, all the images are resized
with a width of 820 pixels and a height of 128 pixels.

Figure 7 and Figure 8 shows the distribution of Covid-19 and non-Covid-19
classes in the dataset. The expected distribution of the dataset is equal
because the classes are collected in similar distributions during the data
collection process. Figure 7 and Figure 8 shows the distribution of classes
after a random split of data between train and test set to avoid an imbalanced
dataset that can lead to inaccurate learning and predictions by the
$cnn_{explain}$. Figure 7 and Figure 8 show a good balance of both the
positive and negative classes, thereby assuring the efficiency of the learning
and testing process.

Figure 6: Distributions of images widths after augmentations. Figure 7:
Classes distribution in train set. Figure 8: Classes distribution in test set.
Figure 9: Loss vs. number of epochs.

To evaluate the accuracy of the model, we have changed the number of epochs
from 1 to 18. Figure 9 shows that initially, both the training and validation
loss is higher, however, it drops significantly after the first few epochs,
indicating an optimal weights adjustment by the network for a correct
prediction. Finally, it fully converges after 8 epochs. Figure 10 shows the
accuracy of the model for the training and validation. Initially,
$cnn_{explain}$ demonstrates low accuracy for both the training and
validation. However, after 8 epochs model is fully learned, thereby showing
good accuracy.

Figure 10: Accuracy vs. number of epochs.

##  V Explainability Results for $cnn_{explain}$

This section presents the explainability results of the $cnn_{explain}$ model
to understand its learning and predictions and internals. Specifically, we
analyze four methods discussed in Section III-A to explain the
$cnn_{explain}$.

###  V-A Features Visualization

This section presents feature visualization in layers to explain the
$cnn_{explain}$ internals. Specifically, it reveals that the last five filters
of the three convolutional layers of the $cnn_{explain}$ as optimal to learn
parameters. Further, there is a visualization of the last dense layer to
interpret the most significant features contributing to classification. These
features distinguish the characteristics of each class for classification with
minimum error.

Figure 11 demonstrates the patterns learned by the first convolutional layer
of the $cnn_{explain}$. This layer contains 16 units. The features learned by
this layer maximize the representation of images to aid the classification of
input images. It can be noted that there is no pattern in the first
convolutional layer that humans can identify more than a single color.
However, these colors are a fundamental basis of the $cnn_{explain}$ to form
more complex ways of understanding spectrograms in complex neural networks.

Figure 11: Convolutional filters of first convolutional layer.

Figure 12 shows the last five filters of the $cnn_{explain}$ in its second
convolutional layer consisting of 32 units of them. Similar to the first
convolutional layer, the patterns learned by the $cnn_{explain}$ are still
simple not to differentiate any clear pattern in a spectrogram. However, in
filter 29, the primary color learned by the filter is red, but there are
spotted areas with a slightly different color. However, filter number 30 has
activated the features like blue color, but a few areas of stronger blue are
mainly located in the center and lower part of the channel. Thus, at this
stage, the appearance of patterns beyond single colors can be observed.

Figure 12: Convolutional filters of second convolutional layer.

Figure 13 presents the filters that the CNN learned in the third and last
convolutional layer with 64 channels. At this stage, most $cnn_{explain}$
filters contain more meaningful information for class distinction. For
example, filters 61, 62, and 64 show different colors and distinct shapes. The
three images show some line patterns that stain the filter. The visualization
shows that the filters in the $cnn_{explain}$ ’s last layer convert from spots
found in previous layers to lines.

Figure 13: Convolutional filters of third convolutional layer.

Figure 14 shows the important input features that $cnn_{explain}$ can
distinguish to classify each class in the last layer. These features can be
seen as a combination of two colors in the spectrograms, i.e., green, and
purple. The first noticeable difference between them is the stained part of
the Covid-19 class, mainly located in the centre of the image. The
localization of this feature is for the non-Covid-19 class with less presence
in the surroundings of the spectrogram. The second difference is that the
stained feature in the non-Covid-19 class appears mainly in purple, whereas
the Covid-19 class has a presence of purple and green colors. The stained
feature has a correlated shape with the patterns visualized in the previous
layers.

Figure 14: Softmax layer.

###  V-B Spectrogram Samples

Figure 15 shows four spectrogram samples that are later used for the remaining
three explainable methods. For simplicity, four spectrograms are taken, with a
balance of classes regarding two spectrograms representing audio of Covid-19
patients and two spectrograms representing audio of patients without Covid-19.
The order of these spectrograms remains fixed throughout the following
sections.

Figure 15: Spectrograms samples.

###  V-C Smooth Gradient (SmoothGrad)

This section analyses the explainability of $cnn_{explain}$ using SmoothGrad.
During the experiments, We have selected 50 smoothed images with a 50% noise
spread level. In the visualization of results, the redder parts in the maps
symbolize the significant presence of the class activated in the spectrogram,
whereas the bluer ones show less. The first image represents the average
gradients towards the Covid-19 class, and the second image shows the average
gradients toward the non-Covid-19 class regardless of the spectrogram class.

Figure 16: SmoothGrad for Covid-19 class.

Figure 16 presents the saliency maps of the SmoothGrad algorithm for the four
images as given in Section V-B towards the Covid-19 class. The SmoothGrad
method seems to underperform the distinction of the classes due to its
inherent property of averaging gradients. The most likely Covid-19 area by the
model is the lower part of the spectrogram in the four spectrograms. However,
not all four spectrograms are in the same class.

Figure 17: SmoothGrad fornon-Covid-19 class.

Figure 17 is the second representation of the SmoothGrad algorithm toward the
non-Covid-19 class. Again, the algorithm seems to highlight the same areas of
the four spectrograms with minor differences. Also, the highlighted areas of
this non-Covid-19 class remain the same as the Covid-19 class. Thus, it seems
like an underperforming task by SmoothGrad.

###  V-D Gradient-weighted Class Activation Mapping (Grad-CAM)

This section presents the results obtained with the GradCAM algorithm.
Similarly, as seen in the last section, the explanation of the model by the
GradCAM focuses Covid-19 and non-Covid-19 classes. Figure 18 shows the four
spectrograms with activations towards Covid-19, where the red colours
represent the areas the Covid-19 relevant features. The first two Covid-19
spectrograms have a slightly less red presence than the last two in the area
with more energy indicating non-Covid-19 class. The first two Covid-19
spectrograms contain a stained feature surrounding the area with more energy,
a change in contrast to the last two spectrograms of non-Covid-19 that are
coloured as blue.

Figure 18: Grad-CAM explanation for Covid-19 class.

Figure 19 shows the four spectrograms with the activation of non-Covid-19
class where the red represent the areas with highlighted non-Covid features.
The first two Covid-19 spectrograms contain the strongest blue areas in the
image’s high-energy parts, representing the less important areas for non-
Covid-19. The last two, non-Covid-19 spectrograms have blue areas as well. As
can be observed, the higher activation of red is dominant than the blue,
representing the correct classification of non-Covid-19 case.

Figure 19: Grad-CAM for non-Covid-19 Class.

###  V-E Local Interpretable Model-Agnostic Explanations (LIME)

Similar to other explainability methods, the results with the LIME algorithm
remain the same. Figure 20 represents the activation of the Covid-19 class and
Figure 21 of non-Covid class for the four spectrograms. Figure 20 shows the
four spectrogram’s output from the LIME algorithm for the Covid-19 class. This
approach highlights stained segments seen in the first image before and after
a high-energy sound, and in the second image at the beginning of the
spectrogram.It can be noted that for the non-Covid spectrograms, the areas of
Covid-19 relevant features appear to have higher frequencies with fewer
frequency segments without clear signs distinction.

Figure 20: LIME for covid-19 class.

Figure 21 shows the results of the LIME algorithm in the four spectrograms
towards the non-Covid-19 class. The algorithm highlights segments with
relatively low stained areas. In case of non-Covid-19 spectrograms, the
algorithm highlights segments that contain waveforms with more energy of sound
as seen in the third image. The fourth image also shows areas at higher
frequencies.

Figure 21: LIME for non-Covid-19 class.

##  VI Conclusion

Most of the deep learning approaches considered for medical decisions do not
come as ’out-of-the-box’, thereby limiting their applicability for domains
such as medical or malware decisions, where wrong decisions can lead to
harmful consequences. In the context of Covid19 detection, full
comprehensibility of the deep learning approach is hard to achieve, although
the facilitated predictions are useful to the practitioner. It is crucial to
validate predictions by providing interpretations of the model and the
contribution of relevant features to classification. XAI can address these
challenges by producing more explainable models. It helps humans to interpret,
trust and understand the emerging generation of machine learning models. Our
work presents the explainabiility of the CNN model called $cnn_{explain}$
using four XAI techniques such as visualization, SmoothGrad, Grad-CAM, and
LIME. These approaches explain the classification of the $cnn_{explain}$ model
for Covid-19 and non-Covid-19. The interpretability provided by these XAI
methods contributes to more reliable decisions by providing insights into an
input space’s relevance and analysis of intermediate relevance filtres at
$cnn_{explain}$ ’s hidden layers. The results by all the XAI methods have
yielded encouraging results from our effort in the explanation of CNN, i.e.,
$cnn_{explain}$ in terms of its processing, network representation, filtre
level explanation, and classification.

Initially, we have evaluated the performance of $cnn_{explain}$ with a
training accuracy of 98.8% after a few epochs for Covid-19 and non-Covid-19.
This ensures that $cnn_{explain}$ model fulfills the requirements of good
accuracy for Covid-19 detection, thereby making the model a good candidate for
the explanation.

The visualization of convolutional filters clearly shows the increasing
complexity of patterns learned by the $cnn_{explain}$ that allows the correct
classification of the classes. Also, the dense layer shows the features that
most likely represent a class by the $cnn_{explain}$. The results show that
there is a stained feature that is clearly present in both classes that has a
positive correlation with the shapes found in the filters. However, the
presence of features is greater in the positive class than in the negative
class. Further, the presence of that feature is more prominent for the purple
color in the positive class, rather than the green in the negative.

The sensitive maps from the SmoothGrad and Grad-CAM algorithm highlight
important features of a sample as does the segments retrieved from the LIME
algorithm. SmoothGrad aims to eliminate noise from the sensitive maps,
however, it underperforms in its application on spectrograms for the use case,
due to its sensitive map showing little distinction for the interpretation of
classes. Finally, the remaining two algorithms show some areas that are
highlighted for the positive Covid-19 class with stained segments, mainly
beside the sounds with high energy. The LIME algorithms seem to highlight
areas with waveform in high-energy areas for the negative non-Covid-19 class.
Unlike the Grad-CAM algorithm that seems to relate to higher frequency areas
for the negative class.

Furthermore, results of the LIME show it as a suitable explanation model for
neural network architectures such as convolutional neural networks, or for
types of classification problems (e.g., Covid-19 and non-Covid-19). More
generally, interpretability by these selected models can contribute to the
design of accurate and efficient classifiers, not only by analyzing and
leveraging the features’ relevance but also through the analysis of
intermediate relevance features “at classifier ’s filtres within hidden
layers”. The results from this research have yielded encouraging results from
our efforts in the explanation of convolutional neural network processing with
a filter-level explanation.

## Acknowledgments

This work is supported by the Atlantic Technological University (ATU), Donegal
Campus, Ireland.

## References

  * [1] A. Das and P. Rad, Opportunities and challenges in explainable artificial intelligence (XAI): A survey. arXiv. preprint. [Online]. arXiv:2006.11371
  * [2] A.M. Antoniadi, Y. Du, Y. Guendouz, L. Wei, C. Mazo, and B.A. Becker, “Current challenges and future opportunities for XAI in machine learning-based clinical decision support systems: a systematic review,” Applied Sciences, vol. 11, pp. 5088, 2021.
  * [3] C. J. Cai, S. Winter, D. Steiner,L. Wilcox, and M. Terry, “SHello AI”: Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making,” in Proc. ACM on Human-computer Interaction Conf., 2019, pp. 1–24.
  * [4] Q. V. Liao, D. Gruen, and S. Miller, “Questioning the AI: informing design practices for explainable AI user experiences,” in Proc. CHI Conf. on Human Factors in Computing Systems, 2020, pp. 1–15.
  * [5] T. Z. Tan, G. S. Ng, and C. Quek, “Improving tractability of Clinical Decision Support system,” in IEEE Proc. Conf. on Neural Networks, 2008, pp. 1997–2002.
  * [6] J. Y. Jeon, W. S. Choi, N.H. Kim, and K.Y. Lee, “Evaluation of scalability and degree of fine-tuning of deep convolutional neural networks for COVID-19 screening on chest X-ray images using explainable deep-learning algorithm,” Journal of Personalized Medicine, vol. 4, no. 10, pp. 213, 2020
  * [7] M. Montebello, M. La Ferla, and D. Seychell, An XAI Approach to Deep Learning Models in the Detection of Ductal Carcinoma in Situ arXiv. preprint. [Online]. arXiv:210
  * [8] Y. Li, Y. and R. Qin, Towards explainable artificial intelligence (XAI) for early anticipation of traffic accidents. arXiv. preprint. [Online]. arXiv:2108.00273
  * [9] I.H. Kim, and C.K. Lee , “A Study of the Classification of IT Jobs Using LSTM and LIME,” in IEEE Proc. Conf. on on Smart Media and Applications, 2020, pp. 248–252.
  * [10] E. Nigri,N. Ziviani, F. Cappabianco, A. Antunes, and A. Veloso , “Explainable deep CNNs for MRI-based diagnosis of Alzheimer’s disease,” in IEEE Proc. Conf. on Neural Networks (IJCNN), 2020, pp. 1–8.
  * [11] J. Y. Jeon, W. S. Choi, N.H. Kim, and K.Y. Lee, “Explaining Deep Learning-Based Traffic Classification Using a Genetic Algorithm,” IEEE Access, vol. 9, pp. 4738–4751, 2020.
  * [12] Z. Ren, Q. Kong, J. Han, M.D. Plumbley, and B. W. Schuller, “CAA-Net: Conditional Atrous CNNs with Attention for Explainable Device-robust Acoustic Scene Classification,” IEEE Transactions on Multimedia. , vol. 23, pp. 4131–4142 , 2021.
  * [13] H. Alshazly, C. Linse, E. Barth, and T. Martinetz, “Explainable covid-19 detection using chest CT scans and deep learning,” IEEE Transactions on Multimedia. , vol. 21, no. 2, pp. 455 , 2021.
  * [14] G. Gong, L. Lin, Z. Wu, Z., R. Xu, S. Kido, and Y. Hu, “Evolutionary Neural Network and Visualization for CNN-based Pulmonary Textures Classification,” in Proc. IEEE 32nd International Conf. on Tools with Artificial Intelligence (ICTAI), 2020, pp. 558–563.
  * [15] Y. Cai, “Computing with Instinct: Rediscovering Artificial Intelligence,” in Springer, 2011, pp. 30.
  * [16] A. Glassner, “Deep Learning: A Visual Approach,” in No Starch Press, 2021, pp. 325.
  * [17] J. Brownlee, “Deep Learning for Computer Vision,” in Machine Learning Mastery, 2019, pp. 109.
  * [18] B. Boashash, “Time-frequency signal analysis and processing: a comprehensive reference,” in Academic press , 2015.
  * [19] V. Bhateja, S. Chandra, Y. D. Zhang, and M. Aradhya, “Intelligent Computing and Communication,” in Springer , 2019, pp. 740.
  * [20] Librosa.org, “Librosa,” in Available from: https://librosa.org/ , [Accessed 22 June 2021].
  * [21] Kaggle.com, “ COVID-19 Cough Recordings,” in Available from: https://www.kaggle.com/himanshu007121/coughclassifier-trial , [Accessed 28 June 2021].

|  Shagufta Henna is lecturer with the Atlantic Technological University,
Donegal, Ireland. She was a post-doctoral researcher with the Waterford
institute of technology, Waterford, Ireland from 2018 to 2019. She received
her doctoral degree in Computer Science from the University of Leicester, UK
in 2013. She is an Associate Editor for IEEE Access,EURASIP Journal on
Wireless Communications and Networking, IEEE Future Directions, and Human-
centric Computing and Information Sciences, Springer. She is senior member of
IEEE. Her current research interests include edge computing, self-supervised
learning, attention in deep learning, representation learning, deep
reinforcement learning, explainable AI, and AI-driven network optimizations.
---|---
Juan Miguel Lopez Alcaraz  is a Master stundent of Big Data Analytics and
Aritificial Intelligence. His research interests include machine learning,
explainable AI, and big data analytics.
---