DFT

    discrete Fourier transform
MVDR

    minimum variance distortionless response
PDF

    probability density function
MMSE

    minimum mean square error
ML

    maximum likelihood
SNR

    signal-to-noise ratio
MAP

    maximum a posteriori
ASR

    automatic speech recognition
POLQA

    perceptual objective listening quality analysis
MOS

    mean opinion score
PESQ

    perceptual evaluation of speech quality
EM

    expectation maximization
STFT

    short-time Fourier transform
PSD

    power spectral density
SI-SDR

    scale-invariant signal-to-distortion ratio
SI-SIR

    scale-invariant signal-to-interference ratio
SI-SAR

    scale-invariant signal-to-artifacts ratio
DNN

    deep neural network

# Nonlinear Spatial Filtering in Multichannel Speech Enhancement

Kristina Tesch, , and Timo Gerkmann The authors are with the Signal Processing
Group, Department of Informatics, Universität Hamburg, 22527 Hamburg, Germany
(e-mail<EMAIL_ADDRESS>timo.gerkmann@uni-hamburg.de).

###### Abstract

The majority of multichannel speech enhancement algorithms are two-step
procedures that first apply a linear spatial filter, a so-called beamformer,
and combine it with a single-channel approach for postprocessing. However, the
serial concatenation of a linear spatial filter and a postfilter is not
generally optimal in the minimum mean square error (MMSE) sense for noise
distributions other than a Gaussian distribution. Rather, the MMSE optimal
filter is a _joint spatial and spectral nonlinear_ function. While estimating
the parameters of such a filter with traditional methods is challenging,
modern neural networks may provide an efficient way to learn the nonlinear
function directly from data. To see if further research in this direction is
worthwhile, in this work we examine the potential performance benefit of
replacing the common two-step procedure with a joint spatial and spectral
nonlinear filter.

We analyze three different forms of non-Gaussianity: First, we evaluate on
super-Gaussian noise with a high kurtosis. Second, we evaluate on
inhomogeneous noise fields created by five interfering sources using two
microphones, and third, we evaluate on real-world recordings from the CHiME3
database. In all scenarios, considerable improvements may be obtained. Most
prominently, our analyses show that a nonlinear spatial filter uses the
available spatial information more effectively than a linear spatial filter as
it is capable of suppressing more than $D-1$ directional interfering sources
with a $D$-dimensional microphone array without spatial adaptation.

###### Index Terms:

Multichannel, speech enhancement, nonlinear spatial filtering, neural networks

##  I Introduction

In our everyday life, we are surrounded by background noise for example
traffic noise or competing speakers. Hence, speech signals that are recorded
in real environments are often corrupted by noise. Speech enhancement
algorithms are employed to recover the target signal from a noisy recording.
This is done by suppressing the background noise or reducing other unwanted
effects such as reverberation. This way, speech enhancement algorithms aim to
improve speech quality and intelligibility. Their fields of application are
manifold and range from assisted listening devices to telecommunication all
the way to automatic speech recognition (ASR) front-ends [1, 2].

If the noisy speech signal is captured by a microphone array instead of just a
single microphone, then not only tempo-spectral properties can be used to
extract the target signal but also spatial information. Spatial filtering aims
at suppressing signal components from other than the target direction. The
filter-and-sum beamforming approach [3, Sec. 12.4.2] achieves this by
filtering the individual microphone signals and adding them. In the frequency
domain, this means to compute the scalar product between a complex weight
vector and the vector of spectral representations of the multichannel noisy
signal. Hence, the beamforming operation is linear with respect to the noisy
input.

The beamforming weights are chosen to optimize some performance measure. For
example, minimizing the noise variance subject to a distortionless constraint
leads to the well-known minimum variance distortionless response (MVDR)
beamformer [4, Sec. 3.6]. The noise suppression capability of such a spatial
filter alone is often not sufficient and a single-channel filter is applied to
the output of the spatial filter to improve the speech enhancement
performance. The second processing stage in this two-step processing scheme is
often referred to as the postfiltering step.

Single-channel speech enhancement has a long research history that has led to
a variety of solutions like the classic single-channel Wiener filter [3, Sec
11.4] or other estimators derived in a statistical framework [5, 6, 7]. Many
recent advances in single-channel speech enhancement are driven by the
modeling capabilities of deep neural networks [8, 9, 10, 11].

(a)

(b)

Figure 1: (a) Illustration of the commonly employed two-step processing using
a linear spatial filter (beamformer) followed by a single-channel postfilter.
(b) Illustration of the nonlinear spatial filter investigated in this paper,
which joins the spatial and spectral processing into a non-separable nonlinear
operation.

It seems convenient to independently develop a spatial filter and a postfilter
and combine them into a two-step procedure afterward as shown in Figure 1. If
the noise follows a Gaussian distribution, this approach can even be regarded
as optimal in the MMSE sense as Balan and Rosca [12] have shown that the MMSE
solution can always be separated into the linear MVDR beamformer and a
postfilter. However, this separability into a linear spatial filter and a
postfilter only holds under the restrictive assumption that the noise is
Gaussian distributed. The work of Hendriks et al. [13] points out that the
MMSE optimal solution for non-Gaussian noise joins the spatial and spectral
processing into a single nonlinear operation. Throughout this work, we call
such an approach a _nonlinear spatial filter_ for brevity even though spectral
processing steps are also included. An illustration is given in Figure 1.

The result of Hendriks et al. reveals that the common two-step multichannel
processing scheme cannot be considered optimal for more general noise
distributions than a Gaussian distribution. This leads to the question if we
should invest in the development of nonlinear spatial filters for example
using DNN. Today, single-channel approaches often use the possibilities of DNN
to learn complex nonlinear estimators directly from data. In contrast, the
field of multichannel speech enhancement is dominated by approaches that use
DNN only for parameter estimation of a beamformer [14, 15] or restrict the
network architecture in a way that a linear spatial processing model is
preserved [16]. Only a few approaches with and without DNN [17, 18, 19] have
been proposed that extend the spatial processing model to be nonlinear. Still,
the questions of how much we can possibly gain by doing this, in which
situations, and also where the benefit of using a nonlinear spatial filter
comes from have not been addressed adequately. These are the questions that we
aim to investigate in this paper.

This work is based on a previous conference publication [20]. In [21] we have
studied related aspects of these questions. Here, we extend our previous work
by more detailed derivations and new analyses that provide some insight into
the functioning of the nonlinear spatial filter. In Section III, we provide a
detailed overview of the theoretical results from a statistical perspective.
We include the previously outlined results and also provide a new simplified
proof for the finding of Balan and Rosca in [12]. We then evaluate the
performance benefit of a nonlinear spatial filter for heavy-tailed noise in
Section IV-A, for an inhomogeneous noise field created by five interfering
human speakers in Section IV-B, and real-world noise recordings in Section
IV-C. In Section V, we investigate the improved exploitation of spatial
information by the nonlinear spatial filter and discuss practical issues of
the used analytic nonlinear spatial filter. Even though nonlinear spatial
filters would most likely be implemented using DNN in the future, in our
analyses we rely on statistical MMSE estimators to provide more general
insights than by using DNN-based nonlinear spatial filters which would be
highly dependent on the network architecture and training data.

##  II Assumptions and notation

We assume that the signals recorded by a $D$-dimensional microphone array
decompose into a target speech and a noise component. For each microphone-
channel $\ell\in\\{1,...,D\\}$, we segment the time-domain signal into
overlapping windows and transform the signal to the frequency domain using the
discrete Fourier transform (DFT) to obtain the DFT coefficients
${Y}_{\ell}(k,i)$ with frequency-bin index $k$ and time-frame index $i$.
Throughout this work, we use segments of length 32 ms with 16 ms shift and
apply the square-root Hann function for spectral analysis and synthesis. By
the additive signal model, the noisy DFT coefficient can be written as the sum
of the clean speech and the noise DFT coefficients
${S}_{\ell}(k,i)\in\mathbb{C}$ and $N_{\ell}(k,i)\in\mathbb{C}$, i.e.,

$\displaystyle{Y}_{\ell}(k,i)$ $\displaystyle=$
$\displaystyle{S}_{\ell}(k,i)+N_{\ell}(k,i).$ (1)

As we model the DFT coefficients to be random variables and assume
independence with respect to the frequency-bin and time-frame index, we drop
the indices $(k,i)$ to simplify the notation. We indicate random variables
with uppercase letters and use lowercase letters for their respective
realization. Furthermore, we assume all DFT coefficients to be zero-mean and
speech and noise to be uncorrelated.

We stack the noisy and noise DFT coefficients into vectors
$\bm{\mathbf{Y}}=[{Y}_{1},{Y}_{2},...,{Y}_{\ell}]^{T}\in\mathbb{C}^{D}$ and
$\bm{\mathbf{N}}=[N_{1},N_{2},...,N_{\ell}]^{T}\in\mathbb{C}^{D}$and obtain
the vector of speech DFT coefficients $\bm{\mathbf{S}}\in\mathbb{C}^{D}$ by
multiplying the clean speech signal coefficient ${S}\in\mathbb{C}$ with the
so-called steering vector $\bm{\mathbf{d}}\in\mathbb{C}^{D}$, which accounts
for the propagation path between the target speaker and the microphones. We
can then rewrite the signal model as

$\displaystyle\bm{\mathbf{Y}}$ $\displaystyle=$
$\displaystyle\bm{\mathbf{d}}{S}+\bm{\mathbf{N}}.$ (2)

The noise correlation matrix is denoted by
$\bm{\mathbf{\Phi}}_{n}=\mathbb{E}[\bm{\mathbf{N}}\bm{\mathbf{N}}^{H}]\in\mathbb{C}^{D\times
D}$ with the statistical expectation operator $\mathbb{E}$ and $(\cdot)^{H}$
denoting the Hermitian transpose. The spectral power of the target speech
signal is given by $\sigma_{s}^{2}=\mathbb{E}[|{S}|^{2}]\in\mathbb{R}^{+}$.
When appropriate, we use the polar representation for complex-valued
quantities, e.g., ${s}=|{s}|e^{j\varphi_{s}}\in\mathbb{C}$, and then let
$\varphi$ denote the phase of the complex number.

##  III Linearity of the optimal spatial filter

In this section, we aim to provide a more complete picture of the nature of
the optimal spatial filter by aggregating existing results and presenting more
straightforward derivations for some of these. We identify the noise
distribution as the key to linearity versus non-linearity of the spatial
filter and also to the separability of spatial and spectral processing.
Accordingly, in our considerations we distinguish the two cases of Gaussian
distributed noise and non-Gaussian distributed noise or, more precisely, noise
that follows a Gaussian mixture distribution.

###  III-A Gaussian Noise

We start with revisiting the results from Balan and Rosca [12] and then
provide a simplified proof that may be easier to follow. We assume that the
vector of noise DFT coefficients $\bm{\mathbf{N}}$ follows a multivariate
complex Gaussian distribution with zero mean and covariance matrix
$\bm{\mathbf{\Phi}}_{n}$, i.e.,
$\bm{\mathbf{N}}\sim\mathcal{CN}(0,\bm{\mathbf{\Phi}}_{n})$. As we employ an
additive signal model, the conditional distribution of the noisy DFT
coefficient vector $\bm{\mathbf{Y}}$ given information on the reference clean
speech DFT is a multivariate complex Gaussian distribution centered around the
vector of clean speech DFT coefficients $\bm{\mathbf{d}}{s}$ with the same
covariance matrix $\bm{\mathbf{\Phi}}_{n}$. The corresponding conditional
probability density function (PDF) is given by [22, Thm. 15.1]

$\displaystyle p_{\bm{\mathbf{Y}}}(\bm{\mathbf{y}}|{s})$ $\displaystyle=$
$\displaystyle\dfrac{1}{\pi^{D}|\bm{\mathbf{\Phi}}_{n}|}\exp\left\\{-(\bm{\mathbf{y}}-\bm{\mathbf{d}}{s})^{H}\bm{\mathbf{\Phi}}_{n}^{-1}(\bm{\mathbf{y}}-\bm{\mathbf{d}}{s})\right\\}.$
(3)

Our goal is to show that the linear MVDR beamformer defined as

$\displaystyle
T\textsubscript{MVDR}(y)=\dfrac{\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{n}^{-1}\bm{\mathbf{y}}}{\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{n}^{-1}\bm{\mathbf{d}}}$
(4)

is the optimal spatial filter with respect to the maximum a posteriori (MAP),
MMSE and maximum likelihood (ML) optimization criterion if the noise follows a
Gaussian distribution.

Balan and Rosca [12] rely on the concept of sufficient statistics to prove the
property in question for the MMSE optimization criterion. In our context, the
MVDR beamformer $T\textsubscript{MVDR}$ is a _sufficient statistic in the
Bayesian sense_ if

$\displaystyle p_{{S}}({s}|\bm{\mathbf{y}})$ $\displaystyle=$ $\displaystyle
p_{{S}}({s}|T\textsubscript{MVDR}(\bm{\mathbf{y}}))$ (5)

holds for every observation $\bm{\mathbf{y}}$ and every prior distribution of
${S}$ [23, Thm. 2.4]. We infer from (5) that all information about ${S}$
contained in the noisy observation is retained in the output of the MVDR
beamformer despite the fact that the MVDR beamformer reduces the dimension of
the multidimensional input to one dimension. Note that the variable of
interest ${S}$ in the above definition is a random variable. In contrast,
$T\textsubscript{MVDR}$ is a _sufficient statistic in the classical sense_ for
the true clean speech DFT coefficient ${s}$, which is not assumed to be a
random variable, if the conditional distribution of the noisy observation
$\bm{\mathbf{Y}}$ given $T\textsubscript{MVDR}(\bm{\mathbf{Y}})$ does not
depend on ${s}$ [24, Def. IV.C.1].

As a first step, Balan and Rosca deduce that the MVDR beamformer is a
sufficient statistic in the classical sense from the Fisher-Neyman
factorization theorem [24, Prop. IV.C.1][25, Cor. 2.6.1], which is applicable
since the conditional PDF of the observation $\bm{\mathbf{Y}}$ given ${S}$ in
(3) can be rewritten as

$\displaystyle p_{\bm{\mathbf{Y}}}(\bm{\mathbf{y}}|{s})$ $\displaystyle=$
$\displaystyle\underbrace{\frac{1}{\pi^{D}|\bm{\mathbf{\Phi}}_{n}|}\exp\\{-\bm{\mathbf{y}}^{H}\bm{\mathbf{\Phi}}_{n}^{-1}\bm{\mathbf{y}}\\}}_{h(\bm{\mathbf{y}})}$
(6)
$\displaystyle\times\underbrace{\exp\left\\{\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{n}^{-1}\bm{\mathbf{d}}\left(2\operatorname*{Re}\left\\{{s}^{*}T\textsubscript{MVDR}(\bm{\mathbf{y}})\right\\}-|{s}|^{2}\right)\right\\}}_{g({s},T\textsubscript{MVDR}(\bm{\mathbf{y}}))}$
$\displaystyle=$ $\displaystyle
h(\bm{\mathbf{y}})g({s},T\textsubscript{MVDR}(\bm{\mathbf{y}}))$
$\displaystyle=$ $\displaystyle h(\bm{\mathbf{y}})g({s},{z}).$

under the Gaussian noise assumption. In the last line of equation (6), we
replaced the random variable $T\textsubscript{MVDR}(\bm{\mathbf{Y}})$ with
${Z}$, i.e.,

$\displaystyle{Z}$ $\displaystyle=$ $\displaystyle
T\textsubscript{MVDR}(\bm{\mathbf{Y}}),$ (7)

and will now continue to use this substitute when it improves the readability.
In a second step, Balan and Rosca conclude that the MVDR beamformer is a
sufficient statistic in the Bayesian sense because any statistic that is
sufficient in the classical sense is also sufficient in the Bayesian sense
[23, Thm. 2.14.2].

We now provide a proof of the $T\textsubscript{MVDR}$ being a sufficient
statistic of ${S}$ in the Bayesian sense, which does not require a reference
to advanced stochastic theorems. For this, we compute a factorization of the
likelihood PDF of ${Z}$ $p_{Z}({z}|{s})$ with ${Z}$ defined in (7) as the
output of the MVDR beamformer for the noisy input $\bm{\mathbf{Y}}$. From the
properties of the multivariate complex Gaussian distribution undergoing a
linear transformation [22, Appx. 15B], we infer that ${Z}$ given ${S}$ is
distributed according to a one-dimensional complex Gaussian distribution with
mean ${s}$ and variance
$(\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{n}^{-1}\bm{\mathbf{d}})^{-1}$, i.e.,

$\displaystyle p_{Z}({z}|{s})$ $\displaystyle=$
$\displaystyle\mathcal{CN}\left({s},(\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{n}^{-1}\bm{\mathbf{d}})^{-1}\right).$
(8)

The corresponding PDF at the output of the beamformer can be factorized as

$\displaystyle p_{Z}({z}|{s})$ $\displaystyle=$
$\displaystyle\underbrace{\frac{\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{n}^{-1}\bm{\mathbf{d}}}{\pi}\exp\\{-\bm{\mathbf{y}}^{H}\bm{\mathbf{\Phi}}_{n}^{-1}\bm{\mathbf{y}}\left|{z}\right|\\}}_{f(\bm{\mathbf{y}})}$
$\displaystyle\IEEEeqnarraymulticol{3}{r}{\times\underbrace{\exp\left\\{\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{n}^{-1}\bm{\mathbf{d}}\left(2\operatorname*{Re}\left\\{{s}^{*}{z}\right\\}-|{s}|^{2}\right)\right\\}}_{g({s},{z})}}$
(9) $\displaystyle=$ $\displaystyle f(\bm{\mathbf{y}})g({s},{z}).$

Using (6) we rewrite the posterior distribution as

$\begin{IEEEeqnarraybox}[]{[}][c]{rCl}p_{{S}}({s}|\bm{\mathbf{y}})&=&\dfrac{p(\bm{\mathbf{y}}|{s})p({s})}{\int_{\mathbb{C}}p(\bm{\mathbf{y}}|{s})p({s})ds}\\\
&=&\dfrac{h(\bm{\mathbf{y}})g({s},{z})p({s})}{\int_{\mathbb{C}}h(\bm{\mathbf{y}})g({s},{z})p({s})ds}.\end{IEEEeqnarraybox}$
(10)

Since the term $h(\bm{\mathbf{y}})$ in the denominator does not depend on the
integration variable ${s}$, this term cancels with the corresponding term in
the numerator. Next, we extend the fraction with the term $f(\bm{\mathbf{y}})$
from (III-A) to obtain

$\begin{IEEEeqnarraybox}[]{[}][c]{rCl}p_{s}({s}|\bm{\mathbf{y}})&=&\dfrac{f(\bm{\mathbf{y}})g({s},{z})p({s})}{\int_{\mathbb{C}}f(\bm{\mathbf{y}})g({s},{z})p({s})ds}\\\
&=&p_{S}({s}|{z})\\\
&=&p_{S}({s}|T\textsubscript{MVDR}(\bm{\mathbf{y}})),\end{IEEEeqnarraybox}$
(11)

which is the identity we wanted to prove (cf. (5)). Consequently, as the
posterior given the noisy observation $\bm{\mathbf{y}}$ equals the posterior
given the output of the MVDR beamformer
$T\textsubscript{MVDR}(\bm{\mathbf{y}})$, we find that

$\displaystyle T\textsubscript{MAP}(\bm{\mathbf{y}})$ $\displaystyle=$
$\displaystyle\operatorname*{arg~{}max~{}}_{{s}\in\mathbb{C}}p_{S}({s}|T\textsubscript{MVDR}(\bm{\mathbf{y}}))$
(12)

holds. The MVDR beamformer reduces its multidimensional input to a single-
channel output and, therefore, the right-hand side of (12) can be seen as a
single-channel postfilter working on the output of the MVDR beamformer. Since
the MMSE estimator complies with the mean of the posterior, a similar
decomposition in a linear spatial filter and a spectral postfilter is given by

$\begin{IEEEeqnarraybox}[]{[}][c]{rCl}T\textsubscript{MMSE}(\bm{\mathbf{y}})&=&\mathbb{E}[{S}|\bm{\mathbf{y}}]\\\
&=&\mathbb{E}[{S}|T\textsubscript{MVDR}(\bm{\mathbf{y}})].\end{IEEEeqnarraybox}$
(13)

Because the relationship (5) holds for all prior distributions of ${S}$, a
decomposition of the MAP and MMSE estimators into a linear spatial filter
followed by a postfilter exist independently from any further assumptions
regarding the prior distribution of the clean speech DFT coefficient.

Finally, we consider the ML estimator. Starting from (6) and exploiting the
monotony of the logarithm and Euler’s formula, we find the representation

$\displaystyle T\textsubscript{ML}(\bm{\mathbf{y}})$ $\displaystyle=$
$\displaystyle\operatorname*{arg~{}max~{}}_{{s}\in\mathbb{C}}p_{\bm{\mathbf{Y}}}(\bm{\mathbf{y}}|{s})$
$\displaystyle=$
$\displaystyle\operatorname*{arg~{}max~{}}_{{s}\in\mathbb{C}}2\operatorname*{Re}\\{{s}^{*}\underbrace{T\textsubscript{MVDR}(\bm{\mathbf{y}})}_{=z}\\}-|{s}|^{2}$
$\displaystyle=$
$\displaystyle\operatorname*{arg~{}max~{}}_{{s}\in\mathbb{C}}2\cdot|{s}|\cdot|z|\cdot\cos(\varphi_{z}-\varphi_{s})-|{s}|^{2}.$

Clearly, this function is maximized when the phase of ${s}$ matches the phase
of $T\textsubscript{MVDR}(\bm{\mathbf{y}})$, as then the cosine function is
maximized. Equating the derivative with respect to $|s|$ to zero and solving
for $|s|$ reveals that the magnitude of the MVDR beamformer maximizes the
likelihood. Thus,
$T\textsubscript{ML}(\bm{\mathbf{y}})=T\textsubscript{MVDR}(\bm{\mathbf{y}})$,
i.e. the MVDR beamformer is the maximum likelihood estimator of the clean
speech DFT coefficient as also stated in [26, Sec. 6.2.1.2].

###  III-B Non-Gaussian Noise

As we have seen, if the noise DFT coefficients follow a Gaussian distribution,
then a linear spatial filter can be considered optimal. However, Hendriks et
al. [13] have shown that this does not need to be the case for non-Gaussian
distributed noise. In their work, they model the noise distribution with a
multivariate complex Gaussian mixture distribution. The $M$ Gaussian mixture
components with respective covariance matrix $\bm{\mathbf{\Phi}}_{m}$,
$m\in\\{1,...,M\\}$, are assumed to be zero-mean such that the conditional PDF
given the clean speech is given by

$\displaystyle p_{\bm{\mathbf{Y}}}(\bm{\mathbf{y}}|{s})$ $\displaystyle=$
$\displaystyle\sum_{m=1}^{M}c_{m}\mathcal{CN}(\bm{\mathbf{d}}{s},\bm{\mathbf{\Phi}}_{m}).$
(15)

with mixture weights $c_{m}$ that sum to one. Hendriks et al. assume that the
amplitude $A_{S}$ and phase $\varphi_{S}$ of the clean speech DFT coefficient
are independent. They model the phase to be uniformly distributed over the
interval $[0,2\pi)$ and assume the amplitude to be generalized-Gamma
distributed ([7, Eq. 1], with $\gamma=2$ and $\beta=\nu/\sigma_{s}^{2}$). The
corresponding PDF

$\displaystyle p_{A_{S}}({a})$ $\displaystyle=$ $\displaystyle
2\dfrac{\left(\dfrac{\nu}{\sigma_{s}^{2}}\right)^{\nu}}{\Gamma(\nu)}a^{2\nu-1}\exp\left\\{-\frac{\nu}{\sigma_{s}^{2}}a^{2}\right\\}\text{
with }\nu>0,~{}a\geq 0$

depends on the speech shape parameter $\nu$, and $\Gamma(\cdot)$ is the Gamma
function. Under these assumptions, Hendriks et al. derive the MMSE estimator

$\displaystyle T\textsubscript{MMSE}(\bm{\mathbf{y}})$ $\displaystyle=$
$\displaystyle\nu\frac{{\displaystyle\sum_{m=1}^{M}}\frac{c_{m}Q_{m}}{|\bm{\mathbf{\Phi}}_{m}|}e^{\left[-\bm{\mathbf{y}}^{H}\bm{\mathbf{\Phi}}_{m}^{-1}\bm{\mathbf{y}}\right]}\frac{\sigma_{s}^{2}T_{\text{MVDR}}^{(m)}(\bm{\mathbf{y}})\mathcal{M}(\nu+1,2,P_{m})}{\nu(\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{m}^{-1}\bm{\mathbf{d}})^{-1}+\sigma_{s}^{2}}}{{\displaystyle\sum_{m=1}^{M}}\frac{c_{m}Q_{m}}{|\bm{\mathbf{\Phi}}_{m}|}e^{\left[-\bm{\mathbf{y}}^{H}\bm{\mathbf{\Phi}}_{m}^{-1}\bm{\mathbf{y}}\right]}\mathcal{M}(\nu,1,P_{m})}$
$\displaystyle\IEEEeqnarraymulticol{3}{C}{T_{\text{MVDR}}^{(m)}(\bm{\mathbf{y}})=\dfrac{\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{m}^{-1}\bm{\mathbf{y}}}{\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{m}^{-1}\bm{\mathbf{d}}},\quad
Q_{m}=(\nu+\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{m}^{-1}\bm{\mathbf{d}}\sigma_{s}^{2})^{-\nu},}$
$\displaystyle\IEEEeqnarraymulticol{3}{C}{\text{and}\quad
P_{m}=\dfrac{\sigma_{s}^{2}\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{m}^{-1}\bm{\mathbf{d}}\left|T_{\text{MVDR}}^{(m)}(\bm{\mathbf{y}})\right|^{2}}{\nu(\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{m}^{-1}\bm{\mathbf{d}})^{-1}+\sigma_{s}^{2}}}$

with $\mathcal{M}(\cdot,\cdot,\cdot)$ being the confluent hypergeometric
function [27, Sec. 9.21]. From (LABEL:eq:mmsegmm) it is apparent that the MMSE
estimator _cannot_ be decomposed in a linear spatial filter and a spectral
postfilter. This is because the linear term $T_{\text{MVDR}}^{(m)}$ as well as
the quadratic term
$\bm{\mathbf{y}}^{H}\bm{\mathbf{\Phi}}_{m}^{-1}\bm{\mathbf{y}}$ depend on the
summation index $m$. The spatial nonlinearity is particularly evident from the
aforementioned quadratic term.

Throughout this work, we compare the results of the optimal spatially
nonlinear MMSE estimator with a classical setup comprised of a linear spatial
filter and (nonlinear) spectral postfilter. Figure 1 provides an illustration
of the compared estimators: part (b) represents the nonlinear spatial filter
$T\textsubscript{MMSE}$ given in (LABEL:eq:mmsegmm) and part (a) corresponds
to a combination of the MVDR beamformer with an MMSE-optimal postfilter. We
now derive the postfilter under the same distributional assumptions as
$T\textsubscript{MMSE}$.

Since the MVDR beamformer is linear, we can infer the distribution of the
beamformer output and observe that it follows a one-dimensional complex
Gaussian mixture distribution with PDF

$\displaystyle p(T\textsubscript{MVDR}(\bm{\mathbf{y}})|{s})$ $\displaystyle=$
$\displaystyle\sum_{m=1}^{M}c_{m}\mathcal{N}_{\mathbb{C}}\bigg{(}{s},\underbrace{\dfrac{\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{n}^{-1}\bm{\mathbf{\Phi}}_{m}\bm{\mathbf{\Phi}}_{n}^{-1}\bm{\mathbf{d}}}{(\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{n}^{-1}\bm{\mathbf{d}})^{2}}}_{\sigma_{m}^{2}})\bigg{)}$
(18)

for an input $\bm{\mathbf{Y}}$ that is distributed according to a multivariate
complex Gaussian mixture distribution. The Gaussian mixture components have
the mean ${s}$ and variance $\sigma_{m}^{2}$, $m\in\\{1,...,M\\}$. Based on
this observation, we compute the MMSE-optimal spectral postfilter using [27,
Eq. 3.339, Eq. 6.643.2, Eq. 9.220.2] and [28, Eq. 10.32.3] and obtain the
estimator

$\displaystyle T\textsubscript{MVDR-MMSE}(\bm{\mathbf{y}})$ $\displaystyle=$
$\displaystyle\IEEEeqnarraymulticol{3}{C}{\nu\frac{{\displaystyle\sum_{m=1}^{M}}\frac{c_{m}Q_{m}}{\sigma_{m}^{2}}e^{\left[-\frac{|T\textsubscript{MVDR}(\bm{\mathbf{y}})|^{2}}{\sigma_{m}^{2}}\right]}\frac{\sigma_{s}^{2}T\textsubscript{MVDR}(\bm{\mathbf{y}})\mathcal{M}(\nu+1,2,P_{m})}{\nu\sigma_{m}^{2}+\sigma_{s}^{2}}}{{\displaystyle\sum_{m=1}^{M}}\frac{c_{m}Q_{m}}{\sigma_{m}^{2}}e^{\left[-\frac{|T\textsubscript{MVDR}(\bm{\mathbf{y}})|^{2}}{\sigma_{m}^{2}}\right]}\mathcal{M}(\nu,1,P_{m})}}$
(19)
$\displaystyle\IEEEeqnarraymulticol{3}{C}{\bm{\mathbf{\Phi}}_{n}=\sum_{m=1}^{M}c_{m}\bm{\mathbf{\Phi}}_{m},\quad\sigma_{m}^{2}=\dfrac{\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{n}^{-1}\bm{\mathbf{\Phi}}_{m}\bm{\mathbf{\Phi}}_{n}^{-1}\bm{\mathbf{d}}}{(\bm{\mathbf{d}}^{H}\bm{\mathbf{\Phi}}_{n}^{-1}\bm{\mathbf{d}})^{2}},}$
$\displaystyle\IEEEeqnarraymulticol{3}{C}{Q_{m}=(\dfrac{1}{\sigma_{m}^{2}}+\dfrac{\nu}{\sigma_{s}^{2}})^{-\nu}\quad\text{and}\quad
P_{m}=\dfrac{\sigma_{s}^{2}\sigma_{m}^{-2}|T\textsubscript{MVDR}(\bm{\mathbf{y}})|^{2}}{\nu\sigma_{m}^{2}+\sigma_{s}^{2}}.}$

that sequentially combines linear spatial processing with MMSE-optimal
spectral postprocessing as depicted in Figure 1.

##  IV Evaluation of the benefit of a nonlinear spatial filter in non-Gaussian
noise

Section III points out that using a nonlinear spatial filter is MMSE-optimal
and, thus, may be beneficial if the noise does not follow a Gaussian
distribution. It is well known that the DFT coefficients of speech are often
better modeled by a more heavy-tailed distribution than a Gaussian if
originating from short-time Fourier transform (STFT) segments with short
duration [6, 29]. Consequently, one may argue that this as well applies to
noise DFT coefficients if the background noise stems from human speakers. In
any case, Martin [29] observed that heavy-tailed distributions also provide a
good fit for DFT coefficients of some types of noise in the one-dimensional
case.

In this section, we investigate the potential of the optimal nonlinear spatial
filter versus the classical separated setup with a linear spatial filter and a
spectral postfilter for noise with a non-Gaussian distribution. Section IV-A
presents our findings for noise that departs from Gaussianity by means of
heavier tails but with a rather simple spatial structure. We published parts
of this analysis and of the analysis in Section IV-C in [20]. However, here we
also include the multichannel Wiener filter for comparison, compute more
detailed performance metrics, and have made changes to the speech power
parameter estimation scheme. In Section IV-B we provide results for noise that
is modeling a spatially more diverse noise field created by five interfering
human speakers and in Section IV-C we evaluate the nonlinear spatial filtering
approach based on real-world noise recordings from the CHiME3 database. Please
find audio examples for all experiments on our website111https://uhh.de/inf-
sp-nonlinear-spatial-filter-tasl2021.

###  IV-A Heavy-tailed noise distribution

In our first experiment, we investigate the performance of the nonlinear
spatial filter $T\textsubscript{MMSE}$ by mixing the target speech signal at
the microphones with multichannel noise that is sampled from a heavy-tailed
Gaussian mixture distribution.

####  IV-A1 Noise distribution model

We construct a Gaussian mixture distribution with an adjustable heavy-
tailedness by combining Gaussian components with scaled versions of the same
covariance matrix. Therefore, we set the $m$th mixture component’s covariance
matrix $\bm{\mathbf{\Phi}}_{m}$ to be

$\displaystyle\bm{\mathbf{\Phi}}_{m}$ $\displaystyle=$
$\displaystyle\dfrac{b^{m-1}}{r}\bm{\mathbf{\Phi}}_{n}$ with r = ∑_m=1^M c_m
b^m-1 (20)

and scaling factor $b\in\mathbb{R}^{+}$. The constant $r$ ensures correct
normalization such that the overall covariance matrix of our scaled Gaussian
mixture distribution remains $\bm{\mathbf{\Phi}}_{n}$.

We rely on the kurtosis to quantify the heavy-tailedness of the scaled
Gaussian mixture distributions. It is a statistical measure that accounts for
the likelihood of the occurrence of outliers [30] and it has been extended for
real-valued multivariate distributions by Mardia [31]. We extend it to
complex-valued random vectors $\bm{\mathbf{X}}\in\mathbb{C}^{n}$ with mean
$\bm{\mathbf{\mu}}$ and covariance $\bm{\mathbf{C}}_{x}$ by defining its
kurtosis to be

$\displaystyle\kappa_{\mathbb{C}}(\bm{\mathbf{X}})$ $\displaystyle=$
$\displaystyle\mathbb{E}\left[(2(\bm{\mathbf{X}}-\bm{\mathbf{\mu}})^{H}\bm{\mathbf{C}}_{x}^{-1}(\bm{\mathbf{X}}-\bm{\mathbf{\mu}}))^{2}\right].$
(21)

A complex-valued $n$-dimensional Gaussian distribution can equivalently be
formulated as a real-valued $2n$-dimensional Gaussian distribution [22, Thm.
15.1]. The additional factor of two in (21) ensures that the same kurtosis
value results for both formulations of the same distribution. Using [32, Sec.
8.2.4], we compute the kurtosis of a vector $\bm{\mathbf{N}}$ distributed
according to a scaled Gaussian mixture distribution to obtain

$\displaystyle\kappa_{\mathbb{C}}(\bm{\mathbf{N}})=2D(2+2D)\underbrace{\sum_{m=1}^{M}c_{m}\dfrac{b^{2(m-1)}}{r^{2}}}_{q}$
(22)

and observe that $\kappa_{\mathbb{C}}(\bm{\mathbf{N}})$ is given by the
kurtosis of a $D$-dimensional complex Gaussian distribution multiplied by a
factor $q$ that depends on the scaling factor $b$ and the number of mixture
components $M$.

####  IV-A2 Experimental setup

In our test scenario, we use five microphones arranged in a linear array with
5 cm spacing and broadside orientation towards the target signal source and
model the propagation path between the target speaker and the microphones
based on time delays only. We perform the evaluation using 48 clean speech
signals taken from the WSJ0 dataset [33] that are balanced between female and
male speakers. The noise DFT coefficients are samples from a scaled Gaussian
mixture distribution with scale factor $b=2$ and a variable number of mixture
components with equal weight $c_{m}=\frac{1}{M}$, $m\in\\{1,...,M\\}$. The
noise covariance matrix $\bm{\mathbf{\Phi}}_{n}$ models a diffuse noise field
with a small portion (factor of 0.05) of additional spatially and spectrally
white noise as in [34, Eq. 27]. The noise and speech are combined such that a
signal-to-noise ratio (SNR) of 0 dB is obtained.

####  IV-A3 Performance evaluation

Figure 2 provides a performance comparison of the jointly spatial and spectral
nonlinear $T\textsubscript{MMSE}$ and the spatially linear
$T\textsubscript{MVDR-MMSE}$ with a nonlinear postfilter. The speech shape
parameter is set to $\nu=0.25$ for both estimators. Furthermore, we display
results obtained with the well-known linear spatial filter
$T\textsubscript{MVDR}$ without a postfilter and the multichannel Wiener
filter $T\textsubscript{MWF}$, which is the MMSE-optimal solution if noise
_and_ speech follow a Gaussian distribution, i.e., $T\textsubscript{MVDR-
MMSE}$ with $\nu=1$ and $M=1$. The performance results are displayed with
respect to the kurtosis factor $q$ on the x-axis indicating an increased
heavy-tailedness of the noise distribution from left to right.

Figure 2: POLQA, SI-SDR, SI-SIR and SI-SAR results for scaled Gaussian mixture
noise distributions with increased heavy-tailedness in diffuse noise.

The plot in the upper left corner shows the performance with respect to the
improvement of the POLQA measure [35], which is the successor of perceptual
evaluation of speech quality (PESQ) [36] and returns the expected mean opinion
score (MOS), which takes values from one (bad) to five (excellent). In any
plot of Figure 2, we are particularly interested in the performance difference
of the $T\textsubscript{MMSE}$ (red) and $T\textsubscript{MVDR-MMSE}$ (blue)
as this gap characterizes the potential performance gain of a nonlinear
spatial filter. For POLQA, we observe an increase of the performance
difference up to 1.1 POLQA score improvement as the noise distribution shifts
towards a more heavy-tailed distribution.

The estimators including a postfilter, $T\textsubscript{MMSE}$,
$T\textsubscript{MVDR-MMSE}$, and $T\textsubscript{MWF}$, require an estimate
of the speech power spectral density (PSD) $\sigma_{s}^{2}$. In contrast to
our previous paper [20], here we do not rely on oracle knowledge of the clean
speech signal to estimate this parameter but obtain an estimate from the noisy
signal based on the cepstral smoothing technique [37]. This results in an
increased performance gap between $T\textsubscript{MMSE}$ and
$T\textsubscript{MVDR-MMSE}$. From this finding, we conclude that a nonlinear
spatial filtering approach is even more beneficial if the performance of the
postfilter decreases due to estimation errors of the spectral power of the
target speech signal.

The next three plots (upper right and second row) display the performance
results for the SI-SDR, SI-SIR, and SI-SAR measures as defined in [38]. We
compute the SI-SDR, SI-SIR, and SI-SAR for segments of length 10 ms without
overlap and include only segments with target speech activity similar to the
computation of the segmental SNR in [39]. The performance results based on the
SI-SDR measure show a similar structure to the ones obtained with POLQA. For
high kurtosis values, we observe a performance gap of 4.5 dB for
$T\textsubscript{MMSE}$ and $T\textsubscript{MVDR-MMSE}$. Furthermore, the
difference between $T\textsubscript{MVDR-MMSE}$ and $T\textsubscript{MWF}$ for
high kurtosis values, which results exclusively from the different postfilter,
is more obvious. The observed performance gaps, in particular the performance
advantage of the nonlinear spatial filter, coincide with our own listening
experience1.

For the computation of the SI-SIR and SI-SAR measure displayed in the second
row, the residual noise is split into interference noise and artifacts. It is
striking to see that the red graph of $T\textsubscript{MMSE}$ runs above the
blue graph of $T\textsubscript{MVDR-MMSE}$ in both plots meaning that the
nonlinear spatial filter achieves better noise reduction and fewer speech
distortions at the same time. The better performance with respect to the SI-
SAR measure is quite notable as we can see that the linear MVDR beamformer
introduces very few speech distortions but its combination with different
postfilters ($T\textsubscript{MVDR-MMSE}$ and $T\textsubscript{MWF}$) still
performs worse than the joint spatial and spectral non-linear processing by
$T\textsubscript{MMSE}$.

###  IV-B Inhomogeneous noise field (interfering speech)

Instead of sampling a Gaussian mixture distribution as in the previous section
or in [21], we now use a setup with five interfering point sources arranged as
illustrated in Figure 3 and, this way, move closer towards realistic noise
scenarios. The estimators $T\textsubscript{MMSE}$ and $T\textsubscript{MVDR-
MMSE}$ have been derived under a Gaussian mixture noise assumption. To be
consistent with this modeling assumption, we require the five interfering
point sources to not be Gaussian distributed or not be simultaneously active
per time-frequency bin, because otherwise the overall noise resulting from the
different interfering sources would also be Gaussian distributed. Choosing
human speakers as interfering sources, this assumption is commonly assumed to
hold and referred to as w-disjoint orthogonality [40].

####  IV-B1 Experimental setup

As can be seen in Figure 3, the target speech source is placed in the
broadside direction of the two-dimensional linear microphone array with 6 cm
microphone spacing. We sample the target speech signal and the interfering
signals from distinct subsets of the WSJ0 dataset. The two-dimensional noise
signal is then obtained by multiplying the interfering speech signals with the
steering vectors $\bm{\mathbf{d}}_{i}$, $i\in\\{0,...,4\\}$, and adding the
individual interfering sources’ signals. The steering vector
$\bm{\mathbf{d}}_{i}$ of the $i$th interfering source positioned at
$\theta_{i}=\frac{\pi}{6}+\frac{2\pi}{5}i$ radians models the relative time
difference of arrival at the microphones. The target speech signal and noise
signal are rescaled to correspond to an SNR of 0 dB.

Figure 3: Illustration of the experiment setup with a two-dimensional linear microphone array, a target speech source in broadside direction and five interfering point sources (human speakers (Section IV-B) or Gaussian bursts (Section V)). | Interfering speech | Gaussian bursts
---|---|---
$\Delta$ POLQA | $0.84~{}\pm~{}0.04$ | $2.64~{}\pm~{}0.08$
$\Delta$ SI-SDR | $4.63~{}\pm~{}0.15$ | $9.92~{}\pm~{}0.30$
$\Delta$ SI-SAR | $3.91~{}\pm~{}0.16$ | $8.39~{}\pm~{}0.26$
$\Delta$ SI-SIR | $6.44~{}\pm~{}0.22$ | $14.95~{}\pm~{}0.46$
ESTOI (noisy) | $0.49~{}\pm~{}0.01$ | $0.57~{}\pm~{}0.02$
ESTOI ($T\textsubscript{MMSE}$) | $0.85~{}\pm~{}0.01$ | $0.94~{}\pm~{}0.00$
ESTOI ($T\textsubscript{MVDR-MMSE}$) | $0.72~{}\pm~{}0.01$ | $0.67~{}\pm~{}0.02$
TABLE I: Performance results (mean and the 95% confidence interval) of the
$T\textsubscript{MMSE}$ and $T\textsubscript{MVDR-MMSE}$ estimators for an
inhomogeneous noise field with interfering speech and Gaussian sources as
described in Section IV-B and Section V respectively.

We now compare the performance of the $T\textsubscript{MMSE}$ and
$T\textsubscript{MVDR-MMSE}$ estimators for the inhomogeneous noise field. For
this, we require estimates of the Gaussian mixture components’ covariance
matrices $\bm{\mathbf{\Phi}}_{m}$ and the mixture weights $c_{m}$. We choose
the number of components equal to the number of interfering sources, i.e.,
$M=5$, and estimate the Gaussian mixture parameters using the expectation
maximization (EM) algorithm [41] applied to overlapping signal segments of
length 250 ms and with an overlap of 50% from the pure noise signal. As
before, we estimate the spectral power of speech using the cepstral smoothing
technique and use a speech shape parameter $\nu=0.25$.

####  IV-B2 Performance evaluation

The first column of Table I displays the performance results for the described
simulation. For the performance measures in the first four rows, preceded with
a $\Delta$ symbol, we report the performance difference between
$T\textsubscript{MVDR-MMSE}$ and $T\textsubscript{MMSE}$ averaged over 48
samples. We observe that the nonlinear spatial filter delivers a considerable
performance gain that amounts to 4.63 dB SI-SDR and a POLQA score of 0.84. The
bottom part of Table I presents ESTOI [42] scores for the noisy signal and the
enhancement results obtained with $T\textsubscript{MMSE}$ and
$T\textsubscript{MVDR-MMSE}$. The ESTOI scores provide a measure of speech
intelligibility. As for the other performance measures, we find that the
nonlinear spatial filter outperforms the combination of a linear spatial
filter and a postfilter as the $T\textsubscript{MMSE}$ estimator yields an
ESTOI score of 0.85 as opposed to the result of 0.72 achieved by
$T\textsubscript{MVDR-MMSE}$.

###  IV-C Real-world CHiME3 noise

Furthermore, we investigate the performance of the nonlinear spatial filter
for real-world noise from the CHiME3 database [43] that has been recorded in
four different environments: a cafeteria, a moving bus, next to a street, and
in a pedestrian area.

####  IV-C1 Experimental setup

The CHiME3 data has been recorded using six microphones that are attached to a
tablet computer. For this experiment, we use the simulated training subset of
the official dataset, which has been created by mixing the recording of real-
world background noise with a spatialized version of WSJ0 utterances. A
detailed description of the data generation process can be found in [43]. We
evaluate on 48 randomly selected samples that are balanced between male and
female speakers.

As before, we require an estimate of the time-varying Gaussian mixture
distribution parameters and estimate them using oracle knowledge of the noise
signal. For this, we apply the EM algorithm to overlapping signal segments of
length 750 ms. For both, $T\textsubscript{MMSE}$ and $T\textsubscript{MVDR-
MMSE}$, we use $\nu=0.25$ and estimate the speech power $\sigma_{s}^{2}$ using
the cepstral smoothing technique. In addition, we need to estimate the
steering vector for the target speaker. For this, we employ oracle knowledge
of the clean speech signal and extract the steering vector estimates as
principal eigenvectors of the time-varying covariance matrix estimates
obtained by recursive smoothing.

####  IV-C2 Performance evaluation

Again, we assess the performance gap between the nonlinear spatial filter
$T\textsubscript{MMSE}$ and the separated setup with a linear spatial filter
and a postfilter $T\textsubscript{MVDR-MMSE}$. Figure 4 displays the SI-SDR
results for these estimators and also the MVDR beamformer
$T\textsubscript{MVDR}$ with respect to the number of mixture components that
have been fitted to a cafeteria background noise using the EM algorithm. While
the performance of the $T\textsubscript{MMSE}$ estimator (red) improves with
increased modeling capabilities of the mixture distribution, neither the
$T\textsubscript{MVDR}$ nor the $T\textsubscript{MVDR-MMSE}$ estimator
benefits from using more mixture components. As a result, we observe a
performance gap of $3.17~{}\text{dB}$ SI-SDR between the best result obtained
with the nonlinear spatial filter and the best result obtained with the
separated setup. This value coincides with the first entry of Table II. For
the POLQA measure displayed in the second row, we find a difference of 0.59
POLQA score. The table shows bigger performance differences for the cafeteria
(CAF) and pedestrian area (PED) noise than the bus (BUS) and street (STR)
noise. We suppose that this reflects that the cafeteria and pedestrian area
noise is less stationary as we hear the most significant differences for
impulse like background noise. The ESTOI scores displayed at the bottom of
Table II indicate that the nonlinear spatial filter is not only beneficial to
the speech quality but also the speech intelligibility. Overall, we conclude
that the Gaussian noise assumption does not seem to be valid for the examined
real-world noise as the nonlinear spatial filter provides a notable benefit
also for these recordings.

Figure 4: SI-SDR results for cafeteria noise with respect to the number of mixture components used to fit the noise distribution. | CAF | BUS | PED | STR
---|---|---|---|---
$\Delta$ SI-SDR | $3.17\pm 0.19$ | $2.48\pm 0.26$ | $3.31\pm 0.24$ | $2.07\pm 0.28$
$\Delta$ POLQA | $0.59\pm 0.06$ | $0.38\pm 0.07$ | $0.56\pm 0.05$ | $0.28\pm 0.04$
ESTOI (noisy) | $0.60\pm 0.03$ | $0.71\pm 0.02$ | $0.56\pm 0.03$ | $0.69\pm 0.03$
ESTOI ($T\textsubscript{MMSE}$) | $0.94\pm 0.01$ | $0.97\pm 0.01$ | $0.93\pm 0.01$ | $0.96\pm 0.01$
ESTOI ($T\textsubscript{MVDR-MMSE}$) | $0.89\pm 0.02$ | $0.95\pm 0.01$ | $0.88\pm 0.02$ | $0.94\pm 0.01$
TABLE II: Performance results (mean and the 95% confidence interval) of the
nonlinear spatial $T\textsubscript{MMSE}$ and linear spatial filter combined
with a postfilter $T\textsubscript{MVDR-MMSE}$ for noise from the CHiME3
databse, which has been fitted with a Gaussian mixture distribution with four
mixture components as described in Section IV-C.

##  V Interpretation: a nonlinear spatial filter enables superior spatial
selectivity

Figure 5: Spectrograms of an example in an inhomogeneous noise field with five
interfering sources emitting Gaussian noise bursts. The second row visualizes
the processing results obtained with $T\textsubscript{MVDR}$,
$T\textsubscript{MVDR-MMSE}$ and $T\textsubscript{MMSE}$ and the top row shows
the clean and noisy spectrograms as well as close-ups of the fine-structure of
a voiced speech segment.

We assume that the performance benefit of the nonlinear spatial filter
reported in Section IV-B and IV-C is due to the more efficient use of spatial
information by the $T\textsubscript{MMSE}$ estimator. Here we support this
conjecture by an experiment that provides an insight into the functioning of
the nonlinear spatial filter.

####  V-1 Experimental setup

We use the same geometric setup as described previously in Section IV-B
(Figure 3) but replace the interfering speech sources with sources that emit
spectrally white Gaussian signals. To match the long-term non-Gaussianity
assumption, only one interfering source emits a signal at a specific time
instance. We implement this by using short (336 ms) non-overlapping Gaussian
bursts for the interfering sources. The so created noise signal can be viewed
as stationary regarding its spectral characteristics except at the segment
boundaries. By applying the EM algorithm to the full-length noise signal we
also model the spatial characteristics as long-term stationary. All other
experiment settings remain unchanged as described before in Section IV-B.

####  V-2 Performance evaluation

The performance results are displayed in the second column of Table I. For
this artificial type of noise, we observe an even greater performance
difference of 9.9 dB SI-SDR and 2.6 POLQA score. In fact, the
$T\textsubscript{MMSE}$ estimator seems to be able to recover the original
signal almost perfectly except from minor residual high-frequency noise while
$T\textsubscript{MVDR-MMSE}$ suffers from clearly audible speech degradation
and residual noise. Audio examples can be found online1.

Figure 5 depicts the spectrograms of the clean and noisy signals in the top
row and the spectrograms of the enhancement results obtained by the
$T\textsubscript{MVDR}$, $T\textsubscript{MVDR-MMSE}$, and
$T\textsubscript{MMSE}$ estimators in the bottom row. The uniform green
coloration of the vertical stripes in the noisy spectrogram reflects the
spectral stationarity. The vertical dark blue lines separate segments with
different spatial properties. While the spatial diversity cannot be seen from
the spectrogram, it becomes visible from the result of the MVDR beamformer
(first in bottom row). Here, the MVDR beamformer suppresses different
frequencies for signal segments with different spatial properties as can be
seen from the displaced horizontal dark blue lines. The described differences
between the $T\textsubscript{MVDR-MMSE}$ (middle) and $T\textsubscript{MMSE}$
(right) estimators’ results are also found in the spectrograms. A close look
reveals that the nonlinear spatial filter preserves much more of the target
signal’s fine structure. Furthermore, a comparison with the spectrogram of the
clean speech signal highlights that it suppresses background noise much better
than the $T\textsubscript{MVDR-MMSE}$ estimator. Residual noise is visible in
the spectrogram only in some segments at a frequency of about 6 kHz.

####  V-3 Discussion

To explain these observations, we examine the covariance matrices
$\bm{\mathbf{\Phi}}_{m}$, $m\in\\{1,...,5\\}$, of the Gaussian mixture noise
distribution estimated with the EM algorithm. In Figure 6 we visualize their
spatial structure based on the directivity pattern [3, Sec. 12.5.2] that they
produce when used as noise correlation matrix in the MVDR beamformer, which is
denoted with $T_{\text{MVDR}}^{(m)}$ in (LABEL:eq:mmsegmm). Furthermore, we
visualize the directivity pattern of the MVDR beamformer. The correlation
matrix $\bm{\mathbf{\Phi}}_{n}$ required to compute $T\textsubscript{MVDR}$ is
related to the mixture component covariance matrices via

$\displaystyle\bm{\mathbf{\Phi}}_{n}$ $\displaystyle=$
$\displaystyle\sum_{m=1}^{M}c_{m}\bm{\mathbf{\Phi}}_{m}.$ (23)

The directivity pattern produced by $T\textsubscript{MVDR}$ is displayed at
the top left followed by visualizations of the five mixture component
covariance matrices. For each of these, a pronounced spatial characteristic
can be observed by means of the horizontal dark lines. On the right side of
the directivity patterns, we indicate the incidence angles $\theta_{i}$ of the
noise sources. We notice that each component’s covariance matrix models one of
the noise sources as apparent from the zero placed in the respective direction
by the MVDR beamformer. The second horizontal line originates from the
symmetry requirements of the directivity pattern, which are determined by the
array geometry.

Figure 6: Directivity patterns of $T\textsubscript{MVDR}$ (top left) and
$T_{\text{MVDR}}^{(m)}$ (MVDR beamformer with noise correlation matrix
$\bm{\mathbf{\Phi}}_{m}$), $m\in\\{1,...,5\\}$. The arrows on the right side
indicate the incidence angle $\theta_{i}$, $i\in\\{0,...,4\\}$ of the $i$th
interfering point source.

In comparison, the directivity pattern of the $T\textsubscript{MVDR}$ (top
left) does not show zeros placed into the directions of the interfering point
sources. Instead, the weighted combination in (23) seems to eliminate some of
the spatial information, which corresponds to the well-known fact that a two-
microphone MVDR beamformer can suppress only one directional interfering
source but not five of them. As a result, only some frequencies are suppressed
for each interfering source as we observed in Figure 5 before.

Figure 6 reveals how much more spatial information can be utilized by
$T\textsubscript{MMSE}$ in comparison with $T\textsubscript{MVDR-MMSE}$, whose
spatial processing relies on an estimate of $\bm{\mathbf{\Phi}}_{n}$ as
visualized in the top left plot. The initial spatial filtering step using the
MVDR beamformer is not capable of suppressing the directional sources and the
remaining noise has to be filtered by the postfilter, which then leads to some
improvements. On the other hand, the nonlinear spatial $T\textsubscript{MMSE}$
estimator can utilize the spatial information provided by the estimates of
covariance matrices $\bm{\mathbf{\Phi}}_{m}$.

One could argue that a time-varying MVDR beamformer $T_{\text{MVDR}}^{(m)}$
with correctly chosen $m$ would suffice to solve the problem on the short
signal segments with noise from a single point source and a complicated
nonlinear approach is not required. However, we must point out that the step
of choosing the ‘right‘ covariance matrix is not required for the nonlinear
spatial filter. Instead, we provide the Gaussian mixture parameters reflecting
the spatial properties of the full utterance and, nevertheless, the
$T\textsubscript{MMSE}$ estimator is capable of suppressing five directional
noise sources with only two microphones without the need for spatial
adaptation. Note that this is an exciting finding as traditional linear
spatial filters can only suppress $D-1$ interfering point sources with $D$
microphones without spatial adaptation [26, Sec. 6.3].

Despite the impressive performance results achieved in this experiment, the
analytic nonlinear spatial filter has some weaknesses: it requires a very
accurate estimation of the spatial and spectral characteristics of the noise
signal and is also computationally quite demanding. In addition to the
presented experiments, we carried out simulations using measured impulse
responses between the microphones and speakers and observed a much lower
benefit from using a nonlinear spatial filter for the experiment with five
interfering Gaussian sources even with access to oracle noise data. This is
because the spatial and spectral diversity of the noise signal increase and
many more mixture components would be required to model the noise accurately
which then results in a data problem. Similarly, estimating the parameters of
the Gaussian mixture from a noisy signal is difficult. We approached this
using masks to identify time-frequency bins that are dominated by noise but
did not obtain reliable estimates this way.

Therefore, we conclude that the analytical estimators allow us to study the
potential of nonlinear spatial filters in principle, but because of high
sensitivity to parameter estimation errors and high computational costs,
practical nonlinear spatial filters may be better implemented using modern
machine learning tools like DNN.

##  VI Conclusions

In a detailed theory overview, we have revisited the fact that the
multichannel MMSE-optimal estimator of the clean speech signal is in general a
jointly spatial-spectral nonlinear filter. Therefore, the state-of-the-art
concatenation of a linear spatial filter and a postfilter is MMSE-optimal only
in the special case that the noise follows a Gaussian distribution. The
experimental section of this paper studied the performance advantage that can
be gained by replacing the generally suboptimal sequential setup with a
nonlinear spatial filter in three different non-Gaussian noise scenarios.

First, we have shown that considerable performance improvements result if the
noise distribution deviates from a Gaussian distribution by an increased
heavy-tailedness as the nonlinear spatial filter enables a higher noise
reduction and lower speech distortions at the same time. Second, we report a
performance benefit of 4.6 dB SI-SDR and of 0.8 POLQA score for an
inhomogeneous noise field created by five interfering speech sources and,
furthermore, we have observed a benefit of about 3.2 dB SI-SDR and 0.6 POLQA
score for the real-world cafeteria noise recordings from the CHiME3 database.
In addition, we have performed experiments that revealed that the nonlinear
spatial filter has some notably increased spatial processing capabilities
allowing for an almost perfect elimination of five Gaussian interfering point
sources with only two microphones.

The presented findings on the performance potential of a nonlinear spatial
filter motivate further research on the implementation of nonlinear spatial
filters, e.g., using DNN to learn the nonlinear spatial filter directly from
data and overcome the parameter estimation issues and other limitations of the
analytic nonlinear spatial filter that we have used for this analysis.

## Acknowledgment

We would like to thank J. Berger and Rohde&Schwarz SwissQual AG for their
support with POLQA.

## References

  * [1] S. Doclo, W. Kellermann, S. Makino, and S. E. Nordholm, “Multichannel Signal Enhancement Algorithms for Assisted Listening Devices: Exploiting spatial diversity using multiple microphones,” _IEEE Signal Processing Magazine_ , vol. 32, no. 2, pp. 18–30, Mar. 2015.
  * [2] F. Weninger, H. Erdogan, S. Watanabe, E. Vincent, J. Le Roux, J. R. Hershey, and B. Schuller, “Speech enhancement with LSTM recurrent neural networks and its application to noise-robust ASR,” in _International Conference on Latent Variable Analysis and Signal Separation (LVA/ICA)_ , Liberec, Czech Republic, Aug. 2015.
  * [3] P. Vary and R. Martin, _Digital speech transmission: enhancement, coding and error concealment_. Chichester, England Hoboken, NJ: John Wiley, 2006.
  * [4] J. Benesty, _Microphone Array Signal Processing_. Berlin: Springer, 2008.
  * [5] Y. Ephraim and D. Malah, “Speech enhancement using a minimum-mean square error short-time spectral amplitude estimator,” _IEEE Transactions on Acoustics, Speech, and Signal Processing_ , vol. 32, no. 6, pp. 1109–1121, Dec. 1984.
  * [6] T. Lotter and P. Vary, “Speech Enhancement by MAP Spectral Amplitude Estimation Using a Super-Gaussian Speech Model,” _EURASIP Journal on Advances in Signal Processing_ , vol. 2005, no. 7, pp. 1110–1126, May 2005.
  * [7] J. S. Erkelens, R. C. Hendriks, R. Heusdens, and J. Jensen, “Minimum Mean-Square Error Estimation of Discrete Fourier Coefficients With Generalized Gamma Priors,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 15, pp. 1741–1752, 2007.
  * [8] Y. Xu, J. Du, L. Dai, and C. Lee, “A regression approach to speech enhancement based on deep neural networks,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 23, no. 1, pp. 7–19, Jan. 2015\.
  * [9] D. Rethage, J. Pons, and X. Serra, “A Wavenet for Speech Denoising,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Apr. 2018, pp. 5069–5073.
  * [10] S. R. Park and J. Lee, “A Fully Convolutional Neural Network for Speech Enhancement,” in _Proc. Interspeech 2017_ , Stockholm, Sweden, 2017, pp. 1993–1997.
  * [11] A. Pandey and D. Wang, “TCNN: Temporal Convolutional Neural Network for Real-time Speech Enhancement in the Time Domain,” in _2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Brighton, United Kingdom, 2019, pp. 6875–6879.
  * [12] R. Balan and J. P. Rosca, “Microphone Array Speech Enhancement by Bayesian Estimation of Spectral Amplitude and Phase,” in _Sensor Array and Multichannel Signal Processing Workshop Proceedings_ , Rosslyn, Virginia, Aug. 2002, pp. 209–213.
  * [13] R. C. Hendriks, R. Heusdens, U. Kjems, and J. Jensen, “On Optimal Multichannel Mean-Squared Error Estimators for Speech Enhancement,” _IEEE Signal Processing Letters_ , vol. 16, pp. 885–888, 2009.
  * [14] J. Heymann, L. Drude, and R. Haeb-Umbach, “Neural network based spectral mask estimation for acoustic beamforming,” in _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Shanghai, China, Mar. 2016, pp. 196–200.
  * [15] X. Xiao, S. Watanabe, H. Erdogan, L. Lu, J. Hershey, M. L. Seltzer, G. Chen, Y. Zhang, M. Mandel, and D. Yu, “Deep beamforming networks for multi-channel speech recognition,” in _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Shanghai, China, 2016, pp. 5745–5749.
  * [16] T. N. Sainath, R. J. Weiss, K. W. Wilson, B. Li, A. Narayanan, E. Variani, M. Bacchiani, I. Shafran, A. Senior, K. Chin, A. Misra, and C. Kim, “Multichannel signal processing with deep neural networks for automatic speech recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 25, no. 5, pp. 965–979, May 2017.
  * [17] H. Lee, H. Y. Kim, W. H. Kang, J. Kim, and N. S. Kim, “End-to-End Multi-Channel Speech Enhancement Using Inter-Channel Time-Restricted Attention on Raw Waveform,” in _Proc. Interspeech 2019_. Graz, Austria: ISCA, Sep. 2019, pp. 4285–4289.
  * [18] B. Tolooshams, R. Giri, A. H. Song, U. Isik, and A. Krishnaswamy, “Channel-Attention Dense U-Net for Multichannel Speech Enhancement,” in _2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Barcelona, Spain, May 2020, pp. 836–840.
  * [19] G. Itzhak, J. Benesty, and I. Cohen, “Nonlinear Kronecker product filtering for multichannel noise reduction,” _Speech Communication_ , vol. 114, pp. 49–59, Nov. 2019.
  * [20] K. Tesch, R. Rehr, and T. Gerkmann, “On Nonlinear Spatial Filtering in Multichannel Speech Enhancement,” in _Proc. Interspeech 2019_ , Graz, Austria, 2019, pp. 91–95.
  * [21] K. Tesch and T. Gerkmann, “Nonlinear spatial filtering for multichannel speech enhancement in inhomogeneous noise fields,” in _2020 IEEE International Conference on Acoustics, Speech and Signal Processing_ , Barcelona, Spain, May 2020, pp. 196–200.
  * [22] S. M. Kay, _Fundamentals Of Statistical Signal Processing_. Pearson, 2009.
  * [23] M. Schervish, _Theory of Statistics_. New York, NY: Springer New York, 1995.
  * [24] H. Poor, _An Introduction to Signal Detection and Estimation_. New York, NY: Springer New York, 1994.
  * [25] E. L. Lehmann, _Testing Statistical Hypotheses_. New York: Springer, 2005.
  * [26] H. L. V. Trees, _Optimum Array Processing: Part IV of Detection, Estimation, and Modulation Theory_. John Wiley & Sons, Apr. 2004.
  * [27] I. S. Gradshteyn and J. M. Ryzhik, _Table of Integrals, Series, and Products_. San Diego: Academic Press, 2000.
  * [28] “NIST Digital Library of Mathematical Functions,” f. W. J. Olver, A. B. Olde Daalhuis, D. W. Lozier, B. I. Schneider, R. F. Boisvert, C. W. Clark, B. R. Miller and B. V. Saunders, eds.
  * [29] R. Martin, “Speech enhancement based on minimum mean-square error estimation and supergaussian priors,” _IEEE Transactions on Speech and Audio Processing_ , vol. 13, no. 5, pp. 845–856, Sep. 2005.
  * [30] P. H. Westfall, “Kurtosis as Peakedness, 1905–2014. R.I.P.” _The American Statistician_ , vol. 68, no. 3, pp. 191–195, 2014.
  * [31] K. V. Mardia, “Measures of Multivariate Skewness and Kurtosis with Applications,” _Biometrika_ , vol. 57, no. 3, pp. 519–530, 1970.
  * [32] K. B. Petersen and M. S. Pedersen, “The Matrix Cookbook,” Nov. 2012.
  * [33] J. S. Garofolo, D. Graff, D. Paul, and D. Pallett, “CSR-I (WSJ0) Complete LDC93S6A,” Linguistic Data Consortium, Philadelphia, May 2007.
  * [34] C. Pan, J. Chen, and J. Benesty, “Performance study of the MVDR beamformer as a function of the source incidence angle,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 22, no. 1, pp. 67–79, Jan. 2014\.
  * [35] “P.863: Perceptual objective listening quality prediction,” International Telecommunication Union, Mar. 2018, iTU-T recommendation. [Online]. Available: https://www.itu.int/rec/T-REC-P.863-201803-I/en
  * [36] “P.862.3: Application guide for objective quality measurement based on Recommendations P.862, P.862.1 and P.862.2,” International Telecommunication Union, Nov. 2007, iTU-T recommendation. [Online]. Available: https://www.itu.int/rec/T-REC-P.862.3-200711-I/en
  * [37] C. Breithaupt, T. Gerkmann, and R. Martin, “A novel a priori SNR estimation approach based on selective cepstro-temporal smoothing,” in _2008 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Las Vegas, USA, Mar. 2008, pp. 4897–4900.
  * [38] J. L. Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR – Half-baked or Well Done?” in _2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Brighton, United Kingdom, May 2019, pp. 626–630.
  * [39] T. Gerkmann and R. C. Hendriks, “Unbiased MMSE-Based Noise Power Estimation With Low Complexity and Low Tracking Delay,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 20, no. 4, pp. 1383–1393, May 2012.
  * [40] O. Yilmaz and S. Rickard, “Blind separation of speech mixtures via time-frequency masking,” _IEEE Transactions on Signal Processing_ , vol. 52, no. 7, pp. 1830–1847, Jul. 2004.
  * [41] C. Bishop, _Pattern recognition and machine learning_. New York: Springer, 2006.
  * [42] J. Jensen and C. H. Taal, “An algorithm for predicting the intelligibility of speech masked by modulated noise maskers,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 24, no. 11, pp. 2009–2022, 2016\.
  * [43] J. Barker, R. Marxer, E. Vincent, and S. Watanabe, “The third ‘CHiME’ speech separation and recognition challenge: Dataset, task and baselines,” in _IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)_ , Scottsdale, USA, Dec. 2015, pp. 504–511.

|  Kristina Tesch (S’20) received a B.Sc and M.Sc in Informatics in 2016 and
2019 from the Universität Hamburg, Hamburg, Germany. She is with the Signal
Processing Group at the Universität Hamburg since 2019 and is currently
working towards a doctoral degree. Her research interests include digital
signal processing and machine learning algorithms for speech and audio with a
focus on multichannel speech enhancement. For her master thesis on
multichannel speech enhancement, she received the award for the best master
thesis at a German Informatics Department from the Fakultätentag Informatik in
2019.
---|---
|  Timo Gerkmann (S’08–M’10–SM’15) studied Electrical Engineering and
Information Sciences at the Universität Bremen and the Ruhr-Universität Bochum
in Germany. He received his Dipl.-Ing. degree in 2004 and his Dr.-Ing. degree
in 2010 both in Electrical Engineering and Information Sciences from the Ruhr-
Universität Bochum, Bochum, Germany. In 2005, he spent six months with Siemens
Corporate Research in Princeton, NJ, USA. During 2010 to 2011 Dr. Gerkmann was
a postdoctoral researcher at the Sound and Image Processing Lab at the Royal
Institute of Technology (KTH), Stockholm, Sweden. From 2011 to 2015 he was a
professor for Speech Signal Processing at the Universität Oldenburg,
Oldenburg, Germany. During 2015 to 2016 he was a Principal Scientist for Audio
& Acoustics at Technicolor Research & Innovation in Hanover, Germany. Since
2016 he is a professor for Signal Processing at the Universität Hamburg,
Germany. His main research interests are on statistical signal processing and
machine learning for speech and audio applied to communication devices,
hearing instruments, audio-visual media, and human-machine interfaces. Timo
Gerkmann serves as an elected member of the IEEE Signal Processing Society
Technical Committee on Audio and Acoustic Signal Processing and as an
Associate Editor of the IEEE/ACM Transactions on Audio, Speech and Language
Processing.
---|---
  *[MMSE]: minimum mean square error
  *[ASR]: automatic speech recognition
  *[MVDR]: minimum variance distortionless response
  *[DNN]: deep neural network
  *[DFT]: discrete Fourier transform
  *[PDF]: probability density function
  *[MAP]: maximum a posteriori
  *[ML]: maximum likelihood
  *[STFT]: short-time Fourier transform
  *[SNR]: signal-to-noise ratio
  *[POLQA]: perceptual objective listening quality analysis
  *[SI-SDR]: scale-invariant signal-to-distortion ratio
  *[SI-SIR]: scale-invariant signal-to-interference ratio
  *[SI-SAR]: scale-invariant signal-to-artifacts ratio
  *[PESQ]: perceptual evaluation of speech quality
  *[MOS]: mean opinion score
  *[PSD]: power spectral density
  *[EM]: expectation maximization