# TDFNet: An Efficient Audio-Visual Speech Separation Model with Top-down
Fusion

Samuel Pegg1,†, Kai Li1,†, Xiaolin Hu1,2,3,∗ $\dagger$ Samuel Pegg and Kai Li
have contributed equally to this work.* Corresponding author. 1\. Department
of Computer Science and Technology, Institute for AI
BNRist, Tsinghua University, Beijing 100084, China
2\. Tsinghua Laboratory of Brain and Intelligence (THBI),
IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing
100084, China
3\. Chinese Institute for Brain Research (CIBR), Beijing 100010, China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Audio-visual speech separation has gained significant traction in recent years
due to its potential applications in various fields such as speech
recognition, diarization, scene analysis and assistive technologies. Designing
a lightweight audio-visual speech separation network is important for low-
latency applications, but existing methods often require higher computational
costs and more parameters to achieve better separation performance. In this
paper, we present an audio-visual speech separation model called Top-Down-
Fusion Net (TDFNet), a state-of-the-art (SOTA) model for audio-visual speech
separation, which builds upon the architecture of TDANet, an audio-only speech
separation method. TDANet serves as the architectural foundation for the
auditory and visual networks within TDFNet, offering an efficient model with
fewer parameters. On the LRS2-2Mix dataset, TDFNet achieves a performance
increase of up to 10% across all performance metrics compared with the
previous SOTA method CTCNet. Remarkably, these results are achieved using
fewer parameters and only 28% of the multiply-accumulate operations (MACs) of
CTCNet. In essence, our method presents a highly effective and efficient
solution to the challenges of speech separation within the audio-visual
domain, making significant strides in harnessing visual information optimally.

###### Index Terms:

Audio-Visual, Multi-Modal, Speech-Separation

##  I Introduction

Figure 1:  Audio-visual speech separation process. From top to bottom: video
frames and mixed inputs, cutting out lip regions from the video, TDFNet speech
separation results.  Figure 2: TDFNet separation pipeline. The audio and video
inputs $\boldsymbol{x}$ and $\boldsymbol{y}$ are encoded by $E_{a}$ and
$E_{v}$ respectively to produce the feature maps $\boldsymbol{a}$ and
$\boldsymbol{v}$, which are sent to the refinement module $R$ to be fused and
then further processed. The mask generator $M$ then takes these refined
features $\boldsymbol{r}$ and generates masks $\boldsymbol{m}_{i}$, which are
multiplied by the encoded audio input $\boldsymbol{a}$ in tern to produce a
separation. Finally, the decoder decodes each of the separated encoded audios.
The figure above uses $n_{spk}=2$ speakers.

Speech separation is the process of extracting distinct audio streams from an
audio recording containing one or more speakers [2]. Consider a scenario where
we have a microphone positioned in the center of a room capturing the voices
of two individuals, A and B. These speakers may talk simultaneously or one
after the other, with varying volume levels and distances from the microphone.
It is crucial to account for all these factors. The objective of a speech
separation model is to split the audio recording into two separate streams,
each containing the audio from a single speaker. Ideally, one output stream
would exclusively contain the voice of speaker A or B.

Speech separation is commonly referred to as the “cocktail party problem” [13,
3]. At social gatherings, such as cocktail parties, our natural inclination is
to concentrate on a particular individual while filtering out the surrounding
conversations. Over the past decade, High quality speech separation has become
increasingly crucial due to the widespread adoption of automated systems and
voice assistants such as Apple’s Siri and home devices such as Amazon Alexa or
Google Home [18, 23, 17, 40, 12, 31].

Methods based on architectures designed for modeling sequences, such as
Recurrent Neural Networks (RNNs) [29], or extracting local patterns such as
Convolutional Neural Networks (CNNs) [30], have proven to be adept at handling
the complexities in speech signals. However, methods relying on auditory
signals alone, known as audio-only speech separation (AOSS) methods, have
become quite saturated in recent years, with new models showing only
incremental performance increases. One notable avenue to bolster the
robustness of speech separation is to integrate multi-modal information [41,
4]. For humans, the integration of auditory information (speech signals) and
visual stimuli (such as lip movements) fundamentally alters our perception of
language [35, 34, 7].

Audio-visual speech separation (AVSS) is similar to speaker extraction [52,
10, 50], which uses one speaker’s voiceprint to create a separation. However,
whereas voice extraction requires enrolling a target speaker in advance and
gathering their voiceprint, visual information can be entirely captured in
real-time, as shown in Figure 1. Visualvoice [9] and CTCNet [24] both use an
encoder-decoder structure to improve the separation performance, but they do
not fully exploit this structure. In addition, integrating audio and visual
information can be computationally expensive, making them less practical for
real-time applications.

In order to efficiently extract and integrate features from different
modalities, we propose a novel AVSS model called TDFNet. It uses encoded audio
and visual information as inputs for a series of TDANet-based blocks[25],
called TDFNet Blocks. These blocks build a hierarchical structure where the
lower levels have higher temporal resolution. Information from different
temporal resolutions moves freely, and is aggregated and fused with the visual
information. We evaluated TDFNet extensively on the LRS2 benchmark dataset to
show that it outperforms various competing baseline models and achieves state-
of-the-art performance using fewer parameters and with a lower computational
cost.

##  II Related Work

###  II-A Audio-Only Speech Separation

Initially, speech separation employed the time-frequency (TF) representation
of the mixed audio, estimated from the waveform using the short-time Fourier
transform (STFT). Pioneering methods utilized matrix factorization[39] and
heuristic techniques[6] to cluster the TF bins of each speaker. However, the
performance of these models was either poor or speaker-dependent. With the
development of deep learning and the introduction of permutation invariant
training (PIT[53]) to solve the permutation problem, the speech separation
space has become increasingly competitive. Researchers soon migrated from the
TF representation of audio to a time domain only representation using a
convolutional encoder, resulting in high performance methods such as
DualPathRNN [29], Sepformer [44], Wavesplit [54], etc. Recently, the multi-
scale speech separation model AFRCNN [16] was proposed as a balance between
efficiency and separation performance. TDANet [25] adds top-down attention
[32, 42] to AFRCNN’s encoder-decoder architecture to obtain state-of-the-art
performance with a greatly reduced computational effort. Both of these models
were inspired by the brain.

Some researchers suggest that the datasets used in the audio-only setting are
often idealized, lacking reverberation, interfering background sounds or
random noise. A recent study [25] found that many top models become average
when faced with these more challenging datasets. Furthermore, audio-only
models encounter significant challenges when confronted with scenarios
involving three or more speakers[14], or when the number of speakers is
unknown[52, 10, 50].

###  II-B Audio-Visual Speech Separation

The audio-visual speech separation field combines audio-only speech separation
with video data to improve performance on noisy and challenging datasets.
Recent neuroscience studies have demonstrated that the human brain effectively
addresses the cocktail party problem by leveraging additional visual cues from
the eyes[26, 43]. Building upon this notion, some researchers[22, 51, 9, 1,
33, 8, 36, 27] have endeavored to incorporate visual information into the
paradigm, aiming to enhance the quality of audio separation. These efforts
have culminated in the current state-of-the-art for audio-visual speech
separation, namely CTCNet[24]. However, CTCNet, due to the complexity of its
multi-scale fusion operations, introduces a substantial amount of computation
which limits its applicability in real-world scenarios. To address this, we
utilize TDANet’s architecture for our visual and auditory feature extraction
networks, which employ progressive multi-scale fusion, effectively reducing
the computational cost of the multi-scale fusion.

##  III TDFNet

Figure 3: The $j$th TDFNet Block. (a) The internals of a single TDFNet block.
(b) The stacked TDFNet blocks with residual connection from the first
iteration.

For a black and white 25 fps video containing $n_{spk}$ speakers, the inputs
for our AVSS pipeline are: a sequence of video frames
$\boldsymbol{y}\in\mathbb{R}^{1\times L_{v}\times H_{in}\times W_{in}}$
containing the lip movements of the desired speaker, and the mixed audio
stream $\boldsymbol{x}\in\mathbb{R}^{1\times L_{a}}$, where $L_{v}$, $H_{in}$,
$W_{in}$ and $L_{a}$ denote the number of video frames, image height, width,
and the length of audio, respectively. We assume that the mono-aural (single
channel) audio of the video $\boldsymbol{x}$ consists of linearly superimposed
voices $\boldsymbol{s}_{i}\in\mathbb{R}^{1\times L_{a}}$ for $i\in[1,n_{spk}]$
such that

$\boldsymbol{x}=\boldsymbol{\epsilon}+\sum_{i=1}^{n_{spk}}\boldsymbol{s}_{i},$
(1)

where $\boldsymbol{\epsilon}\in\mathbb{R}^{1\times L_{a}}$ is some presence of
background noise.

We propose TDFNet, which consists of five main modules: a video encoder
(Section III-A), an audio encoder (Section III-B), a refinement module
(Section III-C), a mask generator (Section III-D) and an audio decoder
(Section III-E). The pipeline of TDFNet is briefly described below (see Figure
2).

  1. 1.

The raw audio and visual signals are sent to respective encoders to obtain
rich feature maps. We denote the audio encoder $E_{a}(\cdot)$ and the video
encoder $E_{v}(\cdot)$:

$\displaystyle\boldsymbol{a}$
$\displaystyle=E_{a}(\boldsymbol{x}),\quad\boldsymbol{a}\in\mathbb{R}^{C_{a}\times
T_{a}},$ $\displaystyle\boldsymbol{v}$
$\displaystyle=E_{v}(\boldsymbol{y}),\quad\boldsymbol{v}\in\mathbb{R}^{C_{v}\times
T_{v}},$

where the $C_{a}$ and $C_{v}$ denote the dimensions of visual and auditory
features. Historically, large values for $C_{a}$ and $C_{v}$ are used as this
leads to a more detailed feature representation. However, this increases the
computational cost of the refinement module. In order to mitigate this cost
while maintaining performance, we design a “bottleneck” by using fewer
channels $B_{a}\leq C_{a}$ and $B_{v}\leq C_{v}$. This is achieved by using 1D
convolutional layers $F_{a}(\cdot)$ and $F_{v}(\cdot)$ with kernel size $1$.
This can be written

$\boldsymbol{a}^{\prime}=F_{a}(\boldsymbol{a}),\quad\boldsymbol{v}^{\prime}=F_{v}(\boldsymbol{v}).$

  2. 2.

The refinement module $R(\cdot,\cdot)$, takes the audio and visual information
and uses the combined information to generate a refined feature map:

$\boldsymbol{r}=R(\boldsymbol{a}^{\prime},\boldsymbol{v}^{\prime}),\quad\boldsymbol{r}\in\mathbb{R}^{B_{a}\times
T_{a}}$

We can see that the output dimensions match the dimensions of the audio input.
This is because the refinement module first fuses the visual information into
the audio information, and then further processes these combined multimedia
features. This will be explained in Section III-C.

  3. 3.

This refined feature map is used to generate masks for each speaker. Let
$M(\cdot)$ denote a mask generating function.

$\\{\boldsymbol{m}_{1},\dots,\boldsymbol{m}_{n_{spk}}\\}=M(\boldsymbol{r}),$

where $\boldsymbol{m}_{i}\in[0,1]^{C_{a}\times T_{a}}$ for all
$i\in[1,n_{spk}]$ and $[0,1]$ is the range of real numbers from 0 to 1. Note
that each mask has the same dimensions as the encoded audio $\boldsymbol{a}$.

  4. 4.

The encoded audio input is multiplied element-wise by each mask in turn,
resulting in the separated encoded audio for each speaker,

$\boldsymbol{z}_{i}=\boldsymbol{a}\odot\boldsymbol{m}_{i},\quad\boldsymbol{z}_{i}\in\mathbb{R}^{C_{a}\times
T_{a}}\;\forall i\in[1,n_{spk}],$

where $\odot$ represents element-wise multiplication.

  5. 5.

The separated audio feature map for each speaker is given to the decoder
$D_{a}(\cdot)$. The decoder returns the separated audios for each speaker as a
waveform,

$\hat{\boldsymbol{s}}_{i}=D_{a}(\boldsymbol{z}_{i}),\quad\hat{\boldsymbol{s}}_{i}\in\mathbb{R}^{1\times
L_{a}}\;\forall i\in[1,n_{spk}].$

###  III-A Video Encoder

The video encoder tales the gray-scale video frames $\boldsymbol{y}$ and uses
the pre-trained lip-reading model used in CTCNet[24], called CTCNet-Lip, to
extract visual features. It consists of a backbone network for extracting
features from the image frames and a classification sub-network for word
prediction. The backbone network includes a 3D convolutional layer and a
standard ResNet-18 network. The frames are convolved with $P$ 3D-kernels with
size $5\times 5\times 1$ and spatial stride size $2\times 2\times 1$ to obtain
a rich feature map. Each feature map is passed through the ResNet-18 network,
and the resulting feature maps are passed to the classification sub-network as
words for word prediction. After lip-reading pre-training, the backbone
network of the CTCNet-Lip is fixed for extracting visual features, and the
classification sub-network is discarded. We encourage readers to read the
original paper for more details.

###  III-B Audio Encoder

The audio encoder $E_{a}$ takes the input audio $\boldsymbol{x}$ and produces
a feature map using a 1D-convolution with $C_{a}$ output channels and kernel
size $K_{a}$, followed by global layer normalization (gLN) [30] and ReLU
activation. Since a stride greater than one is used, this technically changes
the $T_{a}$ dimension to a smaller value, compressing the audio, but for
simplicity we do not change the notation.

###  III-C Refinement Module

The refinement module aims to take the audio and the video embeddings, fuse
them together and then further process and refine the resulting multi-modal
features. This way, we combine the audio and visual information in order to
increase the separation performance. As is seen several times in the audio-
only domain[25, 16], we take an iterative approach. In total, we use $R_{a}$
iterations with fusion occurring $R_{f}$ times, where $R_{f}\leq R_{a}$.

The refinement module consists of three networks that work together in order
to produce the best possible amalgamation of data:

  * •

$\alpha_{j}$, the audio sub-network at iteration $j$.

  * •

$\beta_{j}$, the video sub-network at iteration $j$.

  * •

$\gamma_{j}$, the cross-modal fusion sub-network at iteration $j$.

These three networks do not change the dimensions of their inputs, which is
key for their inter-compatibility. For the fusion iterations, for
$j\in[1,R_{f}]$, one iteration consists of all three modules working together
for form a TDFNet block. For the subsequent audio-only iterations, for
$j\in[R_{f},R_{a}]$, the TDFNet block will will consist of only the audio sub-
network.

Firstly, let $\boldsymbol{a}_{j-1}$ and $\boldsymbol{v}_{j-1}$ be the inputs
for $\alpha_{j}$ and $\beta_{j}$, where the “$0^{\text{th}}$” iteration is
simply taking the outputs of the audio and video “bottleneck” convolutions:

$\boldsymbol{a}_{0}:=\boldsymbol{a}^{\prime}\in\mathbb{R}^{B_{a}\times
T_{a}},\quad\boldsymbol{v}_{0}:=\boldsymbol{v}^{\prime}\in\mathbb{R}^{B_{v}\times
T_{v}}.$ (2)

The first iteration is the most simple. The audio features are passed to the
audio sub-network and the video features are passed to the video sub-network.
The outputs are then passed to the fusion module which fuses the audio
information with the visual information, and the visual information with the
audio information. In the proceeding layers, residual connections are added to
improve training.

$\displaystyle(\boldsymbol{a}_{1},\;\boldsymbol{v}_{1})$
$\displaystyle=\gamma_{1}\left(\alpha_{1}(\boldsymbol{a}_{0}),\;\beta_{1}(\boldsymbol{v}_{0})\right)$
(3) $\displaystyle(\boldsymbol{a}_{j},\;\boldsymbol{v}_{j})$
$\displaystyle=\gamma_{j}\left(\alpha_{j}(\boldsymbol{a}_{j-1}+\boldsymbol{a}_{0}),\;\beta_{j}(\boldsymbol{v}_{j-1}+\boldsymbol{v}_{0})\right)$
(4)

for $j\in[2,R_{f}]$. The interactions can be seen in Figure 3.

After $R_{f}$ fusion iterations, we deem the video and audio signals to be
sufficiently ‘fused’ together into the audio signal
$\boldsymbol{a}_{R_{f}}\in\mathbb{R}^{B_{a}\times T_{a}}$. For subsequent
iterations, the signal is continuously refined using only the audio sub-
network. This simplifies Equation 4 to the one below.

$\boldsymbol{a}_{j}=\alpha_{j}\left(\boldsymbol{a}_{j-1}+\boldsymbol{a}_{0}\right),$
(5)

for $j\in[R_{f}+1,R_{a}]$.

####  III-C1 Audio and Video Sub-Networks

Both $\alpha_{j}$ and $\beta_{j}$ are defined by the same architecture
inspired by TDANet[25] and consists of three main sections:

  1. 1.

Bottom-up Down-sampling

  2. 2.

Recurrent Operator

  3. 3.

Top-down Fusion

The details of these sections will be covered in Section IV. We define

$\displaystyle\boldsymbol{X}_{j}$
$\displaystyle=\alpha_{j}\left(\boldsymbol{a}_{j-1}+\boldsymbol{a}_{0}\right)$
$\displaystyle\in\mathbb{R}^{B_{a}\times T_{a}},$
$\displaystyle\boldsymbol{Y}_{j}$
$\displaystyle=\beta_{j}\left(\boldsymbol{v}_{j-1}+\boldsymbol{v}_{0}\right)$
$\displaystyle\in\mathbb{R}^{B_{v}\times T_{v}},$

as the outputs of the audio and video sub-networks, and hence the two inputs
for the cross-modal fusion sub-network.

####  III-C2 Cross-Modal-Fusion Sub-network

This module is in charge of fusing the audio features into the video features,
and the video features into the audio features. Let $\kappa$ define a
1D-convolution with kernel size $1$ followed by a gLN layer, and let $\phi$
denote nearest neighbor interpolation. $||$ is the concatenation operation
acting along the channel dimension. Wielding these definitions, we can define
the cross-modal fusion sub-network $\gamma_{j}$ that returns two outputs, and
hence we can write the full expression for the $j$th iteration as

$\gamma_{j}(\boldsymbol{X}_{j},\boldsymbol{Y}_{j})=\left(\boldsymbol{a}_{j},\;\boldsymbol{v}_{j}\right),$
(6)

where

$\displaystyle\boldsymbol{a}_{j}$
$\displaystyle=\kappa\left(\boldsymbol{X}_{j}||\phi\left(\boldsymbol{Y}_{j}\right)\right)$
$\displaystyle\in\mathbb{R}^{B_{a}\times T_{a}},$
$\displaystyle\boldsymbol{v}_{j}$
$\displaystyle=\kappa\left(\boldsymbol{Y}_{j}||\phi\left(\boldsymbol{X}_{j}\right)\right)$
$\displaystyle\in\mathbb{R}^{B_{v}\times T_{v}}.$

As we can see, the video features are interpolated to match the audio
dimensions:

$B_{v}\times T_{v}\Longrightarrow B_{v}\times T_{a}.$

The output is concatenated with the audio features and then passed through a
convolution layer to take the dimensions back to the dimensions of the audio
input:

$(B_{a}+B_{v})\times T_{a}\Longrightarrow B_{a}\times T_{a}.$

The video fusion result is similar.

Figure 4: The core architecture of the audio and video sub-networks. The input
is either the audio or the visual features, reduced to the hidden dimension
$D$. The bottom-up down-sampling in the green block uses consecutive
convolutions with stride 2 to compress the data to increasingly small temporal
resolutions. The recurrent operator in the pink block fuses the information to
formulate a global perspective. The top-down fusion in the blue block combines
the global information at different temporal resolutions, and then fuses them
all together into a single feature map.

###  III-D Mask Generator

The mask generator is tasked with taking the output of the refinement module
$\boldsymbol{r}$ and transforming it into a set of masks $\boldsymbol{m}_{i}$
for $i\in[1,n_{spk}]$ and multiplying each mask with the encoded audio
$\boldsymbol{a}$. The mask generator is characterized by a 1D-convolution that
takes the channels from $B_{a}$ to $n_{spk}\times C_{a}$. We then apply an
output gate, which involves the element-wise multiplication of the $Tanh$ and
sigmoid $\sigma$ activations of two convolutions,

$\begin{split}\boldsymbol{X}&=\mathrm{ReLU}\left(\mathrm{Conv}\left(\mathrm{PReLU}(\boldsymbol{r})\right)\right),\\\
\boldsymbol{Y}&=\mathrm{Tanh}\left(\mathrm{Conv}(\boldsymbol{X})\right)\odot\sigma\left(\mathrm{Conv}(\boldsymbol{X})\right).\end{split}$
(7)

Next, we split $\boldsymbol{Y}\in\mathbb{R}^{(n_{spk}\times C_{a})\times
T_{a}}$ across the channels into $n_{spk}$ parts to get the masks
$\boldsymbol{m}_{i}\in\mathbb{R}^{C_{a}\times T_{a}}$ for each speaker $i$.
Finally, we compute

$\boldsymbol{z}_{i}=\boldsymbol{a}\odot\boldsymbol{m}_{i}\quad\text{for
}i\in[1,n_{spk}],$ (8)

to get the separated audios of each speaker, as a feature map.

###  III-E Decoder

The decoder mirrors the audio encoder of TDFNet in terms of stride, padding
and kernel size, and hence we get the equation

$\hat{\boldsymbol{s}}_{i}=\mathrm{TConv}(\boldsymbol{z}_{i})=D_{a}\left(\boldsymbol{z}_{i}\right)\quad\text{for
}i\in[1,n_{spk}],$ (9)

where $\mathrm{TConv}$ is now a 1D transposed convolution that converts the
$n_{spk}$ separated feature maps into $n_{spk}$ waveform audio streams.

##  IV Audio and Video Sub-Network Structure

We have described the entire pipeline of the proposed TDFNet model without
details of the audio and video sub-networks. These modules use a TDANet-like
structure[25] and since the design for the audio and video sub-networks is the
same, we can drop the $a$ and $v$ subscripts from our notation for simplicity.
For more specific details of the TDANet implementation, we refer readers to
the original paper [25]. Here we will give a brief overview, focusing on our
modifications.

The input to the audio and video sub-networks:
$\boldsymbol{a}_{j-1}+\boldsymbol{a}_{0}$ or
$\boldsymbol{v}_{j-1}+\boldsymbol{v}_{0}$, is first bound in place using a
depth-wise convolution, and then converted to a lower “hidden” dimension $D$
using another 1D convolution with kernel size 1. This will be the input for
the main section of TDANet, which can be categorized into three important
phases:

  1. 1.

the bottom-up down-sampling process

  2. 2.

the recurrent operator

  3. 3.

the top-down fusion process

as shown in Figure 4.

#### Bottom-up Down-sampling

For kernel size $k$ and stride $s$, we next define the normalized depth-wise
convolution,

$\mathcal{H}_{k,s}(\boldsymbol{X}):=\mathrm{Norm}\left(\mathrm{DWConv}_{k,s}(\boldsymbol{X})\right).$
(10)

The bottom-up down-sampling process uses stacked $\mathcal{H}_{k,2}$ layers to
obtain the multi-scale set $\boldsymbol{F}$ with different temporal
resolutions:

$\boldsymbol{F}=\left\\{\boldsymbol{F}_{0}\in\mathbb{R}^{D\times
T},\dots,\boldsymbol{F}_{i}\in\mathbb{R}^{D\times\frac{T}{2^{i}}},\dots,\boldsymbol{F}_{q}\in\mathbb{R}^{D\times\frac{T}{2^{q}}}\right\\}.$

#### Recurrent Operator

To extract a global view from these features, we down-sample all elements in
the set $\boldsymbol{F}$ to the dimensions of the smallest element using
adaptive average pooling. Next, we sum all the down-sampled features to
generate the global feature map $\boldsymbol{G}^{in}$:

$\boldsymbol{G}^{in}=\sum_{i=0}^{q}p(\boldsymbol{F}_{i}),\quad\boldsymbol{G}^{in}\in\mathbb{R}^{D\times\frac{T}{2^{q}}}.$
(11)

To allow the model to understand the complex relationships between the
different time steps, a recurrent operator $\mathcal{R}$ is applied along the
temporal dimension of $\boldsymbol{G}^{in}$:

$\displaystyle\boldsymbol{G}^{out}$
$\displaystyle=\mathcal{R}\left(\boldsymbol{G}^{in}\right),\quad\boldsymbol{G}^{out}\in\mathbb{R}^{D\times\frac{T}{2^{q}}}.$
(12)

The recurrent operator refers to a sequence modelling structure such as a
transformer, recurrent neural network (RNN), long-short-term-memory network
(LSTM[15]) or gated recurrent unit (GRU[5]).

For the transformer (see Figure 5), we first use multi-head self-attention
(MHSA)[48] followed by a drop block layer[11] and a feed forward network
(FFN). The FFN consists of three stacked 1D convolutional layers with kernel
sizes {$1$, $k$, $1$}, and number of channels {$D$, $2D$, $D$} respectively.
Residual connections are added at each stage, so for the transformer the
recurrent operator $\mathcal{R}$ is defined as:

$\begin{split}\boldsymbol{G}^{mid}&=\mathrm{MHSA}(\boldsymbol{G}^{in})+\boldsymbol{G}^{in},\\\
\boldsymbol{G}^{out}&=\mathrm{FFN}\left(\boldsymbol{G}^{mid}\right)+\boldsymbol{G}^{mid}.\end{split}$
(13)

Figure 5: Attention mechanism. This is a diagrammatic view of Equation 13.

For the other recurrent operators (RNN, LSTM and GRU) we remove the FFN and
the Drop Block[11] layers, but keep the residual connection. We use
bidirectional RNNs with the hidden dimension set to the input dimension $D$,
hence we also add dropout and a linear projection layer $\mathcal{P}$ from
$2D$ channels back to $D$ channels such that the RNN does not change the
dimensions of $\boldsymbol{G}^{out}$. This alters Equation 13 to:

$\boldsymbol{G}^{out}=\mathcal{P}\left(\text{RNN}\left(\boldsymbol{G}^{in}\right)\right)+\boldsymbol{G}^{in}.$
(14)

#### Top-Down Fusion

Let $I_{k}$ define the Injection Sum [25] with kernel size $k$ (see Figure 6).
Then top-down fusion is defined in two steps. First, the global information
$\boldsymbol{G}^{out}$ is fused with each of the multi-scale local features:

$\boldsymbol{F}^{\prime}_{i}=I_{k}(\boldsymbol{F}_{i},\boldsymbol{G}^{out}),\quad\boldsymbol{F}^{\prime}_{i}\in\mathbb{R}^{D\times\frac{T}{2^{i}}}$
(15)

for $i\in[0,q]$. Next, this multi-scale global-and-local-fusion is collapsed
into a single feature map that has a broad view of the entire input. This is
achieved with an iterative process using an injection sum with kernel size 1:

$\begin{split}\boldsymbol{F}^{\prime\prime}&=I_{1}(\boldsymbol{F}^{\prime}_{q-1},\boldsymbol{F}^{\prime}_{q})+\boldsymbol{F}_{q-1},\\\
\boldsymbol{F}^{\prime\prime}&=I_{1}(\boldsymbol{F}^{\prime}_{q-i-1},\boldsymbol{F}^{\prime\prime})+\boldsymbol{F}_{q-i-1}\quad\text{for
}i\in[1,q-1],\end{split}$ (16)

where after the last iteration,
$\boldsymbol{F}^{\prime\prime}\in\mathbb{R}^{D\times T}$. This differs from
the original TDANet, as in the original implementation $I_{1}$ is replaced
with a different operation. In addition, we have added residual connections
from the multi-scale down-sampled features in order to improve training and
create a UNet-like [38] structure. Both of these changes greatly improved the
consistency and effectiveness of TDANet without increasing the parameters
significantly. Note that in Figure 4 we do not show the residual connections
as it would make the diagram too complex.

Finally, the feature map $\boldsymbol{F}^{\prime\prime}$ is converted from the
hidden dimension $D$ back to the bottleneck dimension $B_{a}$ or $B_{v}$ using
a 1D convolution with kernel size 1, and residual connection is added. This
result is denoted $\boldsymbol{X}_{j}$ for the output of the audio sub-
network, and $\boldsymbol{Y}_{j}$ for the output of the video sub-network, see
Section III-C1.

Figure 6: Architecture for the Injection Sum $I_{k}$ with kernel size $k$ for
two inputs $\boldsymbol{X}$ and $\boldsymbol{Y}$ with different temporal
dimensions.

##  V Experimental Procedures

###  V-A Dataset

Following similar methods [9, 24, 22] in the field, speech separation datasets
were constructed from commonly used audio-visual datasets. The speakers in the
test set of these datasets did not overlap with those in the training and
validation datasets. The raw audio and video frames were obtained using the
FFmpeg tool111https://ffmpeg.org/. A sampling rate of 16 kHz was chosen, and
the model was trained on two seconds of audio and video, equating to audio
vectors of length 32,000 and 50 video frames for the 25 FPS video used.

The LRS2 dataset contains a collection of BBC video clips and three separate
folders for training, validation and testing purposes. In order to adapt this
dataset to an audio-visual dataset, different speakers were randomly selected
two speakers at a time. Their audio signals were mixed using signal-to-noise
rations between -5 and 5 dB. The test set is the same as used in previous
works[9, 24]. In total, the training set contains 11 hours and the validation
set contains 3 hours.

TABLE I: TDFNet hyperparameters. Parameter | Value | Description
---|---|---
$C_{a}$ | 512 | The audio mixture embedding dimension
$K_{a}$ | 21 | The kernel size of the encoder and decoder
$S_{a}$ | 10 | The stride of the encoder and decoder
$B_{a}$ | 512 | The audio bottleneck out channel dimension
$B_{v}$ | 64 | The video bottleneck out channel dimension
$D_{a}$ | 512 | The audio sub-network hidden dimension
$q_{a}$ | 5 | The audio sub-network up-sampling depth
$k_{a}$ | 5 | The audio sub-network kernel size
$h_{a}$ | 512 | The GRU hidden dimension
$R_{a}$ | 16 | The total number of audio sub-network repeats
$D_{v}$ | 64 | The video sub-network hidden dimension
$q_{v}$ | 4 | The video sub-network up-sampling depth
$k_{v}$ | 3 | The video sub-network kernel size
$h_{v}$ | 8 | The number of attention heads
$R_{f}$ | 3 | The total number of video sub-network repeats
TABLE II: Audio sub-network recurrent operator ($R_{a}=4$, $R_{f}=1$). Model | $A_{v}$ | LRS2-2Mix | Params | MACs
---|---|---|---|---
Module | SI-SNRi | SDRi | PESQ | STOI | (M) | (G)
CTCNet[24] |  | 11.1 | 11.6 | 2.90 | 0.895 | 6.3 | 43.6
TDFNet | RNN | 11.6 | 11.9 | 2.95 | 0.903 | 3.6 | 13.2
TDFNet | MHSA | 12.9 | 13.1 | 3.08 | 0.921 | 4.2 | 12.8
TDFNet | LSTM | 13.4 | 13.5 | 3.08 | 0.928 | 6.8 | 15.7
TDFNet | GRU | 13.6 | 13.7 | 3.10 | 0.931 | 5.8 | 15.0
TABLE III: Video sub-network recurrent operator ($R_{a}=16$, $R_{f}=3$).  $\beta_{j}$ | LRS2-2Mix | Params | MACs
---|---|---|---
Module | SI-SNRi | SDRi | PESQ | STOI | (M) | (G)
GRU | 15.3 | 15.4 | 3.19 | 0.942 | 6.6 | 47.2
MHSA | 15.8 | 15.9 | 3.21 | 0.949 | 6.5 | 47.2
TABLE IV: Sharing parameters in the audio sub-network ($R_{a}=4$, $R_{f}=1$) Audio | LRS2-2Mix | Params | MACs
---|---|---|---
Shared | SI-SNRi | SDRi | PESQ | STOI | (M) | (G)
| 12.3 | 12.4 | 3.01 | 0.914 | 22.9 | 15.7
| 13.4 | 13.5 | 3.08 | 0.928 | 6.8 | 15.7
TABLE V: Sharing parameters in the video and cross-modal fusion sub-networks ($R_{a}=16$, $R_{f}=3$). Shared | LRS2-2Mix | Params | MACs
---|---|---|---
SI-SNRi | SDRi | PESQ | STOI | (M) | (G)
| 15.0 | 15.2 | 3.16 | 0.938 | 4.2 | 38.6
| 15.3 | 15.4 | 3.20 | 0.943 | 4.9 | 38.6
TABLE VI: Comparison with SOTA methods ($R_{a}=16$, $R_{f}=3$). The “-” denoted results not reported in the original papers. Model | LRS2-2Mix | Params | MACs
---|---|---|---
SI-SNRi | SDRi | PESQ | STOI | (M) | (G)
uPIT[19] | 3.6 | 4.8 | - | - | 92.7 | -
SuDORM-RF[47] | 9.1 | 9.5 | - | - | 2.7 | -
A-FRCNN[16] | 9.4 | 10.1 | - | - | 6.3 | -
Conv-TasNet[30] | 10.3 | 10.7 | - | - | 5.6 | -
CaffNet-C[22] | - | 10.0 | 0.94 | 0.88 | - | -
CaffNet-C*[22] | - | 12.5 | 1.15 | 0.89 | - | -
Thanh-Dat[46] | - | 11.6 | 3.1 | - | - | -
VisualVoice[9] | 11.5 | 11.8 | 3.00 | - | 77.8 | -
CTCNet[24] | 14.3 | 14.6 | 3.08 | 0.931 | 7.1 | 167.2
TDFNet-small | 13.6 | 13.7 | 3.10 | 0.931 | 5.8 | 15.0
TDFNet (MHSA + Shared) | 15.0 | 15.2 | 3.16 | 0.938 | 4.2 | 38.6
TDFNet-large | 15.8 | 15.9 | 3.21 | 0.949 | 6.5 | 47.2

###  V-B Hyperparameter Settings

Following CTCNet[24] we use 16 total layers with 3 fusion layers. The other
hyperparameters are defined in Table I. The first block shows the encoder and
decoder hyperparameters. The second block shows the audio sub-network
hyperparameters. The third block shows the video sub-network hyperparameters.

For training we used a batch size of 4 and AdamW[28] optimization with a
weight decay of $1\times 10^{-1}$. The initial learning rate used was $1\times
10^{-3}$, but the learning rate value was halved when the validation data set
loss did not decrease for 5 epochs in a row. We also used gradient clipping in
order to limit the maximum $L_{2}$ norm of the gradient to 5. Training was
left running for a maximum of 200 epochs, but early stopping was also applied.
Models were all trained on four servers, three containing 8 NVIDIA 3080 GPUs,
and one containing 8 NVIDIA 3090 GPUs. The model code is implemented using
PyTorch.

###  V-C Loss Function

The loss function used for training is the scale-invariant source-to-noise
ratio (SI-SNR)[21] between the estimated and original signals
$\boldsymbol{s}_{i}$ and $\hat{\boldsymbol{s}}_{i}$ respectively for each
speaker. SI-SNR is defined as

$\text{SI-
SNR}(\boldsymbol{s}_{i},\;\hat{\boldsymbol{s}}_{i})=10\log_{10}\left(\frac{||\boldsymbol{\omega}_{i}\cdot\boldsymbol{s}_{i}||^{2}}{||\hat{\boldsymbol{s}}_{i}-\boldsymbol{\omega}_{i}\cdot\boldsymbol{s}_{i}||^{2}}\right),$
(17)

where $\boldsymbol{\omega}_{i}$ is the result

$\boldsymbol{\omega}_{i}=\frac{\hat{\boldsymbol{s}}_{i}^{T}\boldsymbol{s}_{i}}{\boldsymbol{s}_{i}^{T}\boldsymbol{s}_{i}}.$
(18)

###  V-D Evaluation Metrics

Following recent literature, the scale-invariant signal-to-noise ratio
improvement (SI-SNRi) and signal-to-noise ratio improvement (SDRi) were used
to evaluate the quality of the separated speeches. These metrics were
calculated based on the scale-invariant signal-to-noise ratio (SI-SNR)[21] and
source-to-distortion ratio (SDR)[49]:

$\begin{split}\text{SI-
SNRi}(\boldsymbol{x},\;\boldsymbol{s}_{i},\;\hat{\boldsymbol{s}}_{i})&=\text{SI-
SNR}(\boldsymbol{s}_{i},\;\hat{\boldsymbol{s}}_{i})-\text{SI-
SNR}(\boldsymbol{s}_{i},\;\boldsymbol{x}),\\\
\text{SDRi}(\boldsymbol{x},\;\boldsymbol{s}_{i},\;\hat{\boldsymbol{s}}_{i})&=\text{SDR}(\boldsymbol{s}_{i},\;\hat{\boldsymbol{s}}_{i})-\text{SDR}(\boldsymbol{s}_{i},\;\boldsymbol{x}),\end{split}$
(19)

where

$\text{SDR}(\boldsymbol{s}_{i},\;\hat{\boldsymbol{s}}_{i})=10\log_{10}\left(\frac{||\boldsymbol{s}_{i}||^{2}}{||\boldsymbol{s}_{i}-\hat{\boldsymbol{s}}_{i}||^{2}}\right).$
(20)

We also consider the number of parameters and the MACs. These metrics are
important as they determine the computational complexity and memory
requirements of the models. In both the number of parameters and MACs, a lower
value is preferable. For completeness, we also provide PESQ[37] and STOI[45]
for the main results tables. For these evaluation metrics, a higher value
indicates better performance.

##  VI Results

###  VI-A Ablation studies

In order to evaluate results faster, experimentation was done using a reduced
model setting with 1 fusion layer and 3 audio only layers, totaling 4 layers
($R_{f}=1$, $R_{a}=4$).

####  VI-A1 Different Recurrent Operators

In Table II we examine the effects of using different recurrent operators in
the audio sub-network. The transformer, denoted by MHSA, has a good balance
between the number of parameters, computational complexity and model
performance. It has the lowest number of MAC operations, making the
transformer the most efficient choice for audio separation.

A traditional RNN outperforms CTCNet[24] by a significant margin, but lacks
the huge performance gains of the transformer. However, we can also see that
the RNN model uses significantly less parameters - only 58% of the parameters
used by CTCNet. For these experiments, the RNN, GRU and LSTM models all use
$h_{a}=D_{a}$, the hidden dimension is equal to the input dimension. We found
that increasing the hidden dimension to 2$\times$ the input dimension barely
affected performance, and resulted in a huge increase in the number of
parameters.

Moving on to the LSTM[15] and GRU[5] structures, we can see that both offer
significant gains over the transformer model, and completely outclass the
smaller CTCNet model. As pointed out by Max W. Y. Lam et al.[20], it seems
that an RNN based approach is better for audio, which features a high temporal
correlation, acoustic signal structure, continuities and sequential nature.
Interestingly, even though the GRU model has fewer parameters, it outperforms
the LSTM architecture by a significant margin, while also utilizing fewer MAC
operations. It appears that for this task, the GRU architecture is the optimal
recurrent operator in terms of performance, model size and efficiency. It is
worth noting however that the GRU is not without its downsides. The GRU
structure does use slightly more memory than the LSTM structure of similar
size during training. For readers looking to use larger configurations and who
do not care about model size, the LSTM may be the better choice in order to
train with a larger batch size.

Table III shows the effect of changing the recurrent operator in the video
sub-network, as opposed to the audio sub-network experiments in Table II.
Unlike with the audio sub-network, the MHSA clearly outperforms the GRU
operator for this medium. These two tables combined show why it is important
we use the GRU for the audio sub-network, and MHSA in the video sub-network.

####  VI-A2 Sharing Parameters

In Table IV we experiment with sharing the parameters between the audio
layers. Both TDFNet and CTCNet achieve a smaller model size compared to
VisualVoice[9] by sharing parameters. Specifically, the parameters for the
audio sub-network $\alpha_{j}$ are the same for all $j$, the parameters for
the video sub-network $\beta_{j}$ are the same for all $j$ and the parameters
for the cross-modal fusion sub-network $\gamma_{j}$ are also the same for all
$j$. In PyTorch, this structure is realised by instantiating one TDFNet block,
and passing the data through this same block $R_{a}$ or $R_{f}$ times. In
Table IV we use $R_{f}=1$, so there are no additional layers for the video
sub-network and fusion model to share parameters with. Hence, we experiment
with not sharing the parameters between the audio sub-networks. When we stop
sharing the parameters between the layers of the audio sub-networks, we are
instantiating a new TDFNet block $R_{a}$ times which results in a huge
increase in model size, as seen in Table IV. One might expect this to increase
the performance of the model, but we see quite the exact opposite effect.
Instantiating new layers results in a large drop in performance. This is
likely because parameter sharing allows the TDFNet blocks themselves to act as
an RNN-like structure, and so when we stop sharing parameters we lose this
important effect.

Table V shows the effect of sharing parameters, this time in the video and
fusion layers. If the “shared” column has a “” mark, then the parameters for
$\beta_{j}$ are the same for all $j$, and the parameters for $\gamma_{j}$ are
the same for all $j$: we instantiate one instance of the video and fusion sub-
networks, and pass the data through these layers $R_{f}$ times. We can see
that unlike in the audio sub-network, sharing parameters here decreases
performance. This is likely due to the nature of the fusion operation. In the
first iteration, the video network looks at a pure video feature map generated
from the video encoder. In subsequent iterations, there is the combined audio
signal and the skip connection. It seems that the model likes to use the
subsequent iterations to fuse the different information in different ways, and
thus benefits from instantiating separate layers. This comes at the cost of
increased parameters, but the increase is small and the performance boost is
large. It is also worth noting that the models in Table V use MHSA as the
recurrent operator in both the audio and video sub-networks. If model size is
of most importance, we can see that we can still achieve over 15dB SDRi and
SI-SNRi using only 4.2 million parameters - only 60% of the parameters used by
CTCNet.

###  VI-B Comparison with the state-of-the-arts

In Table VI we can see how TDFNet compares to the competition. We have chosen
the three most interesting results for comparison.

  * •

TDFNet-small is the smaller configuration using only one fusion layer, and
three audio only layers. It comes from the last row of Table II. This model is
small with outstanding performance and a very low computational cost.

  * •

TDFNet (MHSA + Shared) is the version of TDFNet using MHSA as the recurrent
operator in both the audio and video sub-networks, and sharing the video sub-
network parameters. It comes from the first row of Table V. This model
outperforms the SOTA method CTCNet by a significant margin while using only
60% the number of parameters, and 25% the number of MACs.

  * •

TDFNet-large is the final full-bodied version of TDANet using the GRU in the
audio sub-network, and using three separate instances for the video sub-
network. It comes from the second row of Table III. This model outperforms all
other models across all metrics using only 47.2 billion MACs for two seconds
of audio sampled at 16000 Hz. This represents approximately 30% of the MACs
used by CTCNet. At the time of writing this paper, TDFNet is the new SOTA
method in audio-visual speech separation. The SI-SNRi score of 15.8 presents a
10% increase in performance compared to CTCNet.

##  VII Conclusion

Existing multimodal speech separation models are inefficient and have
limitations for real-time tasks. We propose a multi-scale and multi-stage
framework for audiovisual speech separation based on TDANet and CTCNet. This
model can significantly improve the speech separation performance by fusing
features of different modalities several times in the fusion stage and
influencing the feature extraction network of the corresponding modalities
separately. In addition, we explore the impact of different global feature
extraction structures on the performance and find that using GRU for sequence
modeling can substantially improve the performance and reduce the model
computation. Our experiments show that TDFNet outperforms the current state-
of-the-art model CTCNet in several audio separation quality metrics while
using only 30% the number of MACs.

## Acknowledgements

This work was supported in part by the National Key Research and Development
Program of China (grant 2021ZD0200301) and the National Natural Science
Foundation of China (grant 62061136001).

## References

  * [1] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman.  The conversation: Deep audio-visual speech enhancement.  In Interspeech, pages 3244–3248, 2018.
  * [2] Fahimeh Bahmaninezhad, Shi-Xiong Zhang, Yong Xu, Meng Yu, John HL Hansen, and Dong Yu.  A unified framework for speech separation.  arXiv preprint arXiv:1912.07814, 2019.
  * [3] Adelbert W Bronkhorst.  The cocktail party phenomenon: A review of research on speech intelligibility in multiple-talker conditions.  Acta Acustica united with Acustica, 86(1):117–128, 2000.
  * [4] Céline Cappe, Eric M Rouiller, and Pascal Barone.  Multisensory anatomical pathways.  Hearing Research, 258(1-2):28–36, 2009.
  * [5] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  In Empirical Methods in Natural Language Processing (EMNLP), page 1724. Association for Computational Linguistics, 2014.
  * [6] Martin Cooke, John R Hershey, and Steven J Rennie.  Monaural speech separation and recognition challenge.  Computer Speech & Language, 24(1):1–15, 2010.
  * [7] T Donishi, A Kimura, H Imbe, I Yokoi, and Y Kaneoke.  Sub-threshold cross-modal sensory interaction in the thalamus: lemniscal auditory response in the medial geniculate nucleus is modulated by somatosensory stimulation.  Neuroscience, 174:200–215, 2011.
  * [8] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein.  Looking to listen at the cocktail pparty: A speaker-independent audio-visual model for speech separation.  ACM Transactions on Graphics, 37(4):1–11, 2018.
  * [9] Ruohan Gao and Kristen Grauman.  Visualvoice: Audio-visual speech separation with cross-modal consistency.  In Computer Vision and Pattern Recognition Conference (CVPR), pages 15490–15500. IEEE, 2021.
  * [10] Meng Ge, Chenglin Xu, Longbiao Wang, Chng Eng Siong, Jianwu Dang, and Haizhou Li.  SpEx+: A complete time domain speaker extraction network.  In Interspeech, 2020.
  * [11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le.  Dropblock: A regularization method for convolutional networks.  In Conference on Neural Information Processing Systems (NeurIPS), volume 31, 2018.
  * [12] Reinhold Haeb-Umbach, Shinji Watanabe, Tomohiro Nakatani, Michiel Bacchiani, Bjorn Hoffmeister, Michael L Seltzer, Heiga Zen, and Mehrez Souden.  Speech processing for digital home assistants: Combining signal processing with deep-learning techniques.  IEEE Signal Processing Magazine, 36(6):111–124, 2019.
  * [13] Simon Haykin and Zhe Chen.  The cocktail party problem.  Neural Computation, 17(9):1875–1902, 2005.
  * [14] John R Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe.  Deep clustering: Discriminative embeddings for segmentation and separation.  In International Conference on Acoustics, Speech, & Signal Processing (ICASSP), pages 31–35. IEEE, 2016.
  * [15] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural Computation, 9(8):1735–1780, 1997.
  * [16] Xiaolin Hu, Kai Li, Weiyi Zhang, Yi Luo, Jean-Marie Lemercier, and Timo Gerkmann.  Speech separation using an asynchronous fully recurrent convolutional neural network.  In Conference on Neural Information Processing Systems (NeurIPS), 2021.
  * [17] Chanwoo Kim, Anjali Menon, Michiel Bacchiani, and Richard Stern.  Sound source separation using phase difference and reliable mask selection selection.  In International Conference on Acoustics, Speech, & Signal Processing (ICASSP), pages 5559–5563. IEEE, 2018.
  * [18] Chanwoo Kim, Ananya Misra, Kean Chin, Thad Hughes, Arun Narayanan, Tara Sainath, and Michiel Bacchiani.  Generation of large-scale simulated utterances in virtual rooms to train deep-neural networks for far-field speech recognition in google home.  In Interspeech, pages 379–383, 08 2017.
  * [19] Morten Kolbæk, Dong Yu, Zheng-Hua Tan, and Jesper Jensen.  Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(10):1901–1913, 2017.
  * [20] Max WY Lam, Jun Wang, Dan Su, and Dong Yu.  Effective low-cost time-domain audio separation using globally attentive locally recurrent networks.  In IEEE Spoken Language Technology Workshop (SLT), pages 801–808. IEEE, 2021.
  * [21] Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey.  Sdr–half-baked or well done?  In International Conference on Acoustics, Speech, & Signal Processing (ICASSP), pages 626–630. IEEE, 2019.
  * [22] Jiyoung Lee, Soo-Whan Chung, Sunok Kim, Hong-Goo Kang, and Kwanghoon Sohn.  Looking into your speech: Learning cross-modal affinity for audio-visual speech separation.  In Computer Vision and Pattern Recognition Conference (CVPR), pages 1336–1345, 2021.
  * [23] Bo Li, Tara N Sainath, Arun Narayanan, Joe Caroselli, Michiel Bacchiani, Ananya Misra, Izhak Shafran, Hasim Sak, Golan Pundak, et al.  Acoustic modeling for google home.  In Interspeech, pages 399–403, 2017.
  * [24] Kai Li, Fenghua Xie, Hang Chen, Kexin Yuan, and Xiaolin Hu.  An audio-visual speech separation model inspired by cortico-thalamo-cortical circuits.  arXiv preprint arXiv:2212.10744, 2022.
  * [25] Kai Li, Runxuan Yang, and Xiaolin Hu.  An efficient encoder-decoder architecture with top-down attention for speech separation.  In International Conference on Learning Representations (ICLR), 2023\.
  * [26] Yuanqing Li, Fangyi Wang, Yongbin Chen, Andrzej Cichocki, and Terrence Sejnowski.  The effects of audiovisual inputs on solving the cocktail party problem in the human brain: An fmri study.  Cerebral Cortex, 28(10):3623–3637, 2018.
  * [27] Jiuxin Lin, Xinyu Cai, Heinrich Dinkel, Jun Chen, Zhiyong Yan, Yongqing Wang, Junbo Zhang, Zhiyong Wu, Yujun Wang, and Helen Meng.  Av-sepformer: Cross-attention sepformer for audio-visual target speaker extraction.  In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  * [28] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations (ICLR), 2019\.
  * [29] Yi Luo, Zhuo Chen, and Takuya Yoshioka.  Dual-path rnn: Efficient long sequence modeling for time-domain single-channel speech separation.  In International Conference on Acoustics, Speech, & Signal Processing (ICASSP), pages 46–50, 2020.
  * [30] Yi Luo and Nima Mesgarani.  Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(8):1256–1266, 2019.
  * [31] Radek Martinek, Jan Vanus, Jan Nedoma, Michael Fridrich, Jaroslav Frnda, and Aleksandra Kawala-Sterniuk.  Voice communication in noisy environments in a smart house using hybrid lms+ ica algorithm.  Sensors, 20(21):6022, 2020.
  * [32] Kana Mizokuchi, Toshihisa Tanaka, Takashi G Sato, and Yoshifumi Shiraki.  Alpha band modulation caused by selective attention to music enables eeg classification.  Cognitive Neurodynamics, pages 1–16, 2023.
  * [33] Juan F Montesinos, Venkatesh S Kadandale, and Gloria Haro.  A cappella: Audio-visual singing voice separation.  In British Machine Vision Conference (BMVC), 2021.
  * [34] Ryan J Morrill and Andrea R Hasenstaub.  Visual information present in infragranular layers of mouse auditory cortex.  Journal of Neuroscience, 38(11):2854–2862, 2018.
  * [35] Sarah L Pallas, Anna W Roe, and Mriganka Sur.  Visual projections induced into the auditory pathway of ferrets. i. novel inputs to primary auditory cortex (ai) from the lp/pulvinar complex and the topography of the mgn-ai projection.  Journal of Comparative Neurology, 298(1):50–68, 1990.
  * [36] Zexu Pan, Ruijie Tao, Chenglin Xu, and Haizhou Li.  Muse: Multi-modal target speaker extraction with visual cues.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6678–6682. IEEE, 2021.
  * [37] Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra.  Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs.  In International Conference on Acoustics, Speech, & Signal Processing (ICASSP), volume 2, pages 749–752. IEEE, 2001.
  * [38] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer Assisted Interventions (MICCAI), pages 234–241. Springer, 2015.
  * [39] Mikkel N Schmidt and Rasmus Kongsgaard Olsson.  Single-channel speech separation using sparse non-negative matrix factorization.  In Interspeech, volume 2, pages 2–5. Citeseer, 2006.
  * [40] Khairunisa Sharif and Bastian Tenbergen.  Smart home voice assistants: A literature survey of user privacy and security vulnerabilities.  Complex Systems Informatics and Modeling Quarterly, pages 15–30, 10 2020.
  * [41] Linda Smith and Michael Gasser.  The development of embodied cognition: Six lessons from babies.  Artificial Life, 11(1-2):13–29, 2005.
  * [42] Tao Song, Lin Xu, Ziyi Peng, Letong Wang, Cimin Dai, Mengmeng Xu, Yongcong Shao, Yi Wang, and Shijun Li.  Total sleep deprivation impairs visual selective attention and triggers a compensatory effect: Evidence from event-related potentials.  Cognitive Neurodynamics, 17(3):621–631, 2023.
  * [43] Hanne Stenzel, Jon Francombe, and Philip JB Jackson.  Limits of perceived audio-visual spatial coherence as defined by reaction time measurements.  Frontiers in Neuroscience, 13:451, 2019.
  * [44] Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong.  Attention is all you need in speech separation.  In International Conference on Acoustics, Speech, & Signal Processing (ICASSP), pages 21–25. IEEE, 2021.
  * [45] Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen.  A short-time objective intelligibility measure for time-frequency weighted noisy speech.  In International Conference on Acoustics, Speech, & Signal Processing (ICASSP), pages 4214–4217. IEEE, 2010.
  * [46] Thanh-Dat Truong, Chi Nhan Duong, Hoang Anh Pham, Bhiksha Raj, Ngan Le, Khoa Luu, et al.  The right to talk: An audio-visual transformer approach.  In International Conference on Computer Vision (ICCV), pages 1105–1114, 2021.
  * [47] Efthymios Tzinis, Zhepei Wang, and Paris Smaragdis.  Sudo rm -rf: Efficient networks for universal audio source separation.  In Machine Learning for Signal Processing (MLSP), pages 1–6. IEEE, 2020.
  * [48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Conference on Neural Information Processing Systems (NeurIPS), volume 30, 2017.
  * [49] Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte.  Performance measurement in blind audio source separation.  IEEE Transactions on Audio, Speech, and Language Processing, 14(4):1462–1469, 2006.
  * [50] Quan Wang, Hannah Muckenhirn, Kevin Wilson, Prashant Sridhar, Zelin Wu, John Hershey, Rif A Saurous, Ron J Weiss, Ye Jia, and Ignacio Lopez Moreno.  Voicefilter: Targeted voice separation by speaker-conditioned spectrogram masking.  In Interspeech, 2018.
  * [51] Jian Wu, Yong Xu, Shi-Xiong Zhang, Lian-Wu Chen, Meng Yu, Lei Xie, and Dong Yu.  Time domain audio visual speech separation.  In Automatic Speech Recognition & Understanding (ASRU), pages 667–673. IEEE, 2019.
  * [52] Chenglin Xu, Wei Rao, Eng Siong Chng, and Haizhou Li.  Spex: Multi-scale time domain speaker extraction network.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:1370–1384, 2020.
  * [53] Dong Yu, Morten Kolbæk, Zheng-Hua Tan, and Jesper Jensen.  Permutation invariant training of deep models for speaker-independent multi-talker speech separation.  In International Conference on Acoustics, Speech, & Signal Processing (ICASSP), pages 241–245. IEEE, 2017.
  * [54] Neil Zeghidour and David Grangier.  Wavesplit: End-to-end speech separation by speaker clustering.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:2840–2849, 2021.