# D-FaST: Cognitive Signal Decoding
with Disentangled Frequency-Spatial-Temporal Attention

WeiGuo Chen, Changjian Wang, Kele Xu, Yuan Yuan, Yanru Bai and Dongsong Zhang
WeiGuo Chen, Changjian Wang, Kele Xu, Yuan Yuan are with National University
of Defense Technology, Changsha, Hunan, 410000, China. Email:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Bai is with Academy of Medical Engineering and
Translational Medicine, Tianjin University, Tianjin, 300072, China. Email:
<EMAIL_ADDRESS>Zhang is with School of Big Data and Artificial
Intelligence, Xinyang College, Xinyang, Henan, 464000, China. Email:
<EMAIL_ADDRESS>author: Changjian Wang, Kele XuCode is
available at https://github.com/AdFiFi/D-FaST.git

###### Abstract

Cognitive Language Processing (CLP), situated at the intersection of Natural
Language Processing (NLP) and cognitive science, plays a progressively pivotal
role in the domains of artificial intelligence, cognitive intelligence, and
brain science. Among the essential areas of investigation in CLP, Cognitive
Signal Decoding (CSD) has made remarkable achievements, yet there still exist
challenges related to insufficient global dynamic representation capability
and deficiencies in multi-domain feature integration. In this paper, we
introduce a novel paradigm for CLP referred to as Disentangled Frequency-
Spatial-Temporal Attention(D-FaST). Specifically, we present an novel
cognitive signal decoder that operates on disentangled frequency-space-time
domain attention. This decoder encompasses three key components: frequency
domain feature extraction employing multi-view attention, spatial domain
feature extraction utilizing dynamic brain connection graph attention, and
temporal feature extraction relying on local time sliding window attention.
These components are integrated within a novel disentangled framework.
Additionally, to encourage advancements in this field, we have created a new
CLP dataset, MNRED. Subsequently, we conducted an extensive series of
experiments, evaluating D-FaST’s performance on MNRED, as well as on publicly
available datasets including ZuCo, BCIC IV-2A, and BCIC IV-2B. Our
experimental results demonstrate that D-FaST outperforms existing methods
significantly on both our datasets and traditional CSD datasets including
establishing a state-of-the-art accuracy score 78.72% on MNRED, pushing the
accuracy score on ZuCo to 78.35%, accuracy score on BCIC IV-2A to 74.85% and
accuracy score on BCIC IV-2B to 76.81%.

###### Index Terms:

Cognitive Language Processing (CLP), Cognitive Signal Decoding (CSD),
Frequency-spatial-temporal domain attention

##  I Introduction

Figure 1: Conceptual comparison of four brain signal decoding architectures.
(a): The Single-Domain (1D) Architecture primarily focuses on the extraction
of spatial domain information from cognitive signals. (b): The Double-Domain
(2D) Serial Architecture predominantly extracts both spatial and temporal
domain information, either in different orders or simultaneously. (c): The
Triple-Domain (3D) Serial Architecture sequentially extracts information from
the frequency domain, spatial domain, and temporal domain. (d): The Triple-
Domain Disentangled Architecture initially processes cognitive signals through
the frequency and spatial domains, resulting in separate frequency and spatial
features.

Cognitive Signal Decoding (CSD), a fundamental domain within Cognitive
Language Processing (CLP), assumes a pivotal role in the context of few-shot
learning [1], interpretable deep learning-based Natural Language Processing
(NLP) [2, 3, 4], and delving into the intricacies of language physiology in
the human brain, thus contributing to the field of neuro-prosthesis [5, 6].
CSD, particularly when coupled with neuro-imaging techniques such as
Electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI),
has emerged as an indispensable tool for researchers delving into cognitive
science. Among the neuro-imaging modalities, EEG stands out as one of the most
commonly employed methods in CLP due to its high temporal resolution.
Consequently, several deep learning techniques have surfaced as the primary
means of CSD, leading to substantial progress in this domain [7, 8, 9, 10].

EEG signals exhibit intricate characteristics across frequency, spatial, and
temporal domains, particularly in the context of CLP. The question of how to
effectively extract features from these multiple domains and construct
mechanisms for their integration require thorough examination. Currently,
three primary frameworks, as depicted in Fig. 1, have been established based
on the incorporation of information from different domains and fusion methods.
The first framework, referred to as the single-domain (1D) architecture [11,
12, 13], places a significant emphasis on the connectivity of cognitive
signals, predominantly extracting information from the spatial domain. The
second framework, known as the double-domain serial (2D) architecture [14, 8,
15, 16, 17, 9, 18], primarily extracts information from both the spatial and
temporal domains in varying orders [8, 15, 16, 17, 9], or simultaneously [14,
18]. The third framework, the triple-domain serial (3D) architecture,
sequentially extracts information from the frequency domain, spatial domain,
and temporal domain [7]. However, it is noteworthy that when it comes to
extracting information from both the frequency and spatial domains, or from
both the temporal and spatial domains, most models opt for a sequential
approach [7, 16, 19, 14, 17, 18, 20]. These methods may neglect the
observation that the spatial domain shares less relevance but greater
independence with the other two domains, as they are orthogonal in dimension.
In contrast, the frequency domain shares less independence but greater
relevance with the temporal domain, as they offer distinct perspectives on
time series information. Consequently, the sequential extraction of features
from different domains may disrupt the overall extraction process. Fig. 2
intuitively presents the feature distributions of EEGNet [7] under the
original serial framework and the disentangled framework. The frequency-
spatial features obtained by the vanilla EEGNet evidently fail to adequately
represent the task-specific frequency and spatial characteristics inherent in
the data. In contrast, the disentangleded approach effectively encapsulates
these aspects.

Figure 2: t-SNE projections of feature extracted by EEGNet [7] with different
strategies: (a) Serial Framework(Vanilla), (b) Disentangled Framework(Ours).
The dashed circles indicate the range of projected features. The visualization
details can be found in our open source code.

Convolutional Neural Networks (CNNs) have demonstrated notable advantages in
extracting intricate information. Several widely recognized CNN-based models
for Cognitive Signal Decoding (CSD) [7, 21, 11, 16, 19, 22, 14, 13, 17, 9, 18]
are dedicated to enhancing CNNs’ performance in the context of CSD. However,
human brain cognitive processes exhibit substantial contextual relevance and
generally have longer duration compared to other processes, such as Event-
Related Potential (ERP) or Error-Related Negativity potentials (ERN).
Simultaneously, for the sake of facilitating matrix operations, it is
customary to represent signals collected by sensors positioned in three-
dimensional space using two-dimensional multivariate time series. On one hand,
convolution operations, renowned for their local feature extraction
capabilities, encounter difficulties in capturing disrupted adjacency
relationships between nodes. On the other hand, even when nodes are physically
adjacent, convolution operations struggle to effectively capture functional
connections between non-adjacent nodes. Many researchers have sought to
enhance cognitive signal decoders by incorporating Transformers [1, 23, 24,
25, 12, 26], recognizing their proficiency in representing global and
contextual features, and their remarkable progress in NLP, Computer Vision
(CV), and Time Series (TS) domains. However, it is worth noting that most of
these methods simply superimpose Transformer modules onto existing cognitive
signal decoders, often overlooking the overfitting issue that arises from
cognitive signals with limited samples and a low signal-to-noise ratio (SNR),
a challenge stemming from the inherent complexities of Transformers.

In this paper, to address aforementioned issues, we propose D-FaST, a brain
cognitive signal decoder that incorporates in a Dsentangled Frequency-Spatial-
Temporal Attention(Fig. 1(d)). We extensively explore the application of
attention mechanisms in decoding temporal, spatial, and frequency domain
information, as well as various frameworks for integrating these three
domains. We conduct substantial experiments to validate our approach.

The contributions of this paper can be summarized as follows:

  * •

Designing a disentangled frequency-spatial-temporal structure for EEG
processing, which efficiently integrates features from the frequency spatial
and temporal domains and avoids mutual interference between orthogonal
domains.

  * •

Introducing an efficient decoding mechanism based on attention mechanisms for
frequency, spatial, and temporal domains to capture global dynamic and
function-connected feature more effectively, leading to improved EEG
information decoding.

  * •

Conducting extensive experiments on our self-constructed CLP dataset Mandarian
Natural Reading EEG dataset (MNRED), as well as Zurich Cognitive Language
Processing Corpus (ZuCo) and another two classic CSD datasets. The
experimental results demonstrate the effectiveness of our model and achieve
state-of-the-art performance.

The remainder of this paper is organized as follows: Section II presents the
related work. The proposed methodology is illustrated in Section III. The
performance of D-FaST and the visualization analysis are presented in Section
IV. Finally, Section V summarizes the paper.

##  II Related work

###  II-A Cognitive Language Processing

Cognitive Signal Decoding (CSD) primarily relied on traditional machine
learning techniques such as Support Vector Machines (SVM) [27] and Linear
Discriminant Analysis (LDA) [28]. However, with the demonstrated advantages of
CNNs and Recurrent Neural Networks (RNNs), numerous CSD algorithms based on
CNNs and RNNs, such as EEGNet [7], ConvNet [8], and ConvLSTM [29], have been
designed and continue to play a crucial role in various scenarios. As the
field of NLP and CV witnessed the ascension of transformer-based models,
several transformer-based CSD algorithms, such as STAGIN [26] and TTF-Former
[30], have rapidly emerged. Concurrently, multiple datasets have been created
to support CSD research [31, 32, 33]. For instance, the BraVL multimodal
matching dataset [34] combines brain, visual, and linguistic data, enabling
zero-shot decoding of novel visual categories based on recorded human brain
activities through multimodal learning. The ZuCo dataset [31] integrates EEG
and eye-tracking data, capturing participants’ reading of sentences in natural
conditions. In this paper, we introduce the first CLP dataset that employs
Chinese text as stimulus sources, named MNRED.

###  II-B Frequency feature extraction

The method for decoding brain cognitive signals primarily employs two
approaches for frequency feature extraction. One approach utilizes Time-
Frequency Representation (TFR) to express frequency domain information,
encompassing techniques such as the smooth pseudo-Wigner-Ville distribution
(SPWVD), short-time Fourier transform (STFT), continuous wavelet transform
(CWT), and others [19, 22, 35, 36]. The second method entails the extraction
of frequency information from EEG data through convolution operations. For
instance, EEGNet [7] employs convolutional kernels to extract features from
the frequency domain, with kernel sizes set at half the sampling frequency.
Nevertheless, these methods often oversimplify frequency domain features, and
their parameter configurations are constrained by human empirical knowledge,
thus limiting their efficacy in representing spectral information. Notably,
TimesNet [37] transforms 1D time series into a collection of 2D tensors based
on multiple periods, fully exploiting the multi-periodicity present in time
series data. However, applying such a transformation to cognitive signals
poses challenges due to their low signal-to-noise ratio and non-periodic
nature. In this paper, we introduce a novel approach for frequency feature
extraction, involving the use of multi-view attention.

###  II-C Spatial feature extraction

Besides the frequency-domain features, spatial characteristics also represent
another significant aspect of cognitive signals. Cognitive signals are
typically acquired from various brain regions using devices such as EEG caps,
inherently containing spatial information through the data represented by
distinct channels. These signals exhibit functional connectivity (FC) among
different brain regions, often represented as connectivity graphs to encode
the spatial correlations between EEG cap nodes or brain regions. BrainNetCNN
[11] leverages brain connectivity graphs as inputs and models the encoding of
cognitive states through convolutions applied to edge-to-edge, edge-to-node,
and node-to-graph connections. LMDA-Net [16] introduces a channel attention
mechanism to assess the significance of different EEG acquisition nodes in
encoding cognitive states. Nonetheless, the spatial information within the
acquired cognitive signals is inherently two-dimensional, and sometimes even
three-dimensional, making simple convolutions less effective for handling
complex tasks. Approaches such as graph-based node arrangement [21] mitigate
some of the limitations of convolutions by arranging nodes into a two-
dimensional layout based on spatial relationships. However, these approaches
tend to emphasize anatomical connections while neglecting the functional
connectivity (FC) of the brain. Models like BrainGNN [38], IBGNN [39], and
TARDGCN [40] employ Graph Neural Networks (GNN) to model the FC among brain
regions. LOGO [41] has also achieved success in multi-variate time series
prediction using GNN. Transformer-based approaches [42], exemplified by BNT
[12], encode global features of nodes within brain connectivity graphs and
subsequently employ orthogonal clustering methods to compress and extract
high-level features. However, they often primarily consider the static spatial
characteristics of brain cognition. In reality, the process of brain cognition
is dynamic, with historical states significantly influencing the current
cognitive state. Consequently, these approaches struggle to model the dynamic
nature of brain cognitive processes, leading to suboptimal utilization of
spatial domain information. In this paper, we introduce a novel dynamic
connectogram attention mechanism for the extraction of spatial features in a
more dynamic and context-aware manner.

###  II-D Temporal feature extraction

While the representation of cognitive signals from the orthogonal dimensions
of frequency and space domains is sufficiently comprehensive, it is
particularly important to investigate the evolution of cognition in the
temporal dimension, given that cognitive signals are quintessentially
multivariate time series. RNN models [43, 44, 45] excel in extracting features
from such time series data. ConvLSTM [29], which utilizes Long Short-Term
Memory (LSTM) [44] to capture dynamic contextual features from brain cognitive
signals, is another notable approach. Nonetheless, RNNs face challenges
related to parallel computation, leading to heightened computational
complexity and making them less suited for the analysis of brain cognitive
signals sampled at high rates.

BrainNet [46] introduces a self-supervised Bidirectional Contrast Predictive
Coding (BCPC) to pretrain a universal feature encoder for brain cognitive
signals, effectively addressing the issue of low data utilization stemming
from imbalanced EEG data labels. STAGIN [26] excels at extracting contextual
features from dynamic graphs of brain cognitive signals through the
bidirectional encoding capabilities of Transformer structures. However, this
approach necessitates a relatively extended EEG data sampling period, and the
inclusion of Transformer structures introduces computational complexity,
impacting the detailed feature extraction and analytical efficiency of brain
cognitive signals.

Numerous research endeavors have focused on enhancing the efficiency of
Transformers [47, 20, 48, 49]. These studies underscore the effectiveness of
attention-based feature extraction in the temporal domain, while acknowledging
the imperative need to manage computational costs. EEGNet-MSD [25], which
combines EEGNet [7] and Informer [20], offers a simple yet potent approach
with the potential to enhance cognitive signal decoding performance. EmoGT
[50] integrates Graph Convolutional Networks (GCN) with Transformer and
designs a Cross-modal Attention mechanism to establish connections between EEG
data and eye movements. In this paper, we introduce a novel approach: a local
temporal sliding attention mechanism designed for the extraction of temporal
features.

###  II-E Multidomain feature fusion

Existing research suggests that spectral, temporal, and spatial information
play complementary roles in the analysis of cognitive signals, particularly in
interactions between the spatial and spectral domains or the spatial and
temporal domains. Consequently, the prevailing approach is to analyze EEG
signals using multimodal features from multiple dimensions. This necessitates
an efficient feature fusion mechanism for the seamless integration of cross-
domain information. Most popular networks for brain cognitive information
analysis adopt a sequential structure in which features from different domains
are extracted and analyzed in stages. Notable examples include EEGNet [7],
STAGIN [26],MSFEnet [51], CDCN [52] and FBNetGen [15], among others. BrainNet
[46] has also developed a spatio-temporal information alternation fusion
mechanism based on the diffusion property of EEG. TTF-Former [30] incorporates
cross-attention to merge temporal and frequency features. However, as
previously mentioned, there exists a notable degree of independence between
spatial and frequency domain information, as well as between spatial and
temporal domains. The sequential processing within staged structures can lead
to mutual interference between different domain information during the
processing stages, thereby hindering the efficiency of feature extraction. In
this paper, we propose a novel disentangled frequency-spatial-temporal
architecture aimed at seamlessly fusing features from all three domains.

##  III Methodology

Figure 3: The overarching architecture of D-FaST. The dashed boxes delineate
detailed descriptions of the corresponding modules. The three diagrams on the
left provide a comprehensive breakdown of the neural networks within the MVA,
DCA, and LSTA modules. The rightmost section illustrates the interconnections
between these three modules.

###  III-A Problem Definition

The research goal of CSD is training a brain cognitive decoding network
$\mathcal{F}:\mathcal{X}\rightarrow\mathcal{H}$ in which the output
$\mathcal{H}\in\mathbb{R}^{d}$ is a coded representation of cognitive signal
$\mathcal{X}\in\mathbb{R}^{N\times T}$.

Given a set of signal acquisition nodes
$\mathcal{V}=\\{v_{1},v_{2},\cdots,v_{N}\\}$ distributed in the brain area
space, where $N=\lvert\mathcal{V}\rvert$ denotes the number of nodes
represented as sensor channels in EEG data, each node samples EEG data at a
sampling frequency $f$. The collected brain signal data is expressed as
$\mathcal{X}\in\mathbb{R}^{N\times T}$, where $T$ is the number of sampling
time points. It is assumed that the labels of brain cognitive tasks are
represented as cognitive labels $\mathcal{Y}\in\left\\{1,\cdots,C\right\\}$,
and each brain signal sample in the sample set $\\{\mathcal{X}\\}$ corresponds
to a label. A Multi-Layer Perceptron (MLP) transforms $\hat{\mathcal{H}}$ to
logits, where a prediction $\hat{\mathcal{Y}}\in\left\\{1,\cdots,C\right\\}$
can be acquired.

Algorithm 1 Pseudo-code of D-FaST.

Require: Initialized parameters of D-FaST model $\Theta$.

Require: Data set of cognitive signals and corresponding labels
$\\{\mathcal{X},\mathcal{Y}\\}$

D-FaST$(\mathcal{X})$

$\mathcal{Z}_{F}\leftarrow MVA(\mathcal{X})$ $\triangleright$Extract frequency
feature

$\mathcal{Z}_{S}\leftarrow DCA(\mathcal{X})$ $\triangleright$Extract spatial
feature

$\mathcal{Z}_{FS}\leftarrow FUSION(\mathcal{Z}_{F},\mathcal{Z}_{S})^{\top}$
$\triangleright$Fuse

$\mathcal{Z}_{T}\leftarrow LSTA(\mathcal{Z}_{FS})$ $\triangleright$Extract
temporal feature

$\hat{\mathcal{H}}\leftarrow AGGREGATE(\mathcal{Z}_{T})$
$\triangleright$Aggregate

return $\hat{\mathcal{H}}$

TRAIN$(\\{\mathcal{X},\mathcal{Y}\\})$

$\textbf{for }\mathcal{X},\mathcal{Y}\in\\{\mathcal{X},\mathcal{Y}\\}\textbf{
do }$

$\hat{\mathcal{H}}\leftarrow{\textsc{D-FaST}}(\mathcal{X})$
$\triangleright$Forward

$logits\leftarrow MLP(\hat{\mathcal{H}})$ $\triangleright$Classify

$loss\leftarrow CROSS\\_ENTROPY(logits,\mathcal{Y})$

$\triangleright$Calculate loss

$loss.backward()$ $\triangleright$Back-propagate

update($\Theta$) $\triangleright$Update parameters using Adam

end for

One sample data $\mathcal{X}\in\mathbb{R}^{N\times T}$ can be divided into $h$
segments along the temporal axis, with each segment referred to as a time
window corresponding to $w$ sampling time points. Taking the $t^{th}$ time
window as an example, a connectogram can be constructed as
$\mathcal{G}^{t}=\left\\{\mathcal{V},\mathcal{E}^{t}\right\\}$.
$\mathcal{E}^{t}$ Represents the connection relationship between the brain
regions of each sampling node in the $t^{th}$ time window. Such connection
relationship is defined using a triplet
$\left(v_{i},e_{v_{i}v_{j}},v_{j}\right),v_{i},v_{j}\in\mathcal{V}$ where the
weights of the edges are $e_{v_{i}v_{j}}\in\left[0,1\right]$, and
$e_{v_{i}v_{j}}=e_{v_{j}v_{i}}$, indicating that the connection graph
described here is an undirected graph. When $e_{v_{i}v_{j}}=0$, it indicates
no connection between the nodes. Finally, a set of $h$ brain connections
$\mathcal{G}=\left\\{\mathcal{G}^{t}\mid t=1,\cdots,h\right\\}$ is formed.

###  III-B Overview of D-FaST

In this paper, we introduce a novel network called D-FaST, which aims to
enhance the utilization of frequency, spatial, and temporal domains while
improving the effectiveness of structure. Fig. 3 provides an intuitive
overview of D-FaST, while Algorithm 1 describes its overall process using
pseudo-code. D-FaST trains a cognitive signal decoder by applying Multi-View
Attention (MVA), Dynamic Connectogram Attention (DCA) and Local Temporal
Sliding Attention (LTSA). Cognitive signals $\mathcal{X}$ are processed
through MVA and DCA, respectively, yielding frequency feature
$\mathcal{Z}_{F}$ and spatial feature $\mathcal{Z}_{S}$. LSTA extracts
temporal features $\mathcal{Z}_{T}$ from the fused features of
$\mathcal{Z}_{F}$ and $\mathcal{Z}_{S}$. $\mathcal{Z}_{T}$ are then aggregated
to obtain decoded cognitive state $\hat{\mathcal{H}}$. D-FaST avoids the
mutual interference caused by feature differences between frequency and
spatial features by extracting them in a disentangled way.

###  III-C Frequency-Spatial-Temporal Attention

####  III-C1 Multi-View Attention (MVA) for Frequency Feature Extraction

Compared to previous methods that relied on single, experiential frequency-
domain feature extraction [7], this module focuses on the extraction of non-
empirical multi-frequency features and directs the model’s attention towards
significant frequencies. The feature extraction of brain cognitive signals in
the frequency domain is performed using the
$\text{MVA}:\mathcal{X}\rightarrow\mathcal{Z}_{F}\in\mathbb{R}^{k\times
N\times T}$, where $k$ denotes the target number of frequency domain features.
As illustrated in Fig. 4, MVA consists of two components: a multi-view
convolutional structure and frequency attention. The detailed structure of MVA
is depicted in Fig. 3(a).

Multi-View convolution:

Figure 4: Frequency domain information coding process of multi-view attention.

[53] introduced a variety of modular aggregation structures to enhance feature
extraction in a disentangled manner. Similarly, multi-view convolution
transforms $\mathcal{X}\in\mathbb{R}^{N\times T}$ to
$\mathcal{Z}_{MF}\in\mathbb{R}^{k\times N\times T}$. Specifically, multi-view
convolution consists of a superposition of two multi-scales InceptionBlocks:

$\mathcal{Z}_{MF}=\text{InceptionBlock2}\left(\sigma\text{InceptionBlock1}\left(\mathcal{X}\right)\right)$
(1)

where
$\text{InceptionBlock1}:\mathcal{X}\rightarrow\mathcal{Z}_{MF}^{{}^{\prime}}\in\mathbb{R}^{\frac{k}{2}\times
N\times T}$ consists of $k/4$ groups of evenly spaced convolution kernels
ranging from $\left(1,1\right)$ to $\left(\frac{f}{2},1\right)$ with an
interval of $a=\left\lfloor\frac{2f}{k}\right\rfloor$. Each group contains two
convolution kernels, resulting in the extraction of $\frac{k}{2}$ frequency
features in total. $\sigma$ denotes the activation function.

Additionally,
$\text{InceptionBlock2}:\mathcal{Z}_{MF}^{{}^{\prime}}\rightarrow\mathcal{Z}_{MF}\in\mathbb{R}^{k\times
N\times T}$ comprises $k/4$ groups of evenly spaced convolution kernels
ranging from $\left(\frac{f}{2},1\right)$ to $\left(1,1\right)$ with an
interval of $a=\left\lfloor\frac{2f}{k}\right\rfloor$. Each group convolution
includes 4 convolution kernels to extract 4 frequency-domain features from two
inputted frequency-domain features. Modifying the convolution kernel sizes
enhances the richness and hierarchy of the frequency domain feature extraction
process.

Multi-View attention: To further enhance the quality of multi-frequency
features, Multi-View Attention (MVA) assigns attention weights to the
extracted frequency features. This approach facilitates a scientific
investigation into the significance of various frequency information within
brain cognitive signals. Existing models such as SENet [54](Squeeze-and-
Excitation), ECA-Net [55](Efficient Channel Attention), and LMDA-Net [16]
endeavor to elucidate the importance of different channel information through
channel attention mechanisms. Similarly, we propose the design of a MVA,
denoted as $\text{Attention:}\mathcal{Z}_{MF}\rightarrow\mathcal{Z}_{F}$,
which operates as follows:

$\displaystyle\mathcal{Z}_{F}=$
$\displaystyle\text{MVA}\left(\mathcal{X}\right)$ (2) $\displaystyle=$
$\displaystyle\text{DWConv}\left(\text{Attention}\left(\mathcal{Z}_{MF}\right)\right)$
$\displaystyle=$
$\displaystyle\text{DWConv}\left(\mathcal{A}_{F}\mathcal{Z}_{MF}\right)$
$\displaystyle\mathcal{A}_{F}=$
$\displaystyle\text{SE}\left(\mathcal{Z}_{MF}\right)$ $\displaystyle=$
$\displaystyle\text{Sigmoid}\left(\text{Linear}\left(\text{AvgPool2d}\left(\mathcal{Z}_{MF}\right)\right)\right)$

where $\mathcal{A}_{F}\in\left[0,1\right]^{k}$ represents $k$ attention
weights of frequency features of $\mathcal{Z}_{MF}$, and is obtained by
applying one-dimensional convolution followed by two-dimensional average
pooling with $\mathcal{Z}_{MF}$. The attention weights are then further
processed using the sigmoid function. $\text{DWConv}\left(\cdot\right)$ is
used to adjust the output dimension of $\mathcal{Z}_{F}$ in the spatial
domain. As mentioned above, the multi-view convolution kernels in the
InceptionBlock can be adjusted to capture different frequency ranges.
Furthermore, the convolution method is used to obtain local attention, which
reduces the computational cost and pays more attention to the relationship
between adjacent frequencies. In fact, in order to reduce the training
complexity of the model and avoid the overfitting of the model on the data
noise, we also add a pooling layer at the end which is omitted from equation
(2) for the sake of simplify. Similarly, the subsequent pooling layer is
omitted.

####  III-C2 Dynamic Connectogram Attention (DCA) for Spatial Feature
Extraction

Compared to previous static spatial feature representation methods that
focused on the spatial characteristics of physical nodes [11, 16, 12], this
module focuses on the connectivity patterns between virtual regions of
interest and their dynamic characterization. The feature extraction of brain
cognitive signals in the spatial domain is performed using the
$\text{DCA:}\mathcal{X}\rightarrow\mathcal{Z}_{S}\in\mathbb{R}^{k\times
N^{\prime}\times T}$, where $N^{\prime}$ is the number of virtual subspace
nodes. DCA consists of two parts: dynamic connectogram and multi-head dot-
product attention, as shown in Fig. 3(b).

Figure 5: Dynamic connectogram and dynamic connection matrix of each window.

Dynamic Connectogram: Brain cognitive signals exhibit a natural dynamic graph
structure, and the key to various brain functions lies in the connection and
communication between different regions [56]. In order to fuse with features
of other dimensions, DCA first uses one-dimensional convolution to project
$\mathcal{X}\in\mathbb{R}^{N\times T}$ to
$\mathcal{Z}_{S}^{{}^{\prime}}\in\mathbb{R}^{k\times N^{\prime}\times T}$,
$\mathcal{Z}_{S}^{{}^{\prime}}=\left\\{\mathcal{Z}_{S}^{{}^{\prime}t}\mid
t=1,\cdots,h\right\\}$, where
$\mathcal{Z}_{S}^{{}^{\prime}t}\in\mathbb{R}^{k\times N^{\prime}\times w}$,
$w$ is the size of the sliding window; Then DCA calculated the set of dynamic
connection matrices $\mathcal{A}_{S}=\\{\mathcal{A}_{S}^{t}\mid
t=1,\cdots,h\\}$ corresponding to the set of dynamic brain connection graphs
$\mathcal{G}=\left\\{\mathcal{G}^{t}\mid t=1,\cdots,h\right\\}$ with
$\mathcal{Z}_{S}^{{}^{\prime}}$, as shown in Fig. 5, where
$\mathcal{A}_{S}^{t}\in\left[0,1\right]^{k\times N\times N}$ is the connection
matrix corresponding to the $t^{th}$ window, calculated as follows:

$\mathcal{A}_{S}^{t}=\text{Softmax}\left(\text{Sparse}\left(\frac{{Q_{S}^{t}K_{S}^{t}}^{\top}}{\sqrt{T}},\tau\right)\right)$
(3)

where $\text{DWConv}\left(\cdot\right)$ transforms
$\mathcal{Z}_{S}^{{}^{\prime}t}$ to subgraph query matrix
$Q^{t}\in\mathbb{R}^{k\times N^{\prime}\times w}$, and
$\text{InceptionBlock}\left(\cdot\right)$ transforms
$\mathcal{Z}_{S}^{{}^{\prime}t}$ to key matrix $K^{t}\in\mathbb{R}^{k\times
N\times w}$ . The convolution kernel size used in
$\text{DWConv}\left(\cdot\right)$ is $\left(N,1\right)$; $\tau$ denotes the
spatial sparse coefficient. The top $\tau\%$ of the input attention score
matrix is retained by $\text{Sparse}\left(\cdot,\tau\right)$ and the rest
$1-\tau\%$ is assigned to be $-\infty$. After activation function
$\text{Soft}\max$, the edge with insignificant connection is removed. a
scaling operation $1/\sqrt{T}$ is used in the equation to prevent the gradient
from disappearing [57].

Spatial Context Attention: Unlike the multi-head dot-product attention in
Transformer [57] that operates on the embeddings dimension, DCA performs
Spatial Context Attention on the temporal dimension, where dynamic graph
features that corresponds to the aforementioned number of Windows are
extracted. The specific calculation process is as follows:

$\mathcal{Z}_{S}=\mathcal{A}_{S}V_{S}=\sum_{t=1}^{h}{\mathcal{A}_{S}^{t}V_{S}^{t}}$
(4)

Similar to $Q_{S}^{t}$, $\text{InceptionBlock}\left(\cdot\right)$ transforms
$\mathcal{Z}_{S}^{{}^{\prime}t}$ to a value matrix
$V_{S}^{t}\in\mathbb{R}^{k\times N\times w}$; Summation is used here to
aggregate the dynamic information of the sub-graph corresponding to $h$
windows.

Virtual Regions of Interest: In the modeling process mentioned above, we
noticed that the number of nodes $N^{\prime}$ in the subgraph query matrix
$Q^{t}\in\mathbb{R}^{k\times N^{\prime}\times w}$ is defined as the number of
nodes in the subspace. When $N^{\prime}=N$, we can establish the corresponding
relationship between the source node and the target node. When $N^{\prime}\neq
N$($N^{\prime}>N$or$N^{\prime}<N$), such a correspondence cannot be
established. In this case, we can understand $N^{\prime}$ by the concept of a
virtual brain area or virtual node, where $N^{\prime}$ corresponds to nodes of
virtual abstract meaning. The virtual nodes compute the attention-weighted
sums of multiple source nodes and can be seen as representing certain
categories (in terms of spatial features as connections) of the source nodes.
Therefore, they can also be referred to as virtual regions of interest.

Figure 6: Calculation process of temporal attention weight.

####  III-C3 Local Temporal Sliding Attention (LTSA)

Compared to previous time feature extraction networks with large parameter
sizes that disregarded considerations of temporal and spatial complexity [29,
26], which often led to overfitting in small sample scenarios, this module
focuses on utilizing lightweight local networks and attention mechanisms to
achieve equivalent outcomes. The feature extraction of brain cognitive signals
in the temporal domain is performed using the
$\text{LTSA:}\mathcal{Z}_{FS}\in\mathbb{R}^{k\times T\times
N^{\prime}}\rightarrow\mathcal{Z}_{T}\in\mathbb{R}^{k\times T\times
N^{\prime}}$, where
$\mathcal{Z}_{FS}=\text{Fusion}\left(\mathcal{Z}_{F},\mathcal{Z}_{S}\right)^{\top}$
is the fusion of $\mathcal{Z}_{F}$ and $\mathcal{Z}_{S}$ in the frequency
domain and space, which will be introduced in the next subsection. The LTSA
consists of two parts: CNNFormer and local slide-window attention, as depicted
in Fig. 3(c).

CNNFormer: CNNFormer is a Transformer-like model designed for brain cognitive
signals. Similar to DCA, LTSA still utilizes dot-product attention in
Transformer [57]. However, the number of samples of brain cognitive signal
data is relatively small. In order to reduce the number of parameters in the
network and prevent overfitting, LTSA replaces the method of obtaining query,
key and value matrix in Transformer with linear to convolutional operation.
Additionally, using convolution allows the preservation of local timing
information, whereas using full connections would somewhat disrupt such timing
information. The specific equation is calculated as follows:

$\displaystyle\mathcal{Z}_{T}=$ $\displaystyle\mathcal{A}_{T}V_{T}$ (5)
$\displaystyle\mathcal{A}_{T}=$
$\displaystyle\text{Softmax}\left(\frac{Q_{T}K_{T}^{\top}}{\sqrt{N^{\prime}}}+\text{Mask}\right)$

where $\mathcal{A}_{T}\in\mathbb{R}^{k\times T\times T}$ represents the
temporal attention score; $\text{CNN}\left(\cdot\right)$ transforms
$\mathcal{Z}_{FS}$ respectively to $Q_{T},K_{T},V_{T}\in\mathbb{R}^{k\times
T\times N^{\prime}}$; $\text{Mask}\in\mathbb{R}^{T\times T}$ denotes the
attention mask, the value of the mask part is $-\infty$, and the remaining
unmasked part is a diagonal sliding window of size $w$ with a value of $0$, as
shown in Fig. 6. Furthermore, a scaling operation $1/\sqrt{N^{\prime}}$ is
employed to prevent gradient vanishing.

LTSA: Despite the relatively small number of samples in brain cognitive signal
data, the high sampling frequency in EEG and the long sampling time in fMRI
often result in larger samples. If the attention field is not restricted, the
network is likely to learn meaningless long-distance contextual semantics
while neglecting information at close range. To address this issue, LTSA
utilizes local sliding window attention as a means of alleviation. Longformer
[47] proposed several novel and efficient non-global attention mask
mechanisms, achieving favorable outcomes. In this study, we adopt local
sliding window attention and present its corresponding mask matrix, as
depicted in Figure 6.

###  III-D Disentangled Frequency-Spatial Feature Extraction

The existing EEG data processing models can be categorized into three types
based on the type and amount of domain information used: single-domain
structure, double-domain serial structure, and triple-domain serial structure,
as illustrated in Fig. 1 (a), (b), and (c), respectively. The single-domain
architecture generally only extracts spatial domain information from cognitive
signals, with typical models such as BrainNetCNN [11] and BNT [12]. The
double-domain serial structure primarily extracts both spatial and temporal
domain information in different orders. Representative models that extract
spatial information first and then temporal information include FBNetGen [15]
and DeepConvNet [8]. Models that extract temporal information first, followed
by spatial information, include LAMD-Net [16], STAGIN [26], and ShallowConvNet
[8]. The triple-domain serial structure extracts frequency domain information,
spatial domain information, and temporal domain information sequentially, with
EEGNet [7] being the most representative model.

However, these structures fail to capture the differences in relationships
between different domains. The use of serial structures for feature extraction
across different domains often leads to interference, which affects the
effectiveness of feature extraction. To address this, we propose a
disentangled frequency-spatial structure, as illustrated in Fig. 1. In this
disentangled structure, the frequency domain feature module and the spatial
feature module extract features in the frequency and spatial dimensions,
respectively, from the cognitive signals. The results are then fused using the
temporal module, followed by further aggregation to obtain the cognitive state
coding. This can be abstractly expressed as:

$\displaystyle\hat{\mathcal{H}}=$
$\displaystyle\text{Aggregate}\left(\mathcal{Z}_{T}\right)$ (6)
$\displaystyle\mathcal{Z}_{T}=$
$\displaystyle\text{Temporal}\left(\mathcal{Z}_{FS}\right)$
$\displaystyle\mathcal{Z}_{FS}=$
$\displaystyle\text{Fusion}\left(\text{Frequency}\left(\mathcal{X}\right),\text{Spatial}\left(\mathcal{X}\right)\right)^{\top}$

In our approach, the fusion of frequency domain and spatial features is
executed in a parallel fashion using $\text{Fusion}\left(\cdot,\cdot\right)$.
This fusion operation can be implemented as either concatenation
($\text{Concat}\left(\cdot,\cdot\right)$) or addition
($\text{Add}\left(\cdot,\cdot\right)$). The function
$\text{Aggregate}\left(\cdot\right)$ aggregates the tensor $\mathcal{Z}_{T}$
into a one-dimensional representation. This aggregation can be realized
through various techniques such as flattening
($\text{Flatten}\left(\cdot\right)$), mean pooling
($\text{Mean}\left(\cdot\right)$), or employing an attention mechanism
($\text{Attention}\left(\cdot\right)$). The functions
$\text{Temporal}\left(\cdot\right)$, $\text{Temporal}\left(\cdot\right)$, and
$\text{Frequency}\left(\cdot\right)$ correspond to
$\text{LTSA}\left(\cdot\right)$, $\text{MVA}\left(\cdot\right)$, and
$\text{DCA}\left(\cdot\right)$, respectively.

##  IV Experimental Evaluation

In this section, we present an evaluation of the effectiveness of our proposed
D-FaST model through a comprehensive series of experiments. Our study has been
meticulously designed to address the following research questions:

Q1. How does D-FaST perform in comparison to state-of-the-art models featuring
various mechanisms and frameworks when applied to CLP dataset?

Q2. How effectively does the model generalize to previous widely-used
datasets?

Q3. What is the performance of our proposed components, namely, MVA, DCA,
LSTA, and the disentangled framework?

Q4. How do hyperparameters influence the performance?

Q5. To what extent does the trained D-FaST model exhibit interpretability, and
how consistent is it with existing knowledge in the field of neuroscience?

###  IV-A Datasets and Preprocessing

We have selected several brain cognitive model datasets that exhibit strong
cognitive task correlation. The characteristics of the datasets used in our
experiments are summarized in TABLE I.

#### MNRED

MNRED dataset contains 11,624 EEG signals from 30 native speakers of Mandarin
with a gender distribution of 18 males and 12 females, ranged in age from 18
to 25 years. MNRED dataset is a 2-class classification task, and the stimulus
materials encompass two categories: target semantic stimuli and non-target
semantic stimuli, both in the form of a news headline or a brief sentence.
Participants were required to read each stimulus within a 2-second timeframe.
EEG data were collected at a sampling rate of 1100 Hz using a 32-channel
NeuSen W series wireless EEG acquisition system. Data preprocessing involved
referencing to average, resampling the original data to 128 Hz, performing
band-pass filtering from 0.1 to 80 Hz, performing independent component
analysis (ICA) to remove eye blink and movement artifacts.

TABLE I: Data set description. Dataset | MNRED | ZuCo | BCIC IV-2a | BCIC IV-2b
---|---|---|---|---
Size | 11624 | 4478 | 5184 | 6520
Dimension | $30\times 440$ |  $104\times\bigstar$111$\bigstar$: The sampling lengths in ZuCo are inconsistent and exhibit a large variance, which significantly impacts the data quality when either truncating to a specific length or padding the data. | $22\times 577$ | $3\times 513$
Sampling $f$ | 1100Hz | 500Hz | 250Hz | 250Hz
Bandpass filter | [0.5,80] | [0.5,100] | [4,38] | [4,38]
Subjects | 10 | 12 | 9 | 9
Classes | 2 | 9 | 4 | 2
Classes rate | 3:7 | 1 | 1 | 1
Resampling $f$ | 128Hz

#### ZuCo

The ZuCo dataset [31] contains eye-tracking and EEG data from 12 participants,
all native speakers of English, who performed natural reading and relation
extraction tasks on 300 and 407 English sentences from the Wikipedia corpus
[58], as well as sentiment reading on 400 samples from the Stanford Sentiment
Treebank (SST). We choose the Task-Specific Reading (TSR) task and select EEG
signals corresponding to sentences of 10-20 words each. TSR is a ten-class
classification task where participants were instructed to attend to a
particular type of relation in sentences, including award, education,
employer, founder, job title, nationality, political affiliation, visited and
wife.

#### BCIC IV-2A

The BCI Competition IV Dataset 2A (BCIC IV-2A) [59] is a publicly accessible
dataset that captures EEG data from 9 subjects participating in motor
imagination tasks encompassing four distinct categories: left hand, right
hand, foot, and tongue. Data preprocessing procedures involve an initial step
of referencing the original data to 128Hz, following the protocol outlined in
reference [7]. Subsequent steps included band-pass filtering in the frequency
range of 4 to 38Hz, followed by a normalization [60] and European alignment
[61]. For model training and testing, two rounds of data were utilized, each
comprising approximately 288 records. For each record, the temporal segment
following the cue occurrence was extracted, in line with the guidelines
presented in references [7, 8, 14, 62].

#### BCIC IV-2B

The BCI Competition IV Dataset 2B (BCIC IV-2B) [59] comprises EEG data
obtained from 9 subjects participating in two distinct categories of motor
imagination tasks involving the left hand and right hand. The data collection
procedure and filtering techniques applied are consistent with those employed
in BCIC IV-2A. As in the case of BCIC IV-2A, the time segment following the
cue occurrence was extracted from each record [16]. Subsequently, the data
from the 5 rounds for each subject were merged.

TABLE II: Compare experimental results under cross-subject experimental settings on MNRED. The optimal results in the table are highlighted in bold, while the suboptimal results are indicated with an underline. This formatting approach is consistently applied in subsequent tables. Model | Venue | Type | MNRED
---|---|---|---
Accuracy (%) |  AUROC(%) |  Sensitivity (%) |  Specificity (%)
BrainNetCNN [11] | [NeuroImage’17] | 1D | 70.79±0.73 | 59.46±0.68 | 9.60±0.21 | 97.17±1.00
BNT [12] | [NeurIPS’22] | 70.76±0.73 | 61.63±0.69 | 11.45±0.27 | 96.58±1.02
TACNet [14] | [UbiComp’21] | 74.10±0.78 | 73.76±0.82 | 51.45±0.70 | 83.93±0.90
RACNN [13] | [IJCAI’21] | 75.92±0.80 | 57.74±0.64 | 14.39±0.24 | 93.81±0.98
DeepConvNet [8] | [HBM’17] | 2D-Serial | 74.52±0.79 | 78.00±0.85 | 69.61±0.79 | 76.68±0.83
ShallowConvNet [8] | [HBM’17] | 74.19±0.79 | 71.64±0.82 | 43.30±0.58 | 87.61±0.92
FBNetGen [15] | [MIDL’22] | 71.83±0.74 | 65.45±0.72 | 17.58±0.30 | 95.42±1.00
LMDA-Net [16] | [NeuroImage’23] | 76.00±0.82 | 78.12±0.88 | 64.51±0.84 | 80.98±0.88
EEG-ChannelNet [17] | [TPAMI’21] | 74.19±0.78 | 73.20±0.81 | 48.25±0.63 | 85.47±0.90
TCACNet [18] | [IPM’22] | 75.92±0.80 | 74.32±0.84 | 47.32±0.66 | 88.31±0.93
EEGNet [7] | [J Neural Eng’18] | 3D-Serial | 76.06±0.80 | 77.94±0.84 | 57.58±0.73 | 84.13±0.90
D-FaST | [Ours] | 3D-Disentangled | 78.72±0.82 | 81.51±0.85 | 62.20±0.71 | 85.98±0.90
TABLE III: Compare experimental results under cross-subject experimental settings. Number of baseline models is limited due to the unequal length sampling of the dataset. Model | Venue | ZuCo
---|---|---
Accuracy (%) |  AUROC(%)
FBNetGen [15] | [MIDL’22] | 76.82±0.80 | 85.94±1.13
BrainNetCNN [11] | [NeuroImage’17] | 76.64±0.78 | 86.13±1.09
Graph Transformer [42] | [AAAI’21] | 77.66±0.81 | 92.99±0.95
BNT [12] | [NeurIPS’22] | 77.82±0.81 | 92.77±0.95
D-FaST | [Ours] | 78.35±0.79 | 93.19±0.94

###  IV-B Experimental details and evaluation

The experiment is carried out on a working platform configured with four
NVIDIA GeFroce 3090Ti GPUs, and Pytorch is used as the neural network
framework. Firstly, the brain cognitive network is randomly initialized and
then trained end-to-end in a supervised way based on cross entropy loss.

#### Baselines

Several models are meticulously selected for comparative analysis, including
BrainNetCNN [11], BNT [12], DeepConvNet [8], ShallowConvNet, FBNetGen [15],
LMDA-Net [16], EEGNet [7], TACNet [14], RACNN [13], EEG-ChannelNet [17],
SBLEST [9], and TCACNet [18]. It is important to note that the signal
collection length of each sample in the ZuCo dataset is not consistent and
exhibits a highly random distribution. Many models are incapable of handling
variable-length data; therefore, we are only able to evaluate this dataset
using FBNetGen [15], BrainNetCNN [11], Graph Transformer [42], and BNT [12].

#### Evaluation Metrics

In the context of a binary task, as exemplified by MNRED, our evaluation
function encompasses four key indicators: Accuracy, AUROC (Area Under the
Receiver Operating Characteristic curve), Sensitivity, and Specificity. For
multi-class classification tasks, typified by BCIC IV-2A, we employ four
distinct evaluation measures as our evaluation function, namely Accuracy and
the area under the receiver operating characteristic curve (AUROC). The multi-
class AUROC, in particular, adopts a one-to-one approach to systematically
traverse and average all feasible combinations of classes.

#### Cross-Subject Setting

We conduct leave-one-subject-out cross-validation, and finally reported the
mean and standard deviation of experimental performance indicators of all
subjects. We also carry out the cross-subject experiment with 5-fold cross-
validation using stratified sampling strategy, and the relevant results are in
Appendix. -D.

#### Within-Subject Setting

We perform separate 5-fold cross-validation for each subject, selecting the
best value from each fold for evaluation. The mean performance indicators
across the five rounds of experiments are calculated for each subject, and the
average and variance are reported for the results of the nine subjects.

TABLE IV: D-FaST hyperparameter setting on different datasets. Hyper-Parameter | MNRED | ZuCo | BCIC IV-2A | BCIC IV-2B
---|---|---|---|---
$w$ | 16 | 16 | 32 | 3
$\tau$ | 0.6 | 1 | 0.6 | 1
$N^{\prime}$ | 30 | 104 | 22 | 1
mini batch size | 16 | 1 | 32 | 32
epochs | 200 | 100 | 200 | 200
Dropout | 0.1 | 0.5 | 0.5 | 0.5
learning rate | 0.0001$\to$0.00001 | 0.001$\to$0.00001
$k$ | 64
$h$ | 4
weight decay [63] | 0.0001
activation | GeLU
normalization | BatchNorm
schedule | Cosine [64]
optimizer | Adam [65]
TABLE V: Compare experimental results under cross-subject experimental settings. Model | Venue | Type | BCIC IV-2A |  | BCIC IV-2B
---|---|---|---|---|---
Accuracy (%) |  AUROC (%) |  |  Accuracy (%) |  AUROC(%) |  Sensitivity (%) |  Specificity (%)
BrainNetCNN [11] | [NeuroImage’17] | 1D | 35.11±0.41 | 63.01±0.70 |  | - | - | - | -
BNT [12] | [NeurIPS’22] | 33.83±0.41 | 60.59±0.70 |  | - | - | - | -
TACNet [14] | [UbiComp’21] | 50.33±0.66 | 72.35±0.86 |  | 74.05±0.84 | 81.00±0.93 | 74.23±0.92 | 73.87±0.91
RACNN [13] | [IJCAI’21] | 38.39±0.45 | 63.18±0.71 |  | 72.54±0.83 | 78.44±0.91 | 69.04±0.80 | 76.04±0.88
DeepConvNet [8] | [HBM’17] | 2D-Serial | 54.73±0.66 | 79.17±0.87 |  | 74.61±0.83 | 81.83±0.92 | 76.10±0.86 | 73.12±0.86
ShallowConvNet [8] | [HBM’17] | 50.42±0.65 | 73.72±0.87 |  | 69.42±0.84 | 73.41±0.93 | 61.84±0.79 | 77.00±0.93
FBNetGen [15] | [MIDL’22] | 36.36±0.45 | 64.27±0.73 |  | 53.41±0.60 | 53.89±0.62 | 64.01±1.06 | 42.81±0.86
LMDA-Net [16] | [NeuroImage’23] | 52.24±0.67 | 74.10±0.87 |  | 74.81±0.83 | 82.27±0.93 | 77.80±0.85 | 71.82±0.87
EEG-ChannelNet [17] | [TPAMI’21] | 47.74±0.61 | 71.95±0.84 |  | 70.85±0.79 | 78.71±0.90 | 68.56±0.86 | 73.13±0.89
SBLEST [9] | [TPAMI’23] | - | - |  | 67.68±0.09 | 76.58±0.13 | 68.70±0.13 | 66.57±0.22
TCACNet [18] | [IPM’22] | 51.33±0.66 | 73.22±0.87 |  | 73.81±0.83 | 80.12±0.92 | 74.57±0.87 | 73.05±0.87
EEGNet [7] | [J Neural Eng’18] | 3D-Serial | 53.59±0.72 | 74.62±0.90 |  | 73.17±0.82 | 81.79±0.95 | 70.21±0.90 | 76.14±0.91
D-FaST | [Ours] | 3D-Disentangled | 54.96+0.71 | 74.48±0.88 |  | 76.81±0.77 | 83.99±0.84 | 73.89±0.74 | 79.72±0.80
TABLE VI: Compare experimental results under within-subject experimental setting. Model | Venue | Type | BCIC IV-2A |  | BCIC IV-2B
---|---|---|---|---|---
Accuracy (%) |  AUROC(%) |  |  Accuracy (%) |  AUROC(%) |  Sensitivity (%) |  Specificity (%)
BrainNetCNN [11] | [NeuroImage’17] | 1D | 62.52±11.64 | 80.66±9.35 |  | 61.55±5.95 | 62.15±7.56 | 61.09±8.57 | 62.01±12.5
BNT [12] | [NeurIPS’22] | 64.91±13.19 | 82.07±10.38 |  | 60.27±7.10 | 62.11±9.22 | 56.39±15.2 | 64.15±12.3
TACNet [14] | [UbiComp’21] | 74.62±16.28 | 88.18±10.90 |  | 81.69±0.96 | 85.89±1.03 | 79.68±0.95 | 83.71±1.00
RACNN [13] | [IJCAI’21] | - | - |  | 68.45±14.0 | 70.20±16.8 | 71.40±13.3 | 65.50±19.7
DeepConvNet [8] | [HBM’17] | 2D-Serial | 72.24±14.47 | 88.10±10.46 |  | 80.45±11.9 | 85.92±12.3 | 82.26±9.26 | 78.64±15.8
ShallowConvNet [8] | [HBM’17] | 81.69±12.89 | 93.24±7.12 |  | 79.03±14.3 | 83.64±16.6 | 75.20±18.2 | 82.87±13.7
FBNetGen [15] | [MIDL’22] | 72.78±15.93 | 87.44±10.52 |  | 71.27±13.7 | 74.04±17.3 | 71.73±14.3 | 70.82±14.7
LMDA-Net [16] | [NeuroImage’23] | 75.29±17.46 | 88.91±11.44 |  | 81.23±13.4 | 86.33±15.1 | 77.72±17.4 | 84.74±14.1
EEG-ChannelNet [17] | [TPAMI’21] | - | - |  | 74.39±0.89 | 80.75±0.97 | 70.15±0.92 | 78.63±0.99
SBLEST [9] | [TPAMI’23] | - | - |  | 76.45±13.9 | 84.43±15.8 | 74.57±14.2 | 78.57±20.7
TCACNet [18] | [IPM’22] | 75.20±15.59 | 88.84±10.30 |  | 81.21±14.9 | 85.60±17.2 | 78.34±16.7 | 84.07±14.8
EEGNet [7] | [J Neural Eng’18] | 3D-Serial | 81.23±15.65 | 92.37±8.56 |  | 82.60±14.4 | 87.55±15.3 | 81.67±16.2 | 83.54±16.8
D-FaST | [Ours] | 3D-Disentangled | 83.08±13.86 | 92.92±7.42 |  | 83.15±14.2 | 87.29±15.9 | 79.80±19.8 | 86.51±10.5

###  IV-C Performance on CLP datasets (Q1)

We conducted cross-subject and within-subject cognitive classification
experiments on MNRED and ZuCo respectively.

#### Experimental Settings

Hyperparameter settings of D-FaST are summarized in TABLE IV and that of
compared models are summarized in Appendix. -B. For ZuCo dataset, we remove
the LSTA module and only utilized the DCA module for feature extraction due to
the variable length of samples in the dataset. The resulting spatial features
are then flattened and input into a MLP. To fairly compare model performance,
all models use the same optimizer, learning rate and schedule, minibatch size
and number of iterations, and weight decay absorption.

#### Results

As shown in TABLE II, TABLE III, the method D-FaST that we designed achieved
an average accuracy of 78.72% and an AUROC of 81.51% in leave-one-subject-out
cross-validation on binary classification dataset MNRED, an average accuracy
of 78.35% in within-subject on experiment on 9-class classification dataset
ZuCo. The results show that the effect of D-FaST is far superior to other
models. Interestingly, models that use only spatial domain features,
BrainNetCNN [11] and BNT [12], perform poorly on the MNRED dataset. We believe
that EEG data have lower spatial resolution than fMRI data, and that such
models ignore cognitive processes in time and useful information in the
frequency domain and use only limited spatial features. It is worth mentioning
that in order to compare these models more fairly, we conducted experiments
under different frequency domain feature number settings, as depicted in Fig.
7(a). The results show that D-FaST has advantages under different frequency
domain feature number settings. Fig. 9 visualizes D-FaST’s significant
discriminant properties in decoding MNRED.

###  IV-D Generalization ability on traditional datasets (Q2)

We verify the generalization ability of D-FaST against baseline models on
traditional CSD datasets BCIC IV-2A and BCIC IV-2B under different setting:
Cross-Subject and Within-Subject.

#### Experimental Settings

Detailed hyperparameter settings for D-FaST and baseline models can be found
in TABLE IV and Appendix. -B.

#### Results

As shown in TABLE V, in the cross-subject experiment on the BCIC IV-2A and
BCIC IV-2B datasets. D-FaST achieves an average accuracy of 54.96%(+0.23%) and
74.48% AUROC on dataset BCIC IV-2A. D-FaST also pushes average accuracy and
AUROC on BCIC IV-2B to 76.81%(+2.20%) and 83.99%(+1.73%) with Sensitivity
being 73.89% and Specificity being 79.72%(+2.72%).

As shown in TABLE VI, in the within-subject experiment on the BCIC IV-2A,
D-FaST achieves an average accuracy of 83.08%(+1.85%) and an AUROC of 92.92%.
On the BCIC IV-2B dataset, D-FaST achieved an average accuracy of
83.15%(+0.55%). These results outperformed other models significantly. More
experimental results about each subject can be found in TABLE X and TABLE XI.

Additionally, D-FaST showed the second lowest standard deviations in accuracy
(13.86) and AUROC (7.42) when evaluated on the nine subjects, indicating its
higher stability. In contrast, LMDA-Net [16] and TACNet [14], while
potentially achieving optimal performance on specific test set partitions,
lack stability and perform poorly in cross-validation. It is worth mentioning
that increasing the parameter size of EEGNet improved accuracy on the MNRED
dataset but had the opposite effect on BCIC IV-2A and BCIC IV-2B datasets. We
believe that the BCIC IV-2A dataset is comparatively easier than MNRED, and
the overall low accuracy may be due to a low signal-to-noise ratio and poor
data quality in some subjects. Consequently, increasing the parameter size of
EEGNet would cause the model to learn noise and overfit. In contrast, D-FaST
exhibits inherent robustness, as demonstrated in subsequent hyperparameter
sensitivity experiments in Section IV-F, which helps mitigate overfitting to
some extent.

###  IV-E Ablation Study (Q3)

Ablation experiments are carried out for frequency-time-space improvement and
the design of disentangled framework, with EEGNet-large as the baseline. The
experiment carried out a 5-fold cross-validation on MNRED and report the
average accuracy and the standard deviation.

####  IV-E1 Performance Improved By Frequency-Temporal-Spatial Attention

Experiments are carried out on D-FaST using only one module improvement in
frequency, temporal or spatial dimension. The results in TABLE VII indicate
that the model performs better than the baseline when any of the three
improvements are used alone. The performance of combining the three modules
with the disentangled framework is not only better than the experimental setup
of using the three modules alone, but better than combining them with serial
structure.

TABLE VII: The impact of MFA, DSA, CTA, and disentangled frameworks on the model. Method | MFA | DCA | LSTA | Disentangled | Accuracy (%)
---|---|---|---|---|---
Baseline |  |  |  |  | 76.06±0.80
D-FaST | ✓ | ✓ | ✓ | ✓ | 78.72±0.82
✓ | ✓ | ✓ |  | 77.64±0.82
✓ |  |  |  | 76.77±0.81
| ✓ |  |  | 77.10±0.82
|  | ✓ |  | 76.67±0.81

####  IV-E2 Rationality of Disentangled Framework

The disentangled framework and serial framework are compared on D-FaST and
EEGNet-large, respectively. The experiments show that the disentangled
framework performs better on D-FaST and produces similar results to EEGNet-
large. This indicates that frequency and space are not necessarily progressive
relations, and the serial framework may not be the best combination of
frequency and space. The disentangled framework can more fully integrate the
two characteristics.

####  IV-E3 Effect of Fusion Method on Performance

The fusion methods, $\text{Concat}\left(\cdot,\cdot\right)$ makes the
frequency domain dimension features and spatial dimension features side by
side, while $\text{Add}\left(\cdot,\cdot\right)$ superimpose them. The main
difference between the two methods is their gradient backpasses during
training. The performance of these two functions on the MNRED dataset is
compared. As shown in TABLE VIII, the use of superposition is advantageous for
this dataset. Although more complex fusion mechanisms could be designed,
previous studies have found that splicing and direct overlay are usually the
most cost-effective ways without adding a large number of additional
parameters [26, 66].

TABLE VIII: Effect of different fusion methods on model performance on MNRED dataset. Fusion Method |  Accuracy |  AUROC |  Sensitivity |  Specificity
---|---|---|---|---
Concat | 77.01±0.81 | 78.47±0.86 | 56.38±0.73 | 85.99±0.90
Add | 78.72±0.82 | 81.51±0.85 | 62.20±0.71 | 85.98±0.90
TABLE IX: Effect of different aggregate methods on model performance on MNRED dataset. Aggregate Method |  Accuracy |  AUROC |  Sensitivity |  Specificity
---|---|---|---|---
Flatten | 78.72±0.82 | 81.51±0.85 | 62.20±0.71 | 85.98±0.90
Mean | 76.59±0.81 | 77.68±0.85 | 54.24±0.70 | 86.32±0.91
Attention | 76.56±0.81 | 77.82±0.85 | 56.34±0.74 | 85.35±0.90

####  IV-E4 Effect of Aggregate Method on Performance

In order to facilitate the final state to the projection function, the
aggregation function transforms the two-dimensional feature from the time
dimension into an one-dimensional vector. There are three different approaches
used: $\text{Flatten}\left(\cdot\right)$, $\text{Mean}\left(\cdot\right)$ and
$\text{Attention}\left(\cdot\right)$ [26]. We tested the performance of these
three aggregate functions on the MNRED dataset and found that the flatten
approach exhibited superior performance, as depicted in TABLE IX.

###  IV-F Hyperparameter Sensitivity (Q4)

((a))

((b))

((c))

((d))

Figure 7: Dynamic connectogram of negative and positive samples. (a): Performance difference of D-FaST in different frequency domain feature number Settings. (b): Performance difference of D-FaST under the setting of spatial sparsity coefficient and number of subspace nodes. The dashed lines of different colors represent the accuracy changes of different spatial sparsity under each subspace node. The solid black line is the mean value of all the dashed lines. The gray area is the range of accuracy for all spatial sparsity and all subspace node configurations. (c):Performance differences of D-FaST under different window size Settings. (d):Attention Weight of Multi-view. The highlighted bubble indicates the largest attention weight, there $K_{19}$ denote the $9^{th}$ convolution kernel size of InceptionBlock1, $K_{29}$ denote the $9^{th}$ convolution kernel size of InceptionBlock2 and $\mathcal{A}_{F}^{9}$ represents the attention weight of the $9^{th}$ view. TABLE X:  Performance of within subject experiment with different models on BCIC IV-2A. Model | Venue | BCIC IV-2A
---|---|---
A01 | A02 | A03 | A04 | A05 | A06 | A07 | A08 | A09
BrainNetCNN [11] | [NeuroImage’17] | 69.44±5.48 | 53.82±3.70 | 77.43±4.13 | 54.18±7.78 | 43.92±2.54 | 52.95±3.40 | 65.27±6.41 | 75.18±2.90 | 70.49±5.69
BNT [12] | [NeurIPS’22] | 75.16±4.34 | 52.60±4.62 | 77.43±2.13 | 60.08±5.64 | 44.61±4.37 | 50.00±4.54 | 71.53±0.79 | 79.00±1.74 | 73.79±2.81
TACNet [14] | [UbiComp’21] | 81.08±3.98 | 58.51±2.02 | 90.10±3.05 | 71.87±3.21 | 50.88±5.47 | 53.81±5.19 | 90.10±3.35 | 87.67±3.33 | 87.51±2.67
DeepConvNet [8] | [HBM’17] | 78.30±2.20 | 66.31±4.37 | 82.45±5.33 | 76.38±3.90 | 51.73±2.21 | 45.84±5.33 | 80.19±7.84 | 83.50±4.78 | 85.42±2.33
ShallowConvNet [8] | [HBM’17] | 88.38±2.76 | 68.22±4.28 | 90.97±2.51 | 83.00±4.65 | 69.09±2.45 | 59.19±5.81 | 95.84±1.42 | 91.49±1.12 | 89.07±3.15
LMDA-Net [16] | [NeuroImage’23] | 82.29±2.01 | 64.06±5.77 | 93.57±1.91 | 69.09±5.12 | 48.61±3.72 | 51.38±5.90 | 89.06±3.62 | 89.23±3.30 | 90.28±1.15
FBNetGen [15] | [MIDL’22] | 78.47±5.07 | 61.81±2.27 | 86.80±1.45 | 60.60±4.09 | 47.92±3.11 | 56.78±5.11 | 86.12±2.91 | 87.84±1.10 | 88.72±1.63
TCACNet [18] | [IPM’22] | 81.08±4.77 | 60.25±4.05 | 89.06±2.25 | 78.30±3.68 | 50.17±3.71 | 55.21±3.89 | 87.15±1.87 | 87.85±4.13 | 87.68±3.58
EEGNet [7] | [J Neural Eng’18] | 90.63±1.64 | 58.86±2.92 | 95.31±1.32 | 81.24±2.31 | 60.77±5.17 | 63.88±5.46 | 93.06±1.05 | 95.14±1.44 | 92.19±1.73
D-FaST | [Ours] | 90.98±3.27 | 67.36±3.08 | 95.84±1.42 | 83.86±2.40 | 64.58±2.28 | 63.71±3.03 | 94.10±3.01 | 93.24±3.78 | 94.09±1.68
TABLE XI: Performance of within subject experiment with different models on BCIC IV-2B. Model | Venue | BCIC IV-2B
---|---|---
B01 | B02 | B03 | B04 | B05 | B06 | B07 | B08 | B09
RACNN [13] | [IJCAI’21] | 66.00±2.76 | 52.86±1.26 | 56.38±1.96 | 91.63±7.41 | 56.56±4.60 | 61.25±10.7 | 74.94±4.92 | 88.25±7.64 | 68.19±10.1
TACNet [14] | [UbiComp’21] | 75.75±0.77 | 59.50±0.60 | 56.69±0.58 | 97.69±0.98 | 89.50±0.84 | 87.94±0.57 | 84.88±0.65 | 93.31±0.47 | 90.00±0.91
DeepConvNet [8] | [HBM’17] | 73.94±1.98 | 61.71±1.17 | 66.31±0.79 | 97.44±0.26 | 90.94±0.70 | 79.00±1.83 | 81.19±1.28 | 92.00±1.32 | 81.50±0.95
ShallowConvNet [8] | [HBM’17] | 73.84±1.24 | 56.36±1.50 | 56.25±0.82 | 96.84±0.40 | 87.69±0.52 | 82.00±1.05 | 82.56±0.60 | 90.56±0.64 | 85.13±0.84
LMDA-Net [16] | [NeuroImage’23] | 74.56±0.90 | 59.50±2.48 | 61.50±2.14 | 97.81±0.38 | 90.44±1.71 | 84.69±1.94 | 83.50±0.97 | 92.25±1.30 | 86.81±0.81
EEG-ChannelNet [17] | [TPAMI’21] | 63.75±0.65 | 56.07±0.58 | 53.81±0.55 | 96.81±0.87 | 74.00±0.75 | 73.25±4.53 | 75.00±1.25 | 91.81±0.84 | 85.00±1.80
SBLEST [9] | [TPAMI’23] | 72.12±0.73 | 55.93±2.19 | 54.36±1.49 | 92.96±0.34 | 87.96±0.66 | 82.08±0.99 | 76.05±0.70 | 89.52±0.56 | 77.12±0.95
TCACNet [18] | [IPM’22] | 76.50±1.16 | 57.38±0.54 | 56.81±1.28 | 97.63±0.42 | 88.56±1.05 | 87.25±0.84 | 83.38±1.39 | 93.06±0.41 | 90.31±1.80
EEGNet [7] | [J Neural Eng’18] | 75.94±0.38 | 57.50±1.36 | 62.44±1.35 | 98.31±0.17 | 92.86±0.75 | 87.13±0.71 | 84.38±0.49 | 92.31±0.47 | 92.56±0.56
D-FaST | [Ours] | 78.94±1.18 | 59.43±2.31 | 60.94±1.96 | 97.94±0.17 | 93.56±0.93 | 88.50±1.16 | 83.69±0.84 | 92.44±0.46 | 92.94±0.52

Numerous hyperparameters are incorporated in D-FaST. To fully harness the
potential of D-FaST and identify general rules, we conducted a thorough
exploration of the model by varying the collocation hyperparameter settings.

####  IV-F1 Number of Features in the Target Frequency Domain

To prominently showcase the advantages of D-FaST in frequency domain feature
extraction, we compared the accuracy of baseline models [16, 8, 7] using the
same number of frequency domain features on the MNRED dataset. As illustrated
in Fig. 7(a), regardless of the number of features employed in the frequency
domain, D-FaST consistently outperforms the other models. It is worth noting
that while all models demonstrate performance improvement with an increase in
the number of frequency domain features, ShallowConNet and EEGNet approach
saturation, whereas D-FaST still exhibits significant room for improvement.
Taking into consideration the number of parameters, as well as the time and
space complexity associated with training when augmenting the number of
features in the frequency domain, we limited our exploration to a maximum of
128 features. For larger datasets and more complex tasks involving decoding
brain cognitive signals, further increasing the number of features in the
frequency domain can be explored.

####  IV-F2 Spatial Sparsity Coefficient and Number of Subspace Nodes

The spatial sparsity coefficient and number of subspace nodes control the size
and range of the dynamic brain connection graph from both the source and
target nodes. For example, a coefficient of 1 indicates that the target node’s
field of view in each subspace is equivalent to that of all source nodes. With
only one subspace node, all source node features are compressed into a single
target node. In our evaluation on the MNRED dataset, shown in Fig. 7(b), we
tested the model’s performance with 7 subspace nodes and 4 spatial sparsity
settings. Despite the dataset having only 30 brain leads, we also experimented
with 32 and 64 subspace nodes, aligning with the concept of virtual brain
regions mentioned in Section 5. Overall, the model’s performance initially
improves and then declines as the number of subspace nodes increases, reaching
maximum average performance at 16 nodes. Moreover, higher accuracy is observed
with a small number of subspace nodes (e.g., $N^{\prime}=1$) and a larger
spatial sparsity coefficient, while a smaller sparse coefficient (e.g.,
$\tau=0.4$) yields greater accuracy in a larger subspace with more nodes
(e.g., $N^{\prime}=64$). The best performance is achieved when these two
factors are balanced (e.g., $N^{\prime}=8$ & $\tau=0.6$ or $N^{\prime}=16$ &
$\tau=0.8$).

####  IV-F3 Local Temporal Sliding Window

We examined the performance differences of the model across various datasets
at different window size settings, as illustrated in Fig. 7(c). With a window
size of 128, the pooling layer renders the window almost global, implying no
window is used. Consequently, the model’s sensitivity to this hyperparameter
is relatively smaller than that of other hyperparameters. Generally, the model
performs optimally at a window size of 64 and exhibits saturation and a
decline at larger windows. These finding suggests that utilizing a local
temporal sliding window assists the model in efficiently exploring a broader
range of attention.

((a))

((b))

Figure 8: Dynamic connectogram of negative and positive samples. (a): Positive
examples. (b): Negative examples. It is a description of a cognitive process
of the brain consisting of four stages from left to right. The four circular
diagrams at the top correspond to the dynamic brain connectivity of each
stage, with darker colors indicating stronger weights of the corresponding
connection edges. The four heat maps at the bottom correspond to the energy
level of brain regions in each stage, with higher heat indicating higher
energy level. Figure 9: 2D projections of the embeddings of different models
on MNRED by using t-SNE.

###  IV-G Visualization Analysis of Model Behavior (Q5)

####  IV-G1 Multi-view Attention

To provide a more intuitive illustration of the model behavior of multi-view
attention, we visualize $\mathcal{A}_{F}$ in the MVA. Specifically, we input a
batch of 128 randomly sampled test data into the trained model and obtain
$\mathcal{A}_{F}$. We then average it from the MVA module and further average
the attention weights from the same group of convolutional cores, as shown in
Fig. 7(d). For instance, ”49-43”(highlighted in Fig. 7(d)) refers to frequency
domain features of two channels obtained after passing brain cognitive signal
data through a two-dimensional convolution layer with a convolution kernel of
(1,49), and another two-dimensional convolution layer with a convolution
kernel of (1,43). After averaging the attention weights corresponding to these
four channels, the weight is 0.5469. We find that using the ”49-43”
convolution combination, i.e. ”$f/4$-$f/4$”, yields the maximum weight,
followed by ”91-1”(the last combination in Fig. 7(d)), i.e. ”$f/2$-1”. This
suggests that $f/2$ (used in [7]) may not be the best choice for the
perspective of feature extraction in the frequency domain. The MVA captures
the optimal configuration from multiple perspectives in a learnable way. In
fact, the weights obtained by these combinations are not significantly
different from each other, indicating that the model obtains valuable
information from various fields of view.

####  IV-G2 Dynamic Connectogram

In order to provide a more intuitive illustration of the model behavior of
dynamic brain connectomogram attention, we visualize $\mathcal{A}_{S}$ in the
DCA. Fig. 8 visualizes two sets of brain cognitive models generated by subject
6 (randomly selected) when negative and positive data are observed. The main
hyperparameters are set as: $\tau=0.1,N^{\prime}=30,h=4$(all other parameters
are set the same as 4.3). During the initial stage, we note minimal variance
between positive and negative brain topographic maps, and the associated brain
connection maps are similar. This indicates that subjects, having just been
exposed to visual stimuli, hadn’t yet distinguished between stimulus
categories. In the second stage, these differences start to incrementally
increase; by the third stage, notable disparities emerge in both the brain
topography and connection maps. We infer that at this juncture, subjects have
processed the stimulus and formed subjective judgments. In the fourth stage,
although the differences slightly diminish, they continue to persist. This
persistence could be due to subjects’ uncertainty concerning their judgment
after making an initial categorization of the stimulus, leading to continued
variation in the brain connectivity map. In summary, the dynamic brain
connection map and brain topography map maintain a high level of consistency
throughout the cognitive process. This suggests that DCA can dynamically
depict different stages of the brain’s cognitive process and differentiate
between distinct cognitive behaviors.

##  V Conclusion and Analysis

In this article, we introduce a new CLP dataset called MNRED, featuring a
novel paradigm that addresses common issues in brain cognitive signal decoding
tasks. We also propose a brain cognitive signal decoder named D-FaST. By
innovating the coding mechanisms for frequency domain information, spatial
information, and temporal information, as well as designing a decoupled
structure for EEG signal processing that better captures the characteristics
of relationships between different domains of information, we have
significantly enhanced the analysis of EEG signal data. Through experiments
conducted on MNRED, ZuCo, and two classic datasets, BCIC IV-2A and BCIC IV-2B,
we have verified the superior performance of our model, achieving state-of-
the-art results.

## Acknowledgments

The work is supported by the Research Project (BHQ090003000X03).

## References

  * [1] Z. Wang and H. Ji, “Open vocabulary electroencephalography-to-text decoding and zero-shot sentiment classification,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, 2022, pp. 5350–5358.
  * [2] O. Eberle, S. Brandl, J. Pilot, and A. Søgaard, “Do transformer models show similar attention patterns to task-specific human gaze?” in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_ , 2022, pp. 4295–4309.
  * [3] W. Han, J. Qiu, J. Zhu, M. Xu, D. Weber, B. Li, and D. Zhao, “An empirical exploration of cross-domain alignment between language and electroencephalogram,” _arXiv preprint arXiv:2208.06348_ , 2022.
  * [4] X. Ding, B. Chen, L. Du, B. Qin, and T. Liu, “Cogbert: Cognition-guided pre-trained language models,” in _Proceedings of the 29th International Conference on Computational Linguistics_ , 2022, pp. 3210–3225.
  * [5] F. R. Willett, E. M. Kunz, C. Fan, D. T. Avansino, G. H. Wilson, E. Y. Choi, F. Kamdar, M. F. Glasser, L. R. Hochberg, and S. Druckmann, “A high-performance speech neuroprosthesis,” _Nature_ , pp. 1–6, 2023.
  * [6] S. L. Metzger, K. T. Littlejohn, A. B. Silva, D. A. Moses, M. P. Seaton, R. Wang, M. E. Dougherty, J. R. Liu, P. Wu, and M. A. Berger, “A high-performance neuroprosthesis for speech decoding and avatar control,” _Nature_ , pp. 1–10, 2023.
  * [7] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance, “Eegnet: a compact convolutional neural network for eeg-based brain–computer interfaces,” _Journal of neural engineering_ , vol. 15, no. 5, p. 056013, 2018.
  * [8] R. T. Schirrmeister, J. T. Springenberg, L. D. J. Fiederer, M. Glasstetter, K. Eggensperger, M. Tangermann, F. Hutter, and T. Ball, “Deep learning with convolutional neural networks for eeg decoding and visualization,” _Human brain mapping_ , vol. 38, no. 11, pp. 5391–5420, 2017.
  * [9] W. Wang, F. Qi, D. Wipf, C. Cai, T. Yu, Y. Li, Z. Yu, and W. Wu, “Sparse bayesian learning for end-to-end eeg decoding,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [10] S. Gong, K. Xing, A. Cichocki, and J. Li, “Deep learning in eeg: Advance of the last ten-year critical period,” _IEEE Transactions on Cognitive and Developmental Systems_ , vol. 14, no. 2, pp. 348–365, 2022.
  * [11] J. Kawahara, C. J. Brown, S. P. Miller, B. G. Booth, V. Chau, R. E. Grunau, J. G. Zwicker, and G. Hamarneh, “Brainnetcnn: Convolutional neural networks for brain networks; towards predicting neurodevelopment,” _NeuroImage_ , vol. 146, pp. 1038–1049, 2017.
  * [12] X. Kan, W. Dai, H. Cui, Z. Zhang, Y. Guo, and C. Yang, “Brain network transformer,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 25 586–25 599, 2022.
  * [13] Z. Fang, W. Wang, S. Ren, J. Wang, W. Shi, X. Liang, C.-C. Fan, and Z. Hou, “Learning regional attention convolutional neural network for motion intention recognition based on eeg data,” in _Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence_ , 2021, pp. 1570–1576.
  * [14] X. Liu, Q. Hui, S. Xu, S. Wang, R. Na, Y. Sun, X. Chen, and D. Zheng, “Tacnet: Task-aware electroencephalogram classification for brain-computer interface through a novel temporal attention convolutional network,” pp. 660–665, 2021.
  * [15] X. Kan, H. Cui, J. Lukemire, Y. Guo, and C. Yang, “Fbnetgen: Task-aware gnn-based fmri analysis via functional brain network generation,” in _International Conference on Medical Imaging with Deep Learning_. PMLR, 2022, pp. 618–637.
  * [16] Z. Miao, M. Zhao, X. Zhang, and D. Ming, “Lmda-net: A lightweight multi-dimensional attention network for general eeg-based brain-computer interfaces and interpretability,” _NeuroImage_ , p. 120209, 2023.
  * [17] S. Palazzo, C. Spampinato, I. Kavasidis, D. Giordano, J. Schmidt, and M. Shah, “Decoding brain representations by multimodal learning of neural activity and visual features,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 43, no. 11, pp. 3833–3849, 2020.
  * [18] X. Liu, R. Shi, Q. Hui, S. Xu, S. Wang, R. Na, Y. Sun, W. Ding, and D. Zheng, “Tcacnet: Temporal and channel attention convolutional network for motor imagery classification of eeg-based bci,” _Information Processing & Management_, vol. 59, no. 5, p. 103001, 2022.
  * [19] M. Taherisadr, M. Joneidi, and N. Rahnavard, “Eeg signal dimensionality reduction and classification using tensor decomposition and deep convolutional neural networks,” _Electrical Engineering and Systems Science_ , 2019.
  * [20] H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, “Informer: Beyond efficient transformer for long sequence time-series forecasting,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 35, 2021, pp. 11 106–11 115.
  * [21] M. Liu, W. Wu, Z. Gu, Z. Yu, F. Qi, and Y. Li, “Deep learning based on batch normalization for p300 signal detection,” _Neurocomputing_ , vol. 275, pp. 288–297, 2018.
  * [22] S. K. Khare, V. Bajaj, and U. R. Acharya, “Spwvd-cnn for automated detection of schizophrenia patients using eeg signals,” _IEEE Transactions on Instrumentation and Measurement_ , vol. Vol.70, pp. 1–9, 2021.
  * [23] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020, pp. 7871–7880.
  * [24] J. Luo, Y. Wang, S. Xia, N. Lu, X. Ren, Z. Shi, and X. Hei, “A shallow mirror transformer for subject-independent motor imagery bci,” _Computers in Biology and Medicine_ , vol. 164, p. 107254, 2023.
  * [25] R. Fu, Z. Wang, S. Wang, X. Xu, J. Chen, and G. Wen, “Eegnet-msd: A sparse convolutional neural network for efficient eeg-based intent decoding,” _IEEE Sensors Journal_ , 2023.
  * [26] B.-H. Kim, J. C. Ye, and J.-J. Kim, “Learning dynamic graph representation of brain connectome with spatio-temporal attention,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 4314–4327, 2021.
  * [27] D. Chen, S. Wan, and F. Bao, “Epileptic focus localization using discrete wavelet transform based on interictal intracranial eeg.” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. 25, no. 5, pp. 413–425, 2016.
  * [28] B. Kaneshiro, M. Perreau Guimaraes, H.-S. Kim, A. M. Norcia, and P. Suppes, “A representational similarity analysis of the dynamics of object processing using single-trial eeg classification,” _Plos one_ , vol. 10, no. 8, p. e0135697, 2015.
  * [29] Raviraj Joshi, Purvi Goel, Mriganka Sur, and Hema A. Murthy, “Single trial p300 classification using convolutional lstm and deep learning ensembles method,” _Intelligent Human Computer Interaction_ , vol. Vol.11278, pp. 3–15, 2018.
  * [30] X. Li, W. Wei, S. Qiu, and H. He, “Tffransformer for zero-training decoding of two bci tasks,” in _Proceedings of the 30th ACM international conference on multimedia_ , 2022, pp. 51–59.
  * [31] N. Hollenstein, J. Rotsztejn, M. Troendle, A. Pedroni, C. Zhang, and N. Langer, “Zuco, a simultaneous eeg and eye-tracking resource for natural sentence reading,” _Scientific data_ , vol. 5, no. 1, pp. 1–13, 2018.
  * [32] K. Armeni, U. Güçlü, M. van Gerven, and J.-M. Schoffelen, “A 10-hour within-participant magnetoencephalography narrative dataset to test models of language comprehension,” _Scientific Data_ , vol. 9, no. 1, p. 278, 2022.
  * [33] J.-M. Schoffelen, R. Oostenveld, N. H. L. Lam, J. Uddén, A. Hultén, and P. Hagoort, “A 204-subject multimodal neuroimaging dataset to study language processing,” _Scientific Data_ , vol. 6, no. 1, p. 17, 2019.
  * [34] C. Du, K. Fu, J. Li, and H. He, “Decoding visual neural representations by multimodal learning of brain-visual-linguistic features,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [35] A. Kamble, P. H. Ghare, and V. Kumar, “Deep-learning-based bci for automatic imagined speech recognition using spwvd,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 72, pp. 1–10, 2022.
  * [36] Y. Peng, F. Qin, W. Kong, Y. Ge, F. Nie, and A. Cichocki, “Gfil: A unified framework for the importance analysis of features, frequency bands, and channels in eeg-based emotion recognition,” _IEEE Transactions on Cognitive and Developmental Systems_ , vol. 14, no. 3, pp. 935–947, 2022.
  * [37] H. Wu, T. Hu, Y. Liu, H. Zhou, J. Wang, and M. Long, “Timesnet: Temporal 2d-variation modeling for general time series analysis,” in _The Eleventh International Conference on Learning Representations_ , 2022.
  * [38] X. Li, Y. Zhou, N. Dvornek, M. Zhang, S. Gao, J. Zhuang, D. Scheinost, L. H. Staib, P. Ventola, and J. S. Duncan, “Braingnn: Interpretable brain graph neural network for fmri analysis,” _Medical Image Analysis_ , vol. 74, p. 102233, 2021.
  * [39] H. Cui, W. Dai, Y. Zhu, X. Li, L. He, and C. Yang, “Interpretable graph neural networks for connectome-based brain disorder analysis,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VIII_. Springer, 2022, pp. 375–385.
  * [40] W. Li, M. Wang, J. Zhu, and A. Song, “Eeg-based emotion recognition using trainable adjacency relation driven graph convolutional network,” _IEEE Transactions on Cognitive and Developmental Systems_ , vol. 15, no. 4, pp. 1656–1672, 2023.
  * [41] Y. Wang, M. Wu, R. Jin, X. Li, L. Xie, and Z. Chen, “Local-global correlation fusion-based graph neural network for remaining useful life prediction,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2023.
  * [42] V. P. Dwivedi and X. Bresson, “A generalization of transformer networks to graphs,” _AAAIWorkshop on Deep Learning on Graphs: Methods and Applications_ , 2021.
  * [43] T. Mikolov, M. Karafiát, L. Burget, J. Cernocký, and S. Khudanpur, “Recurrent neural network based language model,” in _11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan_. ISCA, 2010, pp. 1045–1048.
  * [44] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, pp. 1735–1780, 1997.
  * [45] J. Chung, C. Gulcehre, K. Cho, and Y. J. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” _NIPS 2014 Workshop on Deep Learning_ , December 01 2014.
  * [46] J. Chen, Y. Yang, T. Yu, Y. Fan, X. Mo, and C. Yang, “Brainnet: Epileptic wave detection from seeg with hierarchical graph diffusion learning,” in _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2022, pp. 2741–2751.
  * [47] I. Beltagy, M. E. Peters, and A. J. a. e.-p. Cohan, “Longformer: The long-document transformer,” p. arXiv:2004.05150, April 01, 2020 2020.
  * [48] H. Cao, Z. Huang, T. Yao, J. Wang, H. He, and Y. Wang, “Inparformer: evolutionary decomposition transformers with interactive parallel attention for long-term time series forecasting,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, 2023, pp. 6906–6915.
  * [49] W. Hua, Z. Dai, H. Liu, and Q. Le, “Transformer quality in linear time,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 9099–9117.
  * [50] W.-B. Jiang, X. Yan, W.-L. Zheng, and B.-L. Lu, “Elastic graph transformer networks for eeg-based emotion recognition,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [51] B. Sun, B. Song, J. Lv, P. Chen, X. Sun, C. Ma, and Z. Gao, “A multiscale feature extraction network based on channel-spatial attention for electromyographic signal classification,” _IEEE Transactions on Cognitive and Developmental Systems_ , vol. 15, no. 2, pp. 591–601, 2023.
  * [52] Z. Gao, X. Wang, Y. Yang, Y. Li, K. Ma, and G. Chen, “A channel-fused dense convolutional network for eeg-based emotion recognition,” _IEEE Transactions on Cognitive and Developmental Systems_ , vol. 13, no. 4, pp. 945–954, 2021.
  * [53] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [54] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [55] Q. Wang, B. Wu, P. F. Zhu, P. Li, W. Zuo, and Q. Hu, “Eca-net: Efficient channel attention for deep convolutional neural networks,” _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 11 531–11 539, 2019.
  * [56] P. Stern, “No neuron is an island,” _Science_ , vol. 378, no. 6619, pp. 486–487, 2022.
  * [57] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [58] A. Culotta, A. McCallum, and J. Betz, “Integrating probabilistic extraction models and data mining to discover relations and patterns in text,” in _Proceedings of the Human Language Technology Conference of the NAACL, Main Conference_ , 2006, pp. 296–303.
  * [59] M. Tangermann, K.-R. Müller, A. Aertsen, N. Birbaumer, C. Braun, C. Brunner, R. Leeb, C. Mehring, K. Miller, G. Mueller-Putz, G. Nolte, G. Pfurtscheller, H. Preissl, G. Schalk, A. Schlögl, C. Vidaurre, S. Waldert, and B. Blankertz, “Review of the bci competition iv,” _Frontiers in Neuroscience_ , vol. 6, 2012.
  * [60] Z. Miao, X. Zhang, C. Menon, Y. Zheng, M. Zhao, and D. Ming, “Priming cross-session motor imagery classification with a universal deep domain adaptation framework,” _Available at SSRN 4185000_ , 2022.
  * [61] H. He and D. Wu, “Transfer learning for brain–computer interfaces: A euclidean space data alignment approach,” _IEEE Transactions on Biomedical Engineering_ , vol. 67, no. 2, pp. 399–410, 2019.
  * [62] F. Lotte, “Signal processing approaches to minimize or suppress calibration time in oscillatory activity-based brain–computer interfaces,” _Proceedings of the IEEE_ , vol. 103, no. 6, pp. 871–890, 2015.
  * [63] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _International Conference on Learning Representations_ , 2018.
  * [64] ——, “Sgdr: Stochastic gradient descent with warm restarts,” in _International Conference on Learning Representations_ , 2016.
  * [65] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [66] Y. Jain, C. I. Tang, C. Min, F. Kawsar, and A. Mathur, “Collossl: Collaborative self-supervised learning for human activity recognition,” in _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ , vol. 6, no. 1, 2022, pp. 1–28.

|  WeiGuo Chen received the BE degree in Computer Science and Technology from
National University of Defense Technology, China, in 2022, where he is
currently pursuing the master’s degree. His research interests include
cognitive intelligence, multimodal learning in brain-computer interface, time
series analysis and natural language processing.
---|---
|  Changjian Wang received his Ph.D. degree in computer science from the
School of Computer, National University of Defense Technology. He is currently
a Professor of the National University of Defense Technology, Changsha, China.
His current research interests include artificial intelligence, big data and
natural language processing.
---|---
|  Kele Xu (Member, IEEE) received the doctorate degree in informatique, les
télécommunications et l’électronique from Paris VI University, Paris, France,
in 2017. He is currently an Associate Professor with the School of Computer
Science, National University of Defense Technology, Changsha, China. His
research interests include audio signal processing, machine learning, and
intelligent software systems.
---|---
|  Yuan Yuan received the PhD degree at National University of Defense
Technology, China. He is now an associate professor at National University of
Defense Technology, China. His research interests focus on supercomputer
systems, AIOPs and system monitoring and diagnosis.
---|---
|  Yanru Bai received the B.Eng. and M.Eng. degrees in biomedical engineering
from Tianjin University, China, and the Ph.D. degree in biomedical engineering
from Nanyang Technological University, Singapore. She is currently an
associate professor with Academy of Medical Engineering and Translational
Medicine, Tianjin University. Her research interests include brain-computer
interface, biomedical signal processing, computational neuroscience, and
neural engineering and rehabilitation.
---|---
|  Dongsong Zhang received the PhD degree in Computer Science and Technology
from National University of Defense Technology, China, in 2012. He currently
is an associate professor in School of Big Data andArtificial Intelligence at
Xinyang College. His research interests are in the area of soft computing
(Neural Network, Fuzzy Logic), Real-Time Systems.
---|---

###  -A Model Details

The algorithm utilizes multi-view convolution and multi-view attention, which
correspond to Eq.1 and Eq. 2, respectively. To ensure that the output and
input of a multi-view convolution share the same time dimension, we apply zero
padding. Furthermore, we introduce a GELU activation function between the two
layers.

These convolution kernels in InceptionBlock corresponding to Eq. 3 and Eq. 4
differ from those in MVA. Although both methods employ convolution kernels of
varying sizes to extract rich features, DCA employs fewer kernels compared to
MVA. Specifically, the convolution kernels in DCA are small
convolutions:$\left[\left(1,1\right),\ \left(1,2\right),\
\left(1,3\right)\right]$.

To control the size of model parameters and enhance operational efficiency, we
set the number of groups of convolutions in Eq. 5 to $E=k\times N^{\prime}$ to
obtai $Q_{T},K_{T},V_{T}$, whose nature are relative to DWConv2d. Without loss
of generality, we still describe them as CNN. Actually, we can modify its
representational power by assigning different values to $E$ and adjusting the
dimensions of $Q_{T},K_{T},V_{T}$. Additionally, we can enhance the
parallelism of matrix multiplication in the algorithm by utilizing multi-head
attention. It is important to note that in order to prevent gradient
vanishing, we have implemented a residual-like dense structure in LSTA. DCA
does not employ residual connections since it already runs in parallel with
MVA.

###  -B Parameter Setting of Baseline Model

TABLE XIII shows specific hyperparameter settings of comparative models [7,
11, 12, 8, 15, 16] reproduced and tuned in this paper. Other parameter
settings not mentioned are kept consistent with the original literature. To
fairly compare model performance, all models use the same optimizer, learning
rate and schedule, minibatch size and number of iterations, and weight decay
absorption.

####  -B1 Number of Model Parameters

TABLE XII shows the number of parameters for the different models. It can be
seen that the number of parameters of the D-FaST model on the MNRED dataset is
much larger than that of other models. Nevertheless, the model does not
overfit due to the large number of parameters, which indicates the robustness
of D-FaST to a certain extent.

####  -B2 Stratified Sampling & Cross Validation

Stratified sampling in training-validation set division often leads to
significant disparities in experimental outcomes. Previous studies [7, 8]
typically separated data into predefined sets, leading to inefficient use of
limited brain signal data and potentially skewed model evaluations. Prior
attempts [16] to reclassify datasets failed to address imbalanced data quality
and were discarded. Our method uses stratified sampling techniques [12] to
balance data category distribution within each cross-validation fold. In
cross-subject analyses, it ensures equal data proportions from different
subjects across partitions. For within-subject variations, it maintains data
distribution equilibrium from different time periods within a dataset. We’ve
also included fixed random seeds in random dataset partitioning to ensure
experiment fairness and reproducibility.

###  -C Visualization detail

Using Fig. 8 as an example, the steps for visualization are as follows:

  * •

we used the training sets of all subjects to train the model under the
conditions $\tau=0.6,N^{\prime}=30,h=4$ (other parameter Settings are
consistent with Section IV-C);

  * •

EEG data $\mathcal{X}^{\star}$ where $N=30,T=440$ is obtained by superimposing
and averaging the cognitive signals of all negative cases in the pre-treated
No.6 (randomly selected) subjects in the test set;

  * •

On the one hand, we evenly divide $\mathcal{X}^{\star}$ into $h=4$ stages, and
then obtain the dynamic energy graph of size $4\times 30\times 110$ and plot
it as a brain topographic map, namely the four graphs at the bottom of Fig. 8;
On the other hand, we input the model $\mathcal{X}^{\star}$ and extract the
set of dynamic connection matrices
$\mathcal{A}_{S}=\\{\mathcal{A}_{S}^{1},\mathcal{A}_{S}^{2},\mathcal{A}_{S}^{3},\mathcal{A}_{S}^{4}\\}$
from the DCA module, respectively select the channels with the largest multi-
view attention in MVA (as described in Section IV-G1), obtain the dynamic
brain connection graph, and draw the directed graph respectively, namely the
four graphs above Fig. 8. In order to highlight the important parts of the
brain connectivity map, we set the sparsity $\tau$ to 0.1 during the test,
that is, each node has a maximum $0.1\times 30=3$ of exits.

###  -D More Results

Although leave-one-subject-out cross-validation is common in CSD, K-fold
cross-validation occupies a very important place in classical machine
learning. We carried out 5-fold cross-validation on MNRED, ZuCo, BCIC IV-2A
and BCIC IV-2B as additional evidence. Results shows in TABLE XIV, TABLE XV
and TABLE XVI.

TABLE XII: Parameter scales for different models. Model | MNRED | ZuCo | BCIC IV-2A | BCIC IV-2B
---|---|---|---|---
BNT [12] | 232K | 1,340K | 161K | 27K
BrainNetCNN [11] | 172K | 503K | 136K | 51K
DeepConvNet [8] | 61K | - | 73K | 46k
ShallowConvNet [8] | 51K | - | 41K | 8K
FBNetGen [15] | 96K | 308K | 80K | 38K
LMDA-Net [16] | 9K | - | 6K | 4K
EEGNet [7] | 726K | - | 36K | 35K
TACNet [14] | 115K | - | 90K | 28K
RACNN [13] | 12,180K | - | 15,227K | 14,200K
EEG-ChannelNet [17] | 2,135K | - | 2,135K | 1,204K
TCACNet [18] | 115K | - | 90K | 28K
Graph Transformer [42] | - | 739K | - | -
D-FaST(ours) | 4,302K | 151K | 672K | 11K
TABLE XIII: Baseline Model hyperparameter settings. Model | Hyper-Parameter
---|---
EEGNet | Num kernels | P1 | D | P2 | Dropout
16 | 4 | 2 | 8 | 0.5
LMDA-Net | Channel depth1 | Ave depth | Depth | K | Channel depth2
product | 5 | 9 | 7 | 16
BNT | Pooling | Freeze center | Sizes | Dim | Orthogonal
(False, True) | True | (N, N/2) | 1024 | True
FBNetGen | Extractor type | Graphgeneration | Embedding size | Window size
cnn | product | 16 | 50
BrainNetCNN | E2E1 | E2E2 | E2N | N2G
(1, 32) | (32, 64) | (64, 1) | (1, 256)
| Num kernels | Dropout
DeepConvNet | 25 | 0.5
ShallowConvNe | 40 | 0.5
TABLE XIV: Compare experimental results under cross-subject (5-fold corss-validation) experimental settings on MNRED. Model | Venue | Type | MNRED
---|---|---|---
Accuracy (%) |  AUROC(%) |  Sensitivity (%) |  Specificity (%)
BrainNetCNN [11] | [NeuroImage’17] | 1D | 73.88±2.48 | 72.15±1.44 | 37.30±8.03 | 81.24±1.52
BNT [12] | [NeurIPS’22] | 74.11±2.37 | 72.30±2.24 | 37.61±10.34 | 80.15±2.26
TACNet [14] | [UbiComp’21] | 79.66±0.73 | 81.68±2.69 | 62.86±7.03 | 86.79±2.23
RACNN [13] | [IJCAI’21] | 79.33±0.80 | 80.57±0.82 | 61.57±0.65 | 86.94±0.89
DeepConvNet [8] | [HBM’17] | 2D-Serial | 77.45±0.64 | 82.90±1.13 | 72.03±3.55 | 79.88±2.58
ShallowConvNet [8] | [HBM’17] | 80.31±1.04 | 84.70±1.17 | 67.33±0.40 | 85.92±1.77
FBNetGen [15] | [MIDL’22] | 78.95±1.80 | 81.35±1.05 | 54.56±6.47 | 86.51±1.96
LMDA-Net [16] | [NeuroImage’23] | 78.60±0.72 | 84.76±0.96 | 72.73±2.58 | 81.13±1.52
EEG-ChannelNet [17] | [TPAMI’21] | 80.88±0.83 | 83.98±0.86 | 60.28±0.63 | 89.83±0.91
TCACNet [18] | [IPM’22] | 80.02±0.96 | 82.12±1.39 | 63.99±1.49 | 86.98±1.32
EEGNet [7] | [J Neural Eng’18] | 3D-Serial | 76.22±1.40 | 83.71±1.92 | 78.82±0.68 | 74.87±2.70
D-FaST | [Ours] | 3D-Disentangled | 82.96±2.04 | 87.43±1.85 | 70.79±5.26 | 88.17±1.80
TABLE XV: Compare experimental results under cross-subject (5-fold corss-validation) experimental settings. Model | Venue | ZuCo
---|---|---
Accuracy (%) |  AUROC(%)
FBNetGen [15] | [MIDL’22] | 70.30±0.74 | 90.56±0.92
BrainNetCNN [11] | [NeuroImage’17] | 85.49±0.87 |  93.87±0.94
Graph Transformer [42] | [AAAI’21] | 87.10±0.88 | 93.75±0.94
BNT [12] | [NeurIPS’22] | 87.45±0.89 | 94.05±0.95
D-FaST | [Ours] | 88.68±0.89 | 92.77±0.94
TABLE XVI: Compare experimental results under cross-subject (5-fold corss-validation) experimental settings. Model | Venue | Type | BCIC IV-2A |  | BCIC IV-2B
---|---|---|---|---|---
Accuracy (%) |  AUROC (%) |  |  Accuracy (%) |  AUROC(%) |  Sensitivity (%) |  Specificity (%)
BrainNetCNN [11] | [NeuroImage’17] | 1D | 54.40±1.10 | 77.34±0.40 |  | - | - | - | -
BNT [12] | [NeurIPS’22] | 55.77±0.95 | 78.77±0.72 |  | - | - | - | -
TACNet [14] | [UbiComp’21] | 65.99±0.67 | 85.25±0.58 |  | 77.04±0.77 | 85.71±0.86 | 73.72±0.77 | 80.37±0.84
RACNN [13] | [IJCAI’21] | - | - |  | 72.07±0.77 | 79.84±0.86 | 69.25±0.75 | 74.89±0.80
DeepConvNet [8] | [HBM’17] | 2D-Serial | 71.95±0.50 | 90.68±0.21 |  | 77.60±0.44 | 86.21±0.64 | 76.24±4.44 | 78.96±4.50
ShallowConvNet [8] | [HBM’17] | 72.38±0.73 | 90.49±0.42 |  | 76.66±0.26 | 85.48±0.08 | 74.72±2.61 | 78.61±2.65
FBNetGen [15] | [MIDL’22] | 67.26±1.58 | 87.14±0.61 |  | 72.05±0.44 | 80.85±0.87 | 72.68±3.97 | 71.42±4.62
LMDA-Net [16] | [NeuroImage’23] | 70.22±1.31 | 88.35±0.63 |  | 79.68±0.26 | 88.00±0.37 | 79.52±2.90 | 79.85±2.71
EEG-ChannelNet [17] | [TPAMI’21] | 61.05±0.63 | 83.18±8.4 |  | 74.98±0.76 | 83.77±0.85 | 73.04±0.77 | 76.92±0.813
SBLEST [9] | [TPAMI’23] | - | - |  | 72.70±0.24 | 81.69±0.16 | 67.21±0.31 | 78.76±0.40
TCACNet [18] | [IPM’22] | 67.50±0.69 | 85.65±0.87 |  | 76.62±0.77 | 85.78±0.86 | 72.08±0.76 | 81.15±0.84
EEGNet [7] | [J Neural Eng’18] | 3D-Serial | 70.52±1.07 | 88.82±0.77 |  | 80.23±0.28 | 88.91±0.24 | 79.08±1.24 | 81.38±1.04
D-FaST | [Ours] | 3D-Disentangled | 74.95±0.66 | 90.56±0.71 |  | 80.72±0.31 | 89.27±0.33 | 79.48±2.33 | 81.96±1.92