# Empirical Interpretation of the Relationship
Between Speech Acoustic Context and
Emotion Recognition

Anna Ollerenshaw, Md. Asif Jalal, Rosanna Milner, Thomas Hain
Speech and Hearing Research Group, Department of Computer Science, University
of Sheffield, UK This work was partly supported by VoiceBase Inc. at the
VoiceBase Research Center which is supported by Liveperson, Inc.

###### Abstract

Speech emotion recognition (SER) is vital for obtaining emotional intelligence
and understanding the contextual meaning of speech. Variations of consonant-
vowel (CV) phonemic boundaries can enrich acoustic context with linguistic
cues, which impacts SER. In practice, speech emotions are treated as single
labels over an acoustic segment for a given time duration. However, phone
boundaries within speech are not discrete events, therefore the perceived
emotion state should also be distributed over potentially continuous time-
windows.

This research explores the implication of acoustic context and phone
boundaries on local markers for SER using an attention-based approach. The
benefits of using a distributed approach to speech emotion understanding are
supported by the results of cross-corpora analysis experiments. Experiments
where phones and words are mapped to the attention vectors along with the
fundamental frequency to observe the overlapping distributions and thereby the
relationship between acoustic context and emotion. This work aims to bridge
psycholinguistic theory research with computational modelling for SER.

###### Index Terms:

emotion recognition, context modelling, speech, attention, computational
paralinguistics, acoustic modelling.

††publicationid: pubid:

##  I Introduction

Speech emotion understanding and recognition (SER) is a complex research area,
with modeling approaches that aim to adapt to speech variability, while
reducing redundancy in acoustic and linguistic perceptual cue recognition.
These approaches are particularly challenging to develop because the target
labels or the perceived emotion states can be considered very subjective and
biased by cultural and linguistic perception differences. Speech emotion,
within the domain of SER, is typically represented by two approaches:
categorical and dimensional. Speech acoustic segments can be treated as a
categorical entity consisting of discrete emotions such as happy, sad, fear,
etc [1]. In the categorical approach, annotators label audio segments as
emotion categories and use them to model speech emotion. The dimensional
approach proposes two fundamental dimensions, valence and arousal, to
represent emotion at a given time [2].

Typically, when a speech emotion corpora is created, each audio segment is
labelled as a specific emotion category by the annotators, and it is assumed
that the whole audio segment signifies that single emotion label [3, 4, 5]. It
is theorised that the perceptual cues for phone boundaries and acoustic
context are ambiguous as they share information for various emotion states [6,
7]. The acoustic stimuli change in speech segments are distributed events and
can therefore overlap. From a psycholinguistic perspective, these distributed,
continuous stimuli transitions constitute theories of human perception of SER
[6, 7]. The context cues can be of different lengths, and the perceptual
acoustic context can be modelled with different length acoustic cues. Work
from [8] shows that speech emotion can be modelled with small acoustic cues
(200 ms). Therefore, the assumption that each acoustic speech segment is
attributed to only one emotion state likely negatively impacts recognition
performance. Multiple sub-emotions can be present depending on the contextual
variation between different segment regions, as shown in Figure 1. This paper
focuses on acoustic perceptual cues and the implication of the length and the
distribution of these cues over speech audio segments for SER.

Figure 1: An example of distributed emotions where labelling an utterance as a
single discrete category could be overlooking other perceived emotions

Previous research for SER mainly focuses on modelling generalised emotion with
different neural network architectures while adapting to speech variability
and reducing redundancy for speaker invariance to improve SER capability [9,
10, 11, 12, 13]. As the focus of current research has shifted towards
embedding modelling and left-right context cues, work by [14] proposed a
spatial representation learning method with CNNs, to model mid to long-term
sequence dependencies. After the advent of the transformer architecture, the
SER models focused more on transformer-based and multi-head fusion-based
modelling approaches [15, 16].

There remains a gap between the psycholinguistic and cognitive theories
regarding speech emotion perceptual cues and the currently developed
computational modelling methods. Research focusing on interpretability is
still underdeveloped for SER models, particularly where the model’s internal
intricacies and representations with the corresponding acoustic segments can
be explained. This work attempts to find a mutual accord with the theories of
speech emotion perception cues across multiple disciplines and bridge the gap
to speech emotion models. By projecting model attention weights across
different time frames (based on various acoustic cues) of the acoustic
segment, the emotion classification is observed to shift. Several corpora have
been considered to demonstrate the task across various types of speech emotion
data (acted, natural, elicited).

This manuscript is organised as follows. Section I introduces the research
domain and the particular research goal of the work, along with some
background information. Section II discusses context modelling and introduces
the idea of overlapping context regions and phone units. Section III presents
the consonant-vowel (CV) boundaries and phonemic overlapped regions and their
significance in speech emotion perception cues and recognition. Section IV
explains the underlying SER model for the interpretation framework and
attention. Section V describes the cross-corpus data, features, experimental
framework, and presents the results and graphs. Section VII discusses the
interpretation of the presented results and suggested directions for the
development of future work.

##  II Context Modelling

Context cues for speech emotion can be described as linguistic and
paralinguistic. The linguistic aspects consist of semantic structure of the
speech segment and the textual meaning. The nonverbal or paralinguistic
aspects provide a rich source of perceptual context cues that facilitates
projecting expressiveness in social discourse in both intra-cultural and
cross-cultural scenarios [17, 18]. Although verbal comprehension mainly
dictates social discourse, perceptual context cues can deliver meaning and
emotion independent of the verbal comprehension using the acoustic changes
that influence the speech delivery [19, 20]. Work in [6] used psychoacoustic
features (such as tempo, prosodic contour, loudness etc.) for modelling
emotion and concluded that different emotional states have different
perceptual cues and that they are subjective to individual contexts despite
having a universal representation of emotion states. Furthermore, the acoustic
contexts are not orthogonal, and the shared information/dimensions represents
the redundant acoustic stimuli which provide context [6, 7]. Naturally, if the
acoustic stimuli changes, the perceptual context cue will also change
accordingly. If the acoustic stimuli are redundant for the cues that define
emotion states, these stimuli share overlapping regions. Typically, a ‘phone’
is regarded as one of the smallest units of an acoustic speech sound. To
explore the implication of the various stimuli regions, the phone boundaries
should be explored. The CV boundaries for context cues are discussed in
Section III.

The authors in [21] have presented left context (referred to as “forward
effects” by the authors), right context (referred to as “backward effects” by
the authors), proximal context and distal acoustic context cues as a in
acoustic events over time. The sensory attention emphasises the change among
these acoustic stimuli, which maximises the potential information for
facilitating speech perception [22]. The stimuli changes at a particular time
over left-right time frames to reflect the emotion state and speech perception
cue at that given point of time. Therefore, it can be assumed that emotion is
a distributed event in acoustic segments, not a single discrete emotion
category. To investigate this hypothesis, a simple computational model of
left-right modelling with attention has been applied in Section IV.

##  III Linguistic Boundaries

Contextual cues, consisting of phonetic aspects for speech, can be used to aid
the determination of the emotional state at a given time. The phonological
forms can have similarities and dissimilarities among the phone boundaries. A
clear distinction has been found between the clusters of vowel and consonant
phone datapoints by work from [23, 24]. The consonant phones play a decisive
role in word meaning comprehension, such that removing initial prosodic
variations in vowel phones (acoustic reduction) enhances word intelligibility
[23]. However, contrasting studies showed that replacing intermittent
consonants with noises or change in emphasis on vowels, increases the
perceived intelligibility of words and sentences to human listeners [25, 26].
It is argued that vowel phones are more responsible for defining the emotional
state of the speech acoustics, and intelligibility due to stressed vowel
regions and wide harmonic variations [24, 27, 28].

Furthermore, the harmonic variations and variations in the pitch within
vowels, change the CV boundaries over time and contextual cues related to
acoustic perception. These continuous perceptual context cues are distributed
over CV boundaries in acoustic segments [27]. Thus it may be possible that at
different left-right time-frames, different regions from the same acoustic
segment may be categorised differently. This can be described as the
relationship between perceptual CV cues with the acoustics, which has been
referred to as acoustic-phonetic context for speech perception [29, 30]. The
aim of this work is to understand the distributed nature of these perceptual
acoustic cues which form intra-linguistic determinism between acoustic
structure and meaning that humans perceive as emotion. Here, meaning and
intelligibility are explored only from acoustic segments as no language model
or external multi-modal data has been used.

##  IV Model architecture: BLSTMATT

Figure 2: The BLSTMATT model pipeline consists of 2 bidirectional-LSTM layers,
with an attention layer and linear classifier

The focus of this work is to explore perceptual acoustic cues and their
relationship to current speech emotion recognition modelling. Developing and
training large-scale SER models is out of the scope of this work as this
approach is to determine the concept of this relationship. As the previously
discussed related theories, regarding speech emotion perception, take into
account past and future context, this can be modelled as a form of left and
right acoustic cues. The chosen modelling approach utilises a bidirectional
long short term memory (LSTM) neural network with a subsequent attention
layer, referred to as BLSTMATT. An overview of the model structure is
displayed in Figure 2.

LSTM networks are unable to exploit the future context and instead they solely
focus on the temporal order of the sequence, whereas bidirectional LSTMs [31]
comprise of an additional layer of hidden connections which allows temporal
information to pass in the opposite direction in order to exploit future and
past contextual information [32]. The hidden connections $\mathbf{h}^{n}$ are
iteratively compiled:

$h^{n}_{t}=\mathcal{H}(\mathcal{W}_{h^{n-1}h^{n}}h_{t}^{n-1}+\mathcal{W}_{h^{n}h^{n}}h^{n}_{t-1}+b_{h}^{n})$
(1)

where $\mathcal{W}$ defines the weight matrices, $\mathcal{H}$ represents the
hidden layer function and $b$ refers to the bias vector. Using this approach,
temporal feature distribution over the sequence can be obtained, which is more
effective for SER tasks [33].

The attention mechanism enables computation of longer-term inter-sequence
dependencies. The additive method for computing attention from [34] is applied
for this approach. Utilising the global mean, the attention mechanism enables
the network to attend to specific parts of itself which in turn captures
global information. The non-linearity $tanh$ is used to multiply the global
mean over the whole temporal vector which computes the positional dependency
of each element. $\mathcal{H}$ denotes the matrix of output vectors from the
LSTM layer, by summing the average time of $\mathcal{H}$ across contextual
modalities, the shared memory matrix $\mathcal{M}$ can be formed by repetition
until it matches the dimension of $\mathcal{H}_{s}$, where $\mathbf{s}$ refers
to the view for the context. Where $T$ refers to iterations, $\mathcal{V}$
denotes the parameters controlling the influence within the view and from the
shared memory:

$\gamma^{(\tau)}=tanh(\mathcal{V}^{(\tau)}_{s1}tanh(\mathcal{H}_{s1}))\cdot
tanh(\mathcal{V}^{(\tau)}_{s2}\mathcal{M}^{(\tau}))$ (2)
$\alpha^{(\tau)}_{s}=\mathcal{V}^{(\tau)T}_{s3}\gamma^{(\tau)}$ (3)

$\mathcal{V}_{s1}$, $\mathcal{V}_{s2}$ and $\mathcal{V}_{s3}$ are parameters
used to compute the attention strength $\alpha$.

The BLSTMATT model setup consists of 2 x 512 dimension hidden layers feeding
into an attention layer, which computes a 128 dimension context vector. For
classification, the network uses a fully-connected linear layer which projects
the attention output. In order to classify over the number of emotions in the
target, the output is normalised with a $softmax$ layer before the loss is
computed.

##  V Experiments

###  V-A Data

The scope for these experiments regards English speaking adult datasets across
three emotion types: one acted dataset, eNTERFACE [3], one natural dataset,
MOSEI [5], and one elicited dataset, IEMOCAP [4]. An overview of the emotion
classifications represented in each dataset are each described briefly below.
For each dataset, the big-six emotions [1] are considered in training and
testing: happy, sad, anger, surprise, disgust and fear.

eNTERFACE (ENT) consists of roughly 1 hour of acted English utterances [3].
The training set is comprised of 38 speakers and the testing set contains the
remaining 5 speakers. The data is split by 8 female speakers and 35 male
speakers from 14 different nations.

IEMOCAP (IEM6) comprises of over 12 hours of US-English utterances from 10
speakers (5 female and 5 male) [4]. There are five dyadic sessions (between
two speakers) which are specifically scripted or contrived to elicit certain
emotions. The training data consists of the first 4 sessions (4 speakers) and
the last session is split for the test set (2 speakers). It is common for
IEMOCAP to be evaluated as four classes: happy, sad, anger and neutral (where
excitement is combined with happy). This test set will be referred to as IEM4.

MOSEI (MOS) is the largest sentiment and emotion dataset with approximately 65
hours of data and more than 1000 speakers [5]. Data is collected from YouTube
and the videos are not specifically designed as an emotion dataset so the
emotional speech is seen as natural. The official training, validation and
test splits for the ACL 2018 conference have been considered, where the
training and validation sets are combined for training. These can be found at
https://github.com/A2Zadeh/CMU-
MultimodalSDK/blob/master/mmsdk/mmdatasdk/dataset/standard_datasets/CMU_MOSEI/cmu_mosei_std_folds.py.

###  V-B Features

Experiments from [35] showed how sequence-based SER systems performed best in
terms of unweighted and weighted accuracy with 23-dimensional log-Mel
filterbank features.

###  V-C Implementation

The BLSTMATT contains two hidden layers of 512 nodes each. The output layer
(size 1024) is passed into the attention mechanism computing a context vector
(size 128), which is projected to 1024 nodes. This is then fed into the
emotion classifier which linearly projects to the 6 classes. The cross-entropy
loss function is applied, which is preceded by a $softmax$ layer. The BLSTMATT
produces a variable length attention vector based on the input segment length,
as mentioned in section IV. The attention vectors have been extracted and
mapped with the phones and words in the input segments to be able to interpret
the acoustic attention.

###  V-D Evaluation

Unweighted accuracy (UA) and the weighted accuracy (WA) are the metrics
typically applied for SER evaluation. The UA calculates accuracy in terms of
the total correct predictions divided by total samples, which gives the same
weight to each class:

$UA=\dfrac{TP+TN}{P+N}$ (4)

where $P$ is the number of correct positive instances (equivalent to $TP+FN$)
and $N$ is the number of correct negative instances (equivalent to $TN+FP$).
As some of the datasets are imbalanced across the emotion classes, see Tables
III, III and III, the WA is calculated which weighs each class according to
the number of samples in that class:

$WA=\dfrac{1}{2}(\dfrac{TP}{P}+\dfrac{TN}{N})$ (5)

Further details regarding the implementation of the scoring scripts can be
found in [34].

###  V-E Acoustic Context

As discussed in Section II and III, the recognition of speech emotion is
hypothesised to be influenced by overlapping perceptual acoustic cues
consisting of variation in the phone boundaries. So, in theory, if the phone
boundaries are shifted, the emotion classification may differ from the
previous predicted emotion state that considered the whole segment. To further
explore this hypothesis, the acoustic context is changed in the following
series of experiments.

Experiments are performed removing frames from the end and beginning of the
original, whole test segments. In Tables III, III and III, this is listed in
the first column labelled ‘skip frames (left-right)’ where a number of frames
are skipped, or removed, from the left and right (left and right context) of
each test segment. For example, 20-200 means 20 frames have been removed from
the left context of each test segment and 200 frames have been removed from
the right context of each test segment. Table III shows the results where
right frames are skipped, Table III shows results where only left frames are
skipped and Table III shows results where both left and right frames are
skipped. If the length of a test utterance is less than the length of context
frames, the test utterance remains unchanged. Therefore, when the skip context
frames become longer, such as 200-100 (that means a total of 300 frames to be
removed), only the test segments with more frames than 300 are used. The
percentage of test corpora that is modified with the context is also reported.
For example, in the SEGS% column, 91.3% means that 8.7% of the test segments
from the corresponding corpora remains the same due to shorter segment length
and 91.3% of the test segments are modified with the corresponding context.
The weighted and unweighted accuracy are reported along with the change in the
context length.

As the experiments consider context length variations, the baseline for this
work is the result when no left or right context is removed. This is the first
line in all Tables with context 0-0. It is the emotion modelling baseline
where one emotion is given for each complete test utterance. For further
details about the validity of the BLSTMATT model, please see work in [35] and
[8].

TABLE I: Cross-corpora results with variable context length, where right
frames are skipped.

| Skip
---
Frames
(left-right)
Unweighted Accuracy (UA %) | Weighted Accuracy (WA %) |  | Percentage of segments (SEGS %)
---
with modified context
Context | ENT | IEM6 | IEM4 | MOS | ENT | IEM6 | IEM4 | MOS | ENT | IEM6 | IEM4 | MOS
0-0 | 93.33 | 69.06 | 88.79 | 73.30 | 88.00 | 64.57 | 63.81 | 54.29 | - | - | - | -
0-30 | 86.89 | 68.73 | 88.28 | 73.55 | 76.40 | 63.82 | 64.44 | 54.76 | 100.0 | 100.0 | 100.0 | 100.0
0-100 | 82.22 | 68.73 | 87.94 | 72.81 | 68.00 | 63.45 | 61.21 | 54.70 | 91.3 | 99.3 | 99.5 | 98.0
0-200 | 86.22 | 68.09 | 86.63 | 71.53 | 75.20 | 62.08 | 61.32 | 53.63 | 40.0 | 77.1 | 80.2 | 89.4

TABLE II: Cross-corpora results with variable context length, where left
frames are skipped.

| Skip
---
Frames
(left-right)
Unweighted Accuracy (UA %) | Weighted Accuracy (WA %) |  | Percentage of segments (SEGS %)
---
with modified context
Context | ENT | IEM6 | IEM4 | MOS | ENT | IEM6 | IEM4 | MOS | ENT | IEM6 | IEM4 | MOS
0-0 | 93.33 | 69.06 | 88.79 | 73.30 | 88.00 | 64.57 | 63.81 | 54.29 | - | - | - | -
20-0 | 91.56 | 69.22 | 88.74 | 73.01 | 84.80 | 64.76 | 63.83 | 54.13 | 100.0 | 100.0 | 100.0 | 100.0
30-0 | 89.33 | 69.66 | 88.68 | 72.97 | 80.80 | 65.03 | 63.83 | 54.18 | 100.0 | 100.0 | 100.0 | 100.0
100-0 | 84.44 | 69.90 | 87.66 | 72.50 | 72.00 | 64.60 | 61.00 | 54.16 | 91.3 | 99.3 | 99.5 | 98.0
200-0 | 87.78 | 69.38 | 87.14 | 71.98 | 78.00 | 63.47 | 61.97 | 54.07 | 40.0 | 77.1 | 80.2 | 89.4
300-0 | 92.89 | 68.61 | 86.86 | 71.11 | 87.20 | 62.89 | 61.66 | 53.45 | 8.7 | 54.5 | 57.8 | 80.2

TABLE III: Cross-corpora results with variable context length, where left and
right frames are skipped.

| Skip
---
Frames
(left-right)
Unweighted Accuracy (UA %) | Weighted Accuracy (WA %) |  | Percentage of segments (SEGS %)
---
with modified context
Context | ENT | IEM6 | IEM4 | MOS | ENT | IEM6 | IEM4 | MOS | ENT | IEM6 | IEM4 | MOS
0-0 | 93.33 | 69.06 | 88.79 | 73.30 | 88.00 | 64.57 | 63.81 | 54.29 | - | - | - | -
20-200 | 86.89 | 68.73 | 87.49 | 71.39 | 76.40 | 63.11 | 62.44 | 53.81 | 38.7 | 71.6 | 73.4 | 87.6
200-100 | 92.67 | 68.25 | 86.58 | 71.30 | 86.80 | 61.83 | 61.42 | 54.00 | 8.7 | 54.5 | 57.8 | 80.2

Figure 3: A happy MOS utterance with no context removed mislabelled as sad
compared to 20 left frames removed correctly labelled as happy, along with the
pitch contour  Figure 4: A happy IEM4 utterance with no context removed
mislabelled as anger compared to 100 right frames removed correctly labelled
as happy, along with the pitch contour

##  VI Results

The experimental results in Tables III, III and III suggest that the SER
results would change when either the left, right or both contexts are changed.
For example, the model tested on MOS has a UA of 73.3% without changing the
context length but upon skipping the context right 100 frames, skip frames
0-100 in Table III, the UA degrades. The same observation of UA degradation
occurs when skipping left frames or both left and right frames, while there is
a slight improvement in UA when skipping 30 right frames. In the case of skip
frames 0-30, removing 30 ms from the end of the segment modifies 100% of the
segments across all the testsets. The results for ENT and IEM4 are worse for
both UA and WA, but for MOS the performance improves. For IEM6, the UA
degrades whereas the WA improves. The majority of the results across all the
datasets degrade upon varying the context length due to the target label,
supplied with those segments, being a fixed discrete emotion category. This
finding corroborates the initial hypotheses that speech emotion is not a fixed
entity that remains the same over the whole audio segment, and that it is
subject to be distributed over different overlapping shorter context queues.

To observe the relationship between the SER results and the hypotheses
regarding the acoustic segments in more detail, the attention weights were
extracted for each test utterance and mapped to the aligned words and phones.
Additionally, the pitch contour was calculated to understand the pitch
correlation with respect to the prosodic utterance using the algorithm found
at https://github.com/google/REAPER. The attention maps for a sample of the
test utterances are presented in Figures 3 and 4: the former from the MOSEI
corpus and the latter from the IEMOCAP corpus. Figure 3 shows that the
attention projection drifts while changing the phone boundaries from the same
audio segment and therefore the emotion state also changes. With context 0-0,
the model incorrectly predicts the emotion sad (attention weights in Figure 3
indicated by red line) whereas removing 20 left frames helps the model
correctly predict the emotion happy (attention weights in Figure 4 indicated
by blue line). The attention weights focus more strongly on different portions
of the test utterance. Similar behaviour can be seen in Figure 4 where
skipping 100 left frames allows the model to make the correct prediction.

##  VII Discussion and Conclusions

In some datasets, such as ENT and IEM4, the SER change is not very prominent,
as can be seen from the results listed for WA and UA. This is potentially due
to several reasons. The first is that some segments in particular datasets
have a length shorter than 200 frames, and these segments remained unchanged
during the context modification. So to properly interpret the results, the
percentage of data that is modified with context at each experiment should be
taken into account. Secondly, the acoustic BLSTMATT model perception is a
direct result of the relationship between the training data and the
corresponding labels given by the annotators, which could have added bias
factors and add to recognition uncertainty. To attempt to mitigate the
inherent biases and to attempt to generalise the model perception cues for
these experiments, the model is trained with four different corpora consisting
of acted, natural and elicited emotions. Consequently, it is argued that the
results corroborate the argument that an continuous approach to emotion
recognition is the optimal strategy based on observed acoustic stimuli shift.
This work is an attempt to bridge the gap that current SER models have, by
explaining the SER model’s internal intricacies and how the representations
correspond with acoustic segments.

Figures 3 and 4 show the attention weight propensity towards the vowel based
regions. This corroborates with the claims from the linguistic and cognitive
theories about speech emotion recognition and CV boundaries, as proposed in
Section III, that consonants play a decisive role in word meaning but vowels
are more responsible for the emotion perception cues as a result of harmonic
variations and stressed regions. The vowels are observed to change the CV
boundaries and the context cues for emotion perception causing many hard
boundaries to be redundant. This suggests the cues for phone boundaries and
acoustic context can share information relative to the perceived emotion
state.

For the IEM6 dataset, when the context lengths were skipped left frames, there
was a slight improvement in UA or WA, while recognition with the MOS and IEM4
datasets improved slightly when context lengths were skipped right frames.
These results highlight that where context cues vary in length, it is possible
for the acoustic segments to contain more than one distinct emotion state. As
the UA and WA vary positively and negatively according to context lengths,
this suggests overlapping regions where the acoustic stimuli are more or less
informative regarding the emotion state. As future speech emotion datasets are
compiled and annotated, if the labels for emotion classes were adjusted to
allow for overlapping categories, this could potentially aid the recognition
performance of current and future developed models. These results and insights
can also be used to modify computational models and mechanisms that are able
to adapt and recognise emotion from various speech domains to be more in-line
with the psycholinguistic theories.

In the current trend of SER models, emotion labels are treated as discrete
labels attributed over a whole segment. The problem explored in this research,
suggests that this approach assumes that an utterance’s global attributes
correlate with the local characteristics over different time frames in the
same segment for learning one discrete emotion category. This is observed to
not be the case most of the time. Vowel-consonant envelopes rapidly change
over time, attributing to different acoustic context. Hence the paralinguistic
cue also changes with acoustic context. The results listed in Section VI
demonstrate this argument. Moreover, by treating acoustic segments and emotion
correspondence as a context-oriented continuous relationship, this should aid
emotion recognition models across languages and dialects due to the
distribution of acoustic boundaries across models trained on various emotion
data. As a result, it could be possible to learn the variability of acoustic
context in speech emotions rather than the variability of acoustic segments in
speech emotions.

Future development of this framework will enable improved emotion modelling by
understanding the intermediate representations and relating audio data with
the computational models. Furthermore, it will help create more accurate
annotations for emotion labels, improving SER corpora generation.

This work argues that discrete categorical emotion classification should not
be the preferred approach to develop future SER models as it has been observed
that emotion cues present as a distributed event, corroborating directly with
cognitive linguistic theory that it is also continuous to recognise. Finding a
suitable approach for accurate modelling of emotion states should be the aim
of future research.

## References

  * [1] P. Ekman, “An argument for basic emotions,” _Cognition & emotion_, vol. 6, no. 3-4, pp. 169–200, 1992.
  * [2] J. A. Russell, “A circumplex model of affect.” _Journal of personality and social psychology_ , vol. 39, no. 6, p. 1161, 1980.
  * [3] O. Martin, I. Kotsia, B. Macq, and I. Pitas, “The enterface’05 audio-visual emotion database,” in _International Conference on Data Engineering Workshops, ICDE, Atlanta, GA, USA, Apr 3-7_ , 2006, p. 8. [Online]. Available: https://doi.org/10.1109/ICDEW.2006.145
  * [4] C. Busso, M. Bulut, C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. Narayanan, “IEMOCAP: interactive emotional dyadic motion capture database,” _Language Resources and Evaluation_ , vol. 42, no. 4, pp. 335–359, 2008. [Online]. Available: https://doi.org/10.1007/s10579-008-9076-6
  * [5] A. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L. Morency, “Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph,” in _Annual Meeting of the Association for Computational Linguistics, ACL, Melbourne, Australia, Jul 15-20_ , 2018, pp. 2236–2246. [Online]. Available: https://www.aclweb.org/anthology/P18-1208/
  * [6] E. Coutinho and N. Dibben, “Psychoacoustic cues to emotion in speech prosody and music,” _Cognition and Emotion_ , vol. 27, no. 4, pp. 658–684, 2013, pMID: 23057507.
  * [7] G. Ilie and W. F. Thompson, “Experiential and cognitive changes following seven minutes exposure to music and speech,” _Music Perception_ , vol. 28, no. 3, pp. 247–264, 2011.
  * [8] M. A. Jalal, R. Milner, and T. Hain, “Empirical interpretation of speech emotion perception with attention based model for speech emotion recognition,” in _Proceedings of Interspeech 2020_. International Speech Communication Association (ISCA), 2020, pp. 4113–4117.
  * [9] B. Schuller, G. Rigoll, and M. Lang, “Hidden Markov model-based speech emotion recognition,” _2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP ’03)._ , vol. 2, pp. 401–404, 2003.
  * [10] S. Zhang, T. Huang, and W. Gao, “Multimodal Deep Convolutional Neural Network for Audio-Visual Emotion Recognition,” in _Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval - ICMR ’16_ , 2016, pp. 281–284.
  * [11] S. Mirsamadi, E. Barsoum, and C. Zhang, “Automatic speech emotion recognition using recurrent neural networks with local attention,” in _2017 IEEE International conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2017, pp. 2227–2231.
  * [12] C.-W. Huang and S. S. Narayanan, “Attention assisted discovery of sub-utterance structure in speech emotion recognition.” in _Interspeech_ , 2016, pp. 1387–1391.
  * [13] Z. Huang, M. Dong, Q. Mao, and Y. Zhan, “Speech emotion recognition using cnn,” in _Proceedings of the 22nd ACM international conference on Multimedia_ , 2014, pp. 801–804.
  * [14] J. Kim, G. Englebienne, K. P. Truong, and V. Evers, “Deep temporal models using identity skip-connections for speech emotion recognition,” in _Proceedings of the 25th ACM international conference on Multimedia_ , 2017, pp. 1006–1013.
  * [15] A. Nediyanchath, P. Paramasivam, and P. Yenigalla, “Multi-head attention for speech emotion recognition with auxiliary learning of gender recognition,” in _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 7179–7183.
  * [16] Z. Lian, B. Liu, and J. Tao, “Ctnet: Conversational transformer network for emotion recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 985–1000, 2021.
  * [17] D. Wilson and T. Wharton, “Relevance and prosody,” _Journal of pragmatics_ , vol. 38, no. 10, pp. 1559–1579, 2006.
  * [18] K. Hoemann, A. N. Crittenden, S. Msafiri, Q. Liu, C. Li, D. Roberson, G. A. Ruark, M. Gendron, and L. Feldman Barrett, “Context facilitates performance on a classic cross-cultural emotion perception task.” _Emotion_ , vol. 19, no. 7, p. 1292, 2019.
  * [19] K. R. Scherer, R. Banse, and H. G. Wallbott, “Emotion inferences from vocal expression correlate across languages and cultures,” _Journal of Cross-cultural psychology_ , vol. 32, no. 1, pp. 76–92, 2001.
  * [20] W. F. Thompson and L.-L. Balkwill, “Decoding speech prosody in five languages,” vol. 2006, no. 158, pp. 407–424, 2006. [Online]. Available: https://doi.org/10.1515/SEM.2006.017
  * [21] C. Stilp, “Acoustic context effects in speech perception,” _Wiley Interdisciplinary Reviews: Cognitive Science_ , vol. 11, no. 1, p. e1517, 2020\.
  * [22] K. R. Kluender, C. E. Stilp, and F. L. Lucas, “Long-standing problems in speech perception dissolve within an information-theoretic perspective,” _Attention, Perception, & Psychophysics_, vol. 81, no. 4, pp. 861–883, 2019\.
  * [23] M. Van de Ven and M. Ernestus, “The role of segmental and durational cues in the processing of reduced words,” _Language and Speech_ , vol. 61, no. 3, pp. 358–383, 2018.
  * [24] D. Fogerty and D. Kewley-Port, “Perceptual contributions of the consonant-vowel boundary to sentence intelligibility,” _The Journal of the Acoustical Society of America_ , vol. 126, no. 2, pp. 847–857, 2009.
  * [25] M. J. Owren and G. C. Cardillo, “The relative roles of vowels and consonants in discriminating talker identity versus word meaning,” _The Journal of the Acoustical Society of America_ , vol. 119, no. 3, pp. 1727–1739, 2006.
  * [26] R. A. Cole, Y. Yan, B. Mak, M. Fanty, and T. Bailey, “The contribution of consonants versus vowels to word recognition in fluent speech,” in _1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings_ , vol. 2. IEEE, 1996, pp. 853–856.
  * [27] P. Ladefoged and D. Broadbent, “Information conveyed by vowels,” _Journal of the Acoustical Society of America_ , 1957.
  * [28] D. Xu, F. Chen, F. Pan, and D. Zheng, “Factors affecting the intelligibility of high-intensity-level-based speech,” _The Journal of the Acoustical Society of America_ , vol. 146, no. 2, pp. EL151–EL157, 2019.
  * [29] F. S. Cooper, P. C. Delattre, A. M. Liberman, J. M. Borst, and L. J. Gerstman, “Some experiments on the perception of synthetic speech sounds,” _The Journal of the Acoustical Society of America_ , 1952.
  * [30] J. L. Miller, “On the internal structure of phonetic categories: A progress report,” _Cognition_ , vol. 50, no. 1-3, pp. 271–285, 1994.
  * [31] A. Graves, N. Jaitly, and A. Mohamed, “Hybrid speech recognition with deep bidirectional LSTM,” in _IEEE Workshop on Automatic Speech Recognition and Understanding, Olomouc, Czech Republic, Dec 8-12,_ , 2013, pp. 273–278. [Online]. Available: https://doi.org/10.1109/ASRU.2013.6707742
  * [32] M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” _IEEE Trans. Signal Processing_ , vol. 45, no. 11, pp. 2673–2681, 1997\. [Online]. Available: https://doi.org/10.1109/78.650093
  * [33] T. Mani Kumar, E. Sanchez, G. Tzimiropoulos, T. Giesbrecht, and M. Valstar, “Stochastic process regression for cross-cultural speech emotion recognition,” _Proc. Interspeech 2021_ , pp. 3390–3394, 2021.
  * [34] R. Beard, R. Das, R. W. Ng, P. K. Gopalakrishnan, L. Eerens, P. Swietojanski, and O. Miksik, “Multi-modal sequence fusion via recursive attention for emotion recognition,” in _Proceedings of the 22nd conference on computational natural language learning_ , 2018, pp. 251–259.
  * [35] R. Milner, M. A. Jalal, R. W. Ng, and T. Hain, “A cross-corpus study on speech emotion recognition,” in _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2019, pp. 304–311.