# Unifying the Discrete and Continuous Emotion labels for Speech Emotion
Recognition

###### Abstract

Traditionally, in paralinguistic analysis for emotion detection from speech,
emotions have been identified with discrete or dimensional (continuous-valued)
labels. Accordingly, models that have been proposed for emotion detection use
one or the other of these label types. However, psychologists like Russell and
Plutchik have proposed theories and models that unite these views, maintaining
that these representations have shared and complementary information. This
paper is an attempt to validate these viewpoints computationally. To this end,
we propose a model to jointly predict continuous and discrete emotional
attributes and show how the relationship between these can be utilized to
improve the robustness and performance of emotion recognition tasks. Our
approach comprises multi-task and hierarchical multi-task learning frameworks
that jointly model the relationships between continuous-valued and discrete
emotion labels. Experimental results on two widely used datasets (IEMOCAP and
MSPPodcast) for speech-based emotion recognition show that our model results
in statistically significant improvements in performance over strong baselines
with non-unified approaches. We also demonstrate that using one type of label
(discrete or continuous-valued) for training improves recognition performance
in tasks that use the other type of label. Experimental results and reasoning
for this approach (called mis-matched training approach) are also presented.

Index Terms—  speech emotion recognition, discrete and continuous labels,
multi-task, hierarchical multi-task

##  1 Introduction

Emotion is a complex entity which can be modelled in different ways. When
defined as a motor response to stimuli, it can be viewed as belonging to a
discrete set; whereas when considered as a subjective feeling, it can be
expressed as a continuous vector in multiple dimensions [1]. Though these two
definitions of emotion appear to be different, they have shared and
complementary information. Psychological models like Plutchik’s wheel of
emotions [2] or Russell’s “arousal-valence” space [3] represent discrete
emotions on and within a circle on a continuous “arousal-valence” plane,
demonstrating that the values of “arousal” and “valence” are correlated to the
perception of discrete emotion.

Speech Emotion Recognition(SER), which refers to the task of identifying the
emotional state of the speaker using speech as input, can be used to predict
both discrete or continuous emotions. The most commonly used continuous
representation for this task comprises three attributes – Valence (V), Arousal
(A), and Dominance(D) (from Russell [4]). Among these three dimensions,
Valence represents the pleasantness of the emotion, Arousal denotes the
intensity of it, and Dominance represents the degree of control over a social
situation. Similarly, a number of discrete emotions have been identified –
happy, sad, angry etc. Translating between these two forms of emotion
representations has been of interest [5, 6, 7, 8], both with audio and textual
input. However conflicting results have been found: while some studies[6] find
that there is little correlation between continuous values and discrete
labels, others [8] find correlations like ‘anger has higher arousal and lower
valence’. Based on the notion that discrete and continuous emotion attributes
are likely related, in this paper, we propose to examine such dependency
relationships.

In order to investigate the relationship between continuous and discrete
attributes, we develop neural network models that jointly predict continuous
and discrete emotion labels. The motivation behind this is two-fold: (a)
holistic models for automatic emotion recognition must be able to capture both
generic and specific notions of emotion contained in the continuous and
discrete emotion labels, and (b) since discrete and continuous emotion labels
may be related to each other, robust and generalizable machine learning models
can be built by leveraging this dependency. We propose a multi-task learning
framework where continuous and discrete emotion labels are predicted together,
but independently of each other. Next, we consider the possibility that
knowledge of the discrete emotion might help predict more accurately the
continuous emotion and vice versa and hence introduce hierarchical multi-task
models that model such a relationship to jointly predict discrete and
continuous emotion.

We further demonstrate that these continuous and discrete labels need not
necessarily be manually annotated within the same corpus to be improve
recognition performance. Of the prevailing annotated datasets for emotion
recognition, very few are annotated for both continuous and discrete
attributes such as MSPPodcast [9], and IEMOCAP[10] while others like MELD[11]
are annotated for discrete attributes only. We demonstrate that even if the
model is trained on continuous emotion labels from MSPPodcast and discrete
labels from IEMOCAP, the proposed Hierarchical Multi-Task approach improves
performance and generalizability. This implies that emotion recognition
datasets can be trained jointly on multiple corpora with different labels. In
summary, this paper makes the following contributions:

  1. 1.

We propose a multi-task learning framework that jointly predicts the
continuous and discrete labels from speech

  2. 2.

We extend this framework to model hierarchical dependencies, where knowledge
of discrete attributes aids continuous prediction and vice versa.

  3. 3.

We demonstrate that the proposed approach can be used when the mis-matched
labels (continuous and discrete) are drawn from different datasets.

##  2 Proposed Approach

Fig. 1: The figure shows model architectures used in this work. The top left
shows the Baseline D model. Baseline C model is similar (not shown) with
intermediate $E_{C}$ and final continuous prediction $\hat{c}$. The top middle
shows the Multi-task model. Top right and bottom figure show the Hierarchical
multi-task model D-C and C-D respectively. SAP stands for self-attention
pooling. MLP stands for multi-layer perceptron. $\hat{y}$ is the discrete
label prediction whereas $\hat{c}$ is the 3-dimensional continuous emotion
prediction.

###  2.1 Encoder Decoder Architecture

To perform SER on input speech, we use an encoder-decoder architecture for the
task. The encoder takes as input a sequence of $N$ content based speech
features $X=[x_{1},x_{2},\cdots x_{N}]$, and produces as its output a sequence
of hidden representations $H=[h_{1},h_{2},....h_{M}]$. These hidden
representations are the input to a decoder that either predicts continuous
emotion $\hat{c}$, discrete emotion $\hat{y}$, or both.

The decoder comprises a temporal self-attentive pooling layer, followed by a
Multi-layer Perceptron (MLP) that extracts task specific embeddings, i.e.,
$E_{D}$ for discrete prediction and $E_{C}$ for continuous prediction. These
embeddings are then passed through the final classification layer that maps to
3 dimensions for continuous prediction (corresponding to V,A,D), or 5
dimensions for discrete prediction (corresponding to the number of discrete
emotion classes). The continuous and discrete baseline models use a similar
neural network architecture, except for the number of output neurons.

Each of the models we describe below, baseline models that perform either
continuous or discrete prediction, and multi-task or hierarchical multi-task
models that predict both discrete and continuous emotion, share the same
encoder structure but with slightly different decoder architectures.

###  2.2 Baseline Models

Baseline Discrete Model
To _independently_ predict the discrete attributes from input $X$, our
encoder-decoder model termed as Baseline D uses a decoder with discrete self-
attentive pooling, MLP which generates the discrete embedding $E_{D}$, and a
discrete classification layer that produces the discrete label $\hat{y}$ as
its output. The model is shown in Fig. 1 (top lefT).

Given the true discrete label $y$, the model is optimized using the multi-
class cross entropy criterion. For a dataset with $A$ utterances, the cross-
entropy loss is computed as:

$\mathcal{L_{\text{disc}}}=-\sum_{i=1}^{A}\sum_{c=1}^{K}y_{i,c}\log(\hat{y}_{i,c})$
(1)

Baseline Continuous Model
To predict the continuous attribute labels _independently_ from the input
speech, our encoder-decoder model, called Baseline C uses a continuous emotion
pooling layer and MLP to predict the continuous emotion
$\hat{c}=[\hat{c}_{Val},\hat{c}_{Aro},\hat{c}_{Dom}]$.

The variables $\hat{c}_{Val},\hat{c}_{Aro},$ and $\hat{c}_{Dom}$ correspond to
the valence, arousal and dominance predictions respectively.

This model is optimized using the Concordance Correlation Coefficient (CCC)
Loss. Given the prediction $\hat{c}$ and the ground truth $c$, the CCC loss is
defined as shown in Equation 2.

$\mathcal{L_{\text{ccc}}}=1-\dfrac{2s_{c\hat{c}}}{s_{c}^{2}+s_{\hat{c}}^{2}+(\bar{c}-\bar{\hat{c}})^{2}}$
(2)

where $s_{c\hat{c}},s_{c}^{2},s_{\hat{c}}^{2},\bar{c},\bar{\hat{c}}$ represent
the covariance between the ground truth and prediction, variance of the ground
truth, variance of the prediction, mean of the groundtruth and mean of the
prediction respectively.The resulting loss for continuous emotion recognition
can be written as the sum of the CCC losses for valence, arousal and dominance
prediction.

###  2.3 Multi-Task Model

Since continuous and discrete representations carry information about the same
emotion, we are interested in understanding if _jointly_ predicting continuous
and discrete attributes improves model performance. To do this, we use a
multi-task architecture to predict both the discrete and continuous emotion
attributes simultaneously. Since we seek to find the commonality between the
discrete and continuous representations, in the multi-task architecture, we
elect to utilize a shared speech encoder that transforms the input speech into
hidden representations $H$ that contains the information that pertains to both
the continuous and discrete emotion attributes,i.e., the shared information.

The Multi-task C,D model predicts both discrete label $\hat{y}$ and continuous
vector $\hat{c}=[\hat{c}_{Val},\hat{c}_{Aro},\hat{c}_{Dom}]$ for every
utterance.

The architecture of the model is shown in Fig. 1 (left). The model uses a
common encoder, but within the decoder- two parallel branches are used to
learn task specific pooling and MLP parameters. The discrete branch (left)
generates a discrete embedding $E_{D}$, which is then used to predict the
discrete emotion $\hat{y}$. Similarly, the continuous branch predicts a
continuous embedding $E_{C}$, which is used to predict the continuous emotion
$\hat{c}$. The model is optimized with the total loss as shown in Equation
(3).

$\mathcal{L_{\text{total}}}=\alpha(\mathcal{L_{\text{ccc}}}^{Val}+\mathcal{L_{\text{ccc}}}^{Dom}+\mathcal{L_{\text{ccc}}}^{Aro})+\beta\mathcal{L_{\text{disc}}}$
(3)

where the values of $\alpha$ and $\beta$ are set to 1 empirically.

###  2.4 Hierarchical Multi-Task Model

Multi-task modelling described in the previous section seeks to utilize the
shared information between discrete and dimensional attributes to predict them
jointly. However, it doesn’t assume any direct dependency between the discrete
and continuous emotion attributes. Based on the hypothesis that knowledge of
discrete emotion attributes would help improve continuous attribute
predictions and vice-versa, we develop hierarchical multi-task models.

In this model, the continuous and discrete emotion prediction branches in the
decoder are used to generate the continuous and discrete embeddings $E_{C}$
and $E_{D}$ respectively, as in the multi-task formulation. However, here, the
predicted continuous emotion, i.e., $\hat{c}$ is not computed solely based on
$E_{C}$, but also on $E_{D}$. In other words, the discrete emotion embedding
is used in conjunction with the continuous embeddding to predict the
continuous emotion. We term this our Hierarchical D-C model since the discrete
embedding is used as auxiliary input to predict the continuous emotion.
Similarly, one can define a Hierarchical C-D model where the continuous
emotion embedding is used as auxiliary input to predict the discrete emotion.

The Hierarchical D-C model is shown in Fig. 1 (right). To perform continuous
emotion prediction with the help of the discrete predictions, discrete emotion
embedding $E_{D}$, (which is used to predict the discrete emotion $\hat{y}$)
is concatenated with the continuous embedding $E_{C}$. This concatenated
embedding $[E_{D}E_{C}]$ is then used to predict the continuous emotion,
$\hat{c}$. Similarly, in the Hierarchical C-D model, in order to utilize
continuous attributes to predict discrete attributes, $E_{C}$ is concatenated
with $E_{D}$ and used to predict $\hat{y}$.

##  3 Experiments

###  3.1 Data

Our experiments are conducted on the MSPPodcast[12] and IEMOCAP [13]. MSP
Podcast is the largest human labelled emotion dataset with 29,965 training
examples and 10,013 test utterances. It is comprised of segments of speech
from podcasts, which means that the speech is expressive and the acoustic
environment less constrained by background noise and other interference. The
MSPPodcast data has labels for three continuous dimensions - valence, arousal
and dominance, and for multiple discrete emotions - of which we use five :
neutral, angry, happy, sad, and disgust. In this work, we report our results
on the balanced test1 evaluation set with 30 male and 30 female speakers. The
second dataset we use is IEMOCAP which is a 20 hour dataset, with annotations
on acted emotion. It comprises of 10 different speakers, 5 male and 5 female.
It is labelled for three continuous dimensions - valence, arousal and
dominance, and for 9 discrete emotions out of which we utilise the ones that
overlap with our MSPPodcast set, i.e., neutral, angry, happy, sad, and
disgust.

###  3.2 Model Hyperparameters

All our models are built using the ESPNet[14] toolkit, and will be publicly
released to encourage further research. We use HUBERT-large [15] embeddings as
the input features and a 4 layer conformer encoder with 64 hidden units for
our models. We freeze the HUBERT-large frontend, and train the conformer and
pooling decoder parameters. We employ separate self-attentive pooling[16]
layers for continuous and discrete emotion prediction so the model can learn
which frames to focus on in order to make emotion predictions. Our decoders
consist of linear projections that map from the encoded output dimension of
768 to 64, 32 and consequently the output sizes of 3 for the continuous
prediction and 5 for the discrete prediction. We use a ReLU activation to
ensure that the predicted continuous attributes are strictly positive and use
LeakyReLU elsewhere. We use a dropout of 0.2 in the decoder. Our models are
trained with the Adam optimizer, and a peak learning rate of 1e-3 for 15,000
warmup steps.

###  3.3 Evaluation Metrics

Discrete emotion recognition is evaluated using F1 or accuracy. For continuous
prediction, we compute the Concordance Correlation Coefficient for each of the
attributes arousal, valence and dominance, and consequently their mean.
Equation (4) shows how the CCC is computed given prediction $\hat{c}$, and
ground truth $c$, where
$s_{c\hat{c}},s_{c}^{2},s_{\hat{c}}^{2},\bar{c},\bar{\hat{c}}$ represent the
covariance between the ground truth and prediction, variance of the ground
truth, variance of the prediction, mean of the groundtruth and mean of the
prediction respectively.

$\mathcal{\text{CCC}}=\dfrac{2s_{c\hat{c}}}{s_{c}^{2}+s_{\hat{c}}^{2}+(\bar{c}-\bar{\hat{c}})^{2}}$
(4)

Table 1: Emotion Recognition Results on IEMOCAP using 5-fold cross-validation:
Concordance Correlation Coefficient- overall,Valence,Activation and Dominance
is reported for continuous emotions, and Unweighted accuracy is reported for
discrete emotion prediction

Model Description | CCC | CCC-V | CCC-A | CCC-D | Acc
---|---|---|---|---|---
Baseline C | 0.580 | 0.548 | 0.606 | 0.566 | -
Baseline D | - | - | - | - | 0.737
Multi-task C,D | 0.603 | 0.571 | 0.669 | 0.567 | 0.723
Hierarchical D-C | 0.667 | 0.660 | 0.717 | 0.625 | 0.744
Hierarchical C-D | 0.648 | 0.651 | 0.694 | 0.599 | 0.749

Table 2: Results on MSPPodcast and IEMOCAP: The second two columns represent the attributes used in training from the IEMOCAP and MSPPodcast datasets respectively. CCC is reported on IEMOCAP Session05 and on the test1 evaluation sets for MSPPodcast respectively. F1 is reported on the test1 evaluation set for MSPPodcast and unweighted accuracy on Session05 for IEMOCAP (we use this because these are standard metrics reported for these datasets). ID | IEMOCAP Train | MSPPodcast Train | IEMOCAP CCC | IEMOCAP Acc | MSPPodcast CCC | MSPPodcast F1
---|---|---|---|---|---|---
1 | - | Cont. | 0.125 | - | 0.524 | -
2 | Cont. | - | 0.327 | - | 0.140 | -
3 | - | Disc. | - | 0.459 | - | 0.325
4 | Disc. | - | - | 0.701 | - | 0.266
5 | Cont. | Cont. | 0.586 | - | 0.524 | -
6 | Disc. | Disc. | - | 0.695 | - | 0.364
7 | Cont. | Disc. | 0.625 | 0.470 | 0.189 | 0.302
8 | Disc. | Cont. | 0.218 | 0.705 | 0.497 | 0.283

###  3.4 Results on IEMOCAP

Table 1 shows our experimental results on the IEMOCAP dataset, where all
results are computed using standard 5-fold cross validation [17]. We observe
that multi-task training improves CCC by 0.02 over the continuous baseline,
however, it does not outperform the discrete baseline. We contend that this is
because both of these predictions are being generated independent of the
other.

The proposed hierarchical models outperform the baselines and multi-task model
on discrete and continuous emotion prediction. Specifically the hierarchical
model that uses discrete attributes to aid the prediction of continuous
attributes, i.e., Hierarchical D-C outperforms the multi-task model and the
continuous baselines on CCC - gaining $0.087$ absolute CCC. This is perhaps
because knowledge of the discrete emotion aids in the prediction of dominance
and arousal. We also observe an absolute $0.8\%$ improvement on accuracy in
Hierarchical D-C. The hierarchical model that uses continuous emotions to help
predict discrete attributes, i.e., Hierarchical C-D outperforms the other 3
models on accuracy. This reaffirms our hypothesis that knowledge of discrete
and continuous emotion attributes can be used to improve performance on
continuous and discrete emotion prediction respectively. We note that the
Hierarchical C-D model improves the recognition of discrete emotion attributes
compared to the multi-task model and the continuous baseline.

###  3.5 Results on MSPPodcast

Table 3: Results on MSPPodcast: Concordance Correlation Coefficient(CCC) -
overall,Valence,Activation and Dominance is reported on the test1 evaluation
set. For discrete emotions, unweighted F1 is reported.

Model Description | CCC | CCC-V | CCC-A | CCC-D | F1
---|---|---|---|---|---
Baseline C | 0.593 | 0.597 | 0.646 | 0.538 | -
Baseline D | - | - | - | - | 0.368
Multi-task C,D | 0.587 | 0.591 | 0.637 | 0.533 | 0.393
Hierarchical D-C | 0.617 | 0.588 | 0.675 | 0.584 | 0.404
Hierarchical C-D | 0.605 | 0.554 | 0.661 | 0.569 | 0.411

Table 3 reports the results of experiments on MSPPodcast. The continuous and
discrete emotion prediction baselines obtain comparable scores to state-of-
the-art approaches[18].

Multi-task training to jointly predict continuous and discrete attributes
improves F1 score over the discrete baseline while slightly degrading
continuous emotion prediction performance. As with the IEMOCAP dataset, the
proposed hierarchical models outperform the baselines and multi-task
prediction models. Specifically, the hierarchical D-C model improves CCC on
continuous attribute prediction over the multi-task and baseline continuous
models. Hierarchical multi-task learning helps learn useful intermediate
representations [19]. Therefore in the hierarchical D-C model, the continuous
prediction task helps learn better discrete emotion representations, thereby
improving discrete prediction as well with respect to the baseline. The
Hierarchical C-D model also improves prediction of discrete emotion over the
other 3 models. This model also outperforms the baseline and multi-task models
on continuous emotion prediction, with gains arising from improved prediction
of arousal and dominance.

###  3.6 Combining IEMOCAP and MSPPodcast

In this section, we attempt to use discrete and continuous labels from
different datasets and analyse gains when training models on multiple datasets
with matched (e.g. discrete-discrete) and mis-matched (e.g. continuous-
discrete) labels. We perform all experiments using a subset of the MSPPodcast
data chosen randomly such that the resulting size is the same as that of the
IEMOCAP training data. This is done in order to be able to make fair
comparisons on transferrability of representations. Table 2 summarizes the
results of our experiment on transferring labels across datasets.

IEMOCAP Discrete Prediction: Consider rows with ID 4,6,8 from Table 2\- we
observe that the best unweighted accuracy on the IEMOCAP test set is obtained
when the model is trained on IEMOCAP discrete labels and MSPPodcast continuous
labels.

IEMOCAP Continuous Prediction: Comparing rows 2,5,7 from Table 2, we observe
that the best CCC on IEMOCAP is achieved when IEMOCAP continuous labels and
MSPPodcast discrete labels are both included in the training.

MSPPodcast Discrete Prediction: From rows 3,6,7 in Table 2, we observe that
the best F-1 score is obtained on MSPPodcast when discrete labels from both
datasets are used, i.e., for row 6 model.

MSPPodcast Continuous Prediction: From the rows 1,5,8 from Table 2, we can
conclude that when the MSPPodcast continuous labels are used alone, the best
performance is achieved.

In conclusion, we observe that using mis-matched labels from the MSPPodcast
data for training improves performance on discrete and continuous emotion
prediction for IEMOCAP. We also note that though discrete labels from IEMOCAP
are transferrable and improve performance on MSPPodcast, the continuous labels
from IEMOCAP do not seem to provide gains on MSPPodcast.

Therefore MSPPodcast discrete and continuous representations help improve
performance on the mismatched label of IEMOCAP, while such transferred
representations from IEMOCAP do not significantly impact MSPPodcast
predictions. We believe this is in part because of the difference in the
nature of annotations across the IEMOCAP and MSPPodcast datasets. For example,
from our analysis we observe that anger is assigned a lower arousal and higher
valence than neutral in IEMOCAP while in MSPPodcast, anger is assigned a
higher arousal and lower valence than neutral. Furthermore, the inter-
annotator agreement for continuous values in MSPPodcast are much lower in the
IEMOCAP, which makes it challenging to obtain gains in MSPPodcast continuous
predictions using transferred representations from IEMOCAP.

##  4 Conclusion and Future Work

Emotion Recognition remains a challenging task. Different datasets are
labelled with varying annotation labels, making it challenging to train large
scale emotion models utilising all the datasets. In this paper, we introduce
hierarchical multi-task learning models that predict discrete or continuous
labels by using continuous or discrete labels respectively.

With our method, we obtain absolute improvements of 1.2 % Accuracy and 4.3
points F-1 for discrete prediction on IEMOCAP and MSPPodcast respectively. On
continuous labels, we improve 0.09 CCC for IEMOCAP, and 0.025 CCC in
MSPPodcast. Furthermore, we also combine IEMOCAP and MSPPodcast with
mismatched emotion annotations, and show that mis-matched labels from
MSPPodcast help performance on IEMOCAP.

## References

  * [1] J.C. Borod,  The Neuropsychology of Emotion,  Affective Science. Oxford University Press, USA, 2000.
  * [2] Robert Plutchik,  “A general psychoevolutionary theory of emotion,”  Theories of emotion, vol. 1, pp. 3–31, 1980.
  * [3] J.A. Russell,  “A circumplex model of affect,”  Journal of personality and social psychology, vol. 39, no. 6, pp. 1161–1178, 1980.
  * [4] James A Russell,  “Culture and the categorization of emotions.,”  Psychological bulletin, vol. 110, no. 3, pp. 426, 1991.
  * [5] Henrik Kessler, Alexander Festini, Harald C Traue, Suzanne Filipic, Michael Weber, and Holger Hoffmann,  “Simplex–simulation of personal emotion experience,”  Affective Computing, pp. 255–270, 2008.
  * [6] Holger Hoffmann, Andreas Scheck, Timo Schuster, Steffen Walter, Kerstin Limbrecht, Harald C Traue, and Henrik Kessler,  “Mapping discrete emotions into the dimensional space: An empirical approach,”  in 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2012, pp. 3316–3320.
  * [7] Sven Buechel and Udo Hahn,  “A flexible mapping scheme for discrete and dimensional emotion representations: Evidence from textual stimuli,”  in CogSci 2017—Proceedings of the 39th Annual Meeting of the Cognitive Science Society, 2017, pp. 180–185.
  * [8] Marián Trnka, Sakhia Darjaa, Marian Ritomskỳ, Róbert Sabo, Milan Rusko, Meilin Schaper, and Tim H Stelkens-Kobsch,  “Mapping discrete emotions in the dimensional space: an acoustic approach,”  Electronics, vol. 10, no. 23, pp. 2950, 2021.
  * [9] Reza Lotfian and Carlos Busso,  “Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings,”  IEEE Transactions on Affective Computing, vol. 10, no. 4, pp. 471–483, 2017.
  * [10] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan,  “Iemocap: interactive emotional dyadic motion capture database.,”  Language Resources and Evaluation, vol. 42, no. 4, pp. 335–359, 2008.
  * [11] Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea,  “MELD: A multimodal multi-party dataset for emotion recognition in conversations,”  in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019, pp. 527–536, Association for Computational Linguistics.
  * [12] R. Lotfian and C. Busso,  “Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings,”  IEEE Transactions on Affective Computing, vol. 10, no. 4, pp. 471–483, October-December 2019.
  * [13] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan,  “Iemocap: Interactive emotional dyadic motion capture database,”  Language resources and evaluation, vol. 42, no. 4, pp. 335–359, 2008.
  * [14] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai,  “ESPnet: End-to-end speech processing toolkit,”  in Proceedings of Interspeech, 2018, pp. 2207–2211.
  * [15] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [16] Pooyan Safari, Miquel India, and Javier Hernando,  “Self-attention encoding and pooling for speaker recognition,”  Proc. Interspeech 2020, pp. 941–945, 2020.
  * [17] Edmilson Morais, Ron Hoory, Weizhong Zhu, Itai Gat, Matheus Damasceno, and Hagai Aronowitz,  “Speech emotion recognition using self-supervised features,”  in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 6922–6926.
  * [18] Sundararajan Srinivasan, Zhaocheng Huang, and Katrin Kirchhoff,  “Representation learning through cross-modal conditional teacher-student training for speech emotion recognition,”  in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 6442–6446.
  * [19] Ramon Sanabria and Florian Metze,  “Hierarchical multitask learning with ctc,”  in 2018 IEEE Spoken Language Technology Workshop (SLT), 2018, pp. 485–490.