# Context-Aware Personality Inference in Dyadic Scenarios:
Introducing the UDIVA Dataset

Cristina Palmero These authors contributed equally to this work. Universitat
de Barcelona  Computer Vision Center  Javier Selva∗ Universitat de Barcelona
Computer Vision Center  Sorina Smeureanu∗ Universitat de Barcelona  Computer
Vision Center  Julio C. S. Jacques Junior  Computer Vision Center  Universitat
Oberta de Catalunya  Albert Clapés  Universitat de Barcelona  Computer Vision
Center  Alexa Moseguí  Universitat de Barcelona  Zejian Zhang  Universitat de
Barcelona  Computer Vision Center  David Gallardo  Universitat de Barcelona
Georgina Guilera  Universitat de Barcelona  David Leiva  Universitat de
Barcelona  Sergio Escalera  Universitat de Barcelona  Computer Vision Center

###### Abstract

This paper introduces UDIVA, a new non-acted dataset of face-to-face dyadic
interactions, where interlocutors perform competitive and collaborative tasks
with different behavior elicitation and cognitive workload. The dataset
consists of 90.5 hours of dyadic interactions among 147 participants
distributed in 188 sessions, recorded using multiple audiovisual and
physiological sensors. Currently, it includes sociodemographic, self- and
peer-reported personality, internal state, and relationship profiling from
participants. As an initial analysis on UDIVA, we propose a transformer-based
method for self-reported personality inference in dyadic scenarios, which uses
audiovisual data and different sources of context from both interlocutors to
regress a target person’s personality traits. Preliminary results from an
incremental study show consistent improvements when using all available
context information.

##  1 Introduction

Human interaction has been a central topic in psychology and social sciences,
aiming at explaining the complex underlying mechanisms of communication with
respect to cognitive, affective, and behavioral perspectives [1, 2]. From a
computational point of view, research in dyadic and small group interactions
enables the development of automatic approaches for detection, understanding,
modeling, and synthesis of individual and interpersonal social signals and
dynamics [3]. Many human-centered applications for good (e.g., early diagnosis
and intervention [4], augmented telepresence [5], and personalized agents [6])
strongly depend on devising solutions for such tasks.

In dyadic interactions, we use verbal and nonverbal communication channels to
convey our goals and intentions [7, 8] while building a common ground [9].
Both interlocutors influence each other based on the cues we perceive [1].
However, the way we perceive, interpret, react, and adapt to them depends on a
myriad of factors. Such factors, which we refer to as context, may include,
but are not limited to: our personal characteristics, either stable (e.g.,
personality [10], cultural background, and other sociodemographic information
[11]) or transient (e.g., mood [12], physiological or biological factors); the
relationship and shared history between both interlocutors; the
characteristics of the situation and task at hand; societal norms; and
environmental factors (e.g., temperature). What is more, to analyze individual
behaviors during a conversation, the joint modeling of both interlocutors is
required due to the existing dyadic interdependencies. While these aspects are
usually contemplated in non-computational dyadic research [13], context- and
interlocutor-aware computational approaches are still scarce, largely due to
the lack of datasets providing contextual metadata in different situations and
populations [14].

Here, we introduce UDIVA, a highly varied multimodal, multiview dataset of
zero- and previous-acquaintance, face-to-face dyadic interactions. It consists
of 188 interaction sessions, where 147 participants arranged in dyads
performed a set of tasks in different circumstances in a lab setting. It has
been collected using multiple audiovisual and physiological sensors, and
currently includes sociodemographic, self- and peer-reported personality,
internal state, and relationship profiling. To the best of our knowledge,
there is no similar publicly available, face-to-face dyadic dataset in the
research field in terms of number of views, participants, tasks, recorded
sessions, and context labels.

As an initial analysis on the UDIVA dataset, we also propose a novel method
for self-reported personality inference in dyadic scenarios. Apart from its
importance in interaction understanding, personality recognition is key to
develop individualized, empathic, intelligent systems [15]. Our proposal is
based on the Video Action Transformer [16], which classifies people’s actions
in a video by taking advantage of the spatiotemporal context around them.
Inspired by [17], we extend query, key, and value from [16] with the other
interlocutor’s scene, audio, and further context metadata. The latter includes
stable and transient characteristics from each interlocutor, as well as
session, task, and relationship information. Finally, we experimentally
evaluate the usefulness of each additional input incrementally, showing
consistent improvements when using all the available context sources and
modalities.

##  2 Related work

Table 1: Publicly available audiovisual human-human (face-to-face) dyadic
interaction datasets. ‘‘Interaction’’, Acted (actors improvising and/or
following an interaction protocol, i.e. given topics/stimulus/tasks), Acted∗
(Scripted), Non-acted (natural interactions in lab environment) or Non-acted∗
(non-acted but guided by interaction protocol); ‘‘F/M’’, number of
participants per gender (Female/Male) or number of participants if gender is
not informed; ‘‘Sess’’, number of sessions; ‘‘Size’’, hours of
recordings;‘‘#Views’’, number of RGB cameras used, and D is RGB+D, E is Ego, M
is Monochrome. The $\phi$ symbol is used to indicate
missing/incomplete/unclear information on the source.

Name / Year |  Focus |  Interaction |  Modality |  Annotations |  F/M |  Sess |  Size |  #Views |  Lang.
---|---|---|---|---|---|---|---|---|---
IEMOCAP [18], 2008 |  Emotion recognition |  Acted∗ & Acted |  Audiovisual, face & hands MoCap. |  Emotions, transcripts, turn-taking |  5/5 |  5 |  $\sim$12h |  2 |  English
CID [19], 2008 |  Speech & conversation analysis |  Non-acted & Non-acted∗ |  Audiovisual |  Speech features, transcripts |  10/6 |  8 |  8h |  1 |  French
HUMAINE† [20, 21], 2011 |  Emotion analysis |  Non-acted∗ |  Audiovisual |  Emotions |  34 |  18 |  $\sim$12h |  4 |  English
MMDB [22], 2013 |  Adult-infant interaction analysis |  Non-acted∗ |  Audiovisual, depth, physiological |  Social cues (gaze, vocal affects, gestures...) |  121 |  160 |  $\sim$13.3h |  8 + 1D |  English
MAHNOB [23], 2015 |  Mimicry |  Non-acted∗ |  Audiovisual, head MoCap. |  Head, face and hand gestures, personality scores (self-reported) |  29/31 |  54 |  11.6h |  2 + 13M |  English
MIT Interview [24], 2015 |  Hirability analysis |  Non-acted∗ |  Audiovisual |  Hirability, speech features, social & behavioral traits, transcripts |  43/26 |  138 |  10.5h |  2 |  English
Creative IT [25], 2016 |  Emotion recognition |  Acted |  Audiovisual, body MoCap. |  Transcripts, speech features, emotion |  9/7 |  8 |  $\sim$1h |  2 |  English
MSP-IMPROV [26], 2017 |  Emotion recognition |  Acted & Non-acted |  Audiovisual |  Turn-taking, emotion |  6/6 |  6 |  9h |  2 |  English
DAMI-P2C [27], 2020 |  Adult-infant interaction analysis |  Non-acted∗ |  Audiovisual |  Emotion, sociodemographics, parenting assessment, child personality (peer-reported) |  38/30 |  65 |  $\sim$21.6h |  1 ϕ |  English
UDIVA (ours), 2020 |  Social interaction analysis |  $\frac{1}{5}$ Non-acted & $\frac{4}{5}$Non-acted∗ |  Audiovisual, heart rate |  Personality scores (self- & peer- reported), sociodemographics, mood, fatigue, relationship type |  66/81 |  188$\times$5 (tasks) |  90.5h |  6 + 2E |  Spanish, Catalan, English
† Here we consider the Green Persuasive and the EmoTABOO [28] databases together. |

This section reviews related work on dyadic scenarios along three axes: social
signals and behaviors in context, personality recognition, and human
interaction datasets.

Social signals and behaviors in context. Dyadic interactions are a rich source
of overt behavioral cues. They can reveal our personal attributes and
cognitive/affective inner states dependent upon the context in which they are
situated. Context can take many forms, and in the case of recognition of an
individual state or attribute, the interaction partner’s attributes and
behaviors can be considered part of the target person’s context. From a
computational perspective, spatiotemporal and multiview information can be
referred to as context as well. For the measurement of interpersonal
constructs (e.g., synchrony [29], rapport [30]), individual social behaviors
(e.g., engagement [31]) and impressions (e.g., dominance [32], empathy [15]),
the joint modeling of both interlocutors and/or other sources of context has
been frequently considered. However, for the task of recognizing individual
attributes such as emotion and personality, context has often been
misrepresented, despite recurrent claims on its importance [33, 34, 35, 36].

Recent years have seen a small surge in interlocutor-aware approaches for
utterance- or turn-based emotion recognition in conversation [37] and
sentiment analysis. Early works were based on handcrafted nonverbal,
spatiotemporal dyadic features [38, 39]. Nowadays, most approaches rely on
deep learning, using conversation transcripts as input with contextualized
word or speaker embeddings [40] and considering past and/or future parts of
the conversation as additional context. Temporal modeling of those feature
representations has been widely performed via recurrent approaches [41], and
more recently with BERT/Transformer-like architectures [42, 43]. Some works
have further proposed to enrich models with additional modalities, such as raw
audiovisual data to enhance the representation of interlocutors’ influences
and dynamics [44, 45], or speech cues in addition to the personality of the
target speaker [46]. Context-aware personality recognition has followed a
similar trend as for emotion, but the literature is even scarcer. We discuss
it next.

Automatic personality recognition. Personality is widely defined as the
manifestation of individual differences in patterns of thought, feeling, and
behavior, that remain relatively stable during time [47]. In the personality
computing field [48], it is usually characterized by the basic Big Five traits
[49] (Openness to Experience, Conscientiousness, Extraversion, Agreeableness,
and Neuroticism), often referred to as OCEAN, based on self-reported
assessments. Most works focus on personality recognition from the individual
point of view, even in a dyadic or small group conversational context [50],
using only features from the target person. Preliminary studies tended to use
handcrafted features from gestures and speech [51], while more recent works
rely on deep learning approaches from raw data [52].

To our knowledge, few methods propose interlocutor- or context-aware methods
for personality recognition. The work of [53] was one of the first, leveraging
turn-taking temporal evolution from transcript features but focusing on
apparent personality recognition (i.e., personality reported by external
observers [54]). With respect to self-reported personality inference in small
group interactions, [55] regressed individual and dyadic features of
personality and social impressions utilizing handcrafted descriptors of
prosody, speech and visual activity. Later, [56] proposed an interlocutor-
modulated recurrent attention model with turn-based acoustic features.
Finally, [57] predicted personality and performance labels by correlation
analysis of co-occurrent key action events, which were extracted from head and
hand pose, gaze and motion intensity features. Regarding context, just one
previous approach added person metadata (e.g., gender, age, ethnicity, and
perceived attractiveness) to audiovisual data [58]. However, their goal was to
better approximate the crowd biases for apparent personality recognition in
one-person videos. Contrary to previous works, we use different sources of
context, including both interlocutors, scene, and task information to infer
personality, using for the first time a video-based transformer adapted to
include audio and further context as metadata.

Human interaction datasets. Research on human behavior and communication
understanding has fostered the creation of a plethora of human interaction
datasets [59, 60, 14, 61]. Here, we focus on publicly available datasets
containing at least audiovisual data, which enable the fusion of multiple
modalities and the creation of more complete representations. In the
literature, we can find examples of rich, non-acted datasets focused on
computer-mediated dyadic settings [62, 63], face-to-face triadic [64, 65], or
small group interactions [66]. A number of TV-based datasets with acted
interactions also exist [67]. However, in such cases, the interlocutors’
internal states are artificially built.

One of the advantages of face-to-face settings is that the full overt
behavioral spectrum can be observed and modeled. Existing publicly available
face-to-face dyadic interaction datasets are summarized in Table 1111The
complete table is included in the supplementary material..

As it can be seen, most of them are limited in the number of participants,
recordings, views, context annotations, language, or purpose. The UDIVA
dataset has been designed with a multipurpose objective and aims at filling
this gap.

##  3 UDIVA dataset

This section introduces the UDIVA dataset (Understanding Dyadic Interactions
from Video and Audio signals), consisting of time-synchronized multimodal,
multiview videos of non-scripted face-to-face dyadic interactions based on
free and structured tasks performed in a lab setup222Additional details
regarding design, participants recruitment, technical setup, and descriptive
statistics will be provided in a follow-up paper..

Figure 1: Recording environment. We used six tripod-mounted cameras, namely
GB: General Rear camera, GF: General Frontal camera, HA: individual High Angle
cameras and FC: individual Frontal Cameras, and two ego cameras E (one per
participant, placed around their neck). a) Position of cameras, general
microphone and participants. b) Example of the time-synchronized 8 views.

###  3.1 Motivation

UDIVA wants to move beyond automatic individual behavior detection and focus
on the development of automatic approaches to study and understand the
mechanisms of influence, perception and adaptation to verbal and nonverbal
social signals in dyadic interactions, taking into account individual and dyad
characteristics as well as other contextual factors. One of our research
questions centers on the feasibility of developing systems able to unravel the
personality and internal processes of an individual by the social signals they
convey, and to understand how interaction partners perceive and react to those
cues directed to them. By publicly releasing the dataset to the research
community, we encourage data sharing and collaboration among different
disciplines, reuse, and repurposing of new research questions.

Figure 2: Examples of the 5 tasks included in the UDIVA dataset from 5
sessions. From left to right: Talk, Lego, Animals, Ghost, Gaze.

###  3.2 Main statistics

The dataset is composed of 90.5h of recordings of dyadic interactions between
147 voluntary participants333Participants gave consent to be recorded and to
share their collected data for research purposes, in compliance with GDPR
https://ec.europa.eu/info/law/law-topic/data-protection_en. (55.1% male) from
4 to 84 years old (mean=31.29), coming from 22 countries (68% from Spain). The
majority of participants were students (38.8%), and identified themselves as
white (84.4%). Participants were distributed into 188 dyadic sessions, with a
participation average of 2.5 sessions/participant (max. 5 sessions). To create
the dyads, three variables were taken into account: 1) gender (Female, Male);
2) age group (Child: 4-18, Young: 19-35, Adult: 36-50, and Senior: 51-84); and
3) relationship among interlocutors (Known, Unknown). Participants were
matched according to their availability and language while trying to enforce a
close-to-uniform distribution among all possible combinations between
variables (60 combinations). A minimum age of 4 years and the ability to
maintain a conversation in English, Spanish or Catalan were the only inclusion
criteria. In the end, the most common interaction group is Male-Male/Young-
Young/Unknown (15%), with 43% of the interactions happening among known
people. Spanish is the majority language of interaction (71.8%), followed by
Catalan (19.7%). Half of the sessions include both interlocutors with Spain as
country of origin.

###  3.3 Questionnaire-based assessments

Prior to their first session, each participant filled a sociodemographic
questionnaire, including: age, gender, ethnicity, occupation, maximum level of
education, and country of origin. To assess personality and/or temperament,
age-dependent standardized questionnaires were administered. In particular,
parents of children up to 8 years old completed the Children Behavior
Questionnaire (CBQ) [68, 69], participants from 9 to 15 years old completed
the Early Adolescent Temperament Questionnaire (EATQ-R) [70], while
participants aged 16 and older completed both the Big Five Inventory (BFI-2)
[47] and the Honesty-Humility axis of the HEXACO personality inventory [71].

All participants (or their parents) completed pre- and post-session mood
([72]) and fatigue (ad hoc 1-to-10 rating scale) assessments. The mood
assessment contained items drawn from the Post Experimental Questionnaire of
Primary Needs (PEQPN [73]). After each session, participants aged 9 and above
completed again the previous temperament/personality and mood questionnaires,
this time rating the individual they interacted with, to provide their
perceived impression. Finally, participants reported the relationship they had
with their interaction partner, if any.

Figure 3: Proposed method to infer self-reported personality (OCEAN) traits
from multimodal synchronized signals and context. Input consists of visual
(face, local context, and extended context chunks), audio (raw chunks), and
metadata (both interlocutors’ characteristics, and session and dyadic
features). Feature extraction is performed by a R(2+1)D network for the visual
chunks and VGGish for audio. The visual features from the R(2+1)D’s 3rd
residual block are concatenated to spatiotemporal encodings (STE). The
VGGish’s audio features and handcrafted metadata features are incorporated to
visual context/query features and the result transformed to the set of Query,
Keys, and Values as input to the Transformer network (Tx). The latter consists
of $N$ Tx layers, each equipped with Local and Extended Context Transformer Tx
units. Such units implement multiheaded attention and provide their updated
queries, which are combined and fed to the next Tx Layer. Finally, the output
of the $N$-th Tx layer is fed to a fully-connected (FC) layer to regress per-
chunk OCEAN scores.

###  3.4 Structure of a dyadic session

Participants were asked to sit at 90∘ to one another around a table (see Fig.
1), to be close enough to perform the administered tasks while facilitating
data acquisition. A session consisted of 5 tasks (illustrated in Fig. 2)
eliciting distinct behaviors and cognitive workload:

Talk. Participants were instructed to talk about any subject during approx. 5
minutes. This task allows analysis of common conversation constructs, such as
turn-taking, synchrony, empathy and quality of interaction, among others.

“Animals” game. Participants asked 10 yes/no questions each to guess the
animal they had on their forehead. Animals were classified into 3 difficulty
levels. This game reveals cognitive processes (e.g., thinking, gaze events).

Lego building. Participants built a Lego together following the instructions
leaflet, ranging between 4 difficulty levels. This task fosters collaboration,
cooperation, joint attention, and leader-follower behaviors, among others.

“Ghost blitz” card game. Participants had to select, from a set of 5 figures,
the one whose color and shape was not shown in a selected card. They played 1
card per turn, competing with each other to be the first at selecting the
correct figure. This task fosters competitive behavior, and allows cognitive
processing speed analysis, among others.

Gaze events. Participants followed directions to look at other’s face, at
static/moving object, or elsewhere, while moving head and eyes. This task
serves as ground truth for gaze gestures and face modeling with varied head
poses.

These tasks were selected along with psychologists due to the variety of
individual and dyadic behaviors they elicit. In particular, Lego structures
have been widely used in observational settings to assess aspects as
communication [74], social skills [75] or teamwork abilities and performance
[76]. Ghost and Animals are examples of board games, proven to be valid
assessments of interpersonal skills [77, 78]. All these aspects are, in turn,
indicators of personality traits like Extraversion, Agreeableness or
Conscientiousness [79]. Cognitive methods, such as the tasks herein used, are
routinely used in personality research [80].

The tasks were explained by a lab proctor prior to each task, who left the
recording room while it was taking place. Only for Gaze the proctor gave the
instructions while participants performed them. Talk was always administered
first as a warm-up, while Gaze was always last. The rest were administered
randomly. The difficulty of Lego and Animals for each session was selected
such that no participants repeated the same Lego or animal twice, while
forcing a uniform distribution on the number of times each item was used for
the total of sessions. To assess their difficulty level, we conducted an
anonymous survey among 19 co-researchers.

###  3.5 Technical setup

The setup consisted of 6 HD tripod-mounted cameras ($1280\times 720$px,
25fps), 1 lapel microphone per participant and an omnidirectional microphone
on the table, as depicted in Fig. 1. Each participant also wore an egocentric
camera ($1920\times 1080$px, 30fps) around their neck and a heart rate monitor
on their wrist. All the capturing devices are time-synchronized and the
tripod-mounted cameras calibrated. See Fig. 1 for an example of the camera
views.

##  4 Personality traits inference

This section provides a first insight into the UDIVA dataset by evaluating it
in a personality traits inference task. We present a transformer-based
context-aware model to regress self-reported personality traits of a target
person during a dyadic interaction. Then, we assess its performance and the
effect of adding several sources of context. Method, evaluation protocol and
results are described next444Additional details are provided in the
supplementary material..

###  4.1 Method description

The attention mechanism of our transformer-based method relates an initial
query, in this case the target person’s face, to the nonverbal behavior of
both interlocutors, the overall scene, and further contextual metadata, and
updates it with relevant context. The process is repeated with the updated
query in consecutive layers to eventually infer the personality (OCEAN)
traits. The proposed method consists of several stages, detailed below. All
components and the information flow among them are illustrated in Fig. 3.

Audiovisual input. Let
$\mathbf{X}_{L},\mathbf{X}_{E}\in[0,255]^{\mathcal{L}\times H\times W\times
3}$ be the pair of time-synchronized full-length videos containing the target
person (local context) and the other interlocutor (extended context),
respectively. We divide them into 32-frame non-overlapping chunks and resize
each chunk’s spatial resolution to $112\times 112$ to obtain, respectively,
$\mathbf{B}_{L},\mathbf{B}_{E}\in[0,255]^{32\times 112\times 112\times 3}$.
The 32 frames of the chunks are sampled with a stride of 2, such that a chunk
corresponds to approx. 2.5 seconds of the original videos. Also, we detect the
target person’s face regions in $\mathbf{X}_{L}$, crop, and re-scale them to
form the face chunk $\mathbf{B}_{F}\in[0,255]^{32\times 112\times 112\times
3}$. As face detector, we use a MobileNet-SSD [81] model pretrained on
Widerface [82]. Apart from the visual data, we define an audio chunk
$\mathbf{b}_{A}\in\mathbb{R}^{132\,300}$ consisting of the raw audio frames
acquired at 44.1 KHz from the general microphone (or one of the lapels if the
general one was not available for that session), and time-synchronized to its
respective video chunk.

Metadata input. Different sources of context are captured in the form of input
metadata, described in Table 2. We consider 2 types of metadata: (1) local
metadata, containing individual context from the target person and session
information; and (2) extended metadata, with individual context from the other
interlocutor and dyadic features.

Feature extraction. First, we normalize the pixel values of
$\\{\mathbf{B}_{F},\mathbf{B}_{L},\mathbf{B}_{E}\\}$ in the range $[0,1]$,
subtracting and dividing them by the mean and standard deviation of the IG-65M
dataset [83]. Then, we feed them to a R(2+1)D network [84] backbone,
pretrained on that same dataset, and save the rich spatiotemporal features
produced by the R(2+1)D’s 3rd convolutional residual stack:
$\mathbf{Z}^{\prime}_{F}=g_{F}(\mathbf{B}_{F};\theta_{F}),\;\mathbf{Z}^{\prime}_{L}=g_{L}(\mathbf{B}_{L};\theta_{C}),\;\mathbf{Z}^{\prime}_{E}=g_{E}(\mathbf{B}_{E};\theta_{C})$,
where $\theta_{F}$ are the weights of the face network $g_{F}(\cdot)$, and
$\theta_{C}$ are shared weights of $g_{L}(\cdot)$ and $g_{E}(\cdot)$ networks.
$\mathbf{Z}^{\prime}_{F},\mathbf{Z}^{\prime}_{L},\mathbf{Z}^{\prime}_{E}\in\mathbb{R}^{16\times
28\times 28\times 128}$ denote the face, local context, and extended context
visual features, respectively. For the audio feature extraction, we use the
VGGish [85] backbone. This VGG-like model, developed specifically for the
audio modality and with pre-trained weights $\theta_{A}$ learned on a
preliminary version of the YouTube-8M [86], provides a feature vector
$\mathbf{a}\in\mathbb{R}^{128}$ encoding information contained in the
$\mathbf{b}_{A}$ chunk: $\mathbf{a}=g_{A}(\mathbf{b}_{A};\theta_{A})$.
Finally, input metadata is normalized according to Table 2, and encoded in
$\mathbf{m}_{L}\in\mathbb{R}^{20}$ and
$\mathbf{m}_{\mathrm{E}}\in\mathbb{R}^{19}$ for local and extended metadata
features, respectively.

Table 2: Description of the different sources of context included as metadata
in the proposed personality inference model.

Context type |  Source |  Value range normalization |  Output size
---|---|---|---
Individual |  Stable (across sessions) |  Age |  Self-reported |  $[17,75]\to[0,1]$ |  1D
|  |  Gender |  Self-reported |  $\\{\text{F},\text{M}\\}\to\\{0,1\\}$ |  1D
|  |  Cultural background |  Self-reported (country of origin) |  Recategorization based on cultural differences [87] |  6D (one-hot encoding)
|  Transient (per session) |  Session index |  Session info. |  $[1,5]\to[0,1]$ |  1D
|  |  Pre-session mood |  Self-reported [72] (8 categories∗, Likert scale) |  $[1,5]\to[0,1]$ (for each category) |  8D
|  |  Pre-session fatigue |  Self-reported (Rating scale) |  $[0^{{\dagger}},10]\to[0,1]$ |  1D
Session |  |  Order of the task
---
within the session
Session info. |  $[1,4]\to[0,1]$ |  1D
|  |  Task difficulty†
---
External survey |  $[0,3]\to[0,1]$ |  1D
Dyadic |  |
---
Interlocutors’
relationship
Self-reported |  $\\{\text{N},\text{Y}\\}\to\\{0,1\\}$ |  1D
∗Categories: good, bad, happy, sad, friendly, unfriendly, tense, and relaxed.
† Sessions with fatigue data missing were assigned a value of 0.
‡ Tasks with no difficulty level associated were assigned a value of 0.

Spatiotemporal encodings (STE). Following other transformer-like
architectures, we need to add positional encodings to our audiovisual feature
embeddings $\mathbf{Z}^{\prime}$, which can be either learned or fixed. We opt
to learn them end-to-end. Being 16 the size of the temporal dimension of the
different $\mathbf{Z}^{\prime}$, we create a vector of zero-centered time
indices
$\mathbf{t}=\left<-\frac{16}{2},-\frac{16}{2}+1,\ldots,\frac{16}{2}-1\right>$.
The temporal encodings are computed by a two-layered network:
$\mathbf{P}^{\prime}_{T}=\mathrm{ReLU}\left(\Theta_{T_{1}}^{\top}\mathrm{ReLU}\left(\Theta_{T_{2}}^{\top}\mathbf{t}\right)\right)$,
where $\Theta_{T_{1}}\in\mathbb{R}^{1\times 20}$ and
$\Theta_{T_{2}}\in\mathbb{R}^{20\times 10}$ are learned weights. The spatial
encodings $\mathbf{P}^{\prime}_{S}$ are computed by a similar encoding
network. Given that $28\times 28$ is the spatial resolution of the features,
we feed to the spatial encoding network a tensor of spatially zero-centered
position indices $\mathbf{S}\in\mathbb{R}^{28\times 28\times 2}$, where
$\mathbf{S}_{i,j}=\left<i-\frac{28}{2},j-\frac{28}{2}\right>,\forall
i,j\in[0,28)$ and weights $\Theta_{S_{1}}\in\mathbb{R}^{2\times 20}$ and
$\Theta_{S_{2}}\in\mathbb{R}^{20\times 10}$. Then, $\mathbf{P}^{\prime}_{T}$
and $\mathbf{P}^{\prime}_{S}$ are reshaped to
$\mathbf{P}_{T}\in\mathbb{R}^{16\times 1\times 1\times 10}$ and
$\mathbf{P}_{S}\in\mathbb{R}^{1\times 28\times 28\times 10}$ and concatenated
together by broadcasting singleton dimensions, i.e.
$\mathbf{P}=\mathbf{P}_{S}\mathbin{\|}\mathbf{P}_{T}$.
$\mathbf{P}\in\mathbb{R}^{16\times 28\times 28\times 20}$ is concatenated to
each of the feature embeddings $\mathbf{Z}^{\prime}$:
$\mathbf{Z}_{F}=\mathbf{Z}^{\prime}_{F}\mathbin{\|}\mathbf{P},\;\mathbf{Z}_{L}=\mathbf{Z}^{\prime}_{L}\mathbin{\|}\mathbf{P},\;\mathbf{Z}_{E}=\mathbf{Z}^{\prime}_{E}\mathbin{\|}\mathbf{P},\;$
resulting in
$\mathbf{Z}_{F},\mathbf{Z}_{L},\mathbf{Z}_{E}\in\mathbb{R}^{16\times 28\times
28\times 148}$. To these features with spatiotemporal encodings, $\mathbf{Z}$,
we will later concatenate metadata and audio to obtain the face query, local
context, and extended context features.

Query Preprocessor (QP). This small module transforms $\mathbf{Z}_{F}$ to a
vector-form:
$\mathbf{f}=\mathrm{QP}(\mathbf{Z}_{F}),\;\mathbf{f}\in\mathbb{R}^{128}$. The
QP consists of a 3D max pooling layer of size $(1,2,2)$ and stride $(1,2,2)$,
a 3D conv layer of size $(1,1,1)$ and $16$ filters, a ReLU activation function
layer, a permutation of dimensions and reshaping so that the temporal
dimensions and the channels are merged into the same dimension, a 2D max
pooling of size $(2,2)$, a 2D conv layer of size $(1,1)$, a ReLU activation
layer, a flattening, and a fully-connected (FC) layer of size $128$, another
ReLU, and a dropout layer.

Multimodality: fusing visuals with audio and metadata. Both local and extended
visual context features along with encodings, $\mathbf{Z}_{L}$ and
$\mathbf{Z}_{E}$, are augmented with audio features. The original
128-dimensional global audio features $\mathbf{a}$ are linearly projected to a
more compact 100-dimensional representation and reshaped to
$\mathbf{A}\in\mathbb{R}^{1\times 1\times 1\times 100}$. Then, the local
context features are simply
$\mathbf{W}_{L}=\mathbf{Z}_{L}\mathbin{\|}\mathbf{A}$. The extended context
features are augmented with the updated audio features and the extended
metadata from the interlocutor, reshaping $\mathbf{m}_{E}\in\mathbb{R}^{19}$
to $\mathbf{M}_{E}\in\mathbb{R}^{1\times 1\times 1\times 19}$ and applying
broadcast concatenation, that is
$\mathbf{W}_{E}=\mathbf{Z}_{E}\mathbin{\|}\mathbf{A}\mathbin{\|}\mathbf{M}_{E}$.
Finally, the face query features $\mathbf{w}_{Q}\in\mathbb{R}^{148}$ are built
from the combination of the QP output along with the target person’s local
metadata: $\mathbf{w}_{Q}=\mathbf{f}\mathbin{\|}\mathbf{m}_{L}$.

Keys, Values, and Query. To obtain the final input to the transformer layers,
we first need to transform local and extended context features into two
different 128-dimensional embeddings (Keys and Values), and also the face
query features into the query embedding of the same size. The Local keys and
Local values are then
$\mathbf{K}_{L}=\mathrm{ReLU}(\mathbf{\Theta}_{K_{L}}^{\top}\mathbf{W}_{L})$
and
$\mathbf{V}_{L}=\mathrm{ReLU}(\mathbf{\Theta}_{V_{L}}^{\top}\mathbf{W}_{L})$
where $\mathbf{\Theta}_{K_{L}},\mathbf{\Theta}_{V_{L}}\in\mathbb{R}^{248\times
128}$, whereas the Extended keys and Extended values are
$\mathbf{K}_{E}=\mathrm{ReLU}(\mathbf{\Theta}_{K_{E}}^{\top}\mathbf{W}_{E})$
and
$\mathbf{V}_{E}=\mathrm{ReLU}(\mathbf{\Theta}_{V_{E}}^{\top}\mathbf{W}_{E})$,
where $\mathbf{\Theta}_{K_{E}},\mathbf{\Theta}_{V_{E}}\in\mathbb{R}^{267\times
128}$. The input Query representation $\mathbf{q}_{0}\in\mathbb{R}^{128}$ is
computed as
$\mathbf{q}_{0}=\mathrm{ReLU}(\mathbf{\Theta}_{Q_{0}}^{\top}\mathbf{w}_{Q}$),
where $\mathbf{\Theta}_{Q_{0}}\in\mathbb{R}^{148\times 128}$.

Transformer network. Our transformer network (Tx) is composed of $N=3$ Tx
layers with 2 Tx units each, one for the local context and another one for the
extended context. The units consist of a multiheaded attention layer with
$H=2$ heads each. Each head computes a separate $128/H$-dimensional linear
projection of the query, the keys, and the values, and applies scaled dot
product attention as in [88]. Then, it concatenates the $H$ outputs, and
linearly projects them back to a new 128-dimensional query. After the
multiheaded attention, the resulting query follows the rest of the pipeline in
the Tx unit (as illustrated in Fig. 3) to obtain the _updated query_. Note
that each unit in the $i$-th layer provides its own updated query, denoted as
$\mathbf{q}_{L_{i}}\in\mathbb{R}^{128}$ and
$\mathbf{q}_{E_{i}}\in\mathbb{R}^{128}$, $0<i\leq N$. These are next
concatenated together and fed to a FC layer to obtain the $i$-th layer’s joint
updated query
$\mathbf{q}_{i}=\mathrm{ReLU}\left(\mathbf{\Theta}_{Q_{i}}^{\top}(\mathbf{q}_{\mathrm{L}_{i}}\mathbin{\|}\mathbf{q}_{\mathrm{E}_{i}})\right)$,
where $\mathbf{\Theta}_{Q_{i}}\in\mathbb{R}^{256\times 128}$. Finally,
$\mathbf{q}_{i}$ is fed as input to the next ($i+1$-th) layer.

Inference. The per-chunk OCEAN traits are obtained by applying a FC layer to
the updated query from the $N$-th (last) layer, i.e.
$\mathbf{y}=\mathbf{\Theta}_{\mathrm{FC}}^{\top}\mathbf{q}_{N}$ where
$\mathbf{\Theta}_{\mathrm{FC}}\in\mathbb{R}^{128\times 5}$. Final per-trait,
per-subject predictions are computed as the median of the chunks predictions
for each participant.

###  4.2 Experimental setup

Table 3: Evaluated scenarios. Mean value baseline (B) obtained from the mean
of the per-trait ground truth labels of the training set; and the proposed
method with/without Local (L) and Extended (E) context, Metadata (m), and
Audio (a) information.

|  Query |  Key and Value
---|---|---
|  Face∗ |  Metadata∗ |  Frame∗ |  Frame‡ |  Metadata‡ | Audio
B | - |  \-  | - | - | - | -
L | ✓ |  \-  | ✓ | - | - | -
Lm | ✓ |  ✓  | ✓ | - | - | -
LE | ✓ |  \-  | ✓ | ✓ | - | -
LEm | ✓ |  ✓  | ✓ | ✓ | ✓ | -
LEa | ✓ |  \-  | ✓ | ✓ | - | ✓
LEam | ✓ |  ✓  | ✓ | ✓ | ✓ | ✓
∗ target person and ‡ interlocutor data.

Table 4: Obtained results on different tasks. Legend: Mean value baseline (B)
obtained from the mean of the per-trait ground truth labels of the training
set; and the proposed method with/without Local (L) and/or Extended (E)
context, Metadata (m), and Audio (a) information.

|  Animals |  Ghost |  Lego | Talk
---|---|---|---|---
| O | C | E | A | N |  Avg | O | C | E | A | N |  Avg | O | C | E | A | N |  Avg | O | C | E | A | N | Avg
B | 0.731 | 0.871 | 0.988 | 0.672 | 1.206 |  0.894  | 0.733 | 0.887 | 0.991 | 0.674 | 1.220 |  0.901  | 0.738 | 0.871 | 0.990 | 0.676 | 1.204 |  0.896  | 0.731 | 0.872 | 0.991 | 0.673 | 1.211 | 0.896
L | 0.742 | 0.879 | 0.955 | 0.674 | 1.133 |  0.877  | 0.744 | 0.891 | 1.010 | 0.677 | 1.242 |  0.913  | 0.723 | 0.852 | 0.917 | 0.676 | 1.164 |  0.866  | 0.769 | 0.769 | 0.997 | 0.664 | 1.177 | 0.875
Lm | 0.721 | 0.874 | 0.946 | 0.684 | 1.154 |  0.876  | 0.759 | 0.859 | 1.027 | 0.642 | 1.208 |  0.899  | 0.725 | 0.798 | 0.857 | 0.618 | 1.101 |  0.820  | 0.743 | 0.798 | 0.962 | 0.636 | 1.168 | 0.861
LE | 0.733 | 0.832 | 0.988 | 0.672 | 1.221 |  0.889  | 0.731 | 0.905 | 0.956 | 0.676 | 1.291 |  0.912  | 0.731 | 0.885 | 0.949 | 0.676 | 1.230 |  0.894  | 0.738 | 0.793 | 0.964 | 0.673 | 1.094 | 0.852
LEm | 0.736 | 0.834 | 0.968 | 0.669 | 1.192 |  0.880  | 0.743 | 0.944 | 0.868 | 0.657 | 1.153 |  0.873  | 0.727 | 0.763 | 0.826 | 0.611 | 1.037 |  0.793 | 0.825 | 0.718 | 0.878 | 0.639 | 1.047 | 0.821
LEa | 0.722 | 0.827 | 0.954 | 0.672 | 1.211 |  0.877  | 0.730 | 0.872 | 0.950 | 0.672 | 1.199 |  0.885  | 0.742 | 0.867 | 0.941 | 0.672 | 1.229 |  0.890  | 0.757 | 0.728 | 0.970 | 0.664 | 1.106 | 0.845
LEam | 0.737 | 0.756 | 0.887 | 0.580 | 1.023 |  0.797 | 0.741 | 0.893 | 0.844 | 0.667 | 1.139 |  0.857 | 0.745 | 0.839 | 0.953 | 0.659 | 1.099 |  0.859  | 0.773 | 0.790 | 0.869 | 0.670 | 0.985 | 0.817

This section describes the experimental setup used to assess the performance
of the personality inference model. The evaluation is performed on all tasks
except Gaze, in which very few personality indicators were present due to the
task design. We use frontal camera views (FC1 and FC2, see Fig. 1), in line
with the proposed methodology.

As personality labels, we use the raw OCEAN scores obtained from the self-
reported BFI-2 questionnaire, converted into z-scores using descriptive data
from normative samples.

Data and splits description. We use the subset of data composed of
participants aged 16 years and above, for which Big-five personality traits
are available (see Sec. 3.3). Subject-independent training, validation and
test splits were selected following a greedy optimization procedure that aimed
at having a similar distribution in each split with respect to participant and
session characteristics, while ensuring that no participants appeared in
different splits.

In terms of sessions and participants, the final splits respectively contain:
116/99 for training, 18/20 for validation, and 11/15 for test. Although the
validation split is larger than the test split, the latter contains a better
trait balance. Since the duration of the videos is not constant throughout
sessions and tasks, in order to balance the number of samples we uniformly
selected around 120 chunks from each stream, based on the median number of
chunks per video. The final sample of chunks contains $94\,960$ instances for
training, $15\,350$ for validation and $7\,870$ for test, distributed among
the 4 tasks.

Evaluation protocol. We follow an incremental approach, starting from the
local context. Six different scenarios are evaluated, summarized in Table 3.
We train one model per scenario and task, since each of the four tasks can
elicit different social signals and behaviors (detailed in Sec. 3.4), which
can be correlated to different degrees with distinct aspects of each
personality trait. Results are evaluated with respect to the Mean Squared
Error between the aggregated personality trait score and associated ground
truth label for each individual in the test set. We also compare the results
to a mean value baseline (‘‘B’’), computed as the mean of the per-trait ground
truth labels of the training set.

###  4.3 Discussion of results

Obtained per-task results for the different scenarios are shown in Table 4. We
discuss some of the findings below.

Effect of including extended (E) visual information. The extended context
contains visual information from the other interlocutor’s behaviors and
surrounding scene, allowing the model to consider interpersonal influences
during a chunk. By comparing ‘‘L’’ vs. ‘‘LE’’ we can observe that, on average,
only Talk benefits from the addition of the extended visual context. Trait-
wise, Extraversion improves for all tasks except for Lego, which performs
worse for all traits. This can be attributed to the fact that the interaction
during this type of collaboration is more slow-paced than in other tasks.
Therefore, interpersonal influences cannot be properly captured during just
one chunk. In contrast, for more natural tasks such as Talk, or fast-moving
games such as Ghost, there are many instant actions-reactions that can be
observed during a single chunk, the effect of which is reflected in the
improved results for those tasks. This motivates the need to extend the model
to capture longer-time interpersonal dependencies, characteristic of human
interactions, across a series of ordered chunks along time, to truly benefit
from this extended information.

Effect of including metadata (m) information. The inclusion of metadata
validates our intuition that personal, task, and dyadic details provide
relevant information to the model to produce overall better predictions,
particularly if the cases ‘‘L’’ vs. ‘‘Lm’’, ‘‘LE’’ vs. ‘‘LEm’’, and ‘‘LEa’’
vs. ‘‘LEam’’ are compared, with the largest improvement observed for Lego
(11.29%, ‘‘LE’’ vs. ‘‘LEm’’ case). Considering the high heterogeneity and
dimensionality of behaviors revealed in an interaction and their multiple
meanings, these concise features appear to be beneficial to better guide the
model and establish meaningful patterns in the data. Nonetheless, a systematic
study would be needed to assess the effect of each feature individually.

Effect of including audio (a) information. From comparing ‘‘LE’’ vs. ‘‘LEa’’
and ‘‘LEm’’ vs. ‘‘LEam’’, we observe that better results are obtained, on
average, for all the tasks when audio information is considered. In line with
previous literature [48], it is clear that paralinguistic acoustic features
are required to better model personality. However, the observed improvement is
smaller for Lego. One plausible reason would be the noise produced by the Lego
pieces while being moved, or by the instructions book while turning its pages
close to the microphones, which would interfere with the learning process. In
the case of more natural routines like Talk, the influence of audio is not as
strong as we would have expected. In contrast, Animals, another speaking-based
task, obtains the best results for almost all traits when audio is considered.
There is one salient difference among these two tasks that may explain this
pattern. The latter elicits more individual covert thinking and cognitive
processes that cannot be entirely observed from the visual modality, so most
of the overt information comes from the spoken conversation. In contrast, the
former elicits a larger range of visual cues which may be more relevant than
acoustic features for certain traits.

Putting everything together. In the last experiment (‘‘LEam’’), the model is
aware of the overall contextual information. We notice that apart from Lego,
for which the audio drawbacks were already commented, all the other tasks seem
to benefit from the provided knowledge, obtaining the lowest error value on
average.

Baseline comparison. We observe that Agreeableness, followed by Openness,
obtain the lowest error among mean value baseline (‘‘B’’) results, indicating
that ground truth labels for such traits are more concentrated. In those
cases, none of the models achieve a substantial improvement over the baseline,
except for Animals, where ‘‘LEam’’ obtains an error of 0.58, the lowest
overall. At the other end we find Neuroticism, which is the trait with most
spread values, but also the one for which we obtain the largest benefits with
the evaluated models. In particular, the largest improvement overall (18.66%)
is given by ‘‘LEam’’ for Talk.

##  5 Conclusion

This paper introduced UDIVA, the largest multiview audiovisual dataset of
dyadic face-to-face non-scripted interactions. To validate part of its
potential, we proposed a multimodal transformer-based method for inferring the
personality of a target person in a dyadic scenario. We incrementally combined
different sources of context (both interlocutors’ scene, acoustic and task
information) finding consistent improvements as they were added, which is
consonant with human interaction research in the psychology field.

UDIVA is currently being annotated with additional labels (e.g.,
transcriptions, continuous action/intention for human-object-human
interaction) to allow for a more holistic analysis of human interaction from
both individual and dyadic perspectives. From a methodological point of view,
we plan to extend the proposed architecture to better capture long-term
discriminative features. Nevertheless, we are releasing this data555The
dataset will be available at
http://chalearnlap.cvc.uab.es/dataset/39/description/. with the purpose of
advancing the research and understanding of human communication from a
multidisciplinary perspective, far beyond personality analysis.

Acknowledgements. This work has been partially supported by the Spanish
projects TIN2015-66951-C2-2-R, RTI2018-095232-B-C22, PID2019-105093GB-I00
(MINECO/FEDER, UE), CERCA Programme/Generalitat de Catalunya, and ICREA under
the ICREA Academia programme. We gratefully thank Alejandro Alfonso, Rubén
Ballester, Rubén Barco, Aleix Casellas, Roger Hernando, Andreu Masdeu, Carla
Morral, Pablo Lázaro, Luis M. Pérez, Italo Vidal, Daniel Vilanova, all HUPBA
members, and all participants for their support in the dataset collection
phase. We also thank the Department of Mathematics and Informatics of
Universitat de Barcelona for granting facilities access, the Data Protection
and Bioethics departments for their assistance, and Noldus IT for loaning part
of the recording setup.

## References

  * [1] Judee K Burgoon, Lesa A Stern, and Leesa Dillman.  Interpersonal adaptation: Dyadic interaction patterns.  Cambridge University Press, 2007.
  * [2] Dawn O Braithwaite and Paul Schrodt.  Engaging theories in interpersonal communication: Multiple perspectives.  Sage Publications, 2014.
  * [3] Alessandro Vinciarelli, Maja Pantic, Dirk Heylen, Catherine Pelachaud, Isabella Poggi, Francesca D’Errico, and Marc Schroeder.  Bridging the gap between social animal and unsocial machine: A survey of social signal processing.  IEEE Transactions on Affective Computing, 3(1):69--87, 2011.
  * [4] Rana El Kaliouby, Rosalind Picard, and Simon Baron-Cohen.  Affective computing and autism.  Annals of the New York Academy of Sciences, 1093(1):228--248, 2006\.
  * [5] Chaitanya Ahuja, Shugao Ma, Louis-Philippe Morency, and Yaser Sheikh.  To react or not to react: End-to-end visual pose forecasting for personalized avatar during dyadic conversations.  In 2019 International Conference on Multimodal Interaction, pages 74--84, 2019.
  * [6] Anna Esposito and Lakhmi C Jain.  Modeling emotions in robotic socially believable behaving systems.  In Toward Robotic Socially Believable Behaving Systems-Volume I, pages 9--14. Springer, 2016.
  * [7] Shrikanth Narayanan and Panayiotis G Georgiou.  Behavioral signal processing: Deriving human behavioral informatics from speech and language.  Proceedings of the IEEE, 101(5):1203--1233, 2013.
  * [8] Alessandro Vinciarelli, Maja Pantic, and Hervé Bourlard.  Social signal processing: Survey of an emerging domain.  Image and Vision Computing, 27(12):1743 -- 1759, 2009.
  * [9] Herbert H Clark, Robert Schreuder, and Samuel Buttrick.  Common ground at the understanding of demonstrative reference.  Journal of verbal learning and verbal behavior, 22(2):245--258, 1983\.
  * [10] Ronen Cuperman and William Ickes.  Big five predictors of behavior and perceptions in initial dyadic interactions: Personality similarity helps extraverts and introverts, but hurts “disagreeables”.  Journal of personality and social psychology, 97(4):667, 2009.
  * [11] Ullica Segerstrale and Peter Molnár.  Nonverbal communication: where nature meets culture.  Routledge, 2018.
  * [12] Gerald L Clore and Jeffrey R Huntsinger.  How emotions inform judgment and regulate thought.  Trends in cognitive sciences, 11(9):393--399, 2007.
  * [13] David A Kenny.  Models of non-independence in dyadic research.  Journal of Social and Personal Relationships, 13(2):279--294, 1996\.
  * [14] Bernd Dudzik, Michel-Pierre Jansen, Franziska Burger, Frank Kaptein, Joost Broekens, Dirk KJ Heylen, Hayley Hung, Mark A Neerincx, and Khiet P Truong.  Context in human emotion perception for automatic affect detection: A survey of audiovisual databases.  In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 206--212. IEEE, 2019.
  * [15] Hye Jeong Park and Jae Hwa Lee.  Looking into the personality traits to enhance empathy ability: A review of literature.  In International Conference on Human-Computer Interaction, pages 173--180, 2020.
  * [16] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman.  Video action transformer network.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [17] Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis-Philippe Morency, and Ehsan Hoque.  Integrating multimodal information in large pretrained transformers.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2359--2369, 2020.
  * [18] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan.  IEMOCAP: Interactive emotional dyadic motion capture database.  Language Resources and Evaluation, 42(4):335, Nov 2008.
  * [19] Philippe Blache, Roxane Bertrand, Gaëlle Ferré, Berthille Pallaud, Laurent Prévot, and Stéphane Rauzy.  The corpus of interactional data: A large multimodal annotated resource.  In Handbook of linguistic annotation, pages 1323--1356. Springer, 2017.
  * [20] Ellen Douglas-Cowie, Roddy Cowie, Ian Sneddon, Cate Cox, Orla Lowry, Margaret Mcrorie, Jean-Claude Martin, Laurence Devillers, Sarkis Abrilian, Anton Batliner, et al.  The humaine database: Addressing the collection and annotation of naturalistic and induced emotional data.  In International conference on affective computing and intelligent interaction, pages 488--500. Springer, 2007.
  * [21] Ellen Douglas-Cowie, Cate Cox, Jean-Claude Martin, Laurence Devillers, Roddy Cowie, Ian Sneddon, Margaret McRorie, Catherine Pelachaud, Christopher Peters, Orla Lowry, et al.  The humaine database.  In Emotion-Oriented Systems, pages 243--284. Springer, 2011.
  * [22] James Rehg, Gregory Abowd, Agata Rozga, Mario Romero, Mark Clements, Stan Sclaroff, Irfan Essa, O Ousley, Yin Li, Chanho Kim, et al.  Decoding children’s social behavior.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3414--3421, 2013.
  * [23] Sanjay Bilakhia, Stavros Petridis, Anton Nijholt, and Maja Pantic.  The mahnob mimicry database: A database of naturalistic human interactions.  Pattern recognition letters, 66:52--61, 2015.
  * [24] Iftekhar Naim, M Iftekhar Tanveer, Daniel Gildea, and Mohammed Ehsan Hoque.  Automated prediction and analysis of job interview performance: The role of what you say and how you say it.  In IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), volume 1, pages 1--6, 2015.
  * [25] Angeliki Metallinou, Zhaojun Yang, Chi-Chun Lee, Carlos Busso, Sharon Carnicke, and Shrikanth Narayanan.  The USC creativeit database of multimodal dyadic interactions: From speech and full body motion capture to continuous emotional annotations.  Lang. Resour. Eval., 50(3):497--521, 2016.
  * [26] C. Busso, S. Parthasarathy, A. Burmania, M. AbdelWahab, N. Sadoughi, and E. Mower Provost.  MSP-IMPROV: An acted corpus of dyadic interactions to study emotion perception.  IEEE Transactions on Affective Computing, 8(1):67--80, January-March 2017.
  * [27] Huili Chen, Yue Zhang, Felix Weninger, Rosalind Picard, Cynthia Breazeal, and Hae Won Park.  Dyadic speech-based affect recognition using dami-p2c parent-child multimodal interaction dataset.  In Proceedings of the 2020 International Conference on Multimodal Interaction, pages 97--106, 2020.
  * [28] Aurélie Zara, Valérie Maffiolo, Jean Claude Martin, and Laurence Devillers.  Collection and annotation of a corpus of human-human multimodal interactions: Emotion and others anthropomorphic characteristics.  In International Conference on Affective Computing and Intelligent Interaction, pages 464--475. Springer, 2007.
  * [29] Emilie Delaherche, Mohamed Chetouani, Ammar Mahdhaoui, Catherine Saint-Georges, Sylvie Viaux, and David Cohen.  Interpersonal synchrony: A survey of evaluation methods across disciplines.  IEEE Transactions on Affective Computing, 3(3):349--365, 2012.
  * [30] Ran Zhao, Tanmay Sinha, Alan W Black, and Justine Cassell.  Socially-aware virtual agents: Automatically assessing dyadic rapport from temporal patterns of behavior.  In International conference on intelligent virtual agents, pages 218--233, 2016.
  * [31] Soumia Dermouche and Catherine Pelachaud.  Engagement modeling in dyadic interaction.  In 2019 International Conference on Multimodal Interaction, pages 440--445, 2019.
  * [32] Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  From facial expression recognition to interpersonal relation prediction.  International Journal of Computer Vision, 126(5):550--569, 2018\.
  * [33] Lisa Feldman Barrett, Batja Mesquita, and Maria Gendron.  Context in emotion perception.  Current Directions in Psychological Science, 20(5):286--290, 2011\.
  * [34] Aidan GC Wright.  Current directions in personality science and the potential for advances through computing.  IEEE Transactions on Affective Computing, 5(3):292--296, 2014.
  * [35] Alessandro Vinciarelli, Anna Esposito, Elisabeth André, Francesca Bonin, Mohamed Chetouani, Jeffrey F Cohn, Marco Cristani, Ferdinand Fuhrmann, Elmer Gilmartin, Zakia Hammal, et al.  Open challenges in modelling, analysis and synthesis of human behaviour in human--human and human--machine interactions.  Cognitive Computation, 7(4):397--413, 2015.
  * [36] Philip Moore.  Do we understand the relationship between affective computing, emotion and context-awareness?  Machines, 5(3):16, 2017.
  * [37] Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy.  Emotion recognition in conversation: Research challenges, datasets, and recent advances.  IEEE Access, 7:100943--100953, 2019.
  * [38] Chi-Chun Lee, Carlos Busso, Sungbok Lee, and Shrikanth S Narayanan.  Modeling mutual influence of interlocutor emotion states in dyadic spoken interactions.  In Tenth Annual Conference of the International Speech Communication Association, 2009.
  * [39] Angeliki Metallinou, Athanasios Katsamanis, and Shrikanth Narayanan.  Tracking continuous emotional trends of participants during affective dyadic interactions using body language and speech information.  Image and Vision Computing, 31(2):137--152, 2013.
  * [40] Zheng Lian, Jianhua Tao, Bin Liu, Jian Huang, Zhanlei Yang, and Rongjun Li.  Context-dependent domain adversarial neural network for multimodal emotion recognition.  Proc. Interspeech 2020, pages 394--398, 2020.
  * [41] Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria.  Dialoguernn: An attentive rnn for emotion detection in conversations.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6818--6825, 2019.
  * [42] Peixiang Zhong, Di Wang, and Chunyan Miao.  Knowledge-enriched transformer for emotion detection in textual conversations.  In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 165--176, 2019.
  * [43] Qingbiao Li, Chunhua Wu, Zhe Wang, and Kangfeng Zheng.  Hierarchical transformer network for utterance-level emotion recognition.  Applied Sciences, 10(13):4447, 2020.
  * [44] Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency.  Multi-attention recurrent network for human communication comprehension.  In AAAI Conference on Artificial Intelligence, volume 2018, page 5642, 2018.
  * [45] Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann.  Conversational memory network for emotion recognition in dyadic dialogue videos.  In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2122--2132, 2018.
  * [46] Jeng-Lin Li and Chi-Chun Lee.  Attention learning with retrievable acoustic embedding of personality for emotion recognition.  In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 171--177. IEEE, 2019.
  * [47] Christopher Soto and Oliver John.  The next big five inventory (bfi-2): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power.  Journal of Personality and Social Psychology, 113:117--143, 07 2017\.
  * [48] Alessandro Vinciarelli and Gelareh Mohammadi.  A survey of personality computing.  IEEE Transaction on Affective Computing, 5(3):273--291, 2014.
  * [49] Robert R. McCrae and Oliver P. John.  An introduction to the five-factor model and its applications.  Journal of Personality, 60(2):175--215, 1992.
  * [50] Oya Aran and Daniel Gatica-Perez.  Cross-domain personality prediction: from video blogs to small group meetings.  In International Conference on Multimodal Interaction, pages 127--130, 2013.
  * [51] Laurent Son Nguyen, Alvaro Marcos-Ramiro, Martha Marrón Romera, and Daniel Gatica-Perez.  Multimodal analysis of body communication cues in employment interviews.  In International Conference on Multimodal Interaction (ICMI), pages 437--444, 2013.
  * [52] Yash Mehta, Navonil Majumder, Alexander Gelbukh, and Erik Cambria.  Recent trends in deep learning based personality detection.  Artificial Intelligence Review, pages 1--27, 2019.
  * [53] Ming-Hsiang Su, Chung-Hsien Wu, and Yu-Ting Zheng.  Exploiting turn-taking temporal evolution for personality trait perception in dyadic conversations.  Transactions on Audio, Speech, and Language Processing, 24(4):733--744, 2016.
  * [54] J. C. S. Jacques Junior, Y. Güçlütürk, M. Perez, U. Güçlü, C. Andujar, X. Baró, H. J. Escalante, I. Guyon, M. A. J. Van Gerven, R. Van Lier, and S. Escalera.  First impressions: A survey on vision-based apparent personality trait analysis.  IEEE Transactions on Affective Computing, pages 1--1, 2019.
  * [55] Sheng Fang, Catherine Achard, and Séverine Dubuisson.  Personality classification and behaviour interpretation: An approach based on feature categories.  In Proceedings of the 18th ACM International Conference on Multimodal Interaction, pages 225--232, 2016.
  * [56] Yun-Shao Lin and Chi-Chun Lee.  Using interlocutor-modulated attention blstm to predict personality traits in small group interaction.  In International Conference on Multimodal Interaction, pages 163--169, 2018.
  * [57] Lingyu Zhang, Indrani Bhattacharya, Mallory Morgan, Michael Foley, Christoph Riedl, Brooke Welles, and Richard Radke.  Multiparty visual co-occurrences for estimating personality traits in group meetings.  In The IEEE Winter Conference on Applications of Computer Vision, pages 2085--2094, 2020.
  * [58] Ricardo Darío Pérez Principi, Cristina Palmero, Julio C Junior, and Sergio Escalera.  On the effect of observed subject biases in apparent personality analysis from audio-visual signals.  IEEE Transactions on Affective Computing, 2019.
  * [59] Michael Kipp, Jean-Claude Martin, Patrizia Paggio, and Dirk Heylen.  Multimodal corpora: from models of natural interaction to systems and applications, volume 5509.  Springer, 2009.
  * [60] Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain.  A review of affective computing: From unimodal analysis to multimodal fusion.  Information Fusion, 37:98--125, 2017.
  * [61] Alexandros Stergiou and Ronald Poppe.  Analyzing human--human interactions: A survey.  Computer Vision and Image Understanding, 188:102799, 2019.
  * [62] Angelo Cafaro, Johannes Wagner, Tobias Baur, Soumia Dermouche, Mercedes Torres Torres, Catherine Pelachaud, Elisabeth André, and Michel Valstar.  The noxi database: multimodal recordings of mediated novice-expert interactions.  In Proceedings of the 19th ACM International Conference on Multimodal Interaction, pages 350--359, 2017.
  * [63] Jean Kossaifi, Robert Walecki, Yannis Panagakis, Jie Shen, Maximilian Schmitt, Fabien Ringeval, Jing Han, Vedhas Pandit, Antoine Toisoul, Bjoern W Schuller, et al.  Sewa db: A rich database for audio-visual emotion and sentiment research in the wild.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019\.
  * [64] Hanbyul Joo, Tomas Simon, Mina Cikara, and Yaser Sheikh.  Towards social artificial intelligence: Nonverbal social signal prediction in a triadic interaction.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [65] Oya Celiktutan, Efstratios Skordos, and Hatice Gunes.  Multimodal human-human-robot interactions (mhhri) dataset for studying personality and engagement.  IEEE Transactions on Affective Computing, 2017.
  * [66] Xavier Alameda-Pineda, Jacopo Staiano, Ramanathan Subramanian, Ligia Batrinca, Elisa Ricci, Bruno Lepri, Oswald Lanz, and Nicu Sebe.  Salsa: A novel dataset for multimodal group behavior analysis.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(8):1707--1720, 2015.
  * [67] Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea.  Meld: A multimodal multi-party dataset for emotion recognition in conversations.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 527--536, 2019.
  * [68] Mary Rothbart, Stephan Ahadi, Karen Hershey, and Philip Fisher.  Investigations of temperament at three to seven years: The children’s behavior questionnaire.  Child development, 72:1394--408, 09 2001.
  * [69] Nuria Osa, Roser Granero, Eva Penelo, Josep Domènech, and Lourdes Ezpeleta.  The short and very short forms of the children’s behavior questionnaire in a community sample of preschoolers.  Assessment, 21, 11 2013.
  * [70] Lesa Ellis and Mary Rothbart.  Revision of the early adolescent temperament questionnaire.  Poster presented at the 2001 Biennal Meeting of the Society for Research in Child Development, 01 2001.
  * [71] Michael Ashton and Kibeom Lee.  The hexaco-60: A short measure of the major dimensions of personality.  Journal of personality assessment, 91:340--5, 07 2009.
  * [72] David Gallardo-Pujol, Antonio Andres-Pueyo, and Alberto Maydeu-Olivares.  Maoa genotype, social exclusion and aggression: An experimental test of a gene-environment interaction.  Genes, brain, and behavior, 12, 10 2012.
  * [73] Kipling D Williams, Cassandra L Govan, Vanessa Croker, Daniel Tynan, Maggie Cruickshank, and Albert Lam.  Investigations into differences between social-and cyberostracism.  Group dynamics: Theory, research, and practice, 6(1):65, 2002.
  * [74] Jennifer Abel and Molly Babel.  Cognitive load reduces perceived linguistic convergence between dyads.  Language and Speech, 60(3):479--502, 2017.
  * [75] Sally Lindsay, Kara Grace Hounsell, and Celia Cassiani.  A scoping review of the role of lego® therapy for improving inclusion and social skills among children and youth with autism.  Disability and health journal, 10(2):173--182, 2017.
  * [76] Riccardo Fusaroli, Johanne S Bjørndahl, Andreas Roepstorff, and Kristian Tylén.  A heart for interaction: Shared physiological dynamics and behavioral coordination in a collective, creative construction task.  Journal of Experimental Psychology: Human Perception and Performance, 42(9):1297, 2016.
  * [77] Willard W Hartup, Doran C French, Brett Laursen, Mary Kathleen Johnston, and John R Ogawa.  Conflict and friendship relations in middle childhood: Behavior in a closed-field situation.  Child Development, 64(2):445--454, 1993.
  * [78] Marion K Underwood, Bertrina L Scott, Mikal B Galperin, Gretchen J Bjornstad, and Alicia M Sexton.  An observational study of social exclusion under varied conditions: Gender and developmental differences.  Child Development, 75(5):1538--1555, 2004.
  * [79] Icek Ajzen.  Attitudes, traits, and actions: Dispositional prediction of behavior in personality and social psychology.  In Advances in experimental social psychology, volume 20, pages 1--63. Elsevier, 1987.
  * [80] Damaris Aschwanden, Mathias Allemand, and Patrick L Hill.  Cognitive methods in personality research.  The Wiley Encyclopedia of Personality and Individual Differences: Measurement and Assessment, pages 49--54, 2020.
  * [81] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications, 2017.
  * [82] Shuo Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Wider face: A face detection benchmark.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [83] Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan.  Large-scale weakly-supervised pre-training for video action recognition.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 12046--12055, 2019.
  * [84] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In IEEE conference on Computer Vision and Pattern Recognition (CVPR), pages 6450--6459, 2018.
  * [85] Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron Weiss, and Kevin Wilson.  Cnn architectures for large-scale audio classification.  In International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2017.
  * [86] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan.  Youtube-8m: A large-scale video classification benchmark.  CoRR, abs/1609.08675, 2016.
  * [87] Yaw M Mensah and Hsiao-Yin Chen.  Global clustering of countries by culture--an extension of the globe study.  Available at SSRN 2189904, 2013.
  * [88] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998--6008. 2017.


# Pop2Piano : Pop Audio-based Piano Cover Generation

###### Abstract

The piano cover of pop music is widely enjoyed by people. However, the
generation task of the pop piano cover is still understudied. This is partly
due to the lack of synchronized {Pop, Piano Cover} data pairs, which made it
challenging to apply the latest data-intensive deep learning-based methods. To
leverage the power of the data-driven approach, we make a large amount of
paired and synchronized {Pop, Piano Cover} data using an automated pipeline.
In this paper, we present Pop2Piano, a Transformer network that generates
piano covers given waveforms of pop music. To the best of our knowledge, this
is the first model to directly generate a piano cover from pop audio without
melody and chord extraction modules. We show that Pop2Piano trained with our
dataset can generate plausible piano covers.

Index Terms—  Piano Cover, Music Arrangement, Transformer, Music
Synchronization

##  1 Introduction

Piano cover refers to a musical piece in which all musical elements of an
existing song are recreated or arranged in the form of a piano performance.
Piano covers of pop music are one of the widely enjoyed forms of music. For
example, people use them for music education purposes, and piano cover
creators have millions of subscribers on media such as Youtube.

In order for a human to create a piano cover, it is necessary to recognize all
musical elements such as melodies, chords or moods from the original audio and
reinterpret them into musically appropriate piano performances. Therefore,
making a piano cover is not an easy task even for humans as it is a creative
task and requires musical knowledge.

There were studies on arranging given pop audio to other instruments [1, 2].
They used multiple external modules to extract explicit musical information,
such as melodies and chords, from the original audio. However, in addition to
this information, piano covers can be made under the influence of various
implicit musical characteristics like the atmosphere of the music and the
composer’s arrangement style. So we believe that the research on end-to-end
conversion between audio and piano covers can also be considered valuable.

Meanwhile, Deep learning has been reported to have excellent performance in
modeling high-dimensional music audio data. To the best of our knowledge,
however, there has been no deep learning study on piano cover modeling using
waveforms from pop music directly. We suspect that this is due to the absence
of large amounts of synchronized {Pop, Piano Cover} data for such modeling,
and also because it is practically difficult for humans to create sufficient
amounts of data for deep learning.

In this study, we introduce a study on the Transformer-based piano cover
generation model Pop2Piano, which generates a piano performance (MIDI) from
the waveform of pop music. Our contribution is summarized as follows:

  * •

We propose a novel approach to piano cover generation. For this, We build
300-hour of synchronized {Pop, Piano Cover} dataset, called Piano Cover
Synchronized to Pop Audio (PSP), and introduce the preprocessing system used
to make this dataset.

  * •

We design Pop2Piano, a Transformer network that generates piano covers from
given audio. It does not use a melody or chord extractor but uses the
spectrogram itself as an input. It also controls the covering style using the
arranger token.

  * •

We upload a list of data and preprocessing codes to reproduce the PSP dataset,
and release an executable Pop2Piano demo on
Colab111https://sweetcocoa.github.io/pop2piano_samples.

Fig. 1: A preprocessing pipeline for synchronizing and filtering paired {Pop,
Piano Cover} audio data.

##  2 Related Work

###  2.1 Automatic Music Transcription

The task most similar to this work in terms of the form of data is automatic
music transcription (AMT). AMT is the task of estimating note information of a
performance from the waveforms of musical instruments. In the piano AMT, [3,
4, 5] used architectures combining CNN for feature extraction and RNN for
modeling note onset and offset and frame. [6] used the Transformer model
(T5-small [7]), and they used the spectrogram of the piano audio as input to
the encoder and generated a MIDI sequence as the output of the decoder.

Multi-instrument AMT is the task of estimating note information for each
instrument from the mixed sounds of several instruments. Cerberus [8] is a
multi-instrument AMT for a determined set of musical instruments using an RNN
structure. MT3 [9] has a structure similar to [6], but it shows that multi-
instrument AMT is possible by mixing datasets for several instruments.

AMT has yielded promising results when there is a large synchronized
instrument-audio dataset. [3, 4, 5, 6] were trained using MAESTRO [10], the
200-hour synchronized piano MIDI-audio dataset. And [8, 9] obtained strong
performance using Slakh2100 [11], the 150-hour multi-instrument MIDI-audio
dataset. Since the task of piano cover generation is similar to AMT in the
sense of input and output, in this study, we build a 300-hour synchronized
piano cover dataset and use it for training.

Although AMT and this study have similarities in converting a given audio into
musical notes, there is an important difference in that AMT has a single
correct answer, and the piano cover generation task does not. For example, the
monophonic vocal of the original song may be expressed as a monophonic note
sequence in the piano cover, but may also be performed as voicing (the
simultaneous vertical placement of related notes at the same time) or doubling
(playing the same notes in different octaves at the same time). The harmony of
the original song may appear as the accompaniment of various textures in the
piano cover, and these textures may also vary depending on the atmosphere of
the original song or the style of the arranger.

Fig. 2: The architecture of our model is an encoder-decoder Transformer. Each
input position for the encoder is one frame of the spectrogram. We
concatenated an embedding vector representing a target arranger style to the
spectrogram. Output midi tokens are autoregressively generated from the
decoder.

###  2.2 Audio-based Instrument Cover

Most audio-based instrument cover studies have used external modules to get
melodies and chords from the audio. Song2Guitar [1] generated guitar cover
from pop audio. They used modules to extract melodies, chords and beats
respectively. Then, a guitar tab score was generated by statistically modeling
the fingering probability. In [2], They extracted melodies, chords, and
choruses using external modules. Then, a piano score was generated using rule-
based methods. Unlike them, our study, Pop2Piano, uses only a beat extractor
and generates piano notes from pop audio directly.

##  3 Dataset

The main bottleneck in modeling raw waveform pairs is a long range of
dependencies, as they are computationally challenging to learn the high-
dimensional semantics of music. An alternative to that is a method of training
short-length waveform segments in pairs. However, in the case of using this
method {Pop, Piano Cover} pairs should be synchronized. If it is not
synchronized, there is a high possibility that the correct answer label is not
included in the given audio segment. And since the collected data was not
synchronized, we designed the preprocessing pipeline to obtain synchronized
{Pop, Piano Cover} data. See Fig 1 for overview.

###  3.1 Preprocessing

####  3.1.1 Synchronizing Paired Music

First, we convert piano cover audio into MIDI, using the piano transcription
model [5]. Second, we roughly align piano MIDI with pop audio. We obtain a
warping path using _SynctoolBox_ [12] and then use it to modify the note
timings of MIDI via linear interpolation. This provides a simple
synchronization between the piano cover and pop audio. Third, The note timings
are quantized into 8th-note units. After extracting beats from pop audio using
_Essentia_ 222https://essentia.upf.edu/, the onset and offset of each note in
MIDI are quantized to the nearest 8th-note beat. If the onsets and offsets of
the quantized notes are the same, the offset is moved to the next beat. In
this way, the entropy of data can be lowered by changing the time unit of a
note from continuous-time (seconds) to quantized time (beats).

####  3.1.2 Filtering Low Quality Samples

There are cases where the data pairs are unsuitable for various reasons, such
as the difference in musical progress or having different keys from the
original song. To filter out these cases, all data with a melody chroma
accuracy [13] of 0.15 or less, or an audio length difference of 20% or more
are discarded. Melody chroma accuracy is calculated between the pitch contour
of the vocal signal extracted from the audio and the top line of the MIDI.

We use _Spleeter_ [14] to separate the vocal signal. Then, to get the melody
contours of pop music, the f0 sequence of the vocal is calculated using
_Librosa_ [15] _pYIN_ [16]. The sample rate is 44100 and the hop length is
1024, and the time resolution is about 23.21ms.

###  3.2 Piano Cover Synchronized to Pop Audio (PSP)

We collect 5989 piano covers from 21 arrangers and corresponding pop songs on
YouTube. We then synchronized and filter {Pop, Piano Cover}. As a result, a
total of 4989 tracks (307 hours) are left, which are used as a training set,
PSP. Note that the piano cover included in the PSP is unique, but the original
song is not. This will help the model train separately the style of the piano
cover according to the arranger conditions and the acoustic characteristics of
the given audio.

##  4 Model

The Pop2Piano model is a Transformer architecture. For each training step, A
style token and a log-Mel spectrogram of 4-beat audio which is sampled
randomly from the dataset are used as the input, and corresponding
synchronized midi messages are used as the target for the decoder. An overview
of this process is shown as Fig 2.

###  4.1 Inputs and Outputs

Pop2Piano uses a log-Mel spectrogram of pop audio as an encoder input. The
sampling rate is 22050, the window size is 4096, and the hop size is 1024. In
addition, the arranger token, which indicates who arranged the target piano
cover, is embedded and appended before the first frame of the spectrogram.
Each step of the decoder output is chosen from the following types of tokens:

Note Pitch

[128 values] Indicates a pitch event for one of the MIDI pitches. But only the
88 pitches corresponding to piano keys are actually used.

Note On/Off

[2 values] Determines whether previous Note Pitch events are interpreted as
note-on or note-off.

Beat Shift

[100 values] Indicates the relative time shift within the segment quantized
into 8th-note beats(half-beats). It will apply to all subsequent note-related
events until the next Beat Shift event. We define the vocabulary with Beat
Shifts up to 50 beats, but because time resets for each segment, In practice
we use only about 10 events of this type.

EOS, PAD

[2 values] Indicates the end of the sequence and the padding of the sequence.

This dictionary is inspired by [6, 9]. For a detailed example of this token’s
interpretation, See Fig 3. For each input, the model autoregressively
generates outputs until an EOS token is generated in the decoder. To generate
a piano cover of pop audio of arbitrary length, in the inference stage, the
audio is sequentially cropped by 4 beats and used as input to the model, then
generated tokens (except for EOS) are concatenated. After that, the relative
beats of the generated tokens are converted into absolute time using the
absolute time information of the beat extracted from the original song and
then converted into a standard MIDI file.

Fig. 3: An example of piano tokenization. the beat shift token means a
relative time shift from that point in time.

###  4.2 Architecture

The Pop2Piano model architecture is T5-small [7] used for [9]. It is a
Transformer network with an encoder-decoder structure. The number of learnable
parameters is about 59M. Unlike [9], the relative positional embedding of the
original T5 is used instead of the absolute positional embedding.
Additionally, A learnable embedding layer is used for embedding the arranger
style.

##  5 Experiment

In this experiment, we check whether Pop2Piano trained with PSP
(Pop2Piano${}_{\text{PSP}}$) can generate a plausible piano cover and verify
that it follows a specific arranger’s style.

###  5.1 Training Setup

We train the model 2000 epoch with a batch size of 32, and the input audio is
randomly extracted from the dataset with 4-beat length audio. The neural
network is optimized using the AdaFactor[17] optimizer and its learning rate
is 0.001.

###  5.2 Generation Result

(a)

(b)

Fig. 4: (a) A Melspectrogram of an input pop audio. (b) A piano roll of output
MIDI notes corresponding to input audio.

Fig. 5: The dots represent the note density of the piano cover generated by
being conditioned with that arranger. The unit is the number of notes per
second.

Fig 4 is an example of a piano cover generated using arbitrary pop music as an
input. The generated sample can be listened to on the
link333https://sweetcocoa.github.io/pop2piano_samples/fig_on_paper.html. The
original song is a complex audio signal mixed with various sounds such as
vocals, bass, and percussion. Nevertheless, it can be seen that the generated
piano cover shows the vocal melody line of the original song as stacked notes,
and also includes a plausible accompaniment that follows the harmony of the
original song. Also, various piano cover samples generated by Pop2piano can be
listened to with the original song on the demo page.

Additionally, we would like to confirm whether the generated piano cover not
only plausibly followed the original song but also covered it in the style of
the arranger we designated. However, the cover style is very implicit, so
there are few ways to evaluate quantitatively. Inspired by the difference in
note densities among arrangers, we measure the note densities of the piano
cover generated by Pop2Piano${}_{\text{PSP}}$ to indirectly verify whether it
generates a piano cover in the target arranger’s style. As a result, Fig 5
shows that the generated piano cover follows the note density of the target
arranger with high linearity of $R^{2}=0.804$.

Original Songs | Arranger | Average MCA
---|---|---
POP909F | Human | 0.395
PSP | Human | 0.3493
POP909F |  Pop2Piano${}_{\text{PSP}}$ | 0.402 $\pm$ 0.021
Table 1: Average Melody Chroma Accuracy (AMCA) of human-made and generated
piano covers. The piano covers of Pop2Piano are generated for all 21 arrangers
of PSP, and its standard deviation according to arrangers are shown in the
table.

We also calculate the MCA used in 3.1.2 for generated piano covers, showing
that they have a similar level of MCA to the dataset even in training without
explicit melodies. See Table 1. Note that a higher MCA doesn’t necessarily
mean a better piano cover. Therefore, a study on how to evaluate the
generation result of a piano cover will also be a good follow-up study.

###  5.3 Subjective Evaluation

We measure the subjective quality of the generated piano covers by evaluating
their naturalness. Since there is no prior study publicly available, we use a
Pop2Piano network trained with POP909 (Pop2Piano${}_{\text{POP909}}$) as a
baseline model. The comparison with this model indicates how the PSP dataset
affected the performance of the Pop2Piano model.

POP909 [18] is a piano MIDI dataset of 909 Chinese pop songs. There are
separate melody, bridge, and accompaniment tracks in this dataset, but we
merge all the tracks for the consistency of the experiment. Like PSP, we
synchronize {Pop, Piano Cover} pairs through our preprocessing pipeline. After
filtering, 817 tracks remain. In the remaining part of this paper, this
preprocessed POP909 dataset is denoted as POP909F.

We conduct a user study. There are 25 participants and no professional
musicians are included. Participants do not know which model has made each
piano cover. We select 10-seconds from the beginning of the chorus of 25 songs
that are not included in the training dataset. For each song, a piano cover
made by a human arranger is used as GT. However, since GT also needs to be
synchronized with the original song, the synchronization pipeline is applied
and that data is denoted as GTF. After listening to the original song and
piano covers, the participants evaluate how naturally the given piano cover
arranged the original song with 1-5 points.

In the listening evaluation, 70% of participants prefer the piano cover
generated by Pop2Piano${}_{\text{PSP}}$ to Pop2Piano${}_{\text{POP909}}$. In
MOS analysis between Pop2Piano${}_{\text{PSP}}$ and
Pop2Piano${}_{\text{POP909}}$, using paired one-sided Wilcoxon test, we reject
H0 at 99% confidence intervals $(0.222,\texttt{inf})$ with $p=3.34e$$-05$. See
Table 2.

(%) |  GTF | Ours | Baseline
---|---|---|---
vs GTF | - | 29.6 | 22.24
vs Ours | 70.4 | - | 36
vs Baseline | 77.76 | 64 | -
MOS | 3.771 | 3.216 | 2.856
Table 2: The winning rate and Mean Opinion Score(MOS) according to the piano
cover model. ”Ours” denotes Pop2Piano${}_{\text{PSP}}$, ”Baseline” denotes
Pop2Piano${}_{\text{POP909}}$.

###  5.4 Limitation

We recognize that some improvements can be made to our model. For instance,
Pop2Piano uses only four-beat length audio for the context of input.
Therefore, features such as melody contour or texture of accompaniment have
less consistency when generating longer than four-beat. Also, time
quantization based on eighth note beats prevents the model from generating
piano covers with other rhythms such as triplets, 16th notes, and trills.

##  6 Conclusion

We present Pop2Piano, a novel study on generating pop piano covers directly
from audio without using melody or chord extraction modules based on a
Transformer network. We collect PSP, 300 hours of paired {Pop, Piano Cover}
datasets, to train the model. And we design a pipeline that synchronizes them
in a form suitable for training neural networks. We open-source the list of
data and code needed to reproduce the PSP. We show and evaluate that
Pop2Piano${}_{\text{PSP}}$ can generate plausible pop piano covers and also
can mimic the style of a specific arranger.

## References

  * [1] Shunya Ariga, Satoru Fukayama, and Masataka Goto,  “Song2guitar: A difficulty-aware arrangement system for generating guitar solo covers from polyphonic audio of popular music.,”  in ISMIR. Suzhou, 2017, pp. 568–574.
  * [2] Hirofumi Takamori, Takayuki Nakatsuka, Satoru Fukayama, Masataka Goto, and Shigeo Morishima,  “Audio-based automatic generation of a piano reduction score by considering the musical structure,”  in International Conference on Multimedia Modeling. Springer, 2019, pp. 169–181.
  * [3] Curtis Hawthorne, Erich Elsen, Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, and Douglas Eck,  “Onsets and frames: Dual-objective piano transcription,”  arXiv preprint arXiv:1710.11153, 2017.
  * [4] Jong Wook Kim and Juan Pablo Bello,  “Adversarial learning for improved onsets and frames music transcription,”  arXiv preprint arXiv:1906.08512, 2019.
  * [5] Qiuqiang Kong, Bochen Li, Xuchen Song, Yuan Wan, and Yuxuan Wang,  “High-resolution piano transcription with pedals by regressing onset and offset times,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3707–3717, 2021.
  * [6] Curtis Hawthorne, Ian Simon, Rigel Swavely, Ethan Manilow, and Jesse Engel,  “Sequence-to-sequence piano transcription with transformers,”  arXiv preprint arXiv:2107.09142, 2021.
  * [7] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu,  “Exploring the limits of transfer learning with a unified text-to-text transformer,”  arXiv preprint arXiv:1910.10683, 2019.
  * [8] Ethan Manilow, Prem Seetharaman, and Bryan Pardo,  “Simultaneous separation and transcription of mixtures with multiple polyphonic and percussive instruments,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 771–775.
  * [9] Josh Gardner, Ian Simon, Ethan Manilow, Curtis Hawthorne, and Jesse Engel,  “Mt3: Multi-task multitrack music transcription,”  arXiv preprint arXiv:2111.03017, 2021.
  * [10] Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck,  “Enabling factorized piano music modeling and generation with the maestro dataset,”  arXiv preprint arXiv:1810.12247, 2018.
  * [11] Ethan Manilow, Gordon Wichern, Prem Seetharaman, and Jonathan Le Roux,  “Cutting music source separation some slakh: A dataset to study the impact of training data quality and quantity,”  in 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2019, pp. 45–49.
  * [12] Meinard Müller, Yigitcan Özer, Michael Krause, Thomas Prätzlich, and Jonathan Driedger,  “Sync toolbox: A python package for efficient, robust, and accurate music synchronization,”  Journal of Open Source Software, vol. 6, no. 64, pp. 3434, 2021\.
  * [13] Colin Raffel, Brian McFee, Eric J Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, Daniel PW Ellis, and C Colin Raffel,  “mir_eval: A transparent implementation of common mir metrics,”  in In Proceedings of the 15th International Society for Music Information Retrieval Conference, ISMIR. Citeseer, 2014.
  * [14] Romain Hennequin, Anis Khlif, Felix Voituret, and Manuel Moussallam,  “Spleeter: a fast and efficient music source separation tool with pre-trained models,”  Journal of Open Source Software, vol. 5, no. 50, pp. 2154, 2020,  Deezer Research.
  * [15] Brian McFee, Alexandros Metsai, Matt McVicar, Stefan Balke, Carl Thomé, Colin Raffel, Frank Zalkow, Ayoub Malek, Dana, Kyungyun Lee, Oriol Nieto, Dan Ellis, Jack Mason, Eric Battenberg, Scott Seyfarth, Ryuichi Yamamoto, viktorandreevichmorozov, Keunwoo Choi, Josh Moore, Rachel Bittner, Shunsuke Hidaka, Ziyao Wei, nullmightybofo, Adam Weiss, Darío Hereñú, Fabian-Robert Stöter, Pius Friesch, Matt Vollrath, Taewoon Kim, and Thassilo,  “librosa/librosa: 0.9.1,” Feb. 2022.
  * [16] Matthias Mauch and Simon Dixon,  “Pyin: A fundamental frequency estimator using probabilistic threshold distributions,”  in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014, pp. 659–663.
  * [17] Noam Shazeer and Mitchell Stern,  “Adafactor: Adaptive learning rates with sublinear memory cost,”  in International Conference on Machine Learning. PMLR, 2018, pp. 4596–4604.
  * [18] Ziyu Wang, Ke Chen, Junyan Jiang, Yiyi Zhang, Maoran Xu, Shuqi Dai, Xianbin Gu, and Gus Xia,  “Pop909: A pop-song dataset for music arrangement generation,”  arXiv preprint arXiv:2008.07142, 2020.