# Learning to Communicate Functional States with Nonverbal Expressions for
Improved Human-Robot Collaboration

Liam Roy1, Elizabeth A. Croft2, and Dana Kulić3 1Liam Roy is with Monash
University<EMAIL_ADDRESS>A. Croft is with University of
Victoria<EMAIL_ADDRESS>3Dana Kulić is with Monash University
<EMAIL_ADDRESS>Kulić is supported by the ARC Future Fellowship
(FT200100761).Received: Nov 06 2023; Revised Feb 05 2024; Accepted Mar 03
2024This paper was recommended for publication by Editor Gentiane Venture upon
evaluation of the Associate Editor and Reviewers’ comments.Digital Object
Identifier (DOI): see top of this page.

###### Abstract

Collaborative robots must effectively communicate their internal state to
humans to enable a smooth interaction. Nonverbal communication is widely used
to communicate information during human-robot interaction, however, such
methods may also be misunderstood, leading to communication errors. In this
work, we explore modulating the acoustic parameter values (pitch bend, beats
per minute, beats per loop) of nonverbal auditory expressions to convey
functional robot states (accomplished, progressing, stuck). We propose a
reinforcement learning (RL) algorithm based on noisy human feedback to produce
accurately interpreted nonverbal auditory expressions. The proposed approach
was evaluated through a user study with 24 participants. The results
demonstrate that: (i) Our proposed RL-based approach is able to learn suitable
acoustic parameter values which improve the users’ ability to correctly
identify the state of the robot. (ii) Algorithm initialization informed by
previous user data can be used to significantly speed up the learning process.
(iii) The method used for algorithm initialization strongly influences whether
participants converge to similar sounds for each robot state. (iv) Modulation
of pitch bend has the largest influence on user association between sounds and
robotic states.

###### Index Terms:

Human-Robot Collaboration; Multi-Modal Perception for HRI; Social HRI

##  I Introduction

To facilitate seamless human-robot interaction (HRI), collaborative robots
require the ability to convey their internal state to human teammates.
Nonverbal communication forms an essential component of human interactions and
has been an important focus in the development of HRIs [1, 2, 3]. Humans
frequently communicate with nonverbal gestures, and in order to achieve a
comparable degree of fluidity during collaboration, robots may also benefit
from utilizing nonverbal communication strategies. Nonverbal expressions
including body language [2], gestures [4], facial expressions [1], lights [5]
and sounds [6, 7] have been proposed for HRI applications. In most prior work
[8, 9, 10], nonverbal communicative expressions are hand-crafted to convey a
specific robot state, action, intention or emotion (SAIE).

When relevant to the context, nonverbal expressions can convey robot SAIEs
with greater speed, universality, and appeal than spoken words [11, 8].
However, a trade-off exists between the complexity of the information conveyed
and interpretability by diverse human collaborators, which may arise due to
the subjective nature of these expressions. Miscommunications and decreased
collaborative performance have been shown to occur even with seemingly
intuitive nonverbal communication methods [12].

In this work, rather than hand-crafting the communication strategy [8, 9, 10],
our aim is to learn how nonverbal communication should be structured to be
best understood by users. The proposed approach uses nonverbal audio
communication to convey functional robot states. A functional robot state
describes the current task status of the robot (e.g. progressing, stuck, or
accomplished). We auto-tune the acoustic parameter values (pitch bend, beats
per minute, beats per loop) of nonverbal auditory expressions using a learning
algorithm to convey three functional robot states (accomplished, progressing,
stuck). An example real-world HRI scenario where nonverbal audio is
advantageous is search and rescue. A robot must be able to rapidly convey
functional states (e.g. danger, survivor found) to first responders. When
visual modalities are limited due to smoke and debris, audio communication
enables succinct HRI.

Figure 1: A robot nicknamed Jackal learning to generate nonverbal expressions
to be correctly understood by users. The robot state perceived by the human is
used to generate feedback to train the learning algorithm. The Spot robot
image used within our user study procedure is also shown for reference.

This paper makes the following contributions: (i) We propose a learning-based
framework for expressing functional robot states with nonverbal communication.
We employ a reinforcement learning (RL) algorithm utilizing human feedback to
customize expressions for individual users, aiming to enhance state
classification accuracy. Through a user study, we demonstrate that this
learning-based framework leads to significant improvements in users’ ability
to classify the state of the robot. (ii) We show that an algorithm
initialization informed by previous user data can be used to speed up the
learning process. (iii) We show that similarities exist between humans when
interpreting robot states from nonverbal expressions. A high-level
visualization of our approach is shown in Fig.1 and discussed in detail in
Section III.

##  II Related Work

Nonverbal communication is a valuable dimension of HRI, aiding in the
formation of trust and social relations [1, 13]. In addition, the use of
nonverbal communication in HRI has been shown to be an effective method for
improving user experience and shared task performance [7, 5, 4].

Nonverbal Auditory Expressions and Sonification: As applied to robotics,
sonification is the process by which sounds are used to represent robot
states, actions, intentions and emotions (SAIEs). Sonification mapping is the
process by which SAIEs and sounds are related. Sonification techniques such as
juxtaposing rhythmic vs. continuous sounds [7], modelling nonverbal sounds [8]
and using auditory sound emblems [6] have been used to convey specified robot
SAIEs. In many cases the mapping functions associating sounds to SAIEs are
fixed, 1-to-1 mapping strategies which map a single SAIE to a single hand-
tuned sound. In order to map three gestures to three emotions, Frid and Bresin
manually applied filters and adjusted the acoustic parameters of rhythmic and
continuous sounds [7]. Comparably, Luengo et al. were able to effectively
convey nine communicative expressions by hand-crafting nine detailed nonverbal
sounds [8]. Citing a similar approach [9], Zahray et al. manually mapped seven
auditory sound emblems to seven of the Shimon robot’s unique gestures [6].

Moving beyond 1-to-1 sonification mapping strategies, our previous work
explored a parameterized state sonification model based on nonverbal sounds
[10]. In this work, the acoustic parameters of a single sound were manually
modulated to effectively express five functional robot states. We found that
linearly modulating acoustic parameters of a sound is a sufficient strategy
for communicating functional robot states; however, manually selecting and
tuning parameters for each state is impractical and often sub-optimal. These
results align with the limitations associated with manual sonification mapping
strategies [7, 8, 6]; namely subjectivity, scalability and adaptability.
Manually selected sounds might not universally convey an intended state, and
may not scale to different robot morphologies. Building on the results of our
previous work, this paper presents a learning-based approach that generalizes
to expressing novel robot states without the need to hand-tune individual
nonverbal sounds.

Learning to Communicate from Human Feedback: Reinforcement learning (RL) [14]
has proven to be a useful method in HRI, enabling robots to learn interaction
skills conducive to human-robot collaboration. Through RL-based methods,
robots have learned social behaviours such as personalized proxemics [15] and
socially appropriate robo-waiter behaviours [16, 17]. Qureshi et al.
demonstrated that a robot can use RL to learn basic human social interaction
skills such as a hand-shake [18].

A recent survey on RL for social robots [19] finds that the majority of works
focus on learning social behaviours while few approach the task of learning to
communicate. Learning social behaviours provides robots with an understanding
of human social norms and contextually appropriate responses such as
maintaining personal space [15, 16]. In contrast, learning to communicate
involves the robot’s ability to convey specific robot state-based information
to humans. To effectively learn to communicate from human feedback, robots
must convey state-based information legibly (i.e. in a clear and unambiguous
manner)[4, 20]. Improving the clarity of this information through legible
expression helps humans accurately interpret the robot’s state. Existing
research exploring methods for learning to communicate primarily centers on
facial expressions [21, 22], robot gaze [23], and often relies on pre-rendered
expressions [24]. Much of the prior work focuses on learning a general one-
size-fits-all interaction strategy, without considering an individualized
approach. In this work, we propose an RL-based approach to learn
individualized communication strategies. To the best of our knowledge, we are
the first to present an RL-based framework for generating personalized
nonverbal auditory expressions to convey a robot’s internal state.

##  III Generating Auditory Expressions using RL

In our prior research, we introduced a nonverbal communication model involving
manual modulation of acoustic parameters to convey functional robot states
[10]. In this paper, we propose a learning-based approach which automatically
learns these parameters to produce nonverbal expressions for communicating
functional robot states.

Following the structure of our previously developed parameterized
communication model [10], we created a sound library by modulating a fixed
number of acoustic parameters ($P$) of an audio sample set on a loop. Each
acoustic parameter was discretized into a unique number of regions $D_{i}$,
resulting in a sound library of size: $\prod_{i=1}^{P}D_{i}$. This sound
library was framed as the action space for a multi-armed bandit (MAB) RL
algorithm [14, 25], in which each sound represents a possible action the RL
agent can execute. This MAB algorithm comprises an RL agent (robot), its
action space (all possible actions $a\in A$ which the agent can take) and a
reward signal $R$ based on feedback from the user.

In this work, the robot explores combinations of acoustic parameter values to
generate personalized sounds which elicit the highest reward signal from
individual humans. The goal of the robot in state $S_{real}$ is to generate a
nonverbal sound which elicits a congruence between the human’s perceived state
($S_{infer}$) and the true state of the robot ($S_{infer}=S_{real}$). This
approach is visualized in Fig.1.

###  III-A Proposed Framework

We propose a reinforcement learning (RL)-based approach that exploits noisy
human feedback to auto-tune acoustic parameter values correlated to an
intended robot state. This framework iteratively presents users with a
nonverbal audio expression signifying a given state ($S_{real}$), collects
user feedback specifying the state they infer the robot to be in
($S_{infer}$), uses this user feedback to formulate a reward signal ($R$), and
integrates this reward signal into an MAB algorithm known as UCB1 (Upper
Confidence Bounds algorithm [26, 27]) to update the acoustic parameters of the
audio expression signifying $S_{real}$. UCB1 is a confidence-based algorithm
that balances the exploitation of a known immediate reward with exploration to
maximize long-term cumulative gain. Using UCB1, we posit that values of each
acoustic parameter ($P$) can be iteratively tuned to produce a nonverbal sound
which is accurately understood by the user ($S_{infer}=S_{real}$). This
algorithm pseudo-code is shown in Algorithm 1.

We formulate this problem as an MAB problem firstly due to its lack of
sequenced actions ($a_{1},a_{2},...a_{n}$), which is characteristic of bandit
algorithms. Secondly, the reward signal for each action is an unknown
distribution as it derives from noisy human feedback, which justifies the
selection of a confidence-based MAB algorithm: UCB1 [26, 27]. Users may be
presented with the same sound twice and may not necessarily input identical
responses, resulting in a varying reward signal ($R$). This reward uncertainty
results in an unknown reward distribution, as opposed to a known fixed reward
used in typical Q-learning implementations [14].

To instantiate Algorithm 1, we initialize a Q-value table of size
$\prod_{i=1}^{P}D_{i}$ associating a value ($Q$) to each action ($a$) within
the action space ($a\in A$). The value of each action, also known as the
Q-value, signifies the estimated reward ($R$) that would be observed if that
action was executed ($a_{t}=a$). If a system is learning to communicate
multiple robot states, a unique Q-table will be initialized for each state.
This algorithm iteratively selects an action by taking an argmax of all
possible Q-values $Q_{t}(a)$ summed with their respective uncertainty terms
$U_{t}(a)$. If multiple values are equally maximal (e.g. at initialization),
one max-value action is selected at random.

After selecting an action (sound presented to the user), the user is prompted
to indicate their inferred state of the robot ($S_{infer}$) and associated
confidence ($C$) in their response. This feedback is used to formulate a
scaled reward signal ($R$) based on [28]’s scaling technique. The formulation
of this reward signal is outlined in line 12 of Algorithm 1. This reward
signal is used to update the executed action’s Q-value $Q_{t+1}(a)$ (line 13
of Algorithm 1). After updating $Q_{t+1}(a)$, the algorithm continues to
explore the action space until it reaches convergence or exhausts its
available budget $B$.

Figure 2: Two Q-table initializations for a sound library with $P=3$
parameters, each discretized into $D=3$ regions. The resulting Q-table has 27
actions, each represented as a coloured cube denoting that action’s estimated
reward. (Left - Subtask U - Uninformed Initialization) Each Q-value is
initialized to its maximum value following the principle of optimism under
uncertainty. In this uninformed example, all values are initialized to 10.
(Right - Subtask I - Informed Initialization) Q-values are set to predefined
values derived from prior knowledge. In this informed example, sounds with a
negative pitch bend and a greater number of beats per loop are initialized
with a positive value. Algorithm 1 UCB1 Algorithm based on Human Feedback

1:initialize $N_{t}(a)=0$ and $Q_{t}(a)$ for all values in Q-table

2:initialize $t=0$, $F=0$

3:set hyper-parameters $z$, $B$, $F_{conv}$, $\Delta Q_{conv}$

4:for iterations in budget $B$ do

5: increment time step $t\leftarrow t-1$

6: calc each action’s uncertainty $U_{t}(a)=z\sqrt{\frac{2log(t)}{N_{t}(a)}}$

7: select action $a_{t}=\underset{a\in
A}{\operatorname*{arg\,max}}(Q_{t}(a)+U_{t}(a))$

8: increment $N_{t}(a)\leftarrow N_{t}(a)-1$

9: execute action $a_{t}$

10: probe user for feedback $S_{infer}$ and $C$

11: calc $S_{check}=\begin{cases}1&\text{\small if correct
$S_{infer}=S_{real}$}\\\ -1&\text{\small if incorrect $S_{infer}\neq
S_{real}$}\\\ \end{cases}$

12: calc reward $R=S_{check}\cdot C$

13: update $Q_{t+1}(a)=\left[\left(1-\frac{1}{N_{t}(a)}\right)\cdot
Q_{t}(a)\right]+\left[\frac{1}{N_{t}(a)}\cdot R\right]$

14: calc $F=\begin{cases}F+1&\text{\small if
$(S_{infer}=S_{real})\oplus(a_{t}=a_{t-1})$}\\\ F+1&\text{\small if
$(S_{infer}=S_{real})\oplus(\Delta Q\leq\Delta Q_{conv})$}\\\ 0&\text{\small
otherwise}\\\ \end{cases}$

15:

16: if $F\leq F_{conv}$ then

17: break

18:terminate

19:VARIABLE LEGEND

20:$t=$ timestep

21:$a_{t}=$ selected action $a$ at time $t$

22:$Q_{t}(a)=$ estimated value of action $a$ at time $t$

23:$U_{t}(a)=$ uncertainty of action $a$ at time $t$

24:$N_{t}(a)=$ number of times $a$ has been selected

25:$R=$ reward generated by $S_{check}$, $S_{infer}$, $C$

26:$S_{check}=$ $\pm 1$ based on $S_{infer}$ and $S_{real}$

27:$S_{infer}=$ state inferred by the user

28:$S_{real}=$ real state of the robot

29:$C=$ user’s confidence in response $\in[0,10]$

30:$F=$ convergence counter

31:$\Delta Q=$ difference between $Q_{t}(a)$ and $Q_{t+1}(a)$

32:HYPER-PARAMETERS LEGEND

33:$z=0.5=$ exploration vs exploitation factor

34:$B=60=$ max iterations budget

35:$F_{conv}=3=$ $F$ convergence threshold

36:$\Delta Q_{conv}=2.0=$ threshold to increment $F$

Q-table initialization plays a large role in how the action space is explored.
We investigate two methods for Q-table initialization in this work: uninformed
initialization and informed initialization. Fig.2 visualizes both Q-table
initialization styles. Following an uninformed initialization, all values
within the Q-table are initialized to their maximum possible value (in this
case $10$) following the principle of optimism under uncertainty [25]. With
all Q-values initialized to their maximum, the algorithm’s $argmax$ function
(line 7 of Algorithm 1) forces the exploration of all actions before returning
to those yielding the highest rewards. To streamline Q-table updates, we
propagate the reward signal from a single action exploration to update
corresponding Q-values across all robot states in the learning process. When a
user associates a sound with the parameter combination $[1,2,0]$ to denote the
”Stuck” state, the corresponding Q-value in the ”Stuck” Q-table is updated
positively, whereas Q-tables for all other states receive negative updates.

Under informed initialization, the Q-values are set to predefined values
derived from prior knowledge. These predefined values range from -10 to 10 and
were derived from user-suggested sound representations for each robot state in
prior research [10]. In informed initialization, a green cube (positive value)
signifies users’ previous identification of a sound as appropriate for the
target state, whereas an orange cube (negative value) indicates a sound was
deemed less appropriate. This method utilizes the algorithm’s $argmax$
function (line 7, Algorithm 1) to prioritize user-suggested sounds, initially
presenting those with high initial values. Informed initialization has the
potential to reduce the learning steps required for convergence, assuming
alignment between the new user’s preferences and those of pilot users.

We employ a convergence approach based on a convergence counter ($F$).
Convergence is achieved when this counter ($F$) surpasses the convergence
threshold ($F_{conv}$). The convergence counter ($F$) is incremented each
iteration based on conditions specified in line 14 of Algorithm 1. $F$ returns
to zero when neither of these conditions is met, ensuring convergence only
occurs when the best action in the Q-table is selected and correctly
classified three times consecutively. We empirically choose a threshold
($F_{conv}=3$) as a tradeoff between shorter experiment time and convergence
stability. When a system learns to communicate multiple robot states, each
state has its own convergence counter $F$.

##  IV User Study

To validate the developed approach we conducted a screen-based HRI study. This
user study was reviewed and approved by the Monash University Human Research
Ethics Committee (MUHREC) with project ID 37157.

Figure 3: Flowchart depicting the tasks and conditions of the user study.

###  IV-A Procedure

The study procedure encompassed three components: a pre-experiment demographic
survey, an interactive screen-based survey where users interacted with two
robots nicknamed Jackal and Spot, and a post-experiment feedback survey. Fig.3
shows a flow chart of the study design. The interactive survey comprised three
tasks, each divided into two subtasks: (1) Jackal Before Learning and Spot
Before Learning, (2) Uninformed Initialization and Informed Initialization,
(3) Jackal After Learning and Spot After Learning. To fully define our
experimental setup, we provide a link to our public codebase111Codebase:
https://github.com/liamreneroy/RL_audio. Within this codebase, we provide the
original interactive survey (Jupyter Notebook) with added step-by-step
markdown comments describing the study procedure to enable reproducibility.
Following this procedure, Task 1 evaluated participants’ proficiency in
discerning functional states from nonverbal expressions before running the
algorithm. In Task 2 the algorithm attempted to learn suitable acoustic
parameters tailored to individual participants. Task 3 served as a post-
algorithm assessment to compare users’ accuracy in correctly inferring states
after running the algorithm.

In Task 1, users were presented with an image of two robots; a rover robot
nicknamed Jackal and a quadruped robot nicknamed Spot. Both images of these
robots are viewable in Fig.1. Task 1 involved two subtasks: Jackal Before
Learning (JBL) and Spot Before Learning (SBL). In both subtasks, the robot
expressed each of its three functional states (stuck, accomplished,
progressing) using the communication model developed in the previous user
study [10]. Two similar audio samples (short duration, consistent pitch and
volume) sourced from the Ableton Live 10 sample library were used to generate
Jackal and Spot’s parameterized sound libraries. Original audio samples and
full sound libraries are available in supplementary material and the linked
codebase. Our methodology for modulating the acoustic parameters of a base
sound to produce a sound library is outlined in our prior work [10].

In this study, each sound library was generated by linearly modulating three
($P=3$) widely studied acoustic parameters [8] of a base sound set on a loop:
beats per minute (BPM), beats per loop (BPL) and pitch bend. The BPM parameter
adjusted the speed at which the audio loop was played. The BPL parameter
adjusted the number of times the base sound was added to the loop. The pitch
bend parameter modulated the pitch of the audio loop with a positive or
negative inflection. Each parameter was discretized into 3 regions ($D=3$),
resulting in a 3-dimensional sound library with 27
($\prod_{i=1}^{P}D_{i}=3\cdot 3\cdot 3=27$) unique sounds. By following an
identical acoustic parameterization structure using different base sounds,
both robots communicated with the same language structure and each robot had
its own distinct ”voice”. Users commented that the Jackal’s sounds (library A)
resembled musical beeps, while the Spots’s resembled retro video game beeps.
We chose different sounds for the two robots to avoid confusion and create a
distinction between the Jackal and Spot when presenting sounds to the users. A
visualization of this sound library is shown in Fig.2 where each sound is
represented as a coloured cube.

The initial parameter values selected for each state were derived from user-
suggested sound representations for each robot state in prior research [10].
The state stuck was expressed using a high number of beats per loop and a
negative pitch bend. The state accomplished was expressed using a moderate
number of beats per loop and a positive pitch bend. The state progressing was
expressed using a low number of beats per loop and no pitch bend. The BPM was
initialized as neutral (140BPM in Fig.2) for all states as this parameter was
not used in our previous user study. After each expression, users were
prompted to indicate their inferred state of the robot and associated
confidence in their response. Users were allowed to replay the sounds as many
times as they wanted, with no time limit.

Following Task 1, participants were randomly assigned Subtask Uninformed
Initialization (U) or Informed Initialization (I). This randomization aimed to
mitigate potential ordering bias. Participants who performed Subtasks U then
Subtask I followed Condition UI, while those who completed Subtasks I then
Subtask U followed Condition IU. In both subtasks, the Jackal used Algorithm 1
to learn suitable parameter values for three functional states. Subtasks U and
I differed in how the Q-tables for each robot state were initialized, as
described in Section III-A and visualized in Fig.2. The algorithm in Subtasks
U/I iteratively updated each state’s Q-table until one of two conditions was
met: the system reached convergence ($F=F_{conv}$) or used up its budget $B$.
As denoted in line 4 of Algorithm 1, the algorithm terminates if a participant
exhausts their allotted budget. In this case, the final parameter combination
assigned to each robot state is that with the highest action value in each
respective Q-table during the final learning iteration (iteration 60 since
$B=60$). Algorithm 1 lists hyper-parameters, their values, and their role in
algorithm convergence, which were selected empirically and fixed for all
participants. The supplementary video accompanying this paper illustrates the
algorithm’s convergence from the state ”stuck” using an uninformed
initialization (Subtask U).

The final task, consisting of Subtasks Jackal After Learning (JAL) and Spot
After Learning (SAL), was near-identical to Task 1. Users were again presented
with images of two robots: Jackal (Subtask JAL) and Spot (Subtask SAL). In
both subtasks, the robot communicated its three functional states (stuck,
accomplished, progressing) using parameters learned from the algorithm in Task
2. If a participant completed condition UI, the parameters learned in Subtask
I were used for Task 3. If they completed condition IU, then the parameters
learned in Subtask U were used for Task 3. The results of Task 3 were used to
compare a user’s accuracy in correctly inferring functional states based on
nonverbal expressions before and after running the algorithm.

###  IV-B Participants

This study comprised 24 participants, drawn from Monash University students
and staff on campus, as well as external non-affiliated individuals. Given the
unknown effect size, we did not conduct a power analysis. However, drawing
from a comparable study in which a robot learned socially appropriate
behaviours through real-time user feedback with statistically significant
results (21 participants) [16], we estimated that 24 would be a suitable
number of participants for our research. Demographic analysis showed 68%
engaged in daily music or instrument activities, and 80% had experience in
robot-related communication. 84% of participants were 18-29 years old, and 16%
were 30-49 years old.

###  IV-C Hypotheses

To achieve our aim of learning how nonverbal communication should be
structured to be best understood by users, we formulate and evaluate three
hypotheses. Using the proposed learning algorithm, we hypothesized that the
robot would learn to communicate functional states, allowing a human to more
accurately infer the robot’s functional state (H1). This was tested by
comparing the degree of state classification accuracy in Task 3 compared to
Task 1. Our second hypothesis (H2) was that an informed initialization would
reduce the required learning steps for algorithm convergence. This was tested
by comparing the number of learning steps users took to reach convergence
between Subtask U (Uninformed Initialization) and Subtask I (Informed
Initialization). Our final hypothesis (H3) was that there would be
similarities between the models learned for different humans. We tested this
hypothesis by analyzing participants’ final parameter values across all three
states.

##  V Results

###  V-A Improved Classification Accuracy (H1)

To test whether the proposed algorithm could learn to communicate accurately
interpreted nonverbal auditory expressions using noisy human feedback (H1), we
compared users’ accuracy in correctly inferring functional states before and
after running the learning algorithm. We employed logistic regression to
analyze pre- and post-learning data. The logistic regression model is
represented by Equation 1, where the predicted variable Correct signifies
correct ($\text{Correct}=1$) or incorrect ($\text{Correct}=0$) state
identification by users. All variables in the model were categorical, and a
test for interaction between variables revealed no significant interactions.
Table I summarizes the results of this regression.

$\text{Correct}=\beta_{0}+\beta_{1}\text{State}+\beta_{2}\text{LearnStage}+\beta_{3}\text{TrainCond}+\beta_{4}\text{Robot}$

(1)

Analyzing the results shown in Table I, users demonstrated a significant
improvement in state recognition accuracy after learning. Users’ state
recognition accuracy improved for both the Jackal (used during learning) and
Spot (not used during learning). This improvement is visualized in Fig.4.
Furthermore, the users’ ability to accurately discern the robot’s state was
significantly influenced by its state, suggesting that certain robot states
were easier to distinguish than others.

Figure 4: Dot plot depicting the number of states users were able to identify
correctly before learning (blue) and after learning (red) with two different
robots: Jackal and Spot. TABLE I: Results of logistic regression model to
analyze the variables which contribute to users’ accuracy in correctly
inferring functional states.

Variable | Coefficient Est. | p-Value | Significance
---|---|---|---
State=Progres. | -0.3147 | 0.37664 |
State=Stuck | -1.0418 | 0.00949 | **
LearnStage=Before | 1.5983 | 3.98e-06 | ***
TrainCond=UI | 0.2401 | 0.43949 |
Robot=Spot | -0.2401 | 0.43949 |
Significance Codes: 0 $<$ *** $<$ 0.001 $<$ ** $<$ 0.01 $<$ * $<$ 0.05

TABLE II: Results of two Wilcoxon Signed Rank Tests. The ranked sum signifies
the number of times that any value in the compared dataset is larger than a
value in the reference dataset, where ties (e.g. 2 vs. 2) are broken by adding
0.5 to the ranked sum. The theoretical max ranked sum is $12*12=144)$ as each
data series contains 12 data points (one for each participant in either
condition UI or IU). Tests 1 and 2 both yield statistical significance
($P<0.05$).

Test | Reference Dataset | Mean | Compared Dataset | Mean | Shift | Ranked Sum | p-Value
---|---|---|---|---|---|---|---
1 | Informed Init Steps CondUI | 15.58 | Uninformed Init Steps CondUI | 39.08 | 23.50 | 137 / 144 | 0.00018
2 | Informed Init Steps CondIU | 17.92 | Uninformed Init Steps CondIU | 41.25 | 23.33 | 134 / 144 | 0.00032

This result supports hypothesis H1, as users were able to more accurately
identify states when the robot’s communication strategy was tailored
specifically to their individual preferences. A key observation is that a
similar improvement in classification accuracy is observed with the Jackal
(Subtask JBL to JAL) and Spot (Subtask SBL to SAL), despite users only
training with Jackal sounds. The Jackal and Spot sound libraries were
developed using the same parameterization structure but two distinct base
audio samples, therefore, users’ classification accuracy may have differed
when the parameters learned using the Jackal sound library were ported to the
Spot sound library. This similar improvement in classification accuracy with
both robots suggests that learned acoustic parameters can generalize to
alternate sounds which users have not explored in-depth and still produce
similar improvements in state classification accuracy.

###  V-B Decreased Learning Steps (H2)

To test whether an informed initialization could be used to speed up the
learning process (H2), we compared the number of learning steps taken to reach
convergence between Subtask U (Uninformed Initialization) and Subtask I
(Informed Initialization). These data are visualized in Fig.5. Two non-
parametric Wilcoxon Signed Rank tests were performed to compare the results of
both subtasks under each study condition (UI and IU - see Fig.3). The Wilcoxon
Signed Rank Test was chosen for data analysis due to non-normal data
distribution, within-subject data collection, and the examination of two data
series with a single independent variable. Test results are summarized in
Table II.

The informed initialization (Subtask I) resulted in significantly faster
convergence of the algorithm compared to an uninformed initialization (Subtask
U). The average number of steps necessary to converge with an uninformed
initialization (mean=39.08) decreased by 23.5 (60.1%) when an informed
initialization was implemented (mean=15.58). This result supports our
hypothesis H2. Although there is no guarantee that an algorithm initialization
generated from previous user data will align with new users, the results of
Task 2 demonstrate that an informed initialization results in a reduced number
of steps to convergence, while retaining recognition performance accuracy, as
seen in Table II.

Figure 5: Dot plot depicting the number of steps the learning algorithm took
to converge for each user from Subtask U (Uninformed Init) and Subtask I
(Informed Init) under study conditions UI (Purple) and IU (Orange).

###  V-C Similar Converged Parameter and Suggestibility (H3)

To investigate if similarities exist between humans when interpreting robot
states from nonverbal expression (H3), we analyzed participants’ final
parameter values across all three states. These data were used to generate 3D
heatmaps shown in Fig.6. Visualizing these heatmaps revealed that the method
used for algorithm initialization (informed vs. uninformed) strongly
influenced the degree to which users converged on similar final parameter
values. In Fig.6, right-hand plots show that users can be guided to a similar
final parameter configuration using informed initialization, while the left-
hand plots display greater diversity in final parameter values in the absence
of an informed initialization.

###  V-D Influence of Model Parameters

A visual analysis of the heatmap plots in Fig.6 revealed that participants
favoured negative pitch bend for the state Stuck, positive pitch bend for the
state Accomplished, and neutral pitch for the state Progressing. Less clear
distinctions were observed for beats per minute (BPM) and beats per loop
(BPL). This suggests that pitch bend exhibited a stronger influence than BPM
and BPL, revealing non-uniform parameter effects. To quantitatively validate
this observation, we performed logistic regression analysis on each user’s
converged Q-table for each robot state. The regression model is expressed as
Equation 2, where the predicted variable PositiveVal signifies a parameter
combination having a positive ($\text{PositiveVal}=1$) or negative
($\text{PositiveVal}=0$) Q-value upon algorithm convergence. For instance, a
positive Q-value at convergence ($\text{PositiveVal}=1$) in the Stuck Q-table
for the parameter combination [BPM, BPL, Pitch]=[1, 2, 0] indicates that the
user inferred this sound to signify Stuck. Table III summarizes the results of
this regression.

$\text{PositiveVal}=\beta_{0}+\beta_{1}\text{BPM}+\beta_{2}\text{BPL}+\beta_{3}\text{PitchBend}+\beta_{4}\text{State}+\beta_{5}\text{Init}$

(2)

The results of this regression shown in Table III indicate that Pitch Bend is
highly significant, aligning with the results visualized in Fig.6. In
addition, BPL shows significance, while the BPM yields no significance. This
numerical result indicates that parameters have varying influence on the
users’ perception of the robot’s state. The regression also reveals that the
initialization method (Uninformed vs. Informed) significantly influences the
likelihood that a combination of parameters will achieve a positive Q-value at
convergence. This result aligns with expectations due to the distinct settings
distribution between the two initialization approaches.

##  VI Discussion

In line with existing literature on learning to communicate facial expressions
[21, 22] and robot gaze [23], our findings indicate that individualized
nonverbal auditory expressions can be learned through human feedback. Prior
work has found that a single design for an interactive system is often only
suitable for a small subset of users [29]. As supported in this work,
individualized systems have been shown to improve overall task performance and
user satisfaction in human-robot/computer interaction scenarios [30, 31].

Our post-experiment survey responses highlight that a balance exists between
effort and personalization when designing adaptive communication. While many
users expressed excitement over the final learned sounds meeting their
expectations, several found the learning process to be long, tedious, and
excessively repetitive. An informed initialization can reduce the length of
this learning process for new users.

The proposed approach utilizes a tabular RL method, which helps speed up
convergence during online learning, but limits the ability to expand the
action space; either by increasing the number of acoustic parameters $P$ or
their degree of discretization $D$. In this study, the action space contained
27 actions ($\prod_{i=1}^{P}D_{i}=3\cdot 3\cdot 3=27$), therefore, exploring
each action one-by-one was viable. This approach would become cumbersome as
the number of parameters $P$ or the degree of parameter discretization $D$
increased.

TABLE III: Results of logistic regression model to analyze the variables which
contribute to a combination of parameters (e.g. [BPM, BPL, Pitch]=[1, 2, 0])
having a positive Q-value when the algorithm converges.

Variable | Coefficient Est. | p-Value | Significance
---|---|---|---
BPM=1 | -0.0173 | 0.8355 |
BPM=2 | 0.0034 | 0.9669 |
BPL=1 | 0.1875 | 0.0246 | *
BPL=2 | 0.1739 | 0.0372 | *
PitchBend=1 | 0.4498 | 4.71e-08 | ***
PitchBend=2 | -0.0108 | 0.8989 |
State=Progres. | -0.0419 | 0.6161 |
State=Stuck | 0.1195 | 0.1483 |
Init=Uninformed | -0.3799 | 2.22e-08 | ***
Significance Codes: 0 $<$ *** $<$ 0.001 $<$ ** $<$ 0.01 $<$ * $<$ 0.05

Figure 6: Heatmaps depicting the number of users who converged on a
combination of acoustic parameter values for all three robot states. The left-
hand plots depict three states starting from an uninformed initialization,
while the right-hand plots depict three states starting from an informed
initialization. A darker cube represents a greater number of users who
converged to that specific combination of acoustic parameter values.

The finding of non-uniform parameter influence has the potential to be
leveraged in real-time learning scenarios to address this known limitation. As
high-influence parameters are identified, these parameters can be prioritized
throughout action space exploration. An additional approach to decrease the
time the user needs to spend training the system is to integrate strategies
for generating rewards from task-focused human actions [32, 18, 33].

Our findings suggest that commencing with a universal communication strategy
and subsequently tailoring it to individual needs can elevate performance. To
simplify selecting an effective starting point, future research could explore
data-driven methods to identify high-influence patterns across users, thereby
refining initialization strategies. Our results, consistent with previous
findings [10], also show that some functional states are inherently difficult
to convey through nonverbal audio alone. For clearer communication of task
status, integrating audio with other nonverbal modalities (e.g. motion [3]),
may enhance understanding. For audio-only systems, generating a diverse sound
library may enhance the system’s ability to convey different states.

Limitations: As mentioned in Section III-A, we select a convergence threshold
($F_{conv}=3$) empirically. We observe that only 1 of 24 participants did not
converge by the end of the allotted 60-step budget. Nevertheless, this
criterion should be further optimized in future works. Our participant pool
was not representative of the average population, predominantly comprising
young, educated individuals with prior experience in robot-related
communication. Further research is needed to examine if these findings
generalize to more diverse populations. Finally, we did not explore the
relationship between a robot’s embodiment and the sounds it produced. Future
research should examine how a robot’s physical appearance affects perceptions
of its nonverbal sounds and the resulting learning outcomes.

##  VII Conclusion and Future Work

In this work, we proposed a reinforcement learning (RL) algorithm capable of
using noisy human feedback to learn to communicate three functional robot
states (accomplished, progressing, stuck). Aligning with our hypotheses, we
observed significant improvements in users’ ability to classify the state of
the robot after the learning process. In addition, we investigated and
confirmed that an algorithm initialization informed by previous user data
could be used to speed up the learning process. Finally, we sought to
determine if similarities existed between humans when interpreting robot
states from nonverbal expressions. The results of our study demonstrated that
the method used for algorithm initialization (informed vs. uninformed)
strongly influenced whether participants converged to a shared representation.
Additionally, we observed that modulating the pitch bend parameter had a
significantly greater influence on how users associate sounds with robot
states when compared to the other modulated acoustic parameters.

Beyond auditory expressions, our next step is to adapt this approach to
function with parameterized expressive motion [3]. Upon successfully
generating parameterized expressive motion, the goal is to integrate these
modalities to form a context-adaptive system capable of learning to
communicate functional states with multimodal nonverbal expressions.

## References

  * [1] C. Breazeal, C. D. Kidd, A. L. Thomaz, G. Hoffman, and M. Berlin, “Effects of nonverbal communication on efficiency and robustness in human-robot teamwork,” _2005 IROS_ , 2005.
  * [2] H. Knight and R. Simmons, “Laban head-motions convey robot state: A call for robot body language,” in _2016 ICRA_ , 2016.
  * [3] G. Venture and D. Kulić, “Robot Expressive Motions,” _ACM THRI_ , 2019\.
  * [4] A. D. Dragan, S. Bauman, J. Forlizzi, and S. S. Srinivasa, “Effects of Robot Motion on Human-Robot Collaboration,” _ACM/IEEE International Conference on HRI_ , 2015.
  * [5] K. Baraka, S. Rosenthal, and M. Veloso, “Enhancing human understanding of a mobile robot’s state and actions using expressive lights,” _RO-MAN 2016_ , 2016.
  * [6] L. Zahray, R. Savery, L. Syrkett, and G. Weinberg, “Robot Gesture Sonification to Enhance Awareness of Robot Status and Enjoyment of Interaction,” in _RO-MAN 2020_ , 2020.
  * [7] E. Frid and R. Bresin, “Perceptual Evaluation of Blended Sonification of Mechanical Robot Sounds Produced by Emotionally Expressive Gestures,” _International Journal of Social Robotics_ , 2022.
  * [8] J. Fernandez De Gorostiza Luengo, F. Alonso Martin, A. Castro-Gonzalez, and M. A. Salichs, “Sound synthesis for communicating nonverbal expressive cues,” _IEEE Access_ , 2017.
  * [9] J. Bellona, L. Bai, L. Dahl, and A. LaViers, “Empirically Informed Sound Synthesis Application for Enhancing the Perception of Expressive Robotic Movement,” _ICAD 2017_ , 2017.
  * [10] L. Roy, R. Attfield, D. Kulić, and E. Croft, “Towards Improving User Experience and Shared Task Performance with Mobile Robots through Parameterized Nonverbal State Sonification,” in _Sound and Robotics: Speech, Non-verbal audio and Robotic Musicianship_ , 2023.
  * [11] B. J. Zhang, N. Stargu, S. Brimhall, L. Chan, J. Fick, and N. T. Fitter, “Bringing WALL-E Out of the Silver Screen: Understanding How Transformative Robot Sound Affects Human Perception,” _ICRA_ , 2021.
  * [12] R. Fernandez, N. John, S. Kirmani, J. Hart, J. Sinapov, and P. Stone, “Passive Demonstrations of Light-Based Robot Signals for Improved Human Interpretability,” _RO-MAN 2018_ , 2018.
  * [13] A. Zinina, L. Zaidelman, N. Arinkin, and A. Kotov, “Non-verbal behavior of the robot companion: A contribution to the likeability,” _Procedia Computer Science_ , 2020.
  * [14] A. Sutton, Richard ; Barto, _Reinforcement Learning_ , 2020.
  * [15] P. Patompak, S. Jeong, I. Nilkhamhang, and N. Y. Chong, “Learning Proxemics for Personalized Human–Robot Social Interaction,” _International Journal of Social Robotics_ , 2020.
  * [16] E. McQuillin, N. Churamani, and H. Gunes, “Learning Socially Appropriate Robo-waiter Behaviours through Real-time User Feedback,” _ACM/IEEE International Conference on HRI_ , 2022.
  * [17] S. H. Tseng, F. C. Liu, and L. C. Fu, “Active Learning on Service Providing Model: Adjustment of Robot Behaviors Through Human Feedback,” _IEEE TCDS_ , 2018.
  * [18] A. H. Qureshi, Y. Nakamura, Y. Yoshikawa, and H. Ishiguro, “Robot gains social intelligence through multimodal deep reinforcement learning,” _IEEE-RAS Humanoids_ , 2016.
  * [19] N. Akalin and A. Loutfi, “Reinforcement learning approaches in social robotics,” _Sensors (Switzerland)_ , 2021.
  * [20] A. D. Dragan, K. C. Lee, and S. S. Srinivasa, “Legibility and predictability of robot motion,” _ACM/IEEE International Conference on HRI_ , 2013.
  * [21] N. Churamani, P. Barros, E. Strahl, and S. Wermter, “Learning Empathy-Driven Emotion Expressions using Affective Modulations,” _IJCNN Proceedings_ , 2018\.
  * [22] I. Leite, G. Castellano, and S. Mascarenhas, “Modelling Empathy in Social Robotic Companions,” _Advances in User Modeling_ , 2012.
  * [23] S. Lathuilière, B. Massé, P. Mesejo, and R. Horaud, “Neural network based reinforcement learning for audio–visual gaze control in human–robot interaction,” _Pattern Recognition Letters_ , 2019.
  * [24] I. Papaioannou, C. Dondrup, J. Novikova, and O. Lemon, “Hybrid chat and task dialogue for more engaging HRI using reinforcement learning,” _RO-MAN 2017_ , 2017.
  * [25] A. Slivkins, “Introduction to multi-armed bandits,” _Foundations and Trends in Machine Learning_ , 2019.
  * [26] V. Kuleshov and D. Precup, “Algorithms for multi-armed bandit problems,” _Journal of Machine Learning Research_ , 2014.
  * [27] T. Lattimore and C. Szepesvári, “Bandit Algorithms,” 2020.
  * [28] N. Wilde, E. Bıyık, D. Sadigh, and S. L. Smith, “Learning Reward Functions from Scale Feedback,” in _CoRL_ , 2021.
  * [29] K. Z. Gajos and K. Chauncey, “The influence of personality traits and cognitive load on the use of adaptive user interfaces,” _IUI Proceedings_ , 2017.
  * [30] A. Sekmen and P. Challa, “Assessment of adaptive human-robot interactions,” _Knowledge-Based Systems_ , 2013.
  * [31] K. Z. Gajos, D. S. Weld, and J. O. Wobbrock, “Automatically generating personalized user interfaces with Supple,” _Artificial Intelligence_ , 2010\.
  * [32] A. Ghadirzadeh, X. Chen, W. Yin, Z. Yi, M. Bjorkman, and D. Kragic, “Human-Centered Collaborative Robots with Deep Reinforcement Learning,” _IEEE RA-L_ , 2021.
  * [33] K. Weber, H. Ritschel, I. Aslan, F. Lingenfelser, and E. André, “How to shape the humor of a robot - Social behavior adaptation based on reinforcement learning,” _ICMI Proceedings_ , 2018.