# Dynamics of deformable straight and curved prolate capsules in simple shear
flow

Xiao Zhang  Department of Chemical and Biological Engineering
University of Wisconsin-Madison, Madison, WI 53706-1691  Wilbur A. Lam
Wallace H. Coulter Department of Biomedical Engineering
Emory University and Georgia Institute of Technology, Atlanta, GA 30332
Department of Pediatrics, Division of Pediatric Hematology/Oncology, Aflac
Cancer and Blood Disorders Center of Children’s Healthcare of Atlanta
Emory University School of Medicine, Atlanta, GA 30322  Winship Cancer
Institute
Emory University, Atlanta, GA 30322  Parker H. Petit Institute of
Bioengineering and Bioscience
Georgia Institute of Technology, Atlanta, GA 30332  Michael D. Graham
Corresponding author. E-mail<EMAIL_ADDRESS>Department of Chemical and
Biological Engineering
University of Wisconsin-Madison, Madison, WI 53706-1691

###### Abstract

This work investigates the motion of neutrally-buoyant, slightly deformable
straight and curved prolate capsules in unbounded simple shear flow at zero
Reynolds number using direct simulations. The curved capsules serve as a model
for the typical crescent-shaped sickle red blood cells in sickle cell disease
(SCD). The effects of deformability and curvature on the dynamics are
revealed. We show that with low deformability, straight prolate spheroidal
capsules exhibit tumbling in the shear plane as their unique asymptotically
stable orbit. This result contrasts with that for rigid spheroids, where
infinitely many neutrally stable Jeffery orbits exist. The dynamics of curved
prolate capsules are more complicated due to a combined effect of
deformability and curvature. At short times, depending on the initial
orientation, slightly deformable curved prolate capsules exhibit either a
Jeffery-like motion such as tumbling or kayaking, or a non-Jeffery-like
behavior in which the director (end-to-end vector) of the capsule crosses the
shear-gradient plane back and forth. At long times, however, a Jeffery-like
quasi-periodic orbit is taken regardless of the initial orientation. We
further show that the average of the long-time trajectory can be well
approximated using the analytical solution for Jeffery orbits with an
effective orbit constant $C_{\textnormal{eff}}$ and aspect ratio
$\ell_{\textnormal{eff}}$. These parameters are useful for characterizing the
dynamics of curved capsules as a function of given deformability and
curvature. As the capsule becomes more deformable or curved,
$C_{\textnormal{eff}}$ decreases, indicating a shift of the orbit towards log-
rolling motion, while $\ell_{\textnormal{eff}}$ increases weakly as the degree
of curvature increases but shows negligible dependency on deformability. As
cell deformability, cell shape, and cell-cell interactions are all
pathologically altered in blood disorders such as SCD, these results will have
clear implications on improving our understanding of the pathophysiology of
hematologic disease.

##  I INTRODUCTION

A healthy human red blood cell (RBC) is a fluid-filled biconcave discoid with
an incompressible membrane comprised of a fluid lipid bilayer tethered to an
elastic spectrin network Mohandas and Evans (1994). In the past decades,
experimental and computational studies have revealed a rich spectrum of
dynamical modes for RBCs in shear flow over a vast domain of parameter space
in terms of membrane mechanics of RBC and flow conditions Goldsmith  _et al._
(1972); Fischer  _et al._ (1978); Abkarian  _et al._ (2007); Skotheim and
Secomb (2007); Dupire  _et al._ (2010); Fedosov  _et al._ (2010, 2011);
Yazdani and Bagchi (2011); Sinha and Graham (2015). In pathological cases
where normal blood flow is altered, the dynamics of diseased RBCs can be very
different from those of healthy ones because of differences in cell
properties. In sickle cell disease (SCD), for example, the intracellular
polymerization of sickle hemoglobin (HbS) leads to an abnormality in the shape
of individual sickle RBCs, but a large variation in cell morphology has been
revealed within the population of sample cells. Byun _et al._ Byun  _et al._
(2012) classified the sickle RBCs into echinocytes, discocyte, and the typical
crescent-shaped irreversibly sickled cells (ISCs) based on their morphological
features. In addition, a general decrease in deformability of sickle RBCs
compared to that of normal RBCs has been determined using various techniques
for the characterization of cell membrane mechanics; the extent of membrane
stiffening varies greatly depending on the states and conditions of individual
sickle RBCs Byun  _et al._ (2012); Messer and Harris (1970); Nash  _et al._
(1984); Evans and Mohandas (1987). The decrease in membrane deformability for
non-ISCs (namely, all sickle RBCs other than ISCs Bertles and Döbler (1969))
is found to be slight under oxygenated conditions Nash  _et al._ (1984), but
substantial upon deoxygenation Messer and Harris (1970), while ISCs exhibit
much lower deformability than non-ISCs in both states Messer and Harris
(1970); Nash  _et al._ (1984); Evans and Mohandas (1987). All these variations
for individual sickle RBCs mentioned above pose challenges to understanding
the dynamics of sickle RBCs in flow using computational approaches. In this
paper, we perform a systematic investigation of the dynamics of single stiff,
fluid-filled curved prolate capsules as a model for the typical crescent-
shaped sickle RBCs in the microcirculation, and show the effects of initial
orientation, membrane deformability and curvature of the capsules on their
dynamical motions.

The model we propose for sickle RBCs in this work derives from spheroidal
capsules, the dynamics of which have been of broad interest. The motion of
isolated neutrally buoyant rigid spheroidal particles in simple shear flow at
zero Reynolds number can be described by trajectories known as Jeffery orbits
Jeffery (1922). During flow the center-of-mass position of the spheroid
remains on its initial streamline with no drift in any direction, while its
major axis of symmetry traces out a closed periodic orbit depending on its
initial orientation. Each orbit corresponds to a specific non-negative orbit
constant $C$, so the quantity $C_{b}=C/(1+C)$ is bounded by zero and one. When
$C=C_{b}=0$, the particle rolls with its major axis aligned with the vorticity
direction, while $C_{b}=1(C=\infty)$ corresponds to a pure tumbling motion in
the shear plane. For any intermediate value of $C_{b}$, the major axis of the
particle undergoes a so-called kayaking motion. Bretherton Bretherton (1962)
later showed that Jeffery’s solution can be applied to describe the dynamics
of any axisymmetric rigid body with fore-aft symmetry.

Generalizing Jeffery’s work, Hinch and Leal Hinch and Leal (1979) derived the
equations for the motion of general (non-axisymmetric) ellipsoids in simple
shear flow at low Reynolds numbers in terms of Euler angles, and found that
the motion of a triaxial ellipsoid with comparable axes follows a doubly
periodic pattern. Yarin _et al_. Yarin  _et al._ (1997) numerically determined
a chaotic behavior for triaxial ellipsoids with one axis significantly longer
than the other two, and proposed an analytical theory to explain the onset of
this chaotic motion. Attempts have also been made to examine the behaviors of
spheroidal capsules when the assumptions on which Jeffery’s theory was based
are not strictly satisfied. Dupont _et al._ Dupont  _et al._ (2013) and
Cordasco and Bagchi Cordasco and Bagchi (2013) recently explored the effects
of membrane deformability on the dynamics of fluid-filled spheroidal capsules.
Through a stability investigation, Dupont _et al._ Dupont  _et al._ (2013)
showed that regardless of the initial orientation, the dynamics of prolate
spheroids in Stokes flow converge towards rolling (with the membrane tank-
treading), wobbling, and oscillating-swinging regimes, respectively, as
deformability increases. The model for the membrane mechanics in this work
describes shear elasticity and area dilatation while neglecting bending
elasticity. Cordasco and Bagchi Cordasco and Bagchi (2013) performed
simulations over a range of parameters for spheroidal capsules using a
membrane model that incorporates shear elasticity, area dilatation and bending
rigidity, and observed that unlike rigid ellipsoids in Stokes flow, capsules
undergo various motions such as precessing and kayaking depending on the
initial shape, deformability, and the ratio of the internal to external fluid
viscosity. In addition, Mao and Alexeev Mao and Alexeev (2014) examined the
effects of fluid inertia and particle inertia on the dynamics of single
prolate and oblate spheroids at low and moderate Reynolds number using
simulations, and revealed that the motion of a single spheroid is dependent on
the particle Reynolds number, aspect ratio, initial orientation and the Stokes
number Lundell and Carlsson (2010).

Other studies have also focused on slender particles such as curved fibers and
revealed more complex behaviors. Wang _et al_. Wang  _et al._ (2012) simulated
the motion of isolated, rigid, neutrally-buoyant, non-Brownian, slightly
curved fibers in simple shear flow at low Reynolds numbers. They revealed that
for some initial orientations the fibers drift in the gradient direction
resulting from the combined effects of the so-called “flipping, scooping, and
spinning” motions, and the drift rate strongly depends on the initial
orientation, aspect ratio, and curvature of the fibers. Crowdy Crowdy (2016)
derived a dynamical system governing the motion of a curved rigid two-
dimensional circular-arc fiber in simple shear, and the analytical solutions
of this dynamical system display the “flipping” and “scooping” motions
observed in Wang  _et al._ (2012). Another example of recent works on objects
with complex dynamics is that of Dutta and Graham Dutta and Graham (2017).
They investigated the dynamics in shear flow of inertialess Miura-patterned
foldable sheets containing predefined crease lines, and found a rich spectrum
of quasi-periodic or periodic motions depending on the initial configuration
of the sheet.

In this paper, we investigate the dynamics of single straight and curved
prolate capsules in unbounded simple shear flow at zero Reynolds number and
discuss the effects of initial orientation, membrane deformability, and degree
of curvature on their dynamics. The rest of the paper is organized as follows:
in Section II we present the model for sickle RBCs and the numerical algorithm
employed to obtain the fluid field; in Section III we first provide a
validation of our numerical simulation, followed by detailed results and
discussion on the dynamics of single straight and curved prolate capsules;
lastly, concluding remarks are presented in Section IV.

##  II MODEL FORMULATION

###  II.1 Model and discretization

Figure 1: Schematic of the 3D orientation of a curved prolate capsule in
unbounded simple shear flow.

We consider an isolated neutrally-buoyant fluid-filled deformable capsule with
a straight or curved prolate rest shape in unbounded simple shear flow (FIG.
1). The shear rate is $\dot{\gamma}$, and the undisturbed flow velocity is
given as $\mathbf{u}^{\infty}=(\dot{\gamma}y,0,0)$. Both the suspending fluid
and the fluid inside the capsule are assumed to be incompressible and
Newtonian with the same viscosity. The prolate spheroidal shape of the capsule
derives from polar stretching and equatorial compression of a spherical
capsule with radius $s$ centered at the origin. The equation for this
transformation is given by

$\mathbf{x}^{p}=\begin{pmatrix}c/s&0&0\\\ 0&a/s&0\\\
0&0&a/s\end{pmatrix}\cdot\mathbf{x}^{s},$ (1)

where $\mathbf{x}^{s}$ and $\mathbf{x}^{p}$ are the positions of a point on
the membrane of the original spherical capsule and the mapped point on the
membrane of the resulting prolate spheroidal capsule with a polar radius $c$
and an equatorial radius $a$ (aspect ratio $r_{p}=c/a$). The resulting prolate
spheroidal capsule is assumed to align with the $x$ axis. To generate a curved
capsule, we then impose a quadratic unidirectional displacement of membrane
points perpendicular to the polar axis of the prolate spheroidal capsule (here
in the $y$ direction),

$x^{c}_{1}=x^{p}_{1},$ (2a)
$x^{c}_{2}=x^{p}_{2}-k\Big{[}x^{p}_{1}-\frac{1}{2}(x^{pl}_{1}+x^{pr}_{1})\Big{]}^{2},$
(2b) $x^{c}_{3}=x^{p}_{3},$ (2c)

where $\mathbf{x}^{pl}$ and $\mathbf{x}^{pr}$ are the positions of the two
ends (left- and right-most membrane points) of the prolate spheroid, and
$\mathbf{x}^{c}=(x^{c}_{1},x^{c}_{2},x^{c}_{3})$ is the position of a membrane
point of the resulting curved prolate capsule mapped with the membrane point
of the prolate spheroid at position
$\mathbf{x}^{p}=(x^{p}_{1},x^{p}_{2},x^{p}_{3})$; $k$ has units of inverse
length, which can be nondimensionalized by $c$ to define the degree of
curvature $K=kc$.

To describe the behavior of the capsule membrane in response to the in-plane
shear elastic force, a membrane model by Skalak _et al._ Skalak  _et al._
(1973) is utilized, in which the strain energy density $W$ is given by

$W_{\mathrm{SK}}=\frac{G}{4}[(I_{1}^{2}+2I_{1}-2I_{2})+C_{a}I_{2}^{2}],$ (3)

where $G$ is the in-plane shear modulus of the membrane, and $C_{a}$ is a
material property that characterizes the energy penalty for area change of the
membrane; the strain invariants $I_{1}$ and $I_{2}$ are functions of the
principal stretch ratios $\lambda_{1}$ and $\lambda_{2}$, defined as

$I_{1}=\lambda_{1}^{2}+\lambda_{2}^{2}-2,\quad
I_{2}=\lambda_{1}^{2}\lambda_{2}^{2}-1.$ (4)

Note that this model predicts a strain hardening behavior of the membrane,
consistent with the experimentally determined response of an RBC membrane to
stretching Sinha and Graham (2015); P. Mills  _et al._ (2004). The
deformability of the capsule is characterized by the nondimensional capillary
number Ca = $\eta\dot{\gamma}c/G$, where $c$ again is half-length of the
straight or curved prolate capsule. In this work the capsules are taken to be
stiff in accordance with the biomechanical properties of the membrane of a
sickle RBC, with Ca lower than 0.2 assuming the shear rate $\dot{\gamma}$ of
blood flow in the microcirculation $\sim 10-10^{3}$ s-1 Lipowsky (2013), the
characteristic length $c$ of a typical sickle RBC $\sim 5-6$ $\mu$m Byun  _et
al._ (2012), the viscosity of plasma $\eta\sim$ 1.71 mPa s Laogun  _et al._
(1980), and the in-plane shear modulus of a sickle RBC membrane $G\sim 29.8$
$\mu$N/m Byun  _et al._ (2012). The bending modulus of the capsule $K_{B}$ is
expressed nondimensionally by $\hat{\kappa}_{B}=K_{B}/c^{2}G$, which is
$O(10^{-4}-10^{-2})$ in the physiological context Evans  _et al._ (2008); Betz
_et al._ (2009); here we set $\hat{\kappa}_{B}=0.04$ for all capsules. In this
work, the natural (spontaneous) shapes for shear and bending elasticities of
the capsule membrane are both chosen to be the same as its rest shape, so that
any in-plane or out-of-plane deformation would lead to an increase in the
membrane energy and an elastic restoring force. The capsule membrane is
discretized into 320 piecewise flat triangular elements, resulting in 162
nodes. We have verified that increasing the number of nodes makes no
difference to the cell dynamics. Based on this discretization, the calculation
of the membrane force density follows the work of Kumar and Graham Kumar and
Graham (2012) and Sinha and Graham Sinha and Graham (2015) using approaches
given by Charrier _et al._ Charrier  _et al._ (1989) for the in-plane shear
force density and Meyer _et al._ Meyer  _et al._ (2002) for the out-of-plane
bending force density, respectively.

The orientation of a curved capsule is defined by the normalized end-to-end
vector p and the unit normal vector n, as illustrated in FIG. 1. In this work,
initial conditions are chosen so that p is initially in the $x$-$z$ plane, and
the initial angle between p and the $x$ axis is denoted as $\alpha$. The
initial angle between n and its projection $\mathbf{n^{\prime}}$ onto the
$x$-$z$ plane is $\beta$ (note that $\mathbf{n^{\prime}}\perp\mathbf{p}$).
Thus the initial orientation of the capsule is given by $\alpha$ and $\beta$.
For example, ($\alpha,\beta$) = (0,$\pi/2$) means that the capsule is
initially in the $x$-$y$ plane with p aligned with the $x$ axis and n with the
$y$ axis. If the capsule is not curved, then $\beta$ is irrelevant to its
orientation and dynamics.

###  II.2 Fluid motion

As noted in Section II.1, the characteristic shear rate $\dot{\gamma}$ of
blood flow in the microcirculation is $\sim$ $10-10^{3}$ s-1 Lipowsky (2013),
the length scale $c$ of a typical sickle RBC $\sim 5-6$ $\mu$m Byun  _et al._
(2012), the viscosity of plasma $\eta$ $\sim$ 1.71 mPa s Laogun  _et al._
(1980), and its density $\rho$ $\sim$ $10^{3}$ kg/$\textrm{m}^{3}$. The
magnitude of the particle Reynolds number
$\textnormal{Re}_{p}=\rho\dot{\gamma}c^{2}/\eta$ obtained using these
parameters is $\sim O(10^{-4}-10^{-2})$, which is assumed to be sufficiently
small so that the fluid motion is governed by the Stokes equation. To
determine the velocity field at each time instant, we employ a boundary
integral method Pozrikidis (1992); Kumar and Graham (2012) for simulation in
unbounded simple shear flow. Once the flow field is obtained, the positions of
the element nodes on the discretized capsule membrane are advanced using a
second-order Adams-Bashforth method with time step $\Delta t=0.02$Ca$l$, where
$l$ is the minimum node-to-node distance. In this work, the midpoint of the p
vector of the capsule is initially placed at the origin of the unbounded
domain.

##  III RESULTS AND DISCUSSION

###  III.1 Model validation

((a))

((b))

Figure 2: Evolution of inclination angle $\phi$ for a stiff prolate spheroid
(Ca = 0.06) with (a) $r_{p}$ = 2.0 and (b) $r_{p}$ = 4.8, respectively. Solid
blue lines are predictions by Jeffery’s theory and red dashed lines are
simulation results.

To validate our model, we compare the numerically determined inclination
profile for a stiff prolate spheroid (Ca = 0.06) with the prediction by
Jeffery’s theory Jeffery (1922),

$\mathrm{tan}\;\phi=r_{p}\mathrm{tan}\bigg{(}\frac{\dot{\gamma}t}{r_{p}+{r_{p}}^{-1}}+\phi_{0}\bigg{)},$
(5)

where $\phi$ is the inclination angle between the major axis of the prolate
spheroid and the $y$ direction, and $\phi_{0}$ its initial value; here
$\phi_{0}=\pi/2$ since the prolate spheroid is initially aligned with the $x$
axis. FIG. 2 shows the evolution of $\phi$ with nondimensionalized time
$\dot{\gamma}t$. A satisfactory agreement between numerical simulation and
Jeffery’s theory (Eq. 5) is observed for the prolate spheroid with either a
low ($r_{p}$ = 2.0) or a high ($r_{p}$ = 4.8) aspect ratio. The very small
discrepancy in each case derives from the non-zero deformability of the
prolate spheroid, and vanishes as Ca approaches zero.

###  III.2 Dynamics of deformable prolate spheroids

((a))

((b))

Figure 3: Trajectories on the unit sphere of (a) p and (b) n for a prolate
spheroid ($r_{p}$ = 4.8, Ca = 0.12) with initial orientation $\alpha=0$ (up to
$\dot{\gamma}t=2000$).

We first consider a stiff prolate spheroidal capsule ($r_{p}$ = 4.8, Ca =
0.12) initially aligned with the $x$ axis, $i.e.$, $\alpha=0$. The long-time
trajectories (up to $\dot{\gamma}t$ = 2000) of the end-to-end vector p and the
normal vector n on the unit sphere are plotted in FIG. 3. In this case, both p
and n appear to trace out a closed circle, indicating that the capsule is
taking a steady symmetric tumbling motion in the $x$-$y$ plane.

((a))

((b))

((c))

((d))

Figure 4: Trajectories on the unit sphere of p ((a) and (b)) and n ((c) and
(d)) for a prolate spheroid ($r_{p}$ = 4.8, Ca = 0.12) with initial
orientation $\alpha=\pi/36$. Figure 5: The evolution of the $p_{z}$ and
$n_{z}$ for a prolate spheroid ($r_{p}$ = 4.8, Ca = 0.12) with initial
orientation $\alpha=\pi/36$.

We then consider a perturbation of the initial orientation by a small angle
$\alpha=\pi/36$. The trajectories of p and n vectors on the unit sphere are
shown in FIG. 4. It can be observed that both $p_{z}$ and $n_{z}$ show an
oscillation around zero (FIG. 5) whose amplitude decays to zero as time
increases. The director p crosses the $x$-$y$ plane back-and-forth (as
indicated by the sign change of $p_{z}$ in FIG. 5), exhibiting non-Jeffery-
like dynamics (note that in Jeffery orbits the director never crosses the
shear-gradient plane). The dynamics of n are much more complicated. The
trajectory reaches a quasi-steady state position on either hemisphere
alternately every time $p_{z}$ changes its sign, as observed in FIG. 4(d), and
the quasi-steady state position gets closer and closer to zero, which is
consistent with the damped oscillation of $n_{z}$ (FIG. 5). All these
behaviors indicate that under small perturbation, the orbit of a slightly
deformable prolate spheroidal capsule evolves back towards the symmetric
tumbling motion in the $x$-$y$ plane, indicating that the symmetric tumbling
orbit in the $x$-$y$ plane is stable.

By further increasing the initial angle $\alpha$ of the prolate spheroid off
the $x$ axis, we observe that for an arbitrary $\alpha$, the orbit of the
prolate spheroid evolves towards the symmetric tumbling in the $x$-$y$ plane
regardless of the initial orientation. This shows that the in-plane symmetric
tumbling is the globally stable state for the prolate spheroidal capsules in
the parameter regime considered.

###  III.3 Dynamics of curved prolate spheroids

((a))

((b))

Figure 6: Trajectories of p (a) and n (b) for a curved prolate capsule with
initial orientation [$\alpha,\beta$] = [0,0] at the early stage of simulation
($0<\dot{\gamma}t<500$).

In this section, we perform a systematic investigation of the dynamics of
curved prolate spheroids at low Ca as described in Section II.1. We first
examine the early-stage dynamics ($0<\dot{\gamma}t\lesssim 500$) of curved
prolate capsules in the case of Ca = 0.12 and $K=0.36$ with various initial
orientations, beginning with the case where p is initially aligned with the
$x$ direction and n with the $z$ direction. Here the trajectory of p traces
out a closed circle on the unit sphere (FIG. 6(a)), while n remains fixed
(FIG. 6(b)), which indicates that the capsule is taking a sideways Jeffery-
like tumbling motion. In the case of [$\alpha,\beta$] = [0,$\pi/6$] (FIG. 7(a)
and 7(d)), p is observed to cross the $x$-$y$ plane back and forth, exhibiting
a wobbling motion which is not Jeffery-like, as shown in FIG. 8(a). We then
consider the case where [$\alpha,\beta$] = [0,$\pi/2$], $i.e.$, the capsule is
completely lying in the $x$-$y$ plane initially, with p aligned with the $x$
direction and n with the $y$ direction. The closed circular orbits on the unit
sphere indicate that the capsule is taking a rigid-body-like symmetric
tumbling motion as its early-stage dynamics (FIG. 7(b) and 7(e)). In the case
[$\alpha,\beta$] = [$\pi/6$,$\pi/3$], a more Jeffery-like orbit is observed
(FIG. 7(c) and 7(f)), in the sense that p does not cross the shear plane. Here
it is noteworthy that in a Jeffery-like orbit, the trajectory of n spans both
hemispheres divided by the shear plane on the unit sphere (FIG. 7(f)), while
it only appears on one hemisphere of the unit sphere in the non-Jeffery-like
orbit observed in the case of [$\alpha,\beta$] = [0,$\pi/6$] described above
(FIG. 7(d)).

((a))

((b))

((c))

((d))

((e))

((f))

Figure 7: Early-stage trajectories on the unit sphere of the p (blue) and n
(red) vectors of a curved prolate capsule with initial orientation
[$\alpha,\beta$] = [0,$\pi/6$] ($0<\dot{\gamma}t<500$) (a,d), [0,$\pi/2$]
($0<\dot{\gamma}t<100$) (b,e), and [$\pi/6$,$\pi/3$] ($0<\dot{\gamma}t<400$)
(c,f), respectively.

((a))

((b))

Figure 8: Time sequence images (front view) of the early-stage (a) and long-
time (b) motions of a curved prolate capsule with initial orientation
[$\alpha,\beta$] = [0,$\pi/6$]. The arrow in each image represents the p
vector at the corresponding time spot.

The early-stage dynamics of the curved prolate capsules described above,
however, are all found to be long-lived transients. FIG. 9 shows the long-time
($\dot{\gamma}t>800$) trajectories of the p vector of curved prolate capsules
with different initial orientations on the unit sphere. The long-time dynamics
of the capsules, regardless of their early-stage orbits, all evolve into a
quasi-periodic kayaking motion that might be described as a “modulated Jeffery
orbit”. This long-time motion is independent of the initial orientation of the
capsules, as indicated by the nearly identical trajectories of p on the unit
sphere (FIG. 9). Qualitatively, these orbits have a fast time scale
corresponding to one cycle of kayaking, with a much longer time scale
describing a slow modulation of the orbit. Each kayaking cycle is slightly
different than the previous, a fact that can be attributed to the dynamics of
n, which are coupled to those of p for a curved particle. FIG. 8(b) shows
snapshots of a long-time orbit.

The two time scales noted above are also observed in the long-time center-of-
mass position profiles shown in FIG. 10, with a shorter time scale
representing one cycle of the kayaking motion described above superimposed
onto a longer time scale corresponding to the orbit modulation. In the
parameter regime considered here, we have seen no evidence of drift in the $y$
(“wall normal”) direction. This is in contrast to the observations of Wang
_et al._ (2012) for curved rigid fibers in the “flipping and scooping” regime;
we believe the difference arises from the modest aspect ratios considered
here, which are much smaller than those studied by Wang  _et al._ (2012). We
do, however, observe a transient phase of spanwise drift in the case of the
capsule initialized to perform a sideways tumbling motion. This vanishes once
the cell reaches its long-time orientational trajectory. Although there is no
drift at long time, both the $y$ and $z$ components of the center of mass
oscillate as the cell kayaks.

((a))

((b))

((c))

Figure 9: Long-time trajectories of p of curved prolate capsules with initial
orientations [$\alpha,\beta$] = [0,$\pi/6$] (a), [0,$\pi/2$] (b), and
[$\pi/6$,$\pi/3$] (c) on the unit sphere ($y$-$z$ view).

((a))

((b))

((c))

Figure 10: Evolution of $y$\- and $z$-components $y_{cm}$ and $z_{cm}$ of the
center-of-mass position of curved prolate capsules with initial orientations
[$\alpha,\beta$] = [0,$\pi/6$] (a), [0,$\pi/2$] (b), and [$\pi/6$,$\pi/3$]
(c). Figure 11: The long-time trajectory (grey dots) of a curved prolate
capsule (Ca = 0.12, $K$ = 0.36) in terms of $\phi$ and $\theta$ and the
average trajectory (black line). Black circles represent the fit of the
average trajectory using Eq. 7.

((a))

((b))

Figure 12: The dependencies of (a) the effective orbit constant
$C_{\textnormal{eff}}$ of the long-time orbit and (b) the effective aspect
ratio $\ell_{\textnormal{eff}}$ on the deformability Ca and the degree of
curvature $K$ of the curved prolate capsules.

Now we vary $\mathrm{Ca}$ and $K$ to see the effects of deformability and
curvature on the long-time dynamics. To illustrate how we characterize these
effects, we first revisit the case of $\mathrm{Ca}=0.12$, $K=0.36$, for which
the orbits in FIG. 9 show the trajectory of p on the unit sphere. The grey
dotted curve in FIG. 11 shows this orbit expressed in spherical coordinates
$\phi$ and $\theta$ based on the coordinate system used in Jeffery’s theory
Jeffery (1922); here $\phi$ is the azimuthal angle with respect to the $y$
axis and $\theta$ is the polar angle with respect to the $z$ axis. With this
representation, we can find a mean trajectory $\bar{\theta}(\phi)$ by
averaging $\theta$ at each $\phi$. This is the black curve in FIG. 11. For a
true Jeffery orbit of a rigid particle, the relation between the two angles
$\phi$ and $\theta$ is given by

$\mathrm{tan}\;\theta=\frac{C\ell}{\big{(}\ell^{2}\;\mathrm{cos}^{2}\phi+\mathrm{sin}^{2}\phi\big{)}^{1/2}},$
(6)

where $C$ and $\ell$ are the orbit constant and aspect ratio for the rigid
spheroid. We have found that in all cases considered here, we can approximate
with good accuracy the curve $\bar{\theta}(\phi)$ as a Jeffery orbit for a
spheroid with an effective orbit constant $C_{\textnormal{eff}}$ and effective
aspect ratio $\ell_{\textnormal{eff}}$. That is, we can use
$C_{\textnormal{eff}}$ and $\ell_{\textnormal{eff}}$ as fitting parameters in
the expression

$\mathrm{tan}\;\bar{\theta}=\frac{C_{\mathrm{eff}}\ell_{\mathrm{eff}}}{\big{(}\ell_{\mathrm{eff}}^{2}\;\mathrm{cos}^{2}\phi+\mathrm{sin}^{2}\phi\big{)}^{1/2}}.$
(7)

The black circles in FIG. 11 show this approximation for $\mathrm{Ca}=0.12$,
$K=0.36$.

We have computed long-time trajectories over a range of $\mathrm{Ca}$ and $K$
and used the resulting values of $C_{\mathrm{eff}}$ and $\ell_{\mathrm{eff}}$
to characterize the dynamics. These results are summarized in FIG. 12. It is
observed that $\ell_{\textnormal{eff}}$ increases weakly as $K$ increases but
shows very little dependency on $\mathrm{Ca}$ (FIG. 12(b)). More importantly,
$C_{\textnormal{eff}}$ decreases as either $\mathrm{Ca}$ or $K$ increases
(FIG. 12(a)), indicating a shift of the orbit towards log-rolling motion with
increasing curvature or deformability.

For completeness, we briefly consider the deviations of the instantaneous
trajectories from the mean curve $\bar{\theta}(\phi)$. The maximum positive
and negative deviations from the mean when $\phi=\pi$ are denoted as
$\Delta\theta^{+}$ and $\Delta\theta^{-}$. The relative deviations
$\Delta\theta/\overline{\theta}$ are found to be insensitive to the parameters
in the range considered here, with
$0.16<\Delta\theta^{+}/\overline{\theta}<0.19$, and
$-0.35<\Delta\theta^{-}/\overline{\theta}<-0.30$.

##  IV CONCLUSION

We investigated the dynamics of neutrally-buoyant, slightly deformable
straight and curved prolate capsules subjected to unbounded simple shear flow
at zero Reynolds number using direct simulations. The curved prolate capsules
in this work serve as a model for the typical crescent-shaped sickle RBCs in
sickle cell disease (SCD), and we aimed to show the effects of initial
orientation, membrane deformability and curvature of the capsules on their
dynamics.

The dynamics of the capsules are simulated using a full model incorporating
both shear and bending elasticities. The parameter regime is based on the
range of experimentally determined values for blood flow in the
microcirculation. We revealed that with low deformability, straight prolate
spheroidal capsules take tumbling in the shear plane as the unique globally
stable orbit independent of the initial orientation. As the trajectories of
the straight prolate capsules transiently evolve towards this in-plane
tumbling, the director can cross the $x$-$y$ plane back and forth, showing a
non-Jeffery-like motion.

Curved prolate capsules exhibit complicated and interesting dynamics due to
the combined effects of membrane deformability $\mathrm{Ca}$ and the curvature
$K$ of the capsules. At short times, a non-Jeffery-like behavior, in which the
director (end-to-end vector) of the capsule crosses the shear-gradient plane
periodically, is observed for specific initial orientations. Steady drift of
the center of mass in the vorticity direction is also observed for some
initial orientations. At long times, however, the trajectories of the curved
capsules, regardless of their initial orientations, all evolve towards a
Jeffery-like quasi-periodic kayaking orbit with the center-of-mass position of
a curved capsule crossing the shear plane periodically instead of remaining in
it. No cross-stream drift is found. The average of the long-time orbit can be
well approximated using the analytical solution for Jeffery orbits, yielding
an effective orbit constant $C_{\textnormal{eff}}$ and aspect ratio
$\ell_{\textnormal{eff}}$. A parameter study further reveals that
$C_{\textnormal{eff}}$ decreases as the capsule becomes more deformable or
curved, indicating a shift of the orbit towards log-rolling motion.

Overall, these results represent an effort to understand the motion of single
sickle RBCs, taking into account the pathologically altered shape as well as
the elastic and bending moduli of these diseased cells. Major aspects of the
pathophysiology of SCD remain poorly understood and currently, there is an
increased focus on how aberrant cell interactions, between sickle RBCs and
other blood cells ($i.e.$ platelets and leukocytes) or between sickle RBCs and
endothelial cells, may contribute to phenomena such as thrombosis or vascular
inflammation that are known to be vital part of the process of SCD. In
particular, the effective orbit constant of a sickle cell moving near a blood
vessel wall may be reflected in the propensity for that cell to collide with
and potentially damage the wall. More broadly, the results of this work
demonstrate how the mechanical properties and distorted cell shape of sickle
RBCs lead to distinct trajectories that may, in turn, cause pathologic
biophysical activation of other cells, and may provide insight into how these
disparate aspects of SCD are ultimately linked. That knowledge will then lead
to improved methods to diagnose complications of and new therapeutic targets
for SCD.

###### Acknowledgements.

This work was supported by NSF Grant No. CBET-1436082 and NIH Grant No.
R21MD011590-01A1.

## References

  * Mohandas and Evans (1994) N. Mohandas and E. Evans, “Mechanical properties of the red cell membrane in relation to molecular structure and genetic defects,” Annu. Rev. Biophys. Biomol. Struct. 23, 787–818 (1994).
  * Goldsmith  _et al._ (1972) H. L. Goldsmith, J. Marlow, and F. C. MacIntosh, “Flow behaviour of erythrocytes. I. Rotation and deformation in dilute suspensions,” Proc. R. Soc. Lond., B, Biol. Sci. 182, 351–384 (1972).
  * Fischer  _et al._ (1978) T. M. Fischer, M. Stohr-Lissen, and H. Schmid-Schonbein, “The red cell as a fluid droplet: tank tread-like motion of the human erythrocyte membrane in shear flow,” Science 202, 894–896 (1978).
  * Abkarian  _et al._ (2007) M. Abkarian, M. Faivre, and A. Viallat, “Swinging of red blood cells under shear flow,” Phys. Rev. Lett. 98, 188302 (2007).
  * Skotheim and Secomb (2007) J. M. Skotheim and T. W. Secomb, “Red blood cells and other nonspherical capsules in shear flow: Oscillatory dynamics and the tank-treading-to-tumbling transition,” Phys. Rev. Lett. 98, 078301 (2007).
  * Dupire  _et al._ (2010) J. Dupire, M. Abkarian, and A. Viallat, “Chaotic dynamics of red blood cells in a sinusoidal flow,” Phys. Rev. Lett. 104, 168101 (2010).
  * Fedosov  _et al._ (2010) D. A. Fedosov, B. Caswell, and G. E. Karniadakis, “A multiscale red blood cell model with accurate mechanics, rheology, and dynamics,” Biophys. J. 98, 2215–2225 (2010).
  * Fedosov  _et al._ (2011) D. A. Fedosov, H. Lei, B. Caswell, S. Suresh, and G. E. Karniadakis, “Multiscale modeling of red blood cell mechanics and blood flow in malaria,” PLoS Comp. Biol. 7, e1002270 (2011).
  * Yazdani and Bagchi (2011) A. Z. K. Yazdani and P. Bagchi, “Phase diagram and breathing dynamics of a single red blood cell and a biconcave capsule in dilute shear flow,” Phys. Rev. E 84, 026314 (2011).
  * Sinha and Graham (2015) K. Sinha and M. D. Graham, “Dynamics of a single red blood cell in simple shear flow,” Phys. Rev. E 92, 042710 (2015).
  * Byun  _et al._ (2012) H. Byun, T. R. Hillman, J. M. Higgins, M. Diez-Silva, Z. Peng, M. Dao, R. R. Dasari, S. Suresh, and Y. Park, “Optical measurement of biomechanical properties of individual erythrocytes from a sickle cell patient,” Acta Biomater. 8, 4130–4138 (2012).
  * Messer and Harris (1970) M. J. Messer and J. W. Harris, “Filtration characteristics of sickle cells: Rates of alteration of filterability after deoxygenation and reoxygenation, and correlations with sickling and unsickling,” J. Lab. Clin. Med. 76, 537–547 (1970).
  * Nash  _et al._ (1984) G. B. Nash, C. S. Johnson, and H. J. Meiselman, “Mechanical properties of oxygenated red blood cells in sickle cell (HbSS) disease,” Blood 63, 73–82 (1984).
  * Evans and Mohandas (1987) E. A. Evans and N. Mohandas, “Membrane-associated sickle hemoglobin: a major determinant of sickle erythrocyte rigidity,” Blood 70, 1443–1449 (1987).
  * Bertles and Döbler (1969) J. F. Bertles and J. Döbler, “Reversible and irreversible sickling: a distinction by electron microscopy,” Blood 33, 884–898 (1969).
  * Jeffery (1922) G. B. Jeffery, “The motion of ellipsoidal particles immersed in a viscous fluid,” Proc. Roy. Soc. Lond. A 102, 161–179 (1922).
  * Bretherton (1962) F. P. Bretherton, “The motion of rigid particles in a shear flow at low Reynolds number,” J. Fluid Mech. 14, 284–304 (1962).
  * Hinch and Leal (1979) E. J. Hinch and L. G. Leal, “Rotation of small non-axisymmetric particles in a simple shear flow,” J. Fluid Mech. 92, 591–607 (1979).
  * Yarin  _et al._ (1997) A. L. Yarin, O. Gottlieb, and I. V. Roisman, “Chaotic rotation of triaxial ellipsoids in simple shear flow,” J. Fluid Mech. 340, 83–100 (1997).
  * Dupont  _et al._ (2013) C. Dupont, A. Salsac, and D. Barthes-Biesel, “Off-plane motion of a prolate capsule in shear flow,” J. Fluid Mech. 721, 180–198 (2013).
  * Cordasco and Bagchi (2013) D. Cordasco and P. Bagchi, “Orbital drift of capsules and red blood cells in shear flow,” Phys. Fluids 25, 091902 (2013).
  * Mao and Alexeev (2014) W. Mao and A. Alexeev, “Motion of spheroid particles in shear flow with inertia,” J. Fluid Mech. 749, 145–166 (2014).
  * Lundell and Carlsson (2010) F. Lundell and A. Carlsson, “Heavy ellipsoids in creeping shear flow: Transitions of the particle rotation rate and orbit shape,” Phys. Rev. E 81, 016323 (2010).
  * Wang  _et al._ (2012) J. Wang, E. J. Tozzi, M. D. Graham, and D. J. Klingenberg, “Flipping, scooping, and spinning: Drift of rigid curved nonchiral fibers in simple shear flow,” Phys. Fluids 24, 123304 (2012).
  * Crowdy (2016) D. Crowdy, “Flipping and scooping of curved 2D rigid fibers in simple shear: The Jeffery equations,” Phys. Fluids 28, 053105 (2016).
  * Dutta and Graham (2017) S. Dutta and M. D. Graham, “Dynamics of Miura-patterned foldable sheets in shear flow,” Soft Matter 13, 2620–2633 (2017).
  * Skalak  _et al._ (1973) R. Skalak, A. Tozeren, R. P. Zarda, and S. Chien, “Strain energy function of red blood-cell membranes,” Biophys. J. 13, 245–280 (1973).
  * P. Mills  _et al._ (2004) J. P. Mills, L. Qie, M. Dao, C. T. Lim, and S. Suresh, “Nonlinear elastic and viscoelastic deformation of the human red blood cell with optical tweezers,” Mech. Chem. Biosyst. 1, 169–180 (2004).
  * Lipowsky (2013) H. H. Lipowsky, “In vivo studies of blood rheology in the microcirculation in an in vitro world: Past, present and future,” Biorheology 50, 3–16 (2013).
  * Laogun  _et al._ (1980) A. A. Laogun, N. O. Ajayi, N. O. Osamo, and L. A. Okafor, “Plasma viscosity in sickle-cell anaemia,” Clin. Phys. Physiol. Meas. 1, 145 (1980).
  * Evans  _et al._ (2008) J. Evans, W. Gratzer, N. Mohandas, K. Parker, and J. Sleep, “Fluctuations of the red blood cell membrane: relation to mechanical properties and lack of ATP dependence,” Biophys. J. 94, 4134–4144 (2008).
  * Betz  _et al._ (2009) T. Betz, M. Lenz, J. Joanny, and C. Sykes, “ATP-dependent mechanics of red blood cells,” Proc. Natl. Acad. Sci. USA 106, 15320–15325 (2009).
  * Kumar and Graham (2012) A. Kumar and M. D. Graham, “Accelerated boundary integral method for multiphase flow in non-periodic geometries,” J. Comput. Phys. 231, 6682–6713 (2012).
  * Charrier  _et al._ (1989) J. M. Charrier, S. Shrivastava, and R. Wu, “Free and constrained inflation of elastic membranes in relation to thermoforming – non-axisymmetric problems,” J. Strain Anal. Eng. Des. 24, 55–74 (1989).
  * Meyer  _et al._ (2002) M. Meyer, M. Desbrun, P. Schroeder, and A. H. Barr, “Discrete differential geometry operators for triangulated 2-manifolds,” Visualization and Mathematics 3, 34–57 (2002).
  * Pozrikidis (1992) C. Pozrikidis,  _Boundary integral and singularity methods for linearized viscous flow_, Cambridge Texts in Applied Mathematics (Cambridge University Press, 1992).


# Establishing Human-Robot Trust through Music-Driven Robotic Emotion Prosody
and Gesture

Richard Savery, Ryan Rose and Gil Weinberg1 1Georgia Tech Center for Music
Technology, Atlanta, GA, USA<EMAIL_ADDRESS>
###### Abstract

As human-robot collaboration opportunities continue to expand, trust becomes
ever more important for full engagement and utilization of robots. Affective
trust, built on emotional relationship and interpersonal bonds is particularly
critical as it is more resilient to mistakes and increases the willingness to
collaborate. In this paper we present a novel model built on music-driven
emotional prosody and gestures that encourages the perception of a robotic
identity, designed to avoid uncanny valley. Symbolic musical phrases were
generated and tagged with emotional information by human musicians. These
phrases controlled a synthesis engine playing back pre-rendered audio samples
generated through interpolation of phonemes and electronic instruments.
Gestures were also driven by the symbolic phrases, encoding the emotion from
the musical phrase to low degree-of-freedom movements. Through a user study we
showed that our system was able to accurately portray a range of emotions to
the user. We also showed with a significant result that our non-linguistic
audio generation achieved an 8% higher mean of average trust than using a
state-of-the-art text-to-speech system.

##  I INTRODUCTION

As co-robots become prevalent at home, work, and in public environments, a
need arises for the development of trust between humans and robots. A meta-
study of human-robot trust [1] has shown that robot-related attributes are the
main contributors to building trust in Human-Robot-Interaction, affecting
trust more than environmental and human related factors. Related research on
artificial agents and personality traits [2, 3] indicates conveying emotions
using subtle non-verbal communication channels such as prosody and gesture is
an effective approach for building trust with artificial agents. These
channels can help convey intentions as well as expressions such as humor,
sarcasm, irony, and state-of-mind, which help build social relationship and
trust.

In this work we developed new modules for Shimi, a personal robotic platform
[4], to study whether emotion-driven non-verbal prosody and body gesture can
help establish affective-based trust in HRI. Our approach is to use music, one
of the most emotive human experiences, to drive a novel system for emotional
prosody [5] and body gesture [6] generation. We propose that music-driven
prosody and gesture generation can provide effective low degrees of freedom
(DoF) interaction that can convey robotic emotional content, avoid the uncanny
valley [7], and help build human-robot trust. Furthermore, trust is highly
dictated by the first impression for both human-human and human-robot
relations [8], implying methods for gaining trust at the start of a
relationship, such as prosody and gesture are crucial.

Figure 1: The musical robot companion Shimi

We address two research questions, firstly, can we use non-verbal prosody
through musical phrases combined with gestures to accurately convey emotions?
We present a novel deep learning based musical voice generation system that
uses the created phrases to generate gesture through musical prosody. In this
way music is central to all interactions presented by Shimi. We evaluate the
effectiveness of the musical audio and generative gesture system for Shimi to
convey emotion, specified by valence-arousal quadrants. Our second research
question is whether emotional conveyance through prosody and gestures driven
by music analysis can increase the level of trust in human-robot-interaction.
For this we conduct a user study to evaluate prosodic audio and gestures
created by our new model in comparison to a baseline text-to-speech system.

##  II Background

###  II-A Trust in HRI

Trust is a key requirement for working with collaborative robots, as low
levels of trust can lead to under-utilization in work and home environments
[9]. A key component of the dynamic nature of trust is created in the first
phase of a relationship [10, 11], while lack of early trust building can
remove the opportunity for trust to develop later on [12]. Lack of trust in
robotic systems can also lead to expert operators bypassing the robot to
complete tasks [13]. Trust is generally categorized into either cognitive
trust or affective trust [14]. Affective trust involves emotional bonds and
personal relationships, while cognitive trust focuses on considerations around
dependability and competence. Perceiving emotion is crucial for the
development of affective trust in human-to-human interaction [15], as it
increases the willingness to collaborate and expand resources bought to the
interactions [16]. Importantly, relationships based on affective trust are
more resilient to mistakes by either party [15], and perceiving an emotional
identity has been shown to be an important contributor for creating believable
and trustworthy interaction [2, 3]. In group interactions, emotional contagion
- where emotion is spread between a group - has been shown to improve
cooperation and trust in team exercises [17].

###  II-B Emotion, Music and Prosody

Emotion conveyance is one of the key elements for creating believable agents
[2], and prosody has been proven to be an effective communication channel to
convey such emotions for humans [18] and robots [19]. On a related front,
music which shares many of the underlying building blocks of prosody such as
pitch, timing, loudness, intonation, and timbre [20], has also been shown to
be a powerful medium to convey emotions [21]. In both music and prosody,
emotions can be classified in a discrete categorical manner (happiness,
sadness, fear, etc.) [22], and through continuous dimensions such as valence,
arousal, and less commonly, dominance, and stance [23, 24]. While some recent
efforts to generate and manipulate robotic emotions through prosody focused on
linguistic robotic communication [19], [25] no known efforts have been made to
use models from music analysis to inform real-time robotic non linguistic
prosody for collaboration, as we propose here.

###  II-C Emotion, Music and Gesture

Human bodily movements are embedded with emotional expression [26, 27], which
can be processed by human “affective channels” [28]. Researchers in the field
of affective computing, have been working on designing machines that can
process and communicate emotions through such channels [29]. Non-conscious
affective channels have been demonstrated to communicate compassion,
awareness, accuracy, and competency, making them vital components of social
interaction with robots [30]. Studies have shown clear correlations between
musical features and movement features, suggesting that a single model can be
used to express emotion through both music and movement [31, 32, 33]. Certain
characteristics of robotic motion have been shown to influence human emotional
response [34, 35]. Additionally, emotional intelligence in robots leads to
facilitated human-robot interactions [36, 25, 37]. Gesture has been used to
accompany speech in robots to help convey affective information [38], however,
to our knowledge, there is no prior work that attempts to integrate physical
gestures and music-driven prosody to convey robotic emotional states.

##  III Shimi and Emotion

###  III-A Prosody

The goal of this project was to create a new voice for Shimi, using musical
audio phrases tagged with an emotion. We aimed to develop a new voice that
could generate phrases in real-time. This was achieved through a multi-layer
system, combining symbolic phrase generation using MIDI controlling a
synthesis playback system.

Figure 2: Angry Speech Pitch and Intensity Figure 3: Calm Speech Pitch and
Intensity

####  III-A1 Dataset and Phrase Generation

To control Shimi’s vocalizations we generate MIDI phrases that drive the
synthesis and audio generation described below and lead the gesture
generation. MIDI is a standard music protocol, where notes are stored by pitch
value with a note on velocity, followed by a note off to mark the end of the
note. With the absence of appropriate datasets we chose to create our own set
of MIDI files tagged with valence and arousal by quadrant. MIDI files were
collected from eleven different improvisers around the United States, each of
whom tagged their recorded files with an emotion corresponding to a quadrant
of the valence/arousal model. Phrases were required to be between 100ms and 6
seconds and each improviser recorded between 50 to 200 samples for each
quadrant. To validate this data we created a separate process whereby the
pitch range, velocities and contour were compared to the RAVDESS [39] data
set, with files removed when the variation was over a manually set threshold.
RAVDESS contains speech files tagged with emotion, Figure 2 and Figure 3
clearly demonstrate the variety of prosody details apparent in the RAVDESS
dataset (created using [40, 41]) and the variation between a calm and angry
utterance of the same phrase.

We chose to use a Recurrent Neural Network, Long Short Term Memory (RNN-LSTM)
as described in [42] to generate musical phrases using this data. RNN-LSTM’s
have been used effectively to generate short melodies, as they are sequential
and consider the input as the output is generated. Other network structures
were considered however we found RNN-LSTM’s very effective for this task when
compared to alternate approaches developed by the authors [43, 44, 45, 46],
more detail is provided in [47]. While using samples directly from the data
was possible, using a neural network allowed infinite variation but also
allowed for generated phrases utilizing all the musical features created by
the improvisers, not just one improviser per sample.

####  III-A2 Audio Creation and Synthesis

The generated MIDI phrases contain symbolic information only without audio. To
create audio we developed a synthesis system to playback the MIDI phrase. As
we desired to create a system devoid of semantic meaning a new vocabulary was
constructed. This was built upon phonemes from the Australian Aboriginal
language Yuwaalaraay a dialect of the Gamilaraay language. Sounds were created
by interpolating four different synthesizer sounds with 28 phonemes.
Interpolation was done using a modified version of WaveNet. These samples are
then time stretched and pitch shifted to match the incoming MIDI file. For a
more detailed technical overview of audio processing read [47].

###  III-B Gestures

In human communication gestures are tightly coupled with speech [48]. Thus,
Shimi’s body language is implemented in the same way, derived from its musical
prosody and leveraging the musical encoding of emotion to express that emotion
physically. Music and movement are correlated, with research finding
commonalities in features between both modes [31]. Additionally, humans
demonstrate patterns in movement that is induced from music [49]. Particular
music-induced movement features are also correlated to perceived emotion in
music [50]. After a musical phrase is generated for Shimi’s voice to sing, the
MIDI representation of that phrase is provided as input to a gesture
generation system. Musical features such as tempo, range, note contour, key,
and rhythmic density are obtained from the MIDI through Python libraries
`pretty_midi` [51] and `music21`111https://github.com/cuthbertLab/music21.
These features are used to create mappings between Shimi’s voice and movement:
for example, pitch contour is used to govern Shimi’s torso forward and
backward movement. Other mappings include beat synchronization across multiple
subdivisions of the beat in Shimi’s foot, and note onset-based movements in
Shimi’s up-and-down neck movement.

After mapping musical features to low-level movements, Shimi’s emotional state
is used to condition the actuation of the movements. Continuous values for
valence and arousal are used to influence the range, speed, and amount of
motion Shimi exhibits. Some conditioning examples include limiting or
expanding the range of motion according to the arousal value, and governing
how smooth motor direction changes are through Shimi’s current valence level.
In some cases, the gestures generated for one degree of freedom are dependent
on another degree of freedom. For example, when Shimi’s torso leans forward,
Shimi’s attached head will be affected as well. As such, to control where
Shimi is looking, any neck gestures need to know the position of the torso. To
accommodate these inter-dependencies, when the gesture system is given input,
each degree of freedom’s movements are generated sequentially and in full,
before being actuated together in time with Shimi’s voice. Video examples of
gesture and audio are available at www.richardsavery.com/shimitrust.

##  IV Methodology

We designed an experiment to identify how well participants could recognize
the emotions shown by our music-driven prosodic and gestural emotion
generator. This part of the experiment aimed to answer our first research
question, can non-verbal prosody combined with gestures accurately portray
emotion. After watching a collection of stimuli, participants completed a
survey measuring the trust rating from each participant. This part of the
experiment was designed to answer the second question, can emotion driven,
non-semantic audio generate trust in a robot.

We hypothesized that through non-semantic prosodic vocalizations accompanied
with low-DoF robotic gesture humans will be able to correctly classify Shimi’s
portrayed emotion as either happy, calm, sad, or angry, with an accuracy
consistent with that of text-to-speech. Our second hypothesis was that we will
see higher levels of trust from the Shimi using non-speech.

###  IV-A Stimuli

Name | Audio | Stochastic | Experimental
---|---|---|---
Audio Only | X |  |
Stochastic Gesture, audio | X | X |
Stochastic Gesture, no audio |  | X |
Experimental Gesture, audio | X |  | X
Experimental Gesture, |  | X | X
no audio |  |  |
TABLE I: Experiment Stimuli Figure 4: Confusion Matrix

The experiment was designed as a between-subjects study, where one group would
hear the audio with the Shimi voice, while the other would hear pre-rendered
text-to-speech. Both groups saw the same gesture and answered the same
prompts. The text-to-speech examples were synchronized in length and emotion
to Shimi’s voice. The stimuli for the Speech Audio experiment used CereProc’s
Meghan voice222https://www.cereproc.com/. CereProc is a state of the art text
to speech engine. The text spoken by Meghan was chosen from the EmoInt Dataset
[52], which is a collection of manually tagged tweets.

###  IV-B Emotion

The generated gestures were either deterministic gestures created using the
previously described system, or deterministic stochastic gestures. Stochastic
gestures were implemented by considering each DoF separately, restricting
their ranges to those implemented in the generative system, and specifying
random individual movement durations up to half of the length of the full
gesture. The random number generator used in these gestures were seeded with
an identifier unique to the stimuli such that they were deterministic between
participants. Gesture stimuli were presented both with and without audio.

###  IV-C Procedure

Participants were gathered from the undergraduate student population at the
Georgia Institute of Technology (N=24). Subjects participated independently,
with the group alternating for each participant, culminating with 12 in each
group. The session began with an introduction to the task of identifying the
emotion displayed by Shimi. Participants responded through a web interface
that controlled Shimi through the experiment and then allowed the user to
select the emotion they thought Shimi was expressing. Stimuli were randomly
ordered for each participant. Table I shows the order of stimuli used, each
category contained 8 stimuli, 2 for each valence arousal quadrant. After
identifying all stimuli participants were directed to a Qualtrics survey to
gather their trust rating.

To measure trust, we used the Trust Perception Scale-HRI [12]. This scale uses
40 questions, each one using a rating scale between 0-100%, to give an average
trust rating per participant. The questions take between 5-10 minutes to
complete and include questions such as how often the robot will be reliable or
pleasant. After completing the trust rating, participants had several open
text boxes to discuss any observations in regards to emotion recognition,
trust or the general experiment. This was the first time trust was mentioned
in the experiment.

##  V Results

Figure 5: Questions with P less than 0.1

###  V-A Gestures and Emotion

After data was collected, two participant’s emotion prediction data was found
to be corrupted due to a problem with the testing interface, reducing the
number of participants in this portion of the study to 22. First, we
considered classification statistics for the isolated predictions of Shimi’s
voice and text-to-speech (TTS) voice. While TTS outperformed Shimi’s voice (F1
score $TTS=0.87$ vs. $Shimi=0.63$), the confusion matrices show errant
predictions in similar scenarios (see figure 4). For example, both audio
classes struggle to disambiguate happy and calm emotions.

Our hope was that adding gestures to accompany the audio would help to
disambiguate emotions. To test that our gestures properly encoded emotion, we
compared predictions for Shimi’s voice accompanied by generated gestures with
predictions accompanied by stochastic gestures, the results of which can also
be seen in figure 4.

While the confusion matrices show a clear prediction improvement in using
generated gestures over stochastic, the results are not statistically
significant. A two-sided T-test provides a p-value of 0.089, which does not
reject the null hypothesis at $\alpha=0.05$. Disambiguities from the audio-
only cases were not mitigated, but the confused emotions changed slightly,
following other gesture and emotion studies [53].

Figure 6: Participants Trust Mean

Some experimental error may have accrued through the mixing of stimuli when
presented to participants. Each stimuli was expected to be independent but
some verbal user feedback expressed otherwise, such as: “the gestures with no
audio seemed to be frequently followed by the same gesture with audio, and it
was much easier to determine emotion with the presence of audio.” The
presentation of stimuli may have led participants to choose an emotion based
on how we ordered stimuli, rather than their perceived emotion of Shimi.

###  V-B Trust

As per the trust scale, a mean percentage for trust was calculated on combined
answers to 40 questions from each participant. A t-test was then run on each
group mean. The average score variation between speech and Shimi audio showed
a significant result (p=0.047), proving the hypothesis. Figure 6 shows the
variation in average scores from all participants. The difference of mean
between groups was 8%. Results from the text entries were positive for the
prosodic voice, and generally neutral or often blank for speech. A common
comment from the participants for the Shimi voice was “Seemed like a
trustworthy friend that I would be fine confiding in.”

Figure 7: Not Significant Trust Results

##  VI Discussion and Future Work

We were able to clearly demonstrate participant recognition of the expected
emotion from Shimi, confirming our first hypothesis. Our model however did not
perform completely as predicted, as audio without gesture lead to the clearest
display of emotion. With a small sample size and a p-value close to being
significant, we were encouraged by qualitative feedback that provided insight
into the shortcomings of the gestures and gave us ideas for future
improvements. For instance, emotions on the same side of the arousal axis were
often hard to disambiguate. One participant noted that “it was generally
difficult to distinguish happy and angry if there were no sounds (similar
situation between sad and calm)”, while another noted “I had some trouble
discerning calm from sad here and there”, and “without speaking, it was
difficult to decipher between anger and excitement”. The general intensity of
the emotion was apparent, however.” Certain movement features led to emotional
connections for the participants, as demonstrated here: “generally, when Shimi
put it’s head down, I was inclined to say it looked sad. When it moved more
violently, particularly by tapping [sic] it’s foot, I was inclined to say it
was angry or happy”, “more forceful movements tended to suggest anger”, and
“When there was more severe motion, I associated that with anger. When the
motion was slower I associated it with sad or calm. If the head was down more
I associated it with sad. And I associated it with happy more when there was
sound and more motion.”

The trust perception scale is designed to give an overall rating, and
independent questions should not necessarily be used to draw conclusions.
However, there were several interesting results indicating further areas of
research. Fig 5 shows all categories with a p value less than 0.10, for which
multiple questions showed significant results with a p value under 0.05).
Shimi’s voice was crafted to be friendly and inviting and as expected received
much higher results for pleasantness and friendliness. Unexpectedly, it also
showed much higher ratings for its perception as being conscious. While
further research is required to confirm the meaning, we believe that the
question on consciousness of Shimi demonstrating a significant result shows
that embodying a robot with a personal prosody (as opposed to human speech)
creates a more believable agent. Figure 7 shows the categories with very
similar distributions of scores. These include Lifelike, A Good Teammate, Have
Errors, Require Maintenance and Openly Communicate. While further research is
needed, this may imply that these features are not primarily associated with
audio. Further work should be done to explore if the same impact can be found
by adjusting audio features of a humanoid robot may also lead to interesting
results.

In other future work we plan to develop experiments with a broader custom
musical data-set across multiple robots. We intend to study emotional
contagion and trust between larger groups of robots across distributed
networks [54], aiming to understand collaboration and trust at a higher level
between multiple robots.

Overall, our trust results were significant and showed that prosody and
gesture can be used to generate higher levels of trust in human-robot
interaction. Our belief that creating a believable agent that avoided uncanny
valley was shown to be correct and was validated through participant comments,
including the open text response: “Shimi seems very personable and expressive,
which helps with trust”.

## References

  * [1] P. A. Hancock, D. R. Billings, K. E. Schaefer, J. Y. Chen, E. J. De Visser, and R. Parasuraman, “A meta-analysis of factors affecting trust in human-robot interaction,” _Human Factors_ , vol. 53, no. 5, pp. 517–527, 2011.
  * [2] M. Mateas, “Artificial intelligence today,” M. J. Wooldridge and M. Veloso, Eds. Berlin, Heidelberg: Springer-Verlag, 1999, ch. An Oz-centric Review of Interactive Drama and Believable Agents, pp. 297–328. [Online]. Available: http://dl.acm.org/citation.cfm?id=1805750.1805762
  * [3] J. Bates, “The role of emotion in believable agents,” _Commun. ACM_ , vol. 37, no. 7, pp. 122–125, Jul. 1994. [Online]. Available: http://doi.acm.org/10.1145/176789.176803
  * [4] M. Bretan, G. Hoffman, and G. Weinberg, “Emotionally expressive dynamic physical behaviors in robots,” _International Journal of Human-Computer Studies_ , vol. 78, pp. 1–16, 2015.
  * [5] R. Adolphs, D. Tranel, and H. Damasio, “Emotion recognition from faces and prosody following temporal lobectomy.” _Neuropsychology_ , vol. 15, no. 3, p. 396, 2001.
  * [6] C. Shan, S. Gong, and P. W. McOwan, “Beyond facial expressions: Learning human emotion from body gestures.” in _BMVC_ , 2007, pp. 1–10.
  * [7] K. F. MacDorman, “Subjective ratings of robot video clips for human likeness, familiarity, and eeriness: An exploration of the uncanny valley,” in _ICCS/CogSci-2006 long symposium: Toward social mechanisms of android science_ , 2006, pp. 26–29.
  * [8] J. Xu and A. Howard, “The impact of first impressions on human- robot trust during problem-solving scenarios,” in _2018 27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN)_ , Aug 2018, pp. 435–441.
  * [9] J. D. Lee and K. A. See, “Trust in automation: Designing for appropriate reliance,” _Human Factors_ , vol. 46, no. 1, pp. 50–80, 2004.
  * [10] P. H. Kim, K. T. Dirks, and C. D. Cooper, “The repair of trust: A dynamic bilateral perspective and multilevel conceptualization,” _Academy of Management Review_ , vol. 34, no. 3, pp. 401–422, 2009.
  * [11] R. E. Miles and W. D. Creed, “Organizational forms and managerial philosophies-a descriptive and analytical review,” _RESEARCH IN ORGANIZATIONAL BEHAVIOR: AN ANNUAL SERIES OF ANALYTICAL ESSAYS AND CRITICAL REVIEWS, VOL 17, 1995_ , vol. 17, pp. 333–372, 1995.
  * [12] K. E. Schaefer, _Measuring Trust in Human Robot Interactions: Development of the “Trust Perception Scale-HRI”_. Boston, MA: Springer US, 2016, pp. 191–218.
  * [13] P. M. Satchell, _Cockpit monitoring and alerting systems_. Routledge, 2016.
  * [14] A. Freedy, E. DeVisser, G. Weltman, and N. Coeyman, “Measurement of trust in human-robot collaboration,” in _Collaborative Technologies and Systems, 2007\. CTS 2007. International Symposium on_. IEEE, 2007, pp. 106–114.
  * [15] D. M. Rousseau, S. B. Sitkin, R. S. Burt, and C. Camerer, “Not so different after all: A cross-discipline view of trust,” _Academy of management review_ , vol. 23, no. 3, pp. 393–404, 1998.
  * [16] T. Gompei and H. Umemuro, “Factors and development of cognitive and affective trust on social robots,” in _International Conference on Social Robotics_. Springer, 2018, pp. 45–54.
  * [17] S. G. Barsade, “The ripple effect: Emotional contagion and its influence on group behavior,” _Administrative science quarterly_ , vol. 47, no. 4, pp. 644–675, 2002.
  * [18] Y. T. Wang, J. Han, X. Q. Jiang, J. Zou, and H. Zhao, “Study of speech emotion recognition based on prosodic parameters and facial expression features,” in _Applied Mechanics and Materials_ , vol. 241. Trans Tech Publ, 2013, pp. 1677–1681.
  * [19] J. Crumpton and C. L. Bethel, “A survey of using vocal prosody to convey emotion in robot speech,” _International Journal of Social Robotics_ , vol. 8, no. 2, pp. 271–285, 2016.
  * [20] A. Wennerstrom, _The music of everyday speech: Prosody and discourse analysis_. Oxford University Press, 2001\.
  * [21] J. Sloboda, “Music: Where cognition and emotion meet,” in _Conference Proceedings: Opening the Umbrella; an Encompassing View of Music Education; Australian Society for Music Education, XII National Conference, University of Sydney, NSW, Australia, 09-13 July 1999_. Australian Society for Music Education, 1999, p. 175.
  * [22] L. Devillers, L. Vidrascu, and L. Lamel, “Challenges in real-life emotion annotation and machine learning based detection,” _Neural Networks_ , vol. 18, no. 4, pp. 407–422, 2005.
  * [23] J. A. Russell, “Emotion, core affect, and psychological construction,” _Cognition and Emotion_ , vol. 23, no. 7, pp. 1259–1283, 2009.
  * [24] A. Mehrabian, “Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament,” _Current Psychology_ , vol. 14, no. 4, pp. 261–292, 1996.
  * [25] C. Breazeal and L. Aryananda, “Recognition of affective communicative intent in robot-directed speech,” _Autonomous robots_ , vol. 12, no. 1, pp. 83–104, 2002.
  * [26] M. Inderbitzin, A. Väljamäe, J. M. B. Calvo, P. F. M. J. Verschure, and U. Bernardet, “Expression of emotional states during locomotion based on canonical parameters,” in _Ninth IEEE International Conference on Automatic Face and Gesture Recognition (FG 2011), Santa Barbara, CA, USA, 21-25 March 2011_. IEEE, 2011, pp. 809–814.
  * [27] H. G. Walbott, “Bodily expression of emotion,” _European Journal of Social Psychology_ , vol. 28, no. 6, pp. 879 – 896, 1998.
  * [28] B. de Gelder, “Towards the neurobiology of emotional body language,” _Nature Reviews Neuroscience_ , vol. 7, pp. 242–249, March 2006.
  * [29] R. W. Picard, “Affective computing,” 1995.
  * [30] M. Scheutz, P. Schermerhorn, and J. Kramer, “The utility of affect expression in natural language interactions in joint human-robot tasks,” in _Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction_. ACM, 2006, pp. 226–233.
  * [31] B. Sievers, L. Polansky, M. Casey, and T. Wheatley, “Music and movement share a dynamic structure that supports universal expressions of emotion,” _Proceedings of the National Academy of Sciences_ , vol. 110, no. 1, pp. 70–75, jan 2013. [Online]. Available: http://www.pnas.org/content/110/1/70
  * [32] G. Weinberg, A. Beck, and M. Godfrey, “Zoozbeat: a gesture-based mobile music studio.” in _NIME_ , 2009, pp. 312–315.
  * [33] G. Weinberg, S. Driscoll, and T. Thatcher, “Jam’aa: a percussion ensemble for human and robotic players,” in _ACM International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 2006), ACM Boston, MA_ , 2006.
  * [34] L. D. Riek, T.-C. Rabinowitch, P. Bremner, A. G. Pipe, M. Fraser, and P. Robinson, “Cooperative gestures: Effective signaling for humanoid robots,” in _Human-Robot Interaction (HRI), 2010 5th ACM/IEEE International Conference on_. IEEE, 2010, pp. 61–68.
  * [35] A. Moon, C. A. Parker, E. A. Croft, and H. Van der Loos, “Design and impact of hesitation gestures during human-robot resource conflicts,” _Journal of Human-Robot Interaction_ , vol. 2, no. 3, pp. 18–40, 2013.
  * [36] H. Kozima and H. Yano, “In search of otogenetic prerequisites for embodied social intelligence,” in _Proceedings of the Workshop on Emergence and Development on Embodied Cognition; International Conference on Cognitive Science_ , 2001, pp. 30–34.
  * [37] G. Castellano, I. Leite, A. Pereira, C. Martinho, A. Paiva, and P. W. McOwan, “Affect recognition for interactive companions: challenges and design in real world scenarios,” _Journal on Multimodal User Interfaces_ , vol. 3, no. 1, pp. 89–98, 2010.
  * [38] M. W. Alibali, S. Kita, and A. J. Young, “Gesture and the process of speech production: We think, therefore we gesture,” _Language and cognitive processes_ , vol. 15, no. 6, pp. 593–613, 2000.
  * [39] S. R. Livingstone and F. A. Russo, “The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English,” _PLOS ONE_ , vol. 13, no. 5, pp. 1–35, 2018. [Online]. Available: https://doi.org/10.1371/journal.pone.0196391
  * [40] Y. Jadoul, B. Thompson, and B. de Boer, “Introducing parselmouth: A python interface to praat,” _Journal of Phonetics_ , vol. 91, pp. 1–15, 11 2018\.
  * [41] P. Boersma and D. Weenink, “Praat: Doing phonetics by computer (version 5.3.51),” 01 2007.
  * [42] R. Savery and G. Weinberg, “Shimon the robot film composer and deepscore,” _Proceedings of Computer Simulation of Musical Creativity_ , p. 5, 2018.
  * [43] M. Bretan, G. Weinberg, and L. Heck, “A unit selection methodology for music generation using deep neural networks,” _arXiv preprint arXiv:1612.03789_ , 2016.
  * [44] R. J. Savery, “An interactive algorithmic music system for edm,” _Dancecult: Journal of Electronic Dance Music Culture_ , vol. 10, no. 1, 2018\.
  * [45] ——, “Algorithmic improvisers,” in _UC Irvine Electronic Theses and Dissertations_ , 2015.
  * [46] R. Savery, M. Ayyagari, K. May, and B. N. Walker, “Soccer sonification: Enhancing viewer experience.” Georgia Institute of Technology, 2019.
  * [47] R. Savery, R. Rose, and G. Weinberg, “Finding Shimi’s voice: fostering human-robot communication with music and a NVIDIA Jetson TX2,” _Proceedings of the 17th Linux Audio Conference_ , p. 5, 2019.
  * [48] D. McNeill, _How language began: Gesture and speech in human evolution_. Cambridge University Press, 2012.
  * [49] P. Toiviainen, G. Luck, and M. R. Thompson, “Embodied Meter: Hierarchical Eigenmodes in Music-Induced Movement,” _Music Perception: An Interdisciplinary Journal_ , vol. 28, no. 1, pp. 59–70, sep 2010. [Online]. Available: http://mp.ucpress.edu/content/28/1/59
  * [50] B. Burger, S. Saarikallio, G. Luck, M. R. Thompson, and P. Toiviainen, “Relationships Between Perceived Emotions in Music and Music-induced Movement,” _Music Perception: An Interdisciplinary Journal_ , vol. 30, no. 5, pp. 517–533, Jun. 2013. [Online]. Available: http://mp.ucpress.edu/cgi/doi/10.1525/mp.2013.30.5.517
  * [51] C. Raffel and D. P. W. Ellis, “Intuitive analysis, creation and manipulation of midi data with pretty_midi,” in _Proceedings of the 15th International Conference on Music Information Retrieval Late Breaking and Demo Papers_ , 2014.
  * [52] S. M. Mohammad and F. Bravo-Marquez, “WASSA-2017 shared task on emotion intensity,” in _Proceedings of the Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA)_ , Copenhagen, Denmark, 2017.
  * [53] A. Lim, T. Ogata, and H. G. Okuno, “Towards expressive musical robots: a cross-modal framework for emotional gesture, voice and music,” _EURASIP Journal on Audio, Speech, and Music Processing_ , vol. 2012, no. 1, p. 3, Jan. 2012. [Online]. Available: https://doi.org/10.1186/1687-4722-2012-3
  * [54] G. Weinberg, “Expressive digital musical instruments for children,” Ph.D. dissertation, Massachusetts Institute of Technology, 1999.