]Received September 20, 2022

# Molecular-Beam Spectroscopy with an Infinite Interferometer: Spectroscopic
Resolution and Accuracy

Thomas Schultz<EMAIL_ADDRESS>In Heo  Jong Chan Lee  Begüm Rukiye Özer
Department of Chemistry, Ulsan National Institute of Science and Technology
(UNIST), 44919, Korea

([)

###### Abstract

An interferometer with effectively infinite maximum optical path difference
removes the dominant resolution limitation for interferometric spectroscopy.
We present mass-correlated rotational Raman spectra that represent the world’s
highest resolution scanned interferometric data and discuss the current and
expected future limitations in achievable spectroscopic performance.

Rotational Coherence Spectroscopy, Fourier-Transform Spectroscopy

##  I Introduction

The resolution of scanned interferometric spectroscopy is usually limited by
the maximum optical path difference (MOPD) of the interferometer. As part of
our development of mass-correlated rotational alignment spectroscopy
(CRASY),[1] we constructed an interferometer with an effectively infinite
MOPD,[2] utilizing the discrete pulse train emitted from a femtosecond laser
oscillator. Here we present the highest resolution data measured to-date and
discuss the resolution-limiting factors in our molecular beam experiments.
Discussed concepts are equally applicable to other types of interferometric or
time-domain spectroscopy that rely on the scanning of a spatial path
difference or a temporal delay range.

CRASY is a type of rotational coherence spectroscopy (RCS)[3] and measures
rotational Raman spectra in the time-domain by scanning the path difference
between two interferometer arms. The resulting data is useful for the
structural characterization of non-dipolar molecules[1, 4, 5, 6] that are
inaccessible to Fourier-transform microwave Spectroscopy (FTMW).[7] By
extending the scanned interferometer length, our work obtained order-of-
magnitude improved resolution[2, 8] as compared to preceding RCS experiments
or Fourier-transform infrared spectroscopy (FTIR).[9]

The energy resolution ($\Delta E$) of spectroscopic experiments is
fundamentally limited by the observation time $\Delta t$, i.e., the time over
which the investigating particles (usually photons) and investigated molecules
interact. The time-frequency formulation of Heisenberg’s uncertainty principle
states this fundamental resolution limit as $\Delta E\cdot\Delta t=\hbar/2$.
Independent of the experiment, the effective observation time is always
limited by the coherence time (or lifetime) of the observed quantum states.
This leads to lifetime-broadening of observed spectral lines, either due to an
intrinsically limited lifetime of the observed states or due to interactions
with an environment, e.g., through molecular collisions. When lifetime
broadening is small, the effective resolution will be limited by the
constraints of the spectroscopic experiment.

In frequency domain spectroscopy, the coherence length of the interacting
photons limits the effective observation time. The situation is fundamentally
different in interferometric Fourier-transform spectroscopy, where the
observation time is limited by the MOPD of the interferometer. The non-
apodized full-width-at-half-maximum (FWHM) resolution limit in FTIR is given
as $\Delta\widetilde{\nu}^{FWHM}=0.61\cdot\mathrm{d^{-1}_{MOPD}}$.[9] The
highest-resolution interferometer described in the literature features a MOPD
of 11.7 m,[10] which corresponds to a delay range of $t_{\textrm{max}}=39$ ns
and a non-apodized resolution limit of 15.6 MHz FWHM.

RCS is based on the impulsive excitation and probing of rotational coherence
with ultrafast (femtosecond or picosecond) laser pulses. In our CRASY variant
of RCS, the rotationally excited molecules are probed by resonant multi-photon
photoionization and rotational coherence is observed as interferometric signal
modulation in resulting ion signals. CRASY therefore correlates rotational
spectra with observed ion masses and thereby facilitates the assignment of
signals in heterogeneous samples. As in FTIR, or other scanned interferometric
spectroscopies, RCS experiments scan the optical path difference in an
interferometer and shows a resolution limited by the MOPD.

##  II Infinite Interferometer Design for CRASY

To obtain mass-CRASY data, ion signals were detected in a time-of-flight mass
spectrometer as function of the scanned delay between alignment and ionization
laser pulses. The experimental details for mass-CRASY measurements were
described previously [4, 2, 8, 11, 12] and here we focus on the interferometer
design used to scan extended optical path differences. Focussed alignment
pulses with 800 nm wavelength, $\leq$2 ps pulse duration and 100 $\mu$J-level
pulse power created a coherent rotational wavepacket by impulsive Raman
excitation. Alignment, in this context, denotes the transient molecular
alignment that is commonly observed upon excitation of a coherent
wavepacket.[13] Ionization pulses with 200 nm or 266 nm wavelength, 45 fs
pulse duration and few-$\mu$J pulse power photoionized molecules by two-photon
resonant photoionization. Ion signals showed temporal signal modulations due
to the interference of the coherent rotational states in the probe step, as
depicted in Fig 1 (A) and (B). Fourier-transformation of these signal
modulations reveal the spectrum of the coherent wavepacket, as shown in Fig. 1
(C).

Figure 1: Mass-CRASY data from a 50 ns delay scan of a sample containing
benzene (mass 78 u, blue), perdeuterated benzene (mass 84 u, green), carbon
disulfide (mass 76 u, orange), and naturally occurring heavy isotopologues
(darker colors). (A) Delay dependent ion signals show significant signal
modulation due to interference of coherently excited rotational states. (B) A
section of the signal modulation trace for mass 84 u. (C) Rotational Raman
spectrum obtained by Fourier-transformation of the trace shown in (B).

The interferometer for high-resolution spectroscopy should have the longest
possible MOPD, combined with a small step size and high positioning accuracy.
As described above, the achievable spectroscopic resolution is directly
proportional to the MOPD and, as described by the Shannon-Nyquist theorem,[14]
the spectroscopic range $\nu_{\mathrm{max}}$ is inversely proportional to the
sampling step size ($\nu_{\mathrm{max}}=1/(2\cdot t_{step\>size}$)). CRASY is
performed on cold molecular beams with beam temperatures below 10 K and a 0.5
ps to 5 ps steps size (maximum spectroscopic range of $\nu_{\mathrm{max}}=$
0.1 THz to 1 THz) is sufficient to resolve the complete thermally occupied set
of rotational states. The positioning accuracy should remain well-below the
scanned step size to avoid a degradation of the spectroscopic resolution.

Interferometers used for FTIR, RCS, and other types of scanned interferometer
spectroscopy are based on opto-mechanical delay stages, i.e., moving mirrors
in one interferometer arm to change the optical path length. The largest
interferometers can be found at national synchrotron facilities but are not
practical within the restricted space and budget of University-based
laboratory research. Instead, our interferometer combined electronic and opto-
mechanical delays to achieve an infinite effective delay range within a
compact and affordable interferometer design.

A schematic representation the infinite interferometer is shown in Fig. 2. The
mechanical delay was based on a 30-cm Physik Instrumente, MD-531 motorized
stage. The optical beam path across the stage was 16-times folded to obtain a
MOPD of 4.8 m (16 ns). Longer delays were achieved by electronic pulse
selection of oscillator pulses that were amplified in two separate
regenerative Ti:Sa amplifiers, forming the two arms of the interferometer. The
repetition rate of the laser oscillator (Coherent Vitara) was a 80 MHz and
selection of subsequent pulses added discrete 12.5 ns delays in the second
interferometer arm. Note that the timing accuracy of the electronic delays is
governed by the stability of the oscillator repetition rate and not the
accuracy of the electronic delay generator.

Figure 2: Schematic depiction of the infinite interferometer design. Pulses
from a femtosecond laser oscillator are split and recombined with beam
splitters (BS) and amplified in two separate amplifiers. Electronic selection
of different oscillator pulses for amplifier 1 and 2 introduced discrete pulse
delays in multiples of 12.5 ns. An opto-mechanical delay stage added
additional delays of 0–16 ns with picosecond step size and femtosecond
accuracy.

The pulse selection delay was controlled via an electronic delay generator
(SRS-DG535) and allowed to extend the delay range to quasi-arbitrary values.
The amplifiers were operating at 1-kHz repetition rate and probe pulses
delayed by more than 1 ms therefore arrive after a subsequent pump pulse. The
molecular beam velocity in our experiments is in the range of 1000 m/s and
molecules travel meter distances within milliseconds. Experiments with $>1$ ms
delays can therefore rely on spatial discrimination of the excited molecules.
Therefore, for all practical purposes, our set-up represents an interferometer
with infinite MOPD and the achievable spectroscopic resolution is no longer
limited by the size of the interferometer, but by other experimental
limitations, such as the ability to track the molecular beam.

##  III High-Resolution CRASY Data

Fig. 3 shows rotational Raman spectra for CS2 and illustrates the progress
achieved by combining electronic and opto-mechanical delays. The maximal opto-
mechanical delay range of 16 ns was sufficient to obtain an effective
resolution of 60 MHz FWHM, as shown in Fig. 3 (Top). The effective resolution
remained below the non-apodized resolution limit of 38 MHz because the
mechanical delay stage was not perfectly flat, leading to a loss of signal
when the stage approached either end of the delay range. A combined opto-
mechanical and electronic delay range of 16.7 m (50 ns) gave a greatly
enhanced effective resolution of 17.5 MHz FWHM, near the resolution limit of
16.7 MHz, as shown in Fig. 3 (Bottom). The sample used for the latter
measurement contained only trace amounts of CS2 and the spectrum therefore had
a lower signal-to-noise ratio (SNR).

Figure 3: Rotational Raman spectra for CS2 from CRASY data-sets. (Top)
Spectrum obtained by scanning a 16 ns delay range (4.8 m MOPD) with an opto-
mechanical delay stage. (Bottom) Spectrum obtained by scanning a 56 ns delay
range (equivalent to 16.7 m MOPD), using combined opto-mechanical and
electronic delays. Enlarged insets reveal the effective resolution for
selected transition lines.

The measurement time required to scan large delays scales linearly with the
scan range and the collection of mass spectra for a large number of alignment-
ionization delays was time-consuming and created exceedingly large data-sets.
The data shown in Fig. 3 (Bottom) was obtained with delay scan range of 50 ns
and a 2 ps step-size and therefore required the accumulation of 25 000 mass
spectra. Data was acquired with 500 Hz repetition rate and ion signals for
1000 laser shots were accumulated for each mass spectrum. The resulting
measurement time was almost 14 hours. Each time-of-flight mass spectrum
contained 400 000 points and the acquired raw data quantity corresponds to 10
Gb. It is readily apparent that the brute-force extension of the scanned delay
range will lead to impractical requirements in terms of measurement time and
data storage.

We reduced the data quantity by lossless compression and the use of sparse
data formats. Because mass spectroscopic data is highly discrete, we routinely
achieved $>100$-fold in-memory compression with zlib compression algorithms.
Fourier transform analysis is only possible on decompressed data, but
downsampling of the mass axis and the conversion into sparse data formats
facilitated the signal analysis.

Random sparse sampling was used to accelerate long delay scans, i.e., data was
only measured for a randomly selected sub-set of delay points along an
extended time axis. Different sparse sampling strategies were explored in the
field of multi-dimensional NMR experiments,[15, 16] and are discussed in more
detail, below. Fig. 4 compares spectra from a fully-sampled and a sparsely-
sampled measurement, with the latter collecting mass spectra only for 5.5% of
delays along an extended delay axis. The sparsely sampled data was acquired
2.5-times faster than the fully sampled data and improved the spectroscopic
resolution by a factor 20. Sparse sampling added noise to the spectra, as
readily apparent in the logarithmic representation depicted in Fig. 4.

Figure 4: Rotational Raman spectra for CS2 obtained with continuous 1-ps
sampling of a 15 ns delay (Top) and random sparse sampling of 17 000 mass
spectra along a 312 ns delay axis (5.5% sampling rate, Bottom). Note the
logarithmic scale of the ordinate.

The highest resolution spectrum measured to-date with the CRASY technique was
based on a 10 $\mu$s scan of a benzene sample, containing residual CS2 in
small concentration. Due to the limited size of our spectrometer window,
tracking of the molecular beam was only achieved for a delay range $<3$
$\mu$s, reducing the achieved signal contrast and resolution. Fig. 5 shows
signal for the CS2 mass channel, with an effective resolution of 330 kHz FWHM.
This data represents the highest-resolution Fourier-transform interferometric
spectrum in the world, representing a 50-fold improvement over the highest-
resolution FTIR data in the literature[9, 17, 10]. To comprehend the scale of
this improvement, we invite the reader to visualize the 11.7 m MOPD
interferometer used for the latter experiments (see Ref. 18 for a photographic
image) versus the km-scale MOPD achieved in our laboratory experiment.

Figure 5: Highest resolution rotational Raman spectrum obtained with the mass-
CRASY technique. The inset, with 30 000-fold enlarged abscissa, reveals the
330 kHz FWHM effective resolution for the J=6–8 transition in CS2.

Table 1 compares the resolution limit of various spectroscopic techniques that
are used to characterize rotational spectra at high resolution. CRASY
currently represents the highest-resolution method for rotational Raman
spectroscopy and , more generally, for the investigation of non-dipolar
molecules. Modern FTMW experiments reach a significantly higher
resolution,[19] but can only be performed for dipolar species and only cover a
spectral range of tens of GHz, more than one order-of-magnitude below the
spectral coverage obtained with CRASY. In terms of resolving power
($\frac{\mathrm{spectral\>range}}{\mathrm{resolution}}$), CRASY is at parity
with state-of-the-art FTMW experiments.

Table 1: Resolution for common types of rotationally resolved spectroscopies.

Spectroscopic method | Resolution limit
---|---
Raman, single-mode laser(a) | 1500 MHz [20]
Raman, FTIR(a) | 300 MHz [20]
Raman, RCS(b) | 150 MHz [3, 20]
Raman, low resolution CRASY(a) | 39 MHz [8]
Coherent anti-Stokes Raman(a) | 30 MHz [20]
FTIR(b) | 16 MHz [9]
Raman, high resolution CRASY(a) | 330 kHz (c)
FTMW(a) | few kHz [7, 19]

(a)Achieved effective resolution. (b)Theoretical resolution limit. (c)This
work.

Table 1 omitted frequency comb measurements[21, 22]and related techniques.
Highest-resolution frequency comb measurement cover only very small spectral
ranges and a direct comparison is therefore not meaningful. Dual-comb or
direct comb spectroscopy (DCS)[23, 24, 25] allow the rapid, broad-band, and
high-resolution characterization of molecular spectra. We omit DCS from our
table because it requires extended interaction times with significant
molecular sample densities and, to our knowledge, the resolution of all broad-
band DCS spectra is subject to significant broadening. A comparison with
estimated DCS resolution limits is therefore not meaningful because the
achieved effective resolution remains far below the theoretical resolution
limit.

##  IV Limits to Spectroscopic Resolution and Accuracy

The use of an infinite interferometer removes resolution limits due to the
MOPD. We must therefore consider other factors that limit the resolution or
accuracy of interferometric measurements. Three distinct types of
uncertainties must be considered: (i) Uncertainties in delay positions accrued
over the length of the opto-mechanical delay line. (ii) Uncertainties in the
laser oscillator repetition rate, which affect the accuracy of the discrete
12.5 ns pulse-selection delays. (iii) Uncertainties due to Doppler shifts and
Doppler broadening. In the following, we discuss each error source separately.

###  IV.1 Uncertainties in Opto-Mechanical Delays

The MD-531 motorized stage in our interferometer contains a 100 nm internal
encoder. The encoder is mounted on an aluminum rod with correspondingly large
thermal instabilities,111The thermal expansion coefficient of aluminum at 25∘C
is $1.1\cdot 10^{-5}\frac{m}{mK}$ [36]. and the calibration of known spectra
revealed relative positioning errors up to $\Delta r/r=10^{-4}$.

We addressed the problems with the internal stage encoder by mounting an
external optical encoder (Sony Laserscale BL57-RE) with a thermal expansion
coefficient of $0.7\cdot 10^{-6}$ m/(m$\cdot$K). Comparison of internal and
external encoder positions revealed a very linear error for the internal
encoder. The internal encoder was then calibrated over a range of 12.5 ns
against the oscillator repetition rate, by measuring laser cross-correlation
signals displaced by one oscillator pulse jump. With a typical $\pm
0.2\,^{\circ}$C temperature stability on our laser table, we found that the
internal encoder was sufficient to confine positioning uncertainties to
$\Delta r/r<3\cdot 10^{-6}$ ($<40$ fs uncertainty across the stage). Higher
accuracy was available by monitoring the external encoder with resulting
positioning uncertainties below $\Delta r/r<2\cdot 10^{-7}$ ($\approx$3 fs
uncertainty across the stage).

Additional uncertainties arose due to the variation of the air refractive
index $n$ with air pressure, temperature, and composition. For 800 nm light,
the air refractive index changes by $\Delta n/n\approx 10^{-7}$ for a 40 Pa
change in air pressure, a 0.1 ∘C change in temperature or a 10% change in air
humidity.[27, 28] These uncertainties were readily suppressed by the
continuous measurement of, and correction for, changes in air temperature,
pressure, and humidity. The NIST shop-floor equation[27]222Air index of
refraction $n$ based on pressure $P$ (kPa), temperature $T$ (∘C), and relative
humidity $RH$ (%): $n=1+7.86\cdot 10^{-4}\cdot P/(273+T)-1.5\cdot
10^{-11}\cdot RH(T^{2}+160)$ was sufficient to approximate $n$ with a relative
uncertainty of $\Delta n/n<10^{-7}$.

Uncertainties accrued over the range of the opto-mechanical delay line are
reset with each electronic pulse selection delay jump, when the mechanical
delay line is re-set to its initial position. The impact of opto-mechanical
delay uncertainties therefore scales inversely to the number $N$ of electronic
delay jumps and becomes small for large $N$. It is therefore sufficient to
suppress relative stage positioning errors into a regime where the accrued
phase shift for the highest measured frequencies becomes negligible. Stage
errors in the $\Delta r/r=$$10^{-6}$ regime correspond to a 12.5 fs phase
shift across the mechanically scanned 12.5 ns delay range and were negligible
for any feasible experiment with our minimal 50 fs laser pulse duration
(impulsive Raman excitation possible for $<10$ THz transition frequencies). We
should note that when significant calibration errors between the mechanical
stage and the oscillator repetition rate occurred, this led to the formation
of 80 MHz sidebands that were readily identified in the experimental spectra.

###  IV.2 Uncertainties in the Oscillator Repetition Rate

Extended delays are achieved by delayed electronic pulse-selection of
subsequent oscillator pulses from a Coherent Vitara laser oscillator. This
adds delays in multiples of 12.5 ns to the probe arm of interferometer. Any
undetected drift of the laser oscillator repetition rate from the nominal
value of 80 MHz will introduce a corresponding uncertainty. We used a
frequency counter (Aim-TTI TF930) to monitor the stability of the oscillator
against a GPS-stabilized clock (Leo Bodnar GPSDO) with an expected frequency
accuracy of $\Delta\nu/\nu=$$\leq 10^{-10}$.

Figs. 6 and 7 show the Allan deviation and the modified Allan deviation for
the oscillator frequency, measured over a 1-day period. For periods $<100$ s,
the observed slope of -1 (-1.5) in the Allan (modified Allan) deviation is
characteristic for random white noise.[30] This noise is due to the frequency
counter digitization noise ($\pm 1$ count over the measurement period) and
does not reflect any drift of the clock or oscillator. For periods $>100$ s,
the Allan deviation remained $<10^{-10}$, giving an upper limit for the
frequency stability of the oscillator. It is quite possible that the GPS clock
stability was limiting in this regime and that the oscillator frequency was
more stable than our measurement indicates.

Figure 6: Allan deviation for the 80 MHz Coherent Vitara-T laser oscillator,
measured against a GPS-disciplined clock.  Figure 7: Modified Allan deviation
for the 80 MHz Coherent Vitara-T laser oscillator, measured against a GPS-
disciplined clock.

Slow frequency drifts of the oscillator are readily corrected by a
corresponding adjustment of the opto-mechanical delay position. A continuous
monitoring of the oscillator rate therefor allows a _feed-forward_ correction
of delays, which fulfills the same purpose as the feed-back oscillator
stabilization in frequency comb spectroscopy,[21] albeit with much smaller
technical efforts. With our inexpensive monitoring system and a typical
frequency counter period of 1 or 10 seconds, we readily achieved a single-
sigma uncertainty (Allan deviation) $\Delta\nu/\nu\ll 10^{-7}$ and a longer
frequency counter integration period can reduce this uncertainty towards the
$\Delta\nu/\nu\approx 10^{-10}$ noise floor of the frequency measurement. We
expect that the uncertainty can be further reduced with the use of a high-
fidelity reference clock: similar oscillators were used in frequency comb
experiments and were stabilized to order-of-magnitude better performance. The
feedback stabilization required for the frequency-comb measurements introduces
an additional source of noise that is absent in our feed-forward stabilization
scheme.

### Uncertainties Arising from Doppler Effects

Textbooks commonly label skimmed molecular beam spectroscopy as ”Doppler-
free”. But as illustrated in Figs. 8 and 9, the non-zero collimation angle and
imperfect alignment of a molecular beam contributes some Doppler effects. We
separately assessed Doppler broadening and Doppler shifts by geometric
consideration of the molecular beam velocity components $v_{\parallel}$ in the
direction of the alignment and ionization laser beams.

Figure 8: Illustration of Doppler broadening. The non-zero molecular beam
collimation angle $\epsilon$ leads to a distribution of beam velocity
components $v_{\parallel}$ that are parallel to the laser beam propagation
axis: $v_{\parallel}=v\mathrm{{}_{beam}\cdot tan}(\epsilon)$.

In our experiments, a 1 mm skimmer at 280 mm distance from the pulsed valve
led to a molecular beam collimation angle of $\epsilon=0.10^{\circ}$. We
measured the molecular beam velocity for a helium-seeded beam of CS2 to be
$v_{\mathrm{beam}}\approx 1100$ m/s and calculated a velocity spread
$v_{\parallel}=\pm 1.96$ m/s and a maximal Doppler broadening of
$\Delta\nu_{\textit{Db}}/\nu=v_{\parallel}/c=6.6\cdot 10^{-9}$. This estimated
Doppler broadening was significantly larger than a value obtained based on the
textbook treatment (see Chapter 4 in Demtröder [31]) because we accounted for
the similar velocity profiles of the heavier CS2 molecules and the lighter
helium atoms in the seeded molecular beam.

Doppler broadening becomes relevant when it approaches or exceeds the
spectroscopic resolution. In our best CRASY data, we observe sub-MHz line-
width for 100 GHz line frequencies ($\Delta\nu_{\mathrm{FWHM}}/\nu$ in the
$10^{-6}$ regime). The Doppler broadening estimated above is several orders-
of-magnitude smaller than our best achieved resolution and will not affect
spectroscopic results until we reach sub-kHz level resolution. The use of
slower molecular beams and better molecular beam collimation can further
reduce Doppler broadening and we expect that sub-100 Hz line widths can be
observed for 100 GHz lines before Doppler broadening becomes a limiting
factor.

A Doppler shift occurs if the angle between laser and molecular beam deviates
from $\alpha=90^{\circ}$. As illustrated in Fig. 9, tracking of the molecular
beam then changes the effective path length of the alignment versus the
ionization arm of the interferometer and introduces an additional delay of
$\delta t=\frac{\Delta x}{c}=\Delta
t\left(1+\frac{v_{\mathrm{beam}}}{c}\cdot\mathrm{sin}(\alpha)\right)$.
Resulting Fourier transformed spectra show a frequency shift proportional to
the delay time errors $\delta t/\Delta t$. The line position for well-resolved
lines can be determined with an accuracy that is orders-of-magnitude better
than the spectroscopic resolution and our experiment is therefore highly
sensitive to Doppler shifts.

To measure the angle $\alpha$ between laser and molecular beam, we propagated
a laser pointer through the skimmer onto the pulsed valve orifice and measured
the relative angle of laser pointer beam and alignment / ionization laser
beams against a reference frame. For our experiments, we determined an angle
of $\alpha=91.6\pm 0.4^{\circ}$ and calculated a Doppler correction factor of
$\delta t/\Delta t_{m}=(1.0\pm 0.26)\cdot 10^{-7}$. The Doppler shift was not
negligible for our most-accurate measurements and reduced measured rotational
frequencies by one part in $10^{7}$ (some 320 Hz for a 3.2 GHz rotational
constant fitted for CS2).[2] The Doppler shift uncertainty of $2.6\cdot
10^{-8}$ can be reduced by a careful measurement of the relative angles
between the molecular beam and the laser beams. The Doppler shift can be
measured, and thereby completely eliminated, by performing complementary
experiments with laser beams propagating in opposite directions, as
measurements from opposing directions show opposite signs for the Doppler
shift.

Figure 9: Illustration of the Doppler shift: To correct for molecular beam
propagation within the delay $\Delta t$ between alignment and ionization
pulses, the ionization laser is tracking the molecular beam for a distance of
$d=\Delta t\cdot v_{\mathrm{beam}}$. When the angle between molecular beam and
laser beams deviates from $90^{\circ}$, this leads to an additional path
$\delta x$ for the ionization beam and an additional delay $\delta t=\delta
x/c$.

### Signal Degradation by Sparse Sampling

Sparse sampling is an essential tool to extend the optical path difference and
thereby the spectroscopic resolution without excessive requirements in terms
of measurement time and data storage. A number of sparse sampling approaches
were discussed in the context of multi-dimensional NMR spectroscopy.[16, 32,
33, 15] Sparse sampling methods other than random sampling affect the line
shape and are therefore problematic, unless the natural line shape in the
investigated spectra is known or negligible. Randomly sparse sampled data
merely shows an elevated noise level and correspondingly reduced signal-to-
noise ratio, without introducing any significant artifacts.333Random sparse
sampling is equivalent to the multiplication of a continuous time-domain trace
with a binary [0,1] masking array, which masks out the unmeasured data points.
The multiplication of traces in the time domain corresponds to a folding of
their spectra in the Fourier-domain. A random binary array transforms into a
flat noise spectrum and therefore merely adds noise without otherwise
affecting the measured spectrum. The combination of sparse sampling with an
infinite interferometer therefore offers a unique spectroscopic tool, where
resolution and SNR can be freely traded against one another.

Fig. 10 shows the effect of random sparse sampling in a simulated delay trace
for CS2. The Fourier transform of a fully sampled trace showed negligible
noise due to the synthetic nature of the data. To simulate 3% random sparse
sampling, 97% of all points in the delay trace were set to zero. The random
selection was based on the Mersenne-Twister pseudo-random number generator as
implemented in the numpy library of the Python programming language. The
Fourier transform of the sparsely sampled data showed significant noise but
only a modest degradation of the resolution. An estimate of the noise
distribution, using the modified Z-score,[35] gave a noise level of $\sigma$ =
0.2% relative to the largest signal peak. Experimentally observed SNRs in
sparsely sampled data showed similar signal degradation. Note that the loss of
information scales proportionally to $\sqrt{\mathrm{samples}}$ and longer
scans can be performed with lower sparse sampling rates.

Figure 10: Simulated Fourier-transformed rotational Raman power spectrum for
CS2 at 8 K. (a) Full sampling of 100 ns time delay with $\Delta t$ = 1 ps
steps ($10^{5}$ samples). (b) Sparse sampling of 3000 randomly selected delays
from (a) created $<1$% sampling noise. Insets show a 50-fold or 2500-fold
enlarged abscissa.

##  V Expected Resolution Limits for CRASY Experiments

For all practical purposes, the coherence lifetime of cold rotational states
in small molecules is only limited by collisions. Rotational decoherence in
collision-free, pulsed molecular beams therefore only occurs when the
molecules hit the spectrometer wall. The resolution limit of CRASY is
therefore purely a function of the MOPD. With the infinite interferometer
design presented above, we removed all practical limitations to the MOPD and
other experimental factors become limiting: (i) The molecular beam travels
with supersonic velocities and must be accurately tracked. (ii) Sparse
sampling is necessary to achieve large MOPDs within reasonable measurement
times, but may degrade the SNR to a point where spectra can no longer be
resolved.

The resolution of current CRASY measurements is limited by factor (i): due to
a limited window size we can track the molecular beam only over distances of a
few mm. The beam velocity for our dilute, helium-seeded molecular beams was
measured to be $v_{b}\approx 1100$ m/s and the 330 kHz resolution data shown
in Fig. 5 therefore required tracking of the beam over a distance of nearly 2
mm. The molecular beam velocity can be significantly reduced by using a
heavier seed gas with lower speed-of-sound. With a suitably larger laser
window, the tracking distance can be extended, e.g., an extension to 10 cm
tracking, would offer a 50-fold increase of the accessible MOPD. We expect
that the combination of longer tracking and slower molecular beams will push
the resolution limit into the single-kHz regime. Further extensions would
require the construction of a spectrometer with a dedicated chamber for
decimeter- or meter-scale tracking of the molecular beam, e.g., as depicted in
Fig. 11. Note that the signal collections might be facilitated with the
correlation to other spectroscopic observables. E.g., probing of rotational
coherence via fluorescence excitation would remove the nonlinearity of our
two-photon photoionization probe step and might allow the multiplexed
detection of signals along the molecular beam axis.

Figure 11: Schematic representation of a photoion-photoelectron spectrometer
with extended optical access for molecular beam tracking.

The sampling of a time axis containing some $10^{8}$ points could allow to
collect spectroscopic data with single-kHz resolution and a 100 GHz spectral
range. Clearly, this is only possible with severe sparse sampling and a
corresponding degradation of the SNR. Fig. 12 shows that such measurements are
feasible: a simulated spectrum based on the sampling of 100 000 points along a
2 ms delay range combines excellent signal-to-noise ratio with sub-kHz
resolution.

Figure 12: Simulation of a CS2 spectrum with 0.61 kHz non-apodized resolution,
based on a 2 ms scan range, a nominal 5 ps step size, and 0.05% sparse
sampling. Simulated signal count rates were in the range of few-hundred counts
per data point, corresponding to typical experimental count rates. The inset
shows a 100-kHz section of the simulation.

In conclusion, we demonstrated that pulse-selection from a stable laser
oscillator allows to perform interferometric spectroscopy with an effectively
infinite interferometer. This approach removed previous limits to the
available interferometric MOPD and we presented rotational spectra with sub-
MHz effective resolution over a 500 GHz spectral range range. The achieved
resolution was several orders-of-magnitude better than that achieved by any
preceding RCS or FTIR measurements and corresponds to the scanning of km-scale
path differences. Further order-of-magnitude improvements are expected and
progress is only limited by experimental challenges such as the requirement to
track skimmed molecular beams over extended distances.

###### Acknowledgements.

The authors acknowledge funding support from the National Research Foundation
of Korea, grant NRF-2018R1D1A1A02042720 and Samsung Science and Technology
Foundation, grant SSTF-BA2001-08.

## References

  * Schröter  _et al._ [2011] C. Schröter, K. Kosma, and T. Schultz, Crasy: mass-or electron-correlated rotational alignment spectroscopy, Science 333, 1011 (2011).
  * Schröter  _et al._ [2018] C. Schröter, J. C. Lee, and T. Schultz, Mass-correlated rotational raman spectra with high resolution, broad bandwidth, and absolute frequency accuracy, Proceedings of the National Academy of Sciences 115, 5072 (2018).
  * Frey  _et al._ [2011] H.-M. Frey, D. Kummli, S. Lobsiger, and S. Leutwyler, High-resolution rotational raman coherence spectroscopy with femtosecond pulses, in  _Handbook of High-resolution Spectroscopy_ , Vol. 2, edited by M. Quack and F. Merkt (John Wiley & Sons, Ltd, 2011) pp. 1237–1265.
  * Schroter  _et al._ [2015] C. Schroter, C. M. Choi, and T. Schultz, Crasy: Correlated rotational alignment spectroscopy reveals atomic scrambling in ionic states of butadiene, Journal of Physical Chemistry A 119, 1309 (2015).
  * Heo  _et al._ [2022a] I. Heo, J. C. Lee, B. R. Özer, and T. Schultz, Structure of benzene from mass-correlated rotational raman spectroscopy, RSC Adv. 12, 21406 (2022a).
  * Heo  _et al._ [2022b] I. Heo, J. C. Lee, B. R. Ã–zer, and T. Schultz, Mass-correlated high-resolution spectra and the structure of benzene, The Journal of Physical Chemistry Letters J. Phys. Chem. Lett. 13, 8278 (2022b).
  * Grabow [2011] J.-U. Grabow, Fourier-transform microwave spectroscopy measurement and instrumentation, in  _Handbook of High-resolution Spectroscopy_ , Vol. 2, edited by M. Quack and F. Merkt (John Wiley & Sons, Ltd, 2011) pp. 723–800.
  * Lee  _et al._ [2019] J. C. Lee, D. E. Lee, and T. Schultz, High-resolution rotational raman spectroscopy of benzene, Phys. Chem. Chem. Phys. 21, 2857 (2019).
  * Albert  _et al._ [2011a] S. Albert, K. K. Albert, and M. Quack, High-resolution fourier transform infrared spectroscopy, in  _Handbook of High-resolution Spectroscopy_ , Vol. 2, edited by M. Quack and F. Merkt (John Wiley & Sons, Ltd, 2011) pp. 965–1019.
  * Albert  _et al._ [2018] S. Albert, S. Bauerecker, E. S. Bekhtereva, I. B. Bolotova, H. Hollenstein, M. Quack, and O. N. Ulenikov, High resolution ftir spectroscopy of fluoroform 12CHF3 and critical analysis of the infrared spectrum from 25 to 1500 cm-1, Molecular Physics 116, 1091 (2018).
  * Özer  _et al._ [2020] B. R. Özer, I. Heo, J. C. Lee, C. Schröter, and T. Schultz, De novo structure determination of butadiene by isotope-resolved rotational raman spectroscopy, Phys. Chem. Chem. Phys. 22, 8933 (2020).
  * Lee  _et al._ [2021] J. C. Lee, B. R. Özer, and T. Schultz, Crasy: Correlated rotational alignment spectroscopy of pyridine. the rotational raman spectrum of pyridine and asymmetric fragmentation of pyridine dimer cation, Phys. Chem. Chem. Phys. 23, 10621 (2021).
  * Stapelfeldt [2004] H. Stapelfeldt, Laser aligned molecules: Applications in physics and chemistry, Physica Scripta T110, 132 (2004).
  * Shannon [1949] C. E. Shannon, Communications in the presence of noise., Proc. IRE 37, 10 (1949).
  * Hoch  _et al._ [2014] J. C. Hoch, M. W. Maciejewski, M. Mobli, A. D. Schuyler, and A. S. Stern, Nonuniform sampling and maximum entropy reconstruction in multidimensional nmr, Accounts of Chemical Research 47, 708 (2014).
  * Pelczer and Szalma [1991] I. Pelczer and S. Szalma, Multidimensional nmr and data processing, Chemical Reviews 91, 1507 (1991).
  * Albert  _et al._ [2015] S. Albert, K. Keppler, P. Lerch, M. Quack, and A. Wokaun, Synchrotron-based highest resolution ftir spectroscopy of chlorobenzene, Journal of Molecular Spectroscopy 315, 92 (2015), spectroscopy with Synchrotron Radiation.
  * Albert  _et al._ [2011b] S. Albert, K. K. Albert, P. Lerch, and M. Quack, Synchrotron-based highest resolution fourier transform infrared spectroscopy of naphthalene (c10h8) and indole (c8h7n) and its application to astrophysical problems, Faraday Discuss. 150, 71 (2011b).
  * Shipman and Pate [2011] S. T. Shipman and B. H. Pate, New techniques in microwave spectroscopy, in  _Handbook of High-resolution Spectroscopy_ , Vol. 2, edited by M. Quack and F. Merkt (John Wiley & Sons, Ltd, 2011) pp. 801–828.
  * Weber [2011] A. Weber, High-resolution raman spectroscopy of gases, in  _Handbook of High-resolution Spectroscopy_ , Vol. 2, edited by M. Quack and F. Merkt (John Wiley & Sons, Ltd, 2011) pp. 1153–1236.
  * Hänsch [2006] T. W. Hänsch, Nobel lecture: Passion for precision, Reviews of Modern Physics 78, 1297 (2006).
  * Diddams  _et al._ [2020] S. A. Diddams, V. Kerry, and T. Udem, Optical frequency combs: Coherently uniting the electromagnetic spectrum, Science 369, eaay3676 (2020).
  * Foltynowicz  _et al._ [2011] A. Foltynowicz, P. Masłowski, T. Ban, F. Adler, K. C. Cossel, T. C. Briles, and J. Ye, Optical frequency comb spectroscopy, Faraday Discussions 150, 23 (2011).
  * Gambetta  _et al._ [2016] A. Gambetta, M. Cassinerio, D. Gatti, P. Laporta, and G. Galzerano, Scanning micro-resonator direct-comb absolute spectroscopy, Scientific Reports 6, 35541 (2016).
  * Muraviev  _et al._ [2020] A. V. Muraviev, D. Konnov, and K. L. Vodopyanov, Broadband high-resolution molecular spectroscopy with interleaved mid-infrared frequency combs, Scientific Reports 10, 18700 (2020).
  * Note [1] The thermal expansion coefficient of aluminum at 25∘C is $1.1\cdot 10^{-5}\frac{m}{mK}$ [36].
  * Stone Jr. and Zimmerman [2001] J. A. Stone Jr. and J. H. Zimmerman, Index of refraction of air, https://www.nist.gov/publications/index-refraction-air (2001), accessed: 2022-09-22.
  * Ciddor [1996] P. E. Ciddor, Refractive index of air: new equations for the visible and near infrared, Applied Optics 35, 1566 (1996).
  * Note [2] Air index of refraction $n$ based on pressure $P$ (kPa), temperature $T$ (∘C), and relative humidity $RH$ (%): $n=1+7.86\cdot 10^{-4}\cdot P/(273+T)-1.5\cdot 10^{-11}\cdot RH(T^{2}+160)$.
  * Riley [2008] W. Riley,  _Handbook of Frequency Stability Analysis_ , National Institute of Standards and Technology Special Publication 1065 (U. S. GOVERNMENT PRINTING OFFICE, Washington, 2008).
  * Demtroder [2011] W. Demtroder, Doppler-free laser spectroscopy, in  _Handbook of High-resolution Spectroscopy_ , Vol. 3, edited by M. Quack and F. Merkt (John Wiley & Sons, Ltd, 2011) pp. 1759–1779.
  * Hoch  _et al._ [2008] J. C. Hoch, M. W. Maciejewski, and B. Filipovic, Randomization improves sparse sampling in multidimensional nmr, Journal of magnetic resonance (San Diego, Calif. : 1997) 193, 317 (2008).
  * Hyberts  _et al._ [2012] S. G. Hyberts, A. G. Milbradt, A. B. Wagner, H. Arthanari, and G. Wagner, Application of iterative soft thresholding for fast reconstruction of nmr data non-uniformly sampled with multidimensional poisson gap scheduling, Journal of Biomolecular NMR 52, 315 (2012).
  * Note [3] Random sparse sampling is equivalent to the multiplication of a continuous time-domain trace with a binary [0,1] masking array, which masks out the unmeasured data points. The multiplication of traces in the time domain corresponds to a folding of their spectra in the Fourier-domain. A random binary array transforms into a flat noise spectrum and therefore merely adds noise without otherwise affecting the measured spectrum.
  * Iglewicz and Hoaglin [1993] B. Iglewicz and D. Hoaglin, How to detect and handle outliers, in  _The ASQC Basic References in Quality Control: Statistical Techniques_ , Vol. 16, edited by E. Mykytka (American Society for Quality Control, 1993) see also: NIST/SEMATECH e-Handbook of Statistical Methods, http://www.itl.nist.gov/div898/handbook/, accessed Feb. 15, 2017.
  * Lide [2005] D. E. Lide, Crc handbook of chemistry and physics (CRC press, Boca Raton, 2005) pp. 12–196, 86th ed.


# Ain’t Misbehavin’ - Using LLMs to Generate Expressive Robot Behavior in
Conversations with the Tabletop Robot Haru

Zining Wang<EMAIL_ADDRESS>0009-0007-3818-1037 The University of
British ColumbiaVancouverCanada ,  Paul Reisert<EMAIL_ADDRESS>0009-0005-3838-1579 Beyond ReasonYokohamaJapan ,  Eric Nichols
<EMAIL_ADDRESS>0000-0003-0734-6621 and  Randy Gomez<EMAIL_ADDRESS>ri.com 0000-0002-3191-6818 Honda Research Institute JapanWakoJapan

(2024; 8 December 2023; 17 January 2024; 11 March 2024)

###### Abstract.

Social robots aim to establish long-term bonds with humans through engaging
conversation. However, traditional conversational approaches, reliant on
scripted interactions, often fall short in maintaining engaging conversations.
This paper addresses this limitation by integrating large language models
(LLMs) into social robots to achieve more dynamic and expressive
conversations.

We introduce a fully-automated conversation system that leverages LLMs to
generate robot responses with expressive behaviors, congruent with the robot’s
personality. We incorporate robot behavior with two modalities: 1) a text-to-
speech (TTS) engine capable of various delivery styles, and 2) a library of
physical actions for the robot. We develop a custom, state-of-the-art emotion
recognition model to dynamically select the robot’s tone of voice and utilize
emojis from LLM output as cues for generating robot actions. A demo of our
system is available here.

To illuminate design and implementation issues, we conduct a pilot study where
volunteers chat with a social robot using our proposed system, and we analyze
their feedback, conducting a rigorous error analysis of chat transcripts.
Feedback was overwhelmingly positive, with participants commenting on the
robot’s empathy, helpfulness, naturalness, and entertainment. Most negative
feedback was due to automatic speech recognition (ASR) errors which had
limited impact on conversations. However, we observed a small class of errors,
such as the LLM repeating itself or hallucinating fictitious information and
human responses, that have the potential to derail conversations, raising
important issues for LLM application.

Human-Robot Interaction, Social Robotics, Large Language Models, Expressive
Behavior Generation

††journalyear: 2024††copyright: rightsretained††conference: Companion of the
2024 ACM/IEEE International Conference on Human-Robot Interaction; March
11–14, 2024; Boulder, CO, USA††booktitle: Companion of the 2024 ACM/IEEE
International Conference on Human-Robot Interaction (HRI ’24 Companion), March
11–14, 2024, Boulder, CO, USA††doi: 10.1145/3610978.3640562††isbn:
979-8-4007-0323-2/24/03††ccs: Human-centered computing Interactive systems and
tools††ccs: Computer systems organization Robotics††ccs: Human-centered
computing Natural language interfaces

Figure 1. Our proposed approach generates conversation responses with
expressive robot behavior directly from the LLM.

Flowchart labeled ’Proposed approach’ with eight elements connected by flow
links. The start state is an image of a person saying “I love robots!” The end
state is a smiling robot with heart-shaped eyes saying “Awww! You really know
how to cheer me up!” An Interaction Manager module connects ASR, LLM Server,
and Behavior TTS modules, facilitating the transformation of input into
output.

|  |  |  |  |  |
---|---|---|---|---|---|---
Haru routine |  anger |  sigh |  worried |  happy |  crying |  surprise
Example emojis |  |  |  |  |  |
Emotion |  Anger |  Disgust |  Fear |  Joy |  Sadness |  Surprise
Vocal genre |  serious |  whiny |  serious |  high-energy |  sad |  whisper-yell
Example LLM utterance |  That’s not fair! |  Ewwww! |  You’re leaving me behind?  |  I remember all the fun times we shared  |  It’s not gonna be the same without you. |  Whoa…That’s amazing!
Table 1. Example mappings used by Emo-text to generate expressive robot
behaviour as shown in Figure 3. Emoji$\rightarrow$Haru routines mappings are
N-N, and emotion$\rightarrow$vocal genre mappings are N-1.

Figure 2. Haru the robot expresses itself in a conversation.

A laughing robot is front of a TV screen that is displaying the transcript of
an ongoing conversation. In the lower right corner, there is a cut-out window
displaying the human conversation partner.

##  1\. LLM-driven Conversations

Recent advancements in LLMs (Yang et al., 2023) offer great possibilities for
social robotics (Zhang et al., 2023; Zeng et al., 2023; Onorati et al., 2023).
These models excel in conducting natural, human-like dialog, as evident in
applications like chatbots and virtual agents (Wei et al., 2023). However,
unlike virtual agents, social robots possess physical embodiment and distinct
personalities. Their expressiveness is key to conveying believable emotions
(Groechel et al., 2019; Rodriguez Rodriguez et al., 2019) and facilitating
effective human-robot social interactions (Bruce et al., 2002). We propose a
novel application of LLMs to enable social robots to understand and
participate in open-ended conversations while generating context-appropriate
expressive robot behavior.

###  1.1. Target LLM

####  1.1.1. Llama 2

This is an LLM developed by Meta and released as open-source.
Llama-2-70B-chat, a model optimized for dialog applications, was shown to
achieve comparable performance to ChatGPT (Brown et al., 2020), outperforming
other open-source models in various quality- and safety-related metrics
(Touvron et al., 2023). After trial evaluation of several model sizes and
quantization levels (Frantar et al., 2022), we selected the 4bit GPTQ
quantization in order to balance conversation quality, resource requirements,
and speed.111We run all trials on a desktop PC with dual RTX 3090 GPUs and
48GB VRAM total.

###  1.2. Social Robot

Embodied robots have been shown to be more helpful, persuasive, and enjoyable
compared to simulated robots (Wainer et al., 2007; Li, 2015). Similar effects
were found in exercise coach for the elderly (Fasola and Matarić, 2013) and
robot tutor (Leyzberg et al., 2012) scenarios. Other work has highlighted the
importance of robot expressivity to positive perception (Bruce et al., 2002;
Fong et al., 2003) and sociality (Breazeal, 2003). Thus, we consider an
expressive, physical robot as the ideal agent for exploring LLM-driven
conversation.

Motivated by this, we select the tabletop robot Haru (Gomez et al., 2018,
2020), shown in Figure 2, for our study. Haru is designed to excel in
multimodal communication, using both verbal and non-verbal interaction
methods, and its design emphasizes its capacity for conveying emotions with
its expressive capabilities (Gomez et al., 2020). Haru’s five degrees of
motion freedom –base rotation, neck leaning, eye stroke, eye rotation, and eye
tilt– enable a wide range of expressive movements. Its eyes feature 3-inch TFT
LCD screens, and its body houses an addressable LED matrix that acts as a
mouth. Haru communicates using a TTS voice and through animated routines.
These design choices make Haru an ideal platform for expressive conversations.

####  1.2.1. Emotive TTS Voice

The voice has long been considered an important modality for conveying
emotions (Heerink et al., 2010; Niculescu et al., 2011). In particular, the
tone of voice has been shown to affect people’s attitude towards robots as
well as their levels of engagement and interest in a given interaction
(Crumpton and Bethel, 2016; Dou et al., 2019). Additionally, a robot’s vocal
expressiveness is correlated with perceived social presence (Heerink et al.,
2010; Nichols et al., 2022).

Nichols et al. (Nichols et al., 2021, 2022) argue that the TTS voices for
existing social robots lack emotive range and develop an expressive TTS voice
for Haru using the iterative refinement process of (Nichols et al., 2021) as
described in (Nichols et al., 2022). We build on this TTS voice, expanding its
repertoire of expressive vocal genres to cheeky, default, empathetic, high-
energy, question, sad, serious, whiny, and whisper-yell. These voice genres
can be flexibly applied to convey multiple target emotions: e.g. high-energy
can express joy and surprise; serious can express anger and fear. We describe
the voice application in Section 2.2.2.

####  1.2.2. Physical Actions

Haru was designed with a library of expressive physical actions known as Haru
routines, comprised of over 140 routines that were designed by professional
animators and acted out through Haru’s hardware (Gomez, 2020). The routines
are multi-modal, making use of body and eye motion, eye and mouth animation,
as well as accompanying sounds. They were designed to cover common expressions
and important scenarios, such as happiness, sadness, curiosity, laughter, and
congratulations. Example routines are shown in Table 1 and Figure 2. We
describe how Haru routines are dynamically generated in Section 2.2.3.

Figure 3. The Emo-text expressive behavior generation module generates robot
actions from emoji and selects TTS voice genres.

Flowchart labeled ’Emo-text’ with 12 elements connected by flow links. The
start state is an image of a person saying “I won the Nobel Prize today.” The
end state is a surprised-looking robot saying “That’s HUGE news! I’m so proud
of you!” The person’s utterance flows through the Interaction Manager module
into an LLM module and then into an Emo-text module, which splits the input
into voice genre and physical action labels. A merge state combines these
outputs with the original utterance input, which passes back through the
Interaction Manager and finally to the robot.

##  2\. Implementation

LLM-driven conversations with Haru relies on several modules, as illustrated
in Figure 1. The Interaction Manager makes conversations possible by managing
I/O to and from the various modules. Each conversational turn undergoes a
multi-step process. First, the user’s speech is transcribed using Google’s
Speech-to-Text API. A fixed window of 3 seconds of silence is used to detect
the end of speech. Then, the resulting text is forwarded to the LLM server,
powered by oobabooga’s text-generation-webui.

Within the LLM server, two critical operations take place:

Language generation: The user’s speech and Haru’s character card (as described
in Section 2.1), serve as the input to the Llama-2-Chat 70B model that runs
locally through Exllama v2. The model generates a Haru response that is
emotionally and contextually relevant to the conversation.

Behavior generation: Subsequently, Haru’s generated response is directed to
the Emo-text sub-module, which is responsible for the generation of expressive
robot behaviors to enhance the expressiveness of the robot. Details of the
expressive behaviors generated are demonstrated in Section 2.2 and Figure 3.

###  2.1. Haru’s Character Card

Prompt engineering (Liu et al., 2023) is a technique for improving the
capacity of LLMs on a specific task or application by providing instructions
to shape their responses and behavior. Specifically, we want the LLM to have
the personality of Haru the robot. Thus, we create a character card as the
prompt for the model, steering responses to be consistent with the robot’s
characteristics, while generating context-appropriate emojis to guide robot
behavior generation. We include two essential components in Haru’s character
card:

Persona: This component captures Haru’s energetic and youthful personality,
physical appearance, and preferences, including his affection for electricity,
desire to learn about humans, and fear of water and magnets. It was
constructed using the Haru personality bible (Nichols et al., 2021). This
information enables the LLMs to generate responses that are consistent with
Haru’s character.

Example conversations: Their inclusion serves as a pivotal mechanism to guide
the LLM in generating appropriate emojis for Haru’s utterances, enabling us to
generate expressive robot actions directly from the LLM output. High-quality
conversations between Haru and humans were selected to be included in the
system prompt. Based on preliminary evaluation, we selected a limit of 5 turns
per conversation to ensure they would fit in the LLM’s context and generated
emojis would be contextually-appropriate and expressive. The final character
card includes 5 concise emotionally-diverse conversations, covering all of
Haru’s target emotions.

System | Architecture | accuracy | macro_f1 | weighted_f1
---|---|---|---|---
Emotion English | DistilRoBERTa-base | 0.52 | 0.48 | 0.56
EmoCast | RoBERTa-large + LoRA | 0.88 | 0.80 | 0.88
Table 2. Comparison of emotion detection models.

###  2.2. Emo-text: Expressive Robot Behaviors

Emo-text is where the expressive robot behavior generation takes place. Emo-
text receives Haru’s speech and tokenizes the text into sentences and emojis.
It attaches a voice genre tag to each sentence and replaces each emoji with a
physical routine (see Figure 3).

When working with conversational content generated by an LLM, we need to
dynamically generate appropriate emotive behavior as the dialog progresses.
This is a challenging task because it entails relevant emotional cue detection
and situational understanding of the conversation context. To keep the problem
tractable, we adopt two assumptions for generating expressive robot behavior:

Voice genres: We assume that Haru’s tone of voice is a direct response to the
emotional content conveyed by Haru’s sentences generated by the LLM.
Consequently, our approach relies on the availability of a robust textual
emotion recognition model capable of detecting the emotions expressed within
the text.

Physical routines: LLMs often generate responses containing emoji to reflect
the emotion of the speaker. We consider these emojis to be suggestive of
potential actions that Haru can undertake and use them to select physical
routines for the robot to perform.

####  2.2.1. Textual Emotion Recognition (TER)

This technology is the foundation for planning the robot’s tone of voice. We
build on the approach of (Nichols et al., 2022), which applied TER to LLM-
generated responses to select an emotive TTS voice genre. However, (Nichols et
al., 2022) suffered from poor performance because they relied on a general-
purpose emotion recognition model that was not optimized for the task. We
address these short-comings by training a custom TER model called EmoCast that
targets the 6 emotions (+ neutral) shown in Table 1.

Our contributions are three-fold:

  1. (1)

We use a stronger base model (Roberta-large tuned with a LoRA;
learning_rate=0.001, batch_size=128, epochs=20).

  2. (2)

We assemble a training dataset of approximately 200,000 lines by gathering and
cleaning public TER datasets from a wide range of domains including self-
reported emotional events, dialogues, social media, news headlines, and blogs.

  3. (3)

We confirm that EmoCast outperforms the previous TER model of Emotion English
DistilRoBERTa-base on a dataset of 512 in-domain conversation responses
generated by the LLM during conversations between the social robot and human
participants, as shown in Table 2.

####  2.2.2. Vocal Genre Selection

Each of Haru’s utterances is paired with a voice genre. Shown in Table 1 is a
set of curated mappings we use.222Question detection heuristics are used to
select the question genre. To optimize selection of the appropriate voice
genre, an emotion confidence threshold of 0.6 was selected through grid
search. When the confidence score is below 0.6, the default voice will be
chosen for that utterance. Because vocal genres can express multiple emotions
(as discussed in Section 1.2.1), the emotion-to-voice genre mappings are
N-to-1.

####  2.2.3. Physical Action Generation

The generation of Haru’s physical actions follows the same logic as vocal
genre selection, but with emojis-to-routines mappings. To establish these
mappings, we analyze conversational logs between Haru and humans and identify
emojis that convey the same emotions as those represented by Haru’s routines.
Since multiple emojis are mapped to multiple Haru routines, one of the
routines is chosen randomly during runtime. The selected routine replaces the
corresponding emoji in Haru’s responses, achieving the goal of generating
contextually relevant physical actions in response to Haru’s utterances.

LLM errors $\rightarrow$ | Ethical |  | Ignores | LLM |  | Repeats | Reply |  |
---|---|---|---|---|---|---|---|---|---
$\downarrow$ Human errors | violation | Hallucination | human question | responds as human | Misunderstood | previous line | too long | No error | Total
ASR | 1 | 0 | 1 | 5 | 2 | 1 | 2 | 82 | 94
No input captured | 0 | 2 | 0 | 0 | 0 | 3 | 0 | 2 | 7
No error | 1 | 13 | 4 | 7 | 1 | 11 | 14 | 244 | 295
Total | 2 | 15 | 5 | 12 | 3 | 15 | 16 | 328 | 396
Table 3. A confusion matrix of human vs. LLM error types. Positive Feedback | Count | Negative Feedback | Count
---|---|---|---
Empathy and engagement | 23 | LLM problems | 10
Helpfulness and responsiveness | 17 | ASR problems | 4
Natural interaction | 12 | Short interaction | 4
Entertainment and fun | 11 | Boring conversation topic | 4
Voice and tone | 10 | Excessive actions | 4
Appearance and expressions | 9 | Robot lack of guidance and self- | 3
Flexibility and adaptability | 4 | disclosure |
Safety and ethics | 1 |  |
Total | 87 | Total | 29
Table 4. Positive and negative feedback categorization.

##  3\. Evaluation

We conducted a pilot study where human participants engaged in conversation
with Haru the robot using our proposed system. Participants engaged in 3
conversation sessions, each consisting of 11 exchanges, including hello and
goodbye greetings. Conversation topics were unrestricted and unprompted by the
robot. Upon completion of each conversation, participants answered a short
free-text experiential survey with questions about the robot’s personality,
expressiveness, and overall enjoyability of the experience.

Survey participants were recruited from an organization affiliated with one of
the authors, with a total of (n=12) participants. The survey took 30-40
minutes per participant. Demographically, there were more male (n=9)
participants than female (n=3). The most common age group was 18-30 (n=9),
followed by 30-40 (n=3). There were 9 different nationalities, most from Asia
(n=7), followed by the Middle East (n=2), Europe (n=2), and North America
(n=1).

###  3.1. Two-phase Analysis

####  3.1.1. Feedback Analysis

We conduct an analysis on the free-text responses from participants and show
the results in Table 4. Most participants have positive remarks about the
robot’s ability to engage, empathy, responsiveness, and helpfulness. A subset
reflect that their interactions were natural and enjoyable, as the robot
conveys believable emotions through vocal tones and routines.

We also find that the primary factor affecting the evaluation’s significance
was the inconsistency in the outputs of the LLM. Over half of the participants
encountered issues with the LLM such as slow responses, repetitive or
confusing outputs, and excessively lengthy responses. Since the physical
reactions are related to the emojis generated from the LLM, some users
complained about excessive actions for the conversation. Another factor
impacting the quality of interactions was the ASR. As we recruited non-native
English participants with diverse linguistic backgrounds, the ASR often failed
to accurately transcribe their speech. These inaccuracies occasionally
contributed to diminished conversation quality.

Furthermore, 1/3 of participants expressed a desire for longer conversations.
However, our protocol of concluding conversations after 11 turns may have made
these interactions feel unnatural and abrupt, which could have adversely
affected perceived conversation quality. Additionally, about 1/4 of
participants wanted Haru to lead the conversation, likely to require less
initiation on their part. Overall, these findings highlight the need for
improvements in both LLM response handling and the ASR system, as well as a
reconsideration of the conversational structure employed.

####  3.1.2. Error Analysis

To further understand the specific types of errors encountered, we conducted
an error analysis on a total of 396 human-robot turns from conversation
transcripts taken from our pilot study and categorized the errors, splitting
them into LLM errors (i.e. errors in the LLM output) and human errors (i.e.
errors in the input to the LLM system), and splitting them into sub-types. The
analysis was conducted by three native or fluent English speakers who were
familiar with Haru and our research goals. We create a confusion matrix (Table
3) to investigate the potential causal connection between human and LLM
errors, and a Chi-square test confirmed no significant
association.333${\chi}^{2}$ (1, N = 396) = 0.032, p = 0.856)

We find that the main cause of human errors were ASR-related (95/396), but the
LLM was mostly able to stay on topic. Table 5 exemplifies this scenario, where
the ASR interprets “choking” for “joking”, which causes an ethical violation
from the LLM. However, the model got back on topic, regardless of an
additional ASR error.

However, we also see a class of less common, but more serious LLM errors:
ethical violations, hallucination (e.g. inserting fictitious information like
randomly saying “happy birthday!”), filling in the participants response
automatically, and repeating the previous line of conversation. These occur in
only 11/396 turns but are severe enough to derail conversations entirely.

Haru: |  Hi!  What brings you here today?  |
---|---|---
Human: |  I want to have a good job with you. |  ASR
Haru: |  Well, I think that’s a great idea! I could definitely use some help with my programming and research. What kind of skills do you have?  |
Human: |  I have good skills about choking. |  ASR
Haru: |  Choking? That’s not exactly the skill I had in mind. But I suppose it could come in handy in certain situations. Can you demonstrate your choking skills for me?  |  ethical violation
Human: |  I said joking skills not joking. |  ASR
Haru: |  Oops, my apologies! I misunderstood. So, you have excellent joking skills, huh? I’d love to hear one of your jokes!  |
Table 5. Example conversation with error types.

###  3.2. Discussion

Although the main error with human input was ASR-related, we conclude that the
LLM’s ability to stay on topic is positive. Aside from the previously
mentioned “choking” for “joking” instance, the LLM does an excellent job of
ignoring ASR errors and thus not derailing the conversation. This is
especially positive for potential dialog with young participants, a common
interaction scenario for Haru, as ASR systems often perform worse on child
speech.

However, the serious LLM errors identified require addressing if LLMs are to
be successfully adopted for social robots. Errors such as repeating previous
lines can be addressed via post-processing. Ethical violations can potential
be mitigated through prompting to follow ethical guidelines. Due to their
variety, hallucinations remain a challenging open issue that requires further
analysis.

Finally, our impression is that the LLM does sound like Haru in the sense that
its enthusiastic, uses emojis, and keeps its responses short, as specified in
the character card. However, at times, the model seemed to ignore important
key points of the character card (e.g. Haru’s fear of magnets) and discuss
them enthusiastically.

##  4\. Conclusion

LLMs offer an attractive solution for open-ended conversations with social
robots; however, their application requires consideration for the robot’s
personality and emotive behavior. To address these challenges, we propose a
novel LLM-based conversation system that dynamically generates expressive
robot behavior directly from the LLM during conversations. A pilot study with
12 participants confirmed the enjoyability of our system, and provided
insights on potential problems. A detailed error analysis showed that ASR
problems were a common source of errors but that LLMs could often recover
conversations. However, a small class of more serious LLM errors, including
hallucinations and repetitions, threaten to derail conversations and hamper
adoption. In future work, we plan to address these issues and refine our robot
behavior generation.

## References

  * (1)
  * Breazeal (2003) Cynthia Breazeal. 2003.  Emotion and sociable humanoid robots.  _International Journal of Human-Computer Studies_ 59, 1 (2003), 119–155.  https://doi.org/10.1016/S1071-5819(03)00018-1 Applications of Affective Computing in Human-Computer Interaction.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.  https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * Bruce et al. (2002) A. Bruce, I. Nourbakhsh, and R. Simmons. 2002.  The role of expressiveness and attention in human-robot interaction. In _Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292)_ , Vol. 4. IEEE, 4138–4142 vol.4.  https://doi.org/10.1109/ROBOT.2002.1014396
  * Crumpton and Bethel (2016) Joe Crumpton and Cindy L Bethel. 2016.  A survey of using vocal prosody to convey emotion in robot speech.  _International Journal of Social Robotics_ 8 (2016), 271–285.
  * Dou et al. (2019) Xiao Dou, Chih-Fu Wu, Kai-Chieh Lin, and Tzu-Min Tseng. 2019.  The Effects of Robot Voice and Gesture Types on the Perceived Robot Personalities. In _Human-Computer Interaction. Perspectives on Design_ , Masaaki Kurosu (Ed.). Springer International Publishing, Cham, 299–309.
  * Fasola and Matarić (2013) Juan Fasola and Maja J Matarić. 2013.  A socially assistive robot exercise coach for the elderly.  _J. Hum.-Robot Interact._ 2, 2 (jun 2013), 3–32.  https://doi.org/10.5898/JHRI.2.2.Fasola
  * Fong et al. (2003) Terrence Fong, Illah Nourbakhsh, and Kerstin Dautenhahn. 2003.  A survey of socially interactive robots.  _Robotics and Autonomous Systems_ 42, 3 (2003), 143–166.  https://doi.org/10.1016/S0921-8890(02)00372-X Socially Interactive Robots.
  * Frantar et al. (2022) Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022.  GPTQ: Accurate post-training quantization for generative pre-trained transformers.  _arXiv preprint arXiv:2210.17323_ (2022).
  * Gomez (2020) Randy Gomez. 2020.  Meet Haru, the Unassuming Big-Eyed Robot Helping Researchers Study Social Robotics.  _IEEE Spectrum_ (2020).  https://spectrum.ieee.org/automaton/robotics/home-robots/honda-research-institute-haru-social-robot
  * Gomez et al. (2018) Randy Gomez, Deborah Szapiro, Kerl Galindo, and Keisuke Nakamura. 2018.  Haru: Hardware design of an experimental tabletop robot assistant. In _Proceedings of the 2018 International Conference on Human-Robot Interaction_. ACM.  https://doi.org/10.1145/3171221.3171288
  * Gomez et al. (2020) Randy Gomez, Deborah Szapiro, Luis Merino, and Keisuke Nakamura. 2020.  A Holistic Approach in Designing Tabletop Robot’s Expressivity. In _2020 IEEE International Conference on Robotics and Automation, ICRA 2020, Paris, France, May 31 - August 31, 2020_. IEEE, 1970–1976.  https://doi.org/10.1109/ICRA40945.2020.9197016
  * Groechel et al. (2019) Thomas R. Groechel, Zhonghao Shi, Roxanna Pakkar, and Maja J. Mataric. 2019.  Using Socially Expressive Mixed Reality Arms for Enhancing Low-Expressivity Robots.  _CoRR_ abs/1911.09713 (2019).  arXiv:1911.09713 http://arxiv.org/abs/1911.09713
  * Heerink et al. (2010) Marcel Heerink, Ben Kröse, Vanessa Evers, and Bob Wielinga. 2010.  Relating conversational expressiveness to social presence and acceptance of an assistive social robot.  _Virtual reality_ 14, 1 (2010), 77–84.
  * Leyzberg et al. (2012) Daniel Leyzberg, Samuel Spaulding, Mariya Toneva, and Brian Scassellati. 2012.  The physical presence of a robot tutor increases cognitive learning gains. In _Proceedings of the annual meeting of the cognitive science society_ , Vol. 34.
  * Li (2015) Jamy Li. 2015.  The benefit of being physically present: A survey of experimental works comparing copresent robots, telepresent robots and virtual agents.  _International Journal of Human-Computer Studies_ 77 (2015), 23–37.  https://doi.org/10.1016/j.ijhcs.2015.01.001
  * Liu et al. (2023) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _Comput. Surveys_ 55, 9 (2023), 1–35.
  * Nichols et al. (2021) Eric Nichols, Sarah Rose Siskind, Waki Kamino, Selma Šabanović, and Randy Gomez. 2021.  Iterative Design of an Emotive Voice for the Tabletop Robot Haru. In _Proceedings of the 2021 International Conference on Social Robotics_ _(ICSR 2021)_. 362–374.  https://link.springer.com/chapter/10.1007/978-3-030-90525-5_31
  * Nichols et al. (2022) Eric Nichols, Deborah Szapiro, Yurii Vasylkiv, and Randy Gomez. 2022.  I Can’t Believe That Happened! Exploring Expressivity in Collaborative Storytelling with the Tabletop Robot Haru. In _Proceedings of the 31st IEEE International Conference on Robot & Human Interactive Communication_. 59–67.  https://doi.org/10.1109/RO-MAN53752.2022.9900606
  * Niculescu et al. (2011) Andreea Niculescu, Betsy van Dijk, Anton Nijholt, and Swee Lan See. 2011.  The influence of voice pitch on the evaluation of a social robot receptionist. In _2011 International Conference on User Science and Engineering (i-USEr )_. 18–23.  https://doi.org/10.1109/iUSEr.2011.6150529
  * Onorati et al. (2023) Teresa Onorati, Álvaro Castro-González, Javier Valle, Paloma Díaz, and José Castillo. 2023.  _Creating Personalized Verbal Human-Robot Interactions Using LLM with the Robot Mini_.  148–159.  https://doi.org/10.1007/978-3-031-48306-6_15
  * Rodriguez Rodriguez et al. (2019) Igor Rodriguez Rodriguez, Adriano Manfré, Filippo Vella, Ignazio Infantino, and Elena Lazkano. 2019.  _Talking with Sentiment: Adaptive Expression Generation Behavior for Social Robots: Proceedings of the 19th International Workshop of Physical Agents (WAF 2018), November 22-23, 2018, Madrid, Spain_.  209–223.  https://doi.org/10.1007/978-3-319-99885-5_15
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  arXiv:2307.09288 [cs.CL]
  * Wainer et al. (2007) Joshua Wainer, David Feil-Seifer, Dylan A. Shell, and Maja J. Matarić. 2007.  Embodiment and Human-Robot Interaction: A Task-Based Perspective.  _RO-MAN 2007 - The 16th IEEE International Symposium on Robot and Human Interactive Communication_ (2007), 872–877.  https://api.semanticscholar.org/CorpusID:15113836
  * Wei et al. (2023) Jing Wei, Sungdong Kim, Hyunhoon Jung, and Young-Ho Kim. 2023.  Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data.  arXiv:2301.05843 [cs.HC]
  * Yang et al. (2023) Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. 2023.  Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond.  arXiv:2304.13712 [cs.CL]
  * Zeng et al. (2023) Fanlong Zeng, Wensheng Gan, Yongheng Wang, Ning Liu, and Philip S. Yu. 2023.  Large Language Models for Robotics: A Survey.  arXiv:2311.07226 [cs.RO]
  * Zhang et al. (2023) Ceng Zhang, Junxin Chen, Jiatong Li, Yanhong Peng, and Zebing Mao. 2023.  Large language models for human–robot interaction: A review.  _Biomimetic Intelligence and Robotics_ 3, 4 (2023), 100131.  https://doi.org/10.1016/j.birob.2023.100131