# Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal
Nuances

Zehui Wu, 1 Ziwei Gong, 1 Lin Ai, 1 Pengyuan Shi,1 Kaan Donbekci, 1 Julia
Hirschberg 1
Department of Computer Science
Columbia University
{zw2804, zg2272, la2734, ps3391<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

This paper introduces a novel approach to emotion detection in speech using
Large Language Models (LLMs). We address the limitation of LLMs in processing
audio inputs by translating speech characteristics into natural language
descriptions. Our method integrates these descriptions into text prompts,
enabling LLMs to perform multimodal emotion analysis without architectural
modifications or addition. We evaluate our approach on two datasets: IEMOCAP
and MELD, demonstrating significant improvements in emotion recognition
accuracy, particularly for high-quality audio data. Our LLM-finetuning
experiments show that incorporating speech descriptions yields a more than 2
percentage point increase in average weighted F1 score on IEMOCAP (from
70.111% to 72.596%). We also compare various LLM architectures and explore the
effectiveness of different feature representations. Our findings highlight the
potential of this approach in enhancing emotion detection capabilities of LLMs
and underscore the importance of audio quality in speech-based emotion
recognition tasks. We’ll release the source code on Github.

Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal
Nuances

Zehui Wu, 1 Ziwei Gong, 1 Lin Ai, 1 Pengyuan Shi,1 Kaan Donbekci, 1 Julia
Hirschberg 1 Department of Computer Science Columbia University {zw2804,
zg2272, la2734, ps3391<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Emotion detection in speech is a crucial component of human-computer
interaction, with applications ranging from customer service to mental health
monitoring. While Large Language Models (LLMs) have shown remarkable
capabilities in processing and understanding text, they lack the ability to
directly process audio inputs. This limitation has hindered their potential in
multimodal emotion recognition tasks that involve both textual and audio data.

Our research addresses this challenge by introducing a novel approach that
enables LLMs to leverage speech characteristics for emotion detection without
requiring architectural modifications. We propose a method that translates
audio features into natural language descriptions, which can be seamlessly
integrated into text prompts for LLMs. This approach bridges the gap between
audio and text modalities, allowing LLMs to perform multimodal emotion
analysis.

In this paper, we evaluate our method using two widely recognized datasets in
the field of emotion recognition: IEMOCAP Busso et al. (2008) and MELD Poria
et al. (2019). We explore the effectiveness of various feature representations
and compare the performance of different LLM architectures. Our study aims to
demonstrate the potential of this approach to enhance the emotion detection
capabilities of LLMs and to provide insights into the factors that influence
its success.

By enabling LLMs to consider both textual and audio-derived information, we
open up new possibilities for more nuanced and accurate emotion recognition
systems. This research not only contributes to the field of affective
computing but also provides a framework for incorporating non-textual
information into language-model-based systems, with potential implications for
a wide range of multimodal AI applications.

##  2 Related Work

###  2.1 LLM for emotion recognition

Recent advancements in Emotion Recognition in Conversation (ERC) have
leveraged Large Language Models (LLMs) to enhance performance and
generalizability. Our work builds upon these developments, particularly the
InstructERC framework Lei et al. (2024), which reformulated ERC as a
generative task using LLMs. InstructERC introduced a retrieval template module
and additional emotion alignment tasks, achieving state-of-the-art results on
common ERC datasets. We extend this approach by incorporating speech
descriptions into the template, aiming to capture crucial paralinguistic
information.

Similar extensions to InstructERC have been explored in recent literature.
BiosERC Xue et al. (2024) enhances the model by extracting and injecting
"biographical information" of speakers into the system, demonstrating improved
performance across multiple datasets. CKERC Fu (2024) takes a different
approach, incorporating commonsense knowledge about interlocutors’ reactions
and intentions to deepen the understanding of speaker information. DialogueLLM
Zhang et al. (2024b) incorporates visual descriptions generated from GPT-4
into the prompt. All of these methods, like our approach, aim to enrich the
input to LLMs with additional contextual information.

Our work differentiates itself by focusing specifically on the integration of
speech characteristics, translated into natural language descriptions. This
approach bridges the gap between audio and text modalities in LLM-based ERC
systems, potentially capturing emotion cues that are present in speech but not
in text alone. By doing so, we aim to combine the strengths of LLMs in
processing textual data with the rich emotional information contained in
speech, offering a novel perspective in the evolving landscape of ERC
research.

###  2.2 Speech Incorporation in LLM

The integration of speech features into LLM-based systems for emotion
recognition and related tasks has been explored in various recent works, each
offering unique approaches and insights. SECap Xu et al. (2023) introduces a
framework that uses LLaMA as a text decoder to generate coherent speech
emotion captions instead of labels. While SECap demonstrates the potential of
LLMs in describing speech emotions, it differs from our approach by requiring
an additional audio encoder (HuBERT) and a bridging network (Q-Former) to
process speech inputs. Similarly, Zhang et al. (2024a) proposes an approach to
integrate speech into LLMs for depression detection by adding acoustic
landmark tokens to the input prompt. This method shares our goal of
incorporating speech information directly into LLM inputs but necessitates
additional parameters and pre-training steps to introduce audio information.

LanSER Gong et al. (2023) takes a different approach by leveraging LLMs to
infer weak emotion labels from speech transcripts, enabling the use of
unlabeled data in SER tasks. Building upon this concept, Santoso et al. (2024)
explores using LLMs for emotional speech annotation by incorporating both
conversation context and textual acoustic feature descriptors. While this
approach shares our interest in combining textual and acoustic information in
the prompt, it primarily uses LLMs for weak label generation rather than
direct emotion recognition training.

Recent research has also explored the application of LLMs in Automatic Speech
Recognition (ASR), demonstrating the versatility of LLMs in speech-processing
tasks. Seed-ASR Bai et al. (2024) introduces an audio-conditioned LLM
framework that incorporates continuous speech representations by developing a
unique audio encoder. Baskar et al. (2024) proposes a speech prefix-tuning
approach with RNNT loss to optimize speech prefixes for LLMs in ASR tasks.
SLAM-ASR Ma et al. (2024) takes a simpler approach, combining off-the-shelf
speech encoders and LLMs with only a trainable linear projector, achieving
state-of-the-art performance on the Librispeech benchmark.

Further extending the capabilities of LLMs in speech processing,
AudioChatLlama Fathullah et al. (2024) presents an end-to-end model that
integrates general-purpose speech abilities into the Llama-2 model. This
approach enables the model to handle audio prompts, perform spoken question
answering, speech translation, and audio summarization, among other tasks.
Similarly, Audio-LLM Li et al. (2024) introduces a method for encoding audio
data into embedded representations that LLMs can comprehend, enabling tasks
such as automatic speech recognition, emotion recognition, and music
captioning. These advancements highlight the growing trend of integrating
comprehensive speech-processing capabilities into LLMs.

Our work distinguishes itself from these approaches in several key aspects:
(a) Direct Integration: We achieve seamless integration without requiring
additional neural network components (e.g., audio encoders or projection
layers) to process speech inputs, thereby reducing computational complexity.
(b) End-to-End Training: We train the LLM directly on combined textual and
speech description inputs, avoiding the need for weak label generation and
additional training stages. (c) Flexibility and Generalizability: Our method
of translating speech features into natural language descriptions allows for
easy integration as a plug-in for other approaches and tasks, offering a
versatile solution for incorporating speech information into LLM-based
systems.

##  3 Methodology

Our novel approach incorporates speech characteristic descriptions in natural
language into the prompt for LLMs. Previous LLM-based approaches have
primarily focused on textual information, neglecting audio signals due to
LLMs’ inability to process direct audio inputs. This omission results in the
loss of crucial emotional cues, as the same sentence can convey different
emotions through variations in audio features such as pitch, volume, and
intonation. Our method bridges this gap by translating audio signals into
natural language descriptions of speech characteristics that are instrumental
in emotion detection.

Our approach offers several advantages in the field of emotion detection.
First, it enables multimodal analysis by seamlessly integrating speech
characteristics into text prompts, allowing LLMs to process audio-derived
information without requiring architectural modifications. This method is
highly compatible with existing LLM infrastructures, facilitating easy
implementation and integration into current systems. Moreover, by preserving
paralinguistic information such as volume, pitch, and speaking rate in our
natural language descriptions, we capture crucial emotional cues that are
often lost in text-only analysis. This enhanced contextual understanding
empowers the LLM to grasp the full spectrum of the utterance, potentially
leading to more accurate and nuanced emotion classification.

###  3.1 Large Language Model Prompt Template

Figure 1: LLM Prompt Template for Emotion Detection: the last bold sentence
with an underline is the target utterance. The orange part denotes outputs
with added speech descriptions. This structured template integrates textual
context and speech characteristics to guide the LLM in performing multimodal
emotion analysis.

Our methodology utilizes a comprehensive LLM Prompt Template designed to
enhance emotion detection capabilities. This template consists of four key
components:

  1. 1.

Instruction: This component sets the stage by positioning the LLM as an expert
in sentiment and emotion analysis, priming it for the task ahead.

  2. 2.

Context: A crucial element that provides the conversational background,
allowing the LLM to understand the broader situation and interpersonal
dynamics at play. We also add speech features behind the last three utterances
to provide additional speech context.

  3. 3.

Speech Descriptions: This innovative addition translates audio signals into
natural language descriptions of speech characteristics. By including details
such as volume, pitch, and speaking rate, we bridge the gap between audio and
text modalities, enabling the LLM to consider paralinguistic features crucial
for accurate emotion detection.

  4. 4.

Question: This component presents the specific task to the LLM, asking it to
select an emotional label for a target utterance from a predefined set of
options. Importantly, it explicitly instructs the model to consider both the
contextual information and the audio features described.

This structured approach ensures that the LLM has access to both textual and
audio-derived information, enabling a more nuanced and accurate emotion
detection process. By incorporating speech characteristics into the prompt, we
allow the LLM to perform multimodal analysis without requiring direct audio
input processing capabilities.

###  3.2 Audio Features

In our approach to emotion detection, we leverage five intuitive audio
features that are easily interpretable by both humans and LLMs. These features
capture essential aspects of speech that convey emotional information. volume
is represented by two sub-features: average volume, which indicates the
overall loudness of speech, and volume variation, which captures dynamic
changes in intensity. Pitch, another crucial indicator of emotion, is
similarly divided into average pitch and pitch variation, allowing us to
detect both the general tone and the modulation of a speaker’s voice. The
fifth feature, speaking rate, represents the speed of speech, offering
insights into the speaker’s emotional state and urgency. By focusing on these
fundamental yet comprehensive audio characteristics, we ensure that the
extracted features are not only rich in emotional content but also readily
understandable by LLMs when described in natural language.

####  3.2.1 Audio Feature Processing

In our approach to audio feature processing, we transform raw numerical
features into categorical representations to enhance interpretability and
facilitate natural language descriptions. This process involves four key
steps:

a) Threshold Calculation: We compute thresholds for each audio feature (such
as average volume, pitch variation, and speaking rate) using quantile-based
segmentation. Depending on the desired granularity, we divide the feature
space into 3, 4, 5, or 6 classes, with thresholds calculated using appropriate
quantiles (e.g., 25th and 75th percentiles for 3 classes, 10th, 25th, 75th,
and 90th percentiles for 5 classes).

b) Speaker-Specific Normalization: Recognizing the individual differences in
speech patterns, we calculate speaker-specific thresholds, falling back to
overall thresholds for less frequent speakers. This approach allows for a more
accurate categorization of speech features relative to each speaker’s typical
patterns.

c) Categorization: Each numerical feature is then categorized based on these
thresholds. For instance, in a 5-class system, a feature value might be
classified as ’very low’, ’low’, ’medium’, ’high’, or ’very high’. This
categorization provides a more intuitive representation of the audio features,
bridging the gap between numerical values and human-interpretable
descriptions.

d) Feature-Specific Descriptions: For each key audio feature (volume, pitch,
speaking rate), we create descriptive phrases based on their categorical
values. For example, "high volume with moderate variation" or "low pitch with
high variation".

####  3.2.2 Audio Impression

Figure 2: Example of Speech Characteristic Description and Derived Impression.

To make the audio features more accessible and meaningful for both human
analysis and LLM processing, we generate natural language impressions based on
the categorized features. Going beyond mere description, we generate
interpretative impressions that suggest potential emotional or psychological
states associated with the observed speech patterns. For instance, "speaking
loudly with significant volume changes" might be interpreted as indicating
"excitement, confidence, or urgency".

To account for the inherent uncertainty in interpreting speech patterns, we
incorporate confidence-adjusting phrases (e.g., "likely", "may") based on the
feature’s proximity to category thresholds. This nuanced approach helps
prevent overconfident interpretations of borderline cases.

The final impression synthesizes information about pitch, volume, and speaking
rate into a coherent, flowing sentence. This integration provides a nuanced
description of the speaker’s vocal characteristics and their potential
implications, offering valuable context for emotion detection tasks.

##  4 Experiments

###  4.1 Dataset

For our experiments, we utilized two widely recognized datasets in the field
of emotion recognition: IEMOCAP and MELD.

IEMOCAP (Interactive Emotional Dyadic Motion Capture): IEMOCAP consists of
dyadic conversations between actors, providing a rich set of emotional
expressions in a controlled setting. It contains approximately 12 hours of
audiovisual data from 10 actors, split into 5 sessions. Each utterance is
annotated with one of nine emotion labels: anger, happiness, excitement,
sadness, frustration, fear, surprise, neutral, and other. IEMOCAP is
particularly valued for its high-quality audio recordings and motion capture
data, making it ideal for multimodal emotion analysis. The dataset includes
both scripted and improvised scenarios, offering a balance between controlled
emotional expressions and more naturalistic interactions.

MELD (Multimodal EmotionLines Dataset): MELD is derived from the popular TV
series "Friends," making it particularly suitable for analyzing emotions in
conversational contexts. It contains 13,708 utterances from 1,433 dialogues,
each annotated with one of seven emotion categories: anger, disgust, fear,
joy, neutral, sadness, and surprise. MELD is multimodal, providing audio,
visual, and textual data for each utterance. The dataset is pre-split into
training (9,989 utterances), validation (1,109 utterances), and test (2,610
utterances) sets. Its multi-party conversational nature and situational
diversity make it an excellent resource for evaluating emotion detection in
complex, realistic scenarios.

For our experiments, we focused on the audio modality from both datasets,
extracting and processing acoustic features as outlined in our methodology. In
the case of the MELD dataset, we implemented speaker-based standardization of
audio features to account for individual vocal characteristics. For the
IEMOCAP dataset, we applied gender-based standardization, recognizing the
physiological differences that can impact vocal features. Additionally, for
IEMOCAP, we confined our analysis to six emotion categories (anger, happiness,
excitement, sadness, frustration, and neutral), aligning with common practices
in the field.

Our utilization of both MELD and IEMOCAP datasets was strategically designed
to evaluate our approach across diverse audio recording environments. MELD,
with its origin in television shows, presents more chaotic and naturalistic
scenarios, while IEMOCAP offers a controlled, studio-based environment. This
dual-dataset approach enables a comprehensive assessment of our emotion
detection method across varied recording conditions.

###  4.2 Primary Results

Method | Text-only | \+ speech des | \+ speech imp | \+ speech des with context
---|---|---|---|---
IEMOCAP | 70.111 $\pm$0.36 | 72.021 $\pm$0.54 (+1.910) | 71.542 $\pm$1.12 (+1.431) |  72.596 $\pm$0.26 (+2.485)
MELD | 67.44 $\pm$0.26 | 67.074 $\pm$0.49 (-0.366) |  67.604 $\pm$0.38 (+0.164) | 67.09 $\pm$0.57 (-0.35)
Table 1: Emotion recognition weighted F1 score (%) on IEMOCAP and MELD
datasets using LoRA fine-tuning on the LLaMA-2-7B-base model. Results are
averaged over three independent runs, reported as mean $\pm$ standard
deviation. Dark green values in parentheses show the improvement over the
Text-only method.

Our results from Table 1 demonstrate that incorporating speech features into
the text input yields performance improvements across both datasets, albeit to
varying degrees. For the IEMOCAP dataset, we observe a substantial improvement
of nearly 2 percentage points when adding speech descriptions to the text
input (from 70.111% to 72.021%). By providing additional speech features for
context, the model improves to 72.596%. (Detailed experiments are in Appendix
A.1) This significant gain underscores the value of integrating acoustic
information in emotion recognition tasks, particularly for high-quality audio
recordings.

In contrast, the MELD dataset does not show improvemnet using speech
descriptions and a modest improvement with the addition of speech impression,
increasing from 67.44% to 67.604%. This disparity in performance between the
two datasets can be attributed to the differences in audio quality and
recording conditions. IEMOCAP, recorded in a controlled studio environment,
provides clean and consistent audio, allowing for more accurate extraction of
speech features. On the other hand, MELD, derived from the TV show "Friends,"
presents more challenging audio conditions. The presence of background
laughter, varying audio quality, overlapping speakers, and short utterance
durations (averaging 2-3 seconds) in MELD likely contribute to less reliable
speech feature extraction, thereby limiting the potential benefits of speech
descriptions.

Interestingly, the use of speech impressions does not yield further
improvements over speech descriptions for either dataset. For IEMOCAP, the
performance (71.542%) falls between that of text-only and text with speech
descriptions, while for MELD, it performs similarly to the text-only baseline
(67.023%). This suggests that our hard-coded impressions, despite being more
interpretative, may introduce noise or inaccuracies that the model struggles
to leverage effectively. The lack of improvement with speech impressions
indicates that allowing the model to learn directly from more objective speech
descriptions may be a more robust approach.

These findings highlight the potential of integrating speech characteristics
into text-based emotion recognition models, particularly when working with
high-quality audio data. However, they also underscore the challenges posed by
real-world audio conditions and the need for careful consideration when
designing multimodal emotion recognition systems.

###  4.3 Model Comparison

Method (IEMOCAP) | Text-only | Text + speech des | Text + speech imp
---|---|---|---
Model |  |  |
Claude Sonnet 3.5 (zero-shot) | 58.99 |  59.23 (+0.24) | 59.11 (+0.12)
LLaMA-2-7b-base (LoRA) | 70.111 $\pm$0.36 |  72.021 $\pm$0.54 (+1.910) | 71.542 $\pm$1.12 (+1.431)
LLaMA-3-8b-base (LoRA) | 70.316 $\pm$0.95 | 71.744 $\pm$0.79 (+1.428) |  71.910 $\pm$0.62 (+1.594)
LLaMA-3-8b-instruct (LoRA) | 70.474 $\pm$0.94 |  72.098 $\pm$0.65 (+1.624) | 71.132 $\pm$0.36 (+0.658)
phi-3-13b-instruct (LoRA) | 69.218 $\pm$0.73 | 70.450 $\pm$0.86 (+1.232) |  70.582 $\pm$0.24 (+1.364)
Table 2: Emotion recognition weighted F1 score (%) on IEMOCAP dataset across
different models and input modalities. Results for LoRA fine-tuned models are
reported as mean $\pm$ standard deviation over three runs. Bold indicates the
best performance for each model. Dark green values in parentheses show the
improvement over the Text-only method.

To evaluate the effectiveness of our approach across different language
models, we conducted experiments using various popular open-sourced LLMs,
including LLaMA-2, LLaMA-3, and Phi-3, and a powerful closed-sourced model,
Claude Sonnet 3.5. Table 2 presents the emotion recognition accuracy on the
IEMOCAP dataset for these models under different input conditions: text-only,
text with speech descriptions, and text with speech impressions.

Our results demonstrate a consistent improvement in performance when
incorporating either speech descriptions or impressions across all models.
This underscores the value of integrating audio-derived features in emotion
recognition tasks, regardless of the underlying language model. For most
models, speech descriptions yield slightly better results than speech
impressions, although the difference is often marginal. This suggests that
more objective, feature-based descriptions may be more reliable for emotion
detection tasks.

Interestingly, increasing model size does not necessarily lead to improved
performance in this task. Despite their stronger performance on public
benchmarks, larger models like LLaMA-3 and Phi-3 do not consistently
outperform the smaller LLaMA-2 model when fine-tuned for emotion detection. In
fact, Phi-3, despite being the largest model tested (13B parameters), performs
worse than both LLaMA-2 and LLaMA-3 across all input conditions. This
indicates that model size alone is not a determining factor for success in
this specific task. While LLaMA-3-8b-instruct achieves the highest accuracy
(72.098%) among all tested models when provided with speech descriptions, the
improvement over LLaMA-2-7b-base (72.021%) is minimal. Additionally, we
observe that using the instruct or base version of these models does not
result in significant performance differences for this task. These findings
suggest that while there may be some benefit to the architectural improvements
in newer models, the gains are not as significant as one might expect given
the increase in model size and the advancements claimed in public benchmarks.
Certain features of the LLaMA architecture may be particularly well-suited for
emotion recognition tasks, but the task-specific nature of emotion detection
may limit the impact of general language model improvements.

The substantial performance gap between the zero-shot Claude model and the
fine-tuned models highlights the importance of task-specific fine-tuning for
emotion recognition. However, it is noteworthy that even in a zero-shot
setting, Claude benefits from the addition of speech features, further
emphasizing the value of multimodal inputs in emotion detection.

These findings emphasize the complexity of model selection for specific tasks
like emotion recognition. While larger, more recent models often excel in
general language understanding tasks, our results show that task-specific
fine-tuning and the integration of multimodal features can sometimes yield
superior results with smaller, specialized models. This underscores the
importance of empirical testing and careful model selection in developing
effective emotion recognition systems.

###  4.4 Evaluation of Speech Features Using Large Language Model (LLM)

Figure 3: Large Language Model (LLM) Prompt Template for speech-feature-only Emotion Detection. Method (IEMOCAP) | Weighted F-1
---|---
Random Guess | 16.67
3-class categories (ML) | 31.0
5-class categories (ML) | 31.333
numerical features (ML) | 32.0
3-class categories (LLM) | 27.602
5-class categories (LLM) | 27.895
speech impression (LLM) | 27.794
Table 3: Comparison of Weighted F-1 Scores for Different Classification
Methods on IEMOCAP

We compared the performance between numerical and categorical acoustic
features using simple ML models in Appendix A.2. Converting numerical features
to categorical ones didn’t significantly reduce performance and the speech
features of IEMOCAP performed significantly better than random guessing. We
found that audio quality is crucial for effective speech-based emotion
recognition. High-quality audio (IEMOCAP) allows for extraction of informative
speech features. Low-quality audio (MELD) makes speech features unreliable for
emotion detection.

The results in Table 3 demonstrate the potential of LLMs in leveraging speech
features for emotion classification. As a reference point, recall that the
5-class categorical representation with ML models achieved a weighted F1 score
of 31.333%, significantly outperforming the random guess baseline of 16.67%.
When using LLMs with natural language descriptions of speech characteristics,
we observe a slight decrease in performance compared to the ML approach. The
5-class categorical representation with LLMs yields a weighted F1 score of
27.895%, while the 3-class representation achieves 27.602%.

This performance gap between ML models and LLMs can be attributed to the
inherent differences in how these models process information. ML models,
designed specifically for discriminative tasks, can directly leverage the
structured, one-hot encoded categorical features. In contrast, LLMs receive
the information as natural language descriptions, which introduces a layer of
abstraction and potential for ambiguity. Despite this, the LLM’s performance
remains substantially above the random guess baseline, indicating that it
successfully extracts meaningful emotional cues from the speech descriptions.

The relatively small difference between 3-class and 5-class representations in
LLM performance (27.602% vs. 27.895%) suggests that increasing the granularity
of speech feature categorization does not significantly impact the LLM’s
ability to discern emotions. This could imply that LLMs are adept at
interpreting broader emotional cues rather than fine-grained acoustic details.

Notably, we also examined the performance of LLMs using only speech
impressions, which are derived from the 5-class categorical speech
descriptions but provide a more interpretative representation of the audio
features. This approach yields a weighted F1 score of 27.794%, slightly lower
than 5-class categorical descriptions. This finding aligns with our earlier
results when combining speech features with textual information, where speech
impressions also showed a marginal decrease in performance compared to direct
speech descriptions. The consistency of this pattern across different
experimental setups suggests that, while speech impressions offer a more
human-interpretable format, they may introduce a level of abstraction or
subjectivity that slightly reduces the model’s ability to accurately classify
emotions. This underscores the delicate balance between feature
interpretability and classification performance in emotion recognition tasks.

While there is some loss of information when transitioning from raw
categorical features to natural language descriptions, the LLM’s ability to
maintain performance well above the baseline demonstrates its potential in
multimodal emotion recognition tasks. This approach opens up possibilities for
more flexible and interpretable emotion recognition systems that can integrate
textual and acoustic information in a unified framework.

##  5 Conclusion

Our study demonstrates that integrating speech characteristics into LLM-based
emotion detection systems significantly improves performance, especially for
high-quality audio data. By translating audio features into natural language
descriptions, we enable LLMs to perform multimodal emotion analysis without
architectural changes.

Key findings show that incorporating speech descriptions enhances emotion
recognition accuracy across different LLM architectures. However, the
effectiveness of this approach heavily depends on audio quality, as evidenced
by the contrasting results between IEMOCAP and MELD datasets.

This research opens up new possibilities for developing more flexible and
interpretable emotion recognition systems that seamlessly integrate textual
and acoustic information. Future work should focus on improving feature
extraction methods for noisy, real-world audio data and exploring more
sophisticated ways of representing speech characteristics in natural language.

Our approach not only enhances the emotion detection capabilities of LLMs but
also provides a framework for incorporating non-textual information into
language model-based systems, with potential implications for a wide range of
multimodal AI applications.

## Acknowledgments

This research is supported in part by the Defense Advanced Research Projects
Agency (DARPA), via the CCU Program contract HR001122C0034. The views,
opinions and/or findings expressed are those of the authors and should not be
interpreted as representing the official views or policies of the Department
of Defense or the U.S. Government.

##  6 Limitation

Our approach to enhancing LLM-based emotion detection shows promise, but it is
important to acknowledge its limitations. A significant constraint is the
method’s heavy reliance on audio quality. As evidenced by the contrasting
results between the IEMOCAP and MELD datasets, our system’s performance
degrades considerably with noisy or low-quality audio input.

Another limitation lies in the current feature extraction process. Our
approach utilizes a limited set of audio features, which may not capture the
full spectrum of emotional nuances present in speech. More complex emotional
cues embedded in prosody, rhythm, or spectral characteristics could be
overlooked, potentially limiting the depth and accuracy of our emotional
analysis. This simplification of audio features may result in a loss of subtle
emotional information that humans can easily discern.

The nature of our training datasets presents another challenge. Our study
primarily relies on acted (IEMOCAP) or scripted (MELD) emotional expressions,
which may not fully represent the complexity and subtlety of emotions in
natural, spontaneous speech. This dataset bias could limit the
generalizability of our findings to real-world scenarios where emotions are
often more nuanced, mixed, or ambiguously expressed. The gap between performed
emotions in our datasets and the diverse, spontaneous emotional expressions in
everyday life may impact the real-world effectiveness of our emotion
recognition system.

Lastly, the computational resources required for this approach pose a
practical limitation. Fine-tuning large language models demands significant
computational power, which may restrict the implementation of this method in
resource-constrained environments. This requirement could limit the
accessibility and widespread adoption of our approach, particularly in
settings where high-performance computing resources are not readily available.

Addressing these limitations in future research will be crucial for developing
more robust, versatile, and widely applicable emotion recognition systems that
can handle the complexities of real-world emotional expressions across various
audio conditions and computational environments.

## References

  * Bai et al. (2024) Ye Bai, Jingping Chen, Jitong Chen, Wei Chen, Zhuo Chen, Chuang Ding, Linhao Dong, Qianqian Dong, Yujiao Du, Kepan Gao, Lu Gao, Yi Guo, Minglun Han, Ting Han, Wenchao Hu, Xinying Hu, Yuxiang Hu, Deyu Hua, Lu Huang, Mingkun Huang, Youjia Huang, Jishuo Jin, Fanliu Kong, Zongwei Lan, Tianyu Li, Xiaoyang Li, Zeyang Li, Zehua Lin, Rui Liu, Shouda Liu, Lu Lu, Yizhou Lu, Jingting Ma, Shengtao Ma, Yulin Pei, Chen Shen, Tian Tan, Xiaogang Tian, Ming Tu, Bo Wang, Hao Wang, Yuping Wang, Yuxuan Wang, Hanzhang Xia, Rui Xia, Shuangyi Xie, Hongmin Xu, Meng Yang, Bihong Zhang, Jun Zhang, Wanyi Zhang, Yang Zhang, Yawei Zhang, Yijie Zheng, and Ming Zou. 2024.  Seed-asr: Understanding diverse speech and contexts with llm-based speech recognition.  _Preprint_ , arXiv:2407.04675.
  * Baskar et al. (2024) Murali Karthick Baskar, Andrew Rosenberg, Bhuvana Ramabhadran, Neeraj Gaur, and Zhong Meng. 2024.  Speech prefix-tuning with rnnt loss for improving llm predictions.  _Preprint_ , arXiv:2406.14701.
  * Busso et al. (2008) Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008.  Iemocap: interactive emotional dyadic motion capture database.  _Language Resources and Evaluation_ , 42:335–359.
  * Fathullah et al. (2024) Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. 2024.  AudioChatLlama: Towards general-purpose speech abilities for LLMs.  In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_ , pages 5522–5532, Mexico City, Mexico. Association for Computational Linguistics.
  * Fu (2024) Yumeng Fu. 2024.  Ckerc : Joint large language models with commonsense knowledge for emotion recognition in conversation.  _Preprint_ , arXiv:2403.07260.
  * Gong et al. (2023) Taesik Gong, Josh Belanich, Krishna Somandepalli, Arsha Nagrani, Brian Eoff, and Brendan Jou. 2023.  Lanser: Language-model supported speech emotion recognition.  _arXiv preprint arXiv:2309.03978_.
  * Lei et al. (2024) Shanglin Lei, Guanting Dong, Xiaoping Wang, Keheng Wang, and Sirui Wang. 2024.  Instructerc: Reforming emotion recognition in conversation with a retrieval multi-task llms framework.  _Preprint_ , arXiv:2309.11911.
  * Li et al. (2024) Dongting Li, Chenchong Tang, and Han Liu. 2024.  Audio-llm: Activating the capabilities of large language models to comprehend audio data.  In _Advances in Neural Networks – ISNN 2024_ , pages 133–142, Singapore. Springer Nature Singapore.
  * Ma et al. (2024) Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, and Xie Chen. 2024.  An embarrassingly simple approach for llm with strong asr capacity.  _Preprint_ , arXiv:2402.08846.
  * Poria et al. (2019) Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019.  Meld: A multimodal multi-party dataset for emotion recognition in conversations.  _Preprint_ , arXiv:1810.02508.
  * Santoso et al. (2024) Jennifer Santoso, Kenkichi Ishizuka, and Taiichi Hashimoto. 2024.  Large language model-based emotional speech annotation using context and acoustic feature for speech emotion recognition.  In _ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 11026–11030.
  * Xu et al. (2023) Yaoxun Xu, Hangting Chen, Jianwei Yu, Qiaochu Huang, Zhiyong Wu, Shixiong Zhang, Guangzhi Li, Yi Luo, and Rongzhi Gu. 2023.  Secap: Speech emotion captioning with large language model.  _Preprint_ , arXiv:2312.10381.
  * Xue et al. (2024) Jieying Xue, Minh Phuong Nguyen, Blake Matheny, and Le Minh Nguyen. 2024.  Bioserc: Integrating biography speakers supported by llms for erc tasks.  _Preprint_ , arXiv:2407.04279.
  * Zhang et al. (2024a) Xiangyu Zhang, Hexin Liu, Kaishuai Xu, Qiquan Zhang, Daijiao Liu, Beena Ahmed, and Julien Epps. 2024a.  When llms meets acoustic landmarks: An efficient approach to integrate speech into large language models for depression detection.  _Preprint_ , arXiv:2402.13276.
  * Zhang et al. (2024b) Yazhou Zhang, Mengyao Wang, Youxi Wu, Prayag Tiwari, Qiuchi Li, Benyou Wang, and Jing Qin. 2024b.  Dialoguellm: Context and emotion knowledge-tuned large language models for emotion recognition in conversations.  _Preprint_ , arXiv:2310.11374.

##  Appendix A Appendix

###  A.1 Speech features for context

Method (IEMOCAP) | Weighted F-1
---|---
text-only | 70.111 $\pm$0.36
\+ speech des | 72.021 $\pm$0.54
\+ volume (all utterances) | 71.226 $\pm$0.25
\+ volume (the last three) | 71.487 $\pm$0.59
\+ pitch (all utterances) | 71.28 $\pm$0.47
\+ pitch (the last three) |  72.595 $\pm$0.26
\+ all features (the last three) | 71.461 $\pm$0.76
Table 4: Comparison of Weighted F-1 Scores for Different Methods of Adding
Speech-Feature Context on IEMOCAP. All experiments in the table use LoRA on
the LLaMA-2-7b-base model.

In our exploration of incorporating speech features into context utterances,
we found that selective application of these features yielded better results
than a blanket approach. As shown in Table 4, adding speech descriptions to
only the last three utterances in the context proved more effective than
applying them to all utterances. This suggests that recent context carries
more weight in emotion recognition. Notably, pitch features outperformed
volume features, with pitch information from the last three utterances
providing a 0.6% improvement over non-contextual speech descriptions,
achieving a weighted F1 score of 72.595%. Interestingly, combining all speech
features for context utterances did not enhance performance, likely due to
information overload. These findings highlight the importance of judicious
feature selection and context consideration in multimodal emotion recognition
tasks.

###  A.2 Evaluation of Speech Features Using Simple Machine Learning Models

To evaluate the effectiveness of our extracted speech features in capturing
emotion-related information, we conducted experiments using various machine
learning models on both numerical and categorical representations of these
features. Table 5 and Table 6 presents the performance of Random Forest (RF),
Support Vector Machine (SVM), and Multi-Layer Perceptron (MLP) models on the
IEMOCAP and MELD datasets.

Features | RF | SVM | MLP | Average
---|---|---|---|---
Random Guess | – | – | – | 0.167(0.167)
Numerical | 0.33(0.31) | 0.33(0.31) | 0.30(0.29) |  0.320(0.303)
3-class | 0.31(0.28) | 0.32(0.30) | 0.30(0.27) | 0.310(0.283)
4-class | 0.28(0.27) | 0.32(0.29) | 0.29(0.27) | 0.297(0.277)
5-class | 0.31(0.29) | 0.33(0.30) | 0.30(0.29) | 0.313(0.293)
6-class | 0.30(0.28) | 0.31(0.29) | 0.29(0.28) | 0.300(0.283)
Table 5: IEMOCAP: Performance comparison of discriminant ML models using different feature representations using weighted F1(Macro F1) scores Features | RF | SVM | MLP | Average
---|---|---|---|---
Random Guess | – | – | – | 0.143(0.143)
Numerical | 0.37(0.15) | 0.34(0.12) | 0.38(0.18) | 0.363(0.150)
3-class | 0.37(0.15) | 0.36(0.13) | 0.37(0.16) |  0.367(0.147)
4-class | 0.36(0.15) | 0.35(0.13) | 0.36(0.14) | 0.357(0.140)
5-class | 0.36(0.16) | 0.35(0.13) | 0.36(0.15) | 0.357(0.147)
6-class | 0.36(0.17) | 0.35(0.13) | 0.35(0.17) | 0.353(0.157)
Table 6: MELD: Performance comparison of discriminant ML models using
different feature representations using weighted F1(Macro F1) scores

Our analysis of the IEMOCAP dataset reveals that the simple five speech
features contain substantial emotion-related information. Both weighted F1 and
macro F1 scores (approximately 0.32 and 0.30, respectively) are nearly double
the random guess baseline of 0.167. This significant improvement over the
baseline indicates that our extracted speech features are effectively
capturing emotional cues. Interestingly, the conversion of numerical features
into categorical features does not result in a substantial loss of emotion-
related information. Moreover, dividing the numerical features into five
classes yields better performance than other granularity levels, with weighted
F1 and macro F1 scores of 0.313 and 0.293, respectively. This suggests that a
moderate level of discretization can reserve clear emotional patterns in
speech features.

The results for the MELD dataset highlight one crucial aspects of speech-based
emotion recognition: the importance of audio quality. While the weighted F1
scores (around 0.36) are higher than the random guess baseline of 0.143, the
macro F1 scores (approximately 0.15) are only marginally better than the
baseline. This discrepancy is crucial to understand: MELD is highly
imbalanced, with neutral emotions comprising about 50% of the dataset. The
high-weighted F1 score is misleading as it primarily reflects the model’s
ability to predict the dominant neutral class, rather than its capacity to
discern emotions from speech features. In contrast, the near-baseline macro F1
score, which treats all classes equally, reveals that the extracted speech
features provide minimal informative value across emotion categories. This
lack of substantial improvement in macro F1 scores indicates that the key
factor behind this poor performance is likely the low quality and challenging
nature of the audio in MELD. This finding underscores the critical importance
of audio quality in obtaining reliable speech features for emotion recognition
tasks. It also explains the modest gains observed when incorporating speech
descriptions in our earlier language model experiments on MELD, contrasting
sharply with the significant improvements seen in IEMOCAP.

In conclusion, these results not only validate the informativeness of our
extracted speech features for emotion detection but also emphasize the crucial
role of audio quality in this process. The stark contrast between IEMOCAP and
MELD results highlights that high-quality audio data, like that in IEMOCAP,
allows for the extraction of informative speech features that significantly
enhance emotion recognition. Conversely, challenging audio conditions, as in
MELD, can render speech features unreliable and minimally informative. This
underscores the need for robust feature extraction methods or pre-processing
techniques when dealing with real-world, noisy audio data. Despite these
challenges, the performance of categorical representations for IEMOCAP
supports our strategy of using discretized speech descriptions in language
model prompts for emotion recognition tasks, providing a foundation for future
improvements in handling diverse audio qualities.


11institutetext:  Aix Marseille Univ., CNRS, CNES, LAM, Marseille, France
22institutetext: Instituto de Astrofśica de Canarias, E-38205 La Laguna,
Tenerife, Spain 33institutetext: Departamento de AstrofÃsica, Universidad de
La Laguna (ULL), E-38206 La Laguna, Tenerife, Spain 44institutetext: Astronomy
Instrumentation Group, University of Cardiff, The Parade, CF24 3AA, United
Kingdom 55institutetext: Núcleo de Astronomía, Facultad de Ingeniería y
Ciencias, Universidad Diego Portales, Av. Ejército 441, Santiago, Chile
66institutetext: Univ. Grenoble Alpes, CNRS, Grenoble INP, Institut Néel,
38000 Grenoble, France 77institutetext: Univ. Grenoble Alpes, CNRS,
LPSC/IN2P3, 38000 Grenoble, France 88institutetext: European Southern
Observatory, Karl Schwarzschild StraÃe 2, 85748 Garching, Germany
99institutetext: European Southern Observatory, Alonso de Cordova 3107,
Vitacura, Santiago, Chile 1010institutetext: Univ. Grenoble Alpes, CNRS, IPAG,
38400 Saint Martin d’Héres, France 1111institutetext: School of Earth and
Space Exploration and Department of Physics, Arizona State University, Tempe,
AZ 85287, USA

# CONCERTO: instrument and status

Alessandro Fasano CONCERTO: instrument and statusCONCERTO: instrument and
statusCONCERTO: instrument and statusCONCERTO: instrument and statusCONCERTO:
instrument and statusCONCERTO: instrument and status<EMAIL_ADDRESS>Peter Ade CONCERTO: instrument and statusCONCERTO: instrument and status
Manuel Aravena CONCERTO: instrument and statusCONCERTO: instrument and status
Emilio Barria CONCERTO: instrument and statusCONCERTO: instrument and status
Alexandre Beelen CONCERTO: instrument and statusCONCERTO: instrument and
status Alain Benoît CONCERTO: instrument and statusCONCERTO: instrument and
status Matthieu Béthermin CONCERTO: instrument and statusCONCERTO: instrument
and status Julien Bounmy CONCERTO: instrument and statusCONCERTO: instrument
and status Olivier Bourrion CONCERTO: instrument and statusCONCERTO:
instrument and status Guillaume Bres CONCERTO: instrument and statusCONCERTO:
instrument and status Martino Calvo CONCERTO: instrument and statusCONCERTO:
instrument and status Andrea Catalano CONCERTO: instrument and statusCONCERTO:
instrument and status Carlos De Breuck CONCERTO: instrument and
statusCONCERTO: instrument and status FranÃois-Xavier Désert CONCERTO:
instrument and statusCONCERTO: instrument and status Carlos Durán CONCERTO:
instrument and statusCONCERTO: instrument and status Thomas Fenouillet
CONCERTO: instrument and statusCONCERTO: instrument and status Jose Garcia
CONCERTO: instrument and statusCONCERTO: instrument and status Gregory Garde
CONCERTO: instrument and statusCONCERTO: instrument and status Johannes Goupy
CONCERTO: instrument and statusCONCERTO: instrument and status Christopher
Groppi CONCERTO: instrument and statusCONCERTO: instrument and status
Christophe Hoarau CONCERTO: instrument and statusCONCERTO: instrument and
status Wenkai Hu CONCERTO: instrument and statusCONCERTO: instrument and
status Guilaine Lagache CONCERTO: instrument and statusCONCERTO: instrument
and status Jean-Charles Lambert CONCERTO: instrument and statusCONCERTO:
instrument and status Jean-Paul Leggeri CONCERTO: instrument and
statusCONCERTO: instrument and status Florence Levy-Bertrand CONCERTO:
instrument and statusCONCERTO: instrument and status Andreas Lundgren
CONCERTO: instrument and statusCONCERTO: instrument and status Juan Macías-
Pérez CONCERTO: instrument and statusCONCERTO: instrument and status Hamdi
Mani CONCERTO: instrument and statusCONCERTO: instrument and status Julien
Marpaud CONCERTO: instrument and statusCONCERTO: instrument and status Philip
Mauskopf CONCERTO: instrument and statusCONCERTO: instrument and status
Alessandro Monfardini CONCERTO: instrument and statusCONCERTO: instrument and
status Giampaolo Pisano CONCERTO: instrument and statusCONCERTO: instrument
and status Nicolas Ponthieu CONCERTO: instrument and statusCONCERTO:
instrument and status Leo Prieur CONCERTO: instrument and statusCONCERTO:
instrument and status Samuel Roni CONCERTO: instrument and statusCONCERTO:
instrument and status Sebastien Roudier CONCERTO: instrument and
statusCONCERTO: instrument and status Damien Tourres CONCERTO: instrument and
statusCONCERTO: instrument and status Carol Tucker CONCERTO: instrument and
statusCONCERTO: instrument and status

###### Abstract

CONCERTO (CarbON CII line in post-rEionization and ReionizaTiOn) is a low-
resolution Fourier transform spectrometer dedicated to the study of star-
forming galaxies and clusters of galaxies in the transparent millimeter
windows from the ground. It is characterized by a wide instantaneous 18.6
arcmin field of view, operates at 130–310 GHz, and was installed on the
12-meter Atacama Pathfinder Experiment (APEX) telescope at 5 100 m above sea
level. CONCERTO’s double focal planes host two arrays of 2 152 kinetic
inductance detectors and represent a pioneering instrument to meet a state-of-
the-art scientific challenge. This paper introduces the CONCERTO instrument
and explains its status, shows the first CONCERTO spectral maps of Orion, and
describes the perspectives of the project.

##  1 Introduction

Modern study of the reionization and post-reionization epochs of the Universe
and galaxy-cluster characterization requires the design of ambitious
instrumentation in the millimeter (mm) domain. Next-generation satellites
dedicated to mm wavelengths are currently being studied for the coming
decades. In the meantime, the international community is developing ground-
based experiments to improve our knowledge of the topics mentioned and define
new technologies that the satellite experiments would ultimately adopt. The
study of the reionization and post-reionization epochs of the Universe can be
pursued via the measurement of [CII] line emission lagache . [CII] is among
the brightest lines originating from star-forming galaxies and is a reliable
tracer of star formation on a global scale. It is red-shifted in the
transparent mm atmospheric windows at $z>5.2$ kovetz ; concerto and observed
via 3-D line intensity mapping. This technique adds the spectral information
in the third dimension in addition to the two sky coordinates, and follows the
line evolution with redshift. It measures the angular fluctuations in sky
brightness rather than solving the single galaxy structure and analyzes the
[CII] distribution on different spatial scales. [CII] intensity mapping is
also one of the main goals of CCAT-prime choi and TIME crites .

Galaxy clusters are now commonly observed at mm wavelengths via the
Sunyaev–Zelâdovich effect (SZ). The complementary mm information, together
with traditional X-ray observations, permits a more detailed comprehension of
cluster structure and its physical characterization, and adds cosmological
interest to galaxy-cluster science. Determining the distribution of clusters
of galaxies as a function of mass and redshift has proven to be a powerful
cosmological tool, particularly for inferring the matter content and dynamics
of the Universe. Again, a strategy for studying galaxy clusters requires a
multi-frequency capability by aiming to fully separate the different
components and extract physical information (pressure, temperature, density,
and line-of-sight velocity).

The mapping of [CII] at high redshift and galaxy clusters via SZ can be
achieved at same electromagnetic frequencies. Both scientific targets demand
high mapping speed combined with a multi-frequency capability. CONCERTO was
conceived to fulfill these needs; it requires a wide field-of-view (FoV)
Fourier transform spectrometer (FTS) operating at mm wavelengths.

This paper presents the CONCERTO instrument in Sect. 2. In Sect. 3, we present
the first CONCERTO spectral maps of the Orion Nebula. Section 4 illustrates
the project’s timeline, status, and perspectives.

##  2 The instrument and its main systematic effects

The CONCERTO instrument generates more than 16k spectra per second during
observations by performing on-the-fly (otf) scanning at tens of arcseconds per
second with an instantaneous FoV of 18.6 arcmin (see fasano_aa, ;
fasano_nika2,  for more details of the otf scanning technique while
continuously recording with the FTS). First, we must maximize the mapping
speed to guarantee the scientifically required coverage of a few square
degrees of sky demanded by the [CII] intensity mapping. Secondly, the sampling
rate has to be fast enough to compensate for atmospheric fluctuations.
CONCERTO comprises a fast FTS that records a full spectrum of the projected
FoV in sub-second time intervals, where the knee point of 1/f atmospheric
noise arises at $\lesssim 1$ Hz. A linear motor creates the optical path
difference (OPD) within the FTS by oscillating the position of a roof mirror,
and its stroke length defines the spectral resolution. This design creates an
interference figure called an interferogram. The single detector recovers all
the spectral information by scanning the interferogram. In fact, rather than
the single detector efficiency, the gain is obtained by adopting large format
arrays, in other words, by increasing the mapping speed. The FTS permits the
exploitation of large-array formats multiplexing the electromagnetic
frequencies. For CONCERTO, 4 304 kinetic inductance detectors (KIDs; doyle, )
are projected onto the sky and distributed in two focal planes. These are
orthogonally positioned facing the two sides of the last FTS polarizer which
differentiates the optical path in a transmitted and reflected direction
corresponding to the two focal planes. Two arrays, the low-frequency (LF) and
the high-frequency (HF) array, are installed at the focal plane positions. The
two arrays share the same design but are distinguishable by two different
filtering chains, resulting in overlapping bandpasses that do not cover the
same electromagnetic regions.

The instrumental characteristics of CONCERTO measured during the 2021
commissioning campaign on the sky are summarized in Table 1.

Table 1: Main characteristics of CONCERTO instrument coupled to the APEX telescope. fasano_spie Illuminated telescope primary mirror diameter | 11 m
---|---
Field-of-view diameter | 18.6 arcmin
Photometric beam widths HF $\mid$ LF |  $\sim$30 $\mid$ $\sim$35 arcsec
Absolute spectral resolution | settable and down to $\sim$1.5 GHz
Frequency range HF $\mid$ LF | 195–310 $\mid$ 130–270 GHz
Pixels on sky HF $\mid$ LF | 2 152 $\mid$ 2 152
Instrument geometrical throughput | 2.5$\times$10-3 sr m2
Single pixel geometrical throughput | 1.16$\times$10-6 sr m2
Data rate | 128 MB/s

The complexity of such an instrument demands a precise characterization of its
subsystems, a situation which presents peculiar challenges to the
interpretation of systematic effects, which, for the most part, have never
been tackled before owing to the pioneering nature of the project. These
problems must be understood, and innovative solutions have to be found. In
CONCERTO, one particular effect concerns the instrument: the roof mirror
motion that introduces the OPD in the FTS generates both a wind flow and a
vibrational resonance in the central beamsplitter membrane. This effect
translates into a complex perturbation of the OPD as a consequence of the
large (480$\times$800 mm2) beamsplitter area, an elliptic 50 $\mu$m-thick
membrane of polyimide on which lie 50 $\mu$m of copper wires with a pitch of
100 $\mu$m. This systematic was observed soon after the first commissioning
campaign and represented the main issue to be tackled during CONCERTO’s
commissioning. We needed a solution in order to monitor and mitigate this
effect. In December 2021, we installed a laser sensor pointing to the center
of the membrane to record the real-time OPD variations. In April 2022, we
improved the system with two active speakers programmed to generate counter
waves that cancel the vibrations. At the beginning of the commissioning, we
adopted the fastest acquisition rate utilized so far by recording a full
(forward/backward) motor stroke at 2.5 Hz. Soon after identifying the membrane
vibration issue, we slowed the motion to 1.9 Hz to reduce the impact of
vibration intensity and better sample the interferogram. Finally, we used the
slowest motion together with the activation of the counter system. The results
of these three set-ups are shown in Fig. 1, where the vibration measurement is
expressed in terms of power spectral density (PSD). The highest speed results
in a higher vibration frequency; in contrast, slowing down the motor stroke
motion drags the oscillation to a lower frequency with a reduced amplitude.
Finally, we successfully managed to mitigate the systematic effect by
exploiting a lower speed and the counter system.

---
Figure 1: Power spectral density (PSD) of the beamsplitter oscillation vs
frequency. In black is the 1536-point sampling (2.5 Hz); in blue is the
2048-point one (1.9 Hz); in red is the 2048-point configuration with the
counter system active. The main figure displays the zoom on the frequency
range of interest (46–48 Hz), while the subfigure on the top left shows the
whole frequency-domain range (up to 1906.5 Hz). fasano_spie

##  3 Spectral maps of Orion

CONCERTO was opened to the scientific community to observe the millimeter
Universe, together with the two main scientific drivers. In particular, we
present here an observation of the Orion Nebula in spectral mode. This project
is part of CONCERTO open time, program number 110.23NK.001, in the ESO period
P110. Orion is a reference region for extragalactic astronomy. It epitomizes
star-forming regions and is a template for starburst galaxies. The observing
program aims to measure dust emissivity in the mm domain galliano . The
central region of Orion, including the Trapezium McCaughrean , has an extent
of a quarter of a square degree. The NIKA2 map of the region shows a large
extended part of the map above a brightness level of 37 MJy/sr at 250 GHz
Ajeddig . With two hours of observation on-field, a 1-$\sigma$ sensitivity of
0.2 in the emissivity index per 30-arcsecond beam (CONCERTO-like) was
obtained.

The Orion Nebula was observed for 2 hours with 12 scans covering an area of
30$\times$60 arcmin2. The same configuration as for galaxy cluster
observations was adopted by setting the CONCERTO FTS stroke at 30 mm. The data
were calibrated using Uranus emission and bandpass measurements. The
calibrated photometric map is shown in Fig. 2. At this stage, this preliminary
calibration was cross-checked with Planck 217 GHz map. The CONCERTO map
reveals the center of Orion BN/KL, Orion South, the Orion Bar, and the North-
South main filament. The Orion Bar is conspicuous in the visible, as
demonstrated by Hubble Space Telescope observations. We find that, spectrally,
the Bar is dominated by free-free emission, while other regions are dominated
by Galactic dust emission. The spectral cube enables us to display Orion with
slices at different frequencies (see Fig. 2), covering 140 to 310 GHz where we
avoid contamination by the water vapor line emission at 180 GHz. These
spectral maps represent a sample of CONCERTO’s spectral mapping capabilities
of extended regions.

Figure 2: Spectral maps of Orion 30$\times$60 arcmin2. From left to right and
top to bottom: photometric map of Orion. Bins of 6 GHz around 200, 246, and
260 GHz central frequencies. These results demonstrate CONCERTO’s capability
of producing spectral maps of extended regions.

##  4 Timeline, status and perspectives

CONCERTO was granted an ERC advanced grant (grant agreement No78812) starting
in January 2019. The program was funded to design, fabricate, install, and
observe with the CONCERTO instrument at the APEX telescope gusten . The
instrument was shipped on 1 March 2021, and the team started the installation
on 6 April at the 12-m APEX telescope, located at an altitude of 5 100 m on
the Llano de Chajnantor in Northern Chile.111https://www.eso.org/public/teles-
instr/apex After four days (10 April), the cryogenic cooling down started and
reached nominal temperature at the focal plane two days after (12 April). The
technical commissioning then started. On 2 May, we performed the first
extended-source observation (the Crab
Nebula222https://www.eso.org/public/unitedkingdom/announcements/ann21010). The
observations started two days later remotely from France monfardini . The
commissioning phase was performed until the end of June 2021 catalano . The
instrument started its regular scientific program in July 2021.

CONCERTO was dismounted from the APEX telescope in May 2023 and is back in the
NÃel Institut (Grenoble, France), in the laboratory where it was designed,
fabricated, and assembled. About two years after the shipment, CONCERTO was
used for $\sim$793 hours in the [CII] intensity mapping program and for
$\sim$465 hours in 11 programs (6 SZ, 3 interstellar medium, and 2 evolved
stars). We stocked 50 days of data (174 TB with a compression factor equal to
5). The collaboration has entered the phase of data exploitation to achieve
the scientific targets. Several aspects are being addressed to refine the data
processing (e.g., the OPD fine reconstruction, the FTS reference
characterization, the KID off-resonance response, and the atmospheric emission
contamination). We are also exploiting our development of CONCERTO instrument
model, a fundamental tool to properly handle the various systematic effects in
terms of interpretation, comprehension, and development of analysis tools to
mitigate their effect. Description of the CONCERTO instrument model will be
the topic of a forthcoming paper.

In conclusion, this paper has presented the CONCERTO project and instrument,
and the delivery of the first spectral maps. Although preliminary in nature,
these observations clearly show CONCERTO’s considerable potential in the mm
domain. These maps, with finer calibration and further study, will be
presented in future publications.

_Acknowledgements._ The KID arrays described in this paper have been produced
at the PTA Grenoble microfabrication facility. CONCERTO project and this work
have been supported by the LabEx FOCUS ANR-11-LABX-0013, the European Research
Council (ERC) under the European Unionâs Horizon 2020 research and innovation
program (project CONCERTO, grant agreement No 788212), and the Excellence
Initiative of Aix-Marseille University-A*Midex, a French âInvestissements
dâAvenirâ program.

## References

  * (1) G. Lagache, Proceedings IAU Symposium 333, 228 (2018).
  * (2) Kovetz, E. D. et al., 2017, arXiv e-prints, arXiv:1709.09066 (2017)
  * (3) CONCERTO collaboration, A&A 642, 60 (2020)
  * (4) S. K. Choi et al., J. Low Temp. Phys. 199, 1089 (2020)
  * (5) A. T. Crites et al., Proc. SPIE, SPIE Conf. Ser. 9153, 91531W (2014)
  * (6) A. Fasano et al., A&A 656, A116 (2021)
  * (7) A. Fasano et al., European Physical Journal Web of Conferences 257, 17 (2022)
  * (8) S. Doyle et al., Millimeter, Submillimeter, and Far-Infrared Detectors and Instrumentation for Astronomy V 7741, 77410M (2010)
  * (9) A. Fasano et al., Millimeter, Submillimeter, and Far-Infrared Detectors and Instrumentation for Astronomy XI 12190, 9 (2022)
  * (10) F. Galliano et al., ARA&A 56, 673 (2018)
  * (11) M. J. McCaughrean et al., Astronomical Journal 108, 1382 (1994)
  * (12) H. Ajeddig et al., European Physical Journal Web of Conferences 257, 2 (2022)
  * (13) R. GÃsten et al., A&A 454, L13 (2006)
  * (14) A. Monfardini et al., Journal of Low Temperature Physics 209, 751 (2022)
  * (15) A. Catalano et al., European Physical Journal Web of Conferences 257, 10 (2022)