# Evaluating Large Language Models Using Contrast Sets: An Experimental
Approach

Manish Sanwal
University of Texas at Austin
<EMAIL_ADDRESS>

###### Abstract

In the field of Natural Language Inference (NLI), particularly for multi-input
text classification tasks, Cross-Entropy Loss is commonly used as a general
error metric. Although effective as a training benchmark, this metric does not
adequately assess a model’s understanding of language entailments. In this
work, we propose a novel approach for creating a contrast set for the Stanford
Natural Language Inference (SNLI) dataset (Bowman et al., 2015). Our method
involves automatically replacing verbs, adverbs, and adjectives with their
synonyms, maintaining the original sentence meaning. This approach helps
determine whether a model truly comprehends the language or merely identifies
recurring patterns for making predictions. We utilized the ELECTRA-small model
(Clark et al., 2020) for our investigation. While the model exhibits an 89.9%
accuracy on the standard SNLI dataset, its performance drops to 72.5% on our
contrast set—a significant 17% decrease. This finding prompted an in-depth
analysis to understand the underlying learning patterns of the model.
Subsequently, we enhanced the model’s robustness by fine-tuning it with a
contrast training dataset tailored for SNLI, resulting in an improved accuracy
of 85.5% on contrast sets. These experiments underscore the necessity for more
balanced datasets in NLI tasks that account for varied linguistic expressions.
We anticipate that our findings will inspire further development of
comprehensive datasets, fostering the advancement of more nuanced and
effective NLI models.

##  1 Introduction

Language entailment tasks have long stood as pivotal challenges in the field
of natural language processing (NLP). With the advent of deep learning
techniques, particularly the development of transformer models Vaswani et al.
(2017), the landscape of these tasks has undergone a significant
transformation. State-of-the-art models are now achieving scores that rival,
and in some cases surpass, human evaluators. This remarkable progress,
however, predominantly hinges on conventional evaluation metrics, chiefly
accuracy against established validation datasets.

Despite these advances, a growing body of literature Jia and Liang, 2017
(Gardner et al., 2020) etc. suggests that these models, while effective, might
succeed by capitalizing on superficial patterns prevalent in test examples
rather than attaining a profound understanding of language nuances. This
phenomenon raises critical questions about the depth and authenticity of the
language understanding these models claim to achieve.

In our research, we introduce a novel method for assessing Natural Language
Inference (NLI) models. This approach uniquely incorporates the use of a
contrast set alongside traditional validation datasets. It is specifically
applied to the Stanford Natural Language Inference (SNLI) dataset (Bowman et
al., 2015), which challenges models to analyze a given premise and hypothesis,
determining if their relationship is one of entailment, contradiction, or
neutrality. Our innovative approach, which integrates contrast sets into the
evaluation process, adds a new dimension to the scrutiny of NLI models.

Our methodology aims to rigorously test models not just on their ability to
predict these relationships in standard settings but also on their robustness
and adaptability to nuanced, contextually varied scenarios presented in the
contrast set. This dual-layered evaluation framework is designed to probe the
models’ deeper language understanding capabilities, moving beyond mere pattern
recognition to a more authentic and comprehensive comprehension of linguistic
constructs.

Building on the premise of our initial investigation, the second phase of our
experiment pivoted towards enhancing the model’s performance on the contrast
set. Recognizing the potential limitations in the current training regimes, we
hypothesized that a more nuanced and diverse training dataset could
significantly bolster the model’s ability to handle the complexity inherent in
our contrast set.

To test this hypothesis, we employed a technique known as "Data Augmentation."
This approach involved creating an expanded training dataset, which we refer
to as the "contrast training set." This new set was designed to mimic the
diversity and complexity of the contrast set, thereby exposing the model to a
broader spectrum of linguistic scenarios during its training phase. Our
objective was to not just train the model on standard examples but to also
fine-tune it on examples that closely align with the challenging scenarios
presented in the contrast set.

The results of this data augmentation strategy were promising. By training the
model on this enriched dataset, we observed a notable improvement in its
performance. Specifically, the model’s accuracy on the contrast set surged to
an impressive 87.5%. This marked increase underscored the effectiveness of our
data augmentation approach, highlighting its potential as a powerful tool for
enhancing the sophistication and adaptability of NLI models.

This experiment demonstrates the significance of training data diversity in
developing models that not only perform well on standard benchmarks but also
possess a robust understanding of nuanced language contexts. By integrating
these more complex and varied training examples, we have taken a step towards
creating models that more closely mirror the depth and flexibility of human
language comprehension.

##  2 Model and the Dataset

###  2.1 The Model

For our investigation, we utilized the ELECTRA-small model (Clark et al.,
2020). Originally, the Electra model was conceived as a discriminator capable
of differentiating between "real" input tokens and "fake" ones generated by
another neural network, akin to the discriminator in a Generative Adversarial
Network (GAN). Remarkably, at a small scale, ELECTRA demonstrates robust
performance even on a single GPU. When scaled up, it achieves state-of-the-art
results on the SQuAD 2.0 dataset (Rajpurkar et al., 2018). Additionally, this
model can be fine-tuned for various downstream tasks, including
classification, sequence tagging, and language entailment.

####  2.1.1 Standard Evaluation Method

For a classification problem with $C$ classes, the cross-entropy loss for a
single instance can be expressed as:

$\text{CrossEntropy}=-\sum_{c=1}^{C}y_{c}\log(p_{c})$ (1)

In this context, $y_{c}$ denotes a binary indicator, assuming values of either
0 or 1, to establish whether the class label $c$ accurately represents the
classification of a given observation. Concurrently, $p_{c}$ represents the
predicted probability that the observation is correctly classified under class
$c$. Considering a dataset comprising $N$ instances, the aggregate cross-
entropy loss is typically expressed as the mean loss across all instances.

$\text{Error}=\frac{1}{N}\sum_{n=1}^{N}\text{CrossEntropy}(y_{nc},p_{nc})$ (2)

In this equation, $y_{nc}$ and $p_{nc}$ refer to the true label and the
predicted probability for class $c$ in instance $n$, respectively.

###  2.2 The DataSet

The SNLI corpus, in its 1.0 version (Bowman et al., 2015), comprises a dataset
of 570,000 pairs of sentences, all written in English and manually annotated
for a balanced classification task. The sentences are categorized into three
labels: entailment, contradiction, and neutral, which are fundamental to the
study of Natural Language Inference (NLI), also referred to as Recognizing
Textual Entailment (RTE). The dataset primarily sources its language from
English-speaking Flickr users and crowdworkers on Amazon Mechanical Turk. Each
data entry in the corpus includes a premise and a hypothesis, both represented
as strings, along with an integer label that indicates the category of the
sentence pair.

####  2.2.1 Data Fields

premise: This is a string that serves as a basis for evaluating the validity
of the hypothesis.
hypothesis: A string that is to be assessed for its truthfulness, falsehood,
or indeterminacy in relation to the premise.
label: An integer used to categorize the relationship between the premise and
the hypothesis. A value of 0 signifies that the hypothesis is entailed by the
premise, 1 indicates neither entailment nor contradiction between the premise
and hypothesis, and 2 denotes that the hypothesis contradicts the premise.

⬇

1{

2 "premise": "A person on a horse jumps over a broken down airplane.",

3 "hypothesis": "A person is training his horse for a competition.",

4 "label": 1

5}

##  3 Contrast set Evaluation

###  3.1 General Framework

A model that depends on basic cues but lacks a deep understanding of language
can perform effectively in terms of average accuracy, particularly when these
cues are often predictive. (Weissenborn et al., 2017) suggest that numerous
SQuAD questions can be resolved through elementary heuristics, specifically
those focusing on type and keyword-matching. To assess if current models have
acquired knowledge beyond these rudimentary patterns, we propose a contrast
set designed to challenge inadequate models by modifying test examples.
Consider following example.

Original Premise: One little toddler wearing blue pajamas and a little girl
wearing red pajamas opening presents on christmas day., Original Hypothesis:
Two small children are unhappy with the opening of their presents., Original
Label: 1 Updated Hypothesis: Two minor children are dejected with the opening
of their presents. Predicted Label: 0

In the above case a human evaluator can easily understand the updated
hypothesis and can predict that the hypothesis neither entail nor contradicts
the premise but the model predicts that the hypothesis entails the premise.
Lets define a contrast set $A$ to be a function that takes in a premise $p$,
and hypothesis $h$ and optionally with model $f$ to generates a contrasting
example ($p$, $h^{\\\ ^{\prime}}$).

$(p,h^{\\\ ^{\prime}})=A(p,h,f)$ (3)

As we have a model $f$ to predict the probability for each class using the
contrast example

$p_{nc}=f(p,h^{\\\ ^{\prime}})$ (4)

and with the predict probability $p_{nc}$ we can define contrast cross entropy
error as

$\text{Contrast
Error}=\frac{1}{N}\sum_{n=1}^{N}\text{CrossEntropy}(y_{nc},p_{nc})$ (5)

###  3.2 Contrast set creation

Generating contrast sets in the field of natural language processing presents
a significant challenge, a stark contrast to the relatively simpler process in
image classification. In image classification, one can create a contrasting
example by introducing a minimal, almost imperceptible amount of noise to the
original input, as described by (Goodfellow et al., 2015). These minute
adjustments, while not altering the fundamental semantics or meaning of the
image, are capable of influencing the predictions of models, particularly
those that are excessively sensitive to changes that do not affect the
underlying semantic content.

In the realm of natural language processing, the comparable strategy would
involve paraphrasing the original input text, a concept explored by (Madnani
and Dorr, 2010). Paraphrasing aims to rewrite the text while preserving its
original meaning, effectively creating a semantic mirror of the original input
that differs only in its phrasing or structure. However, this task is fraught
with complexities. High-precision paraphrase generation is a sophisticated
endeavor, as it requires maintaining the delicate balance of meaning in the
face of linguistic modifications. Unlike in image processing, where the
alterations are subtle and non-semantic, changes in language, even minor ones,
often lead to a shift in meaning. Thus, most attempts to edit or modify a
sentence in the pursuit of paraphrase generation inadvertently result in a
change in its original meaning, presenting a substantial hurdle in the
development of robust natural language processing models that can reliably
handle such semantically-preserving transformations.

In our experiment, we employed a straightforward approach, focusing on the
automated substitution of select keywords in the hypothesis with their
synonyms. This method, while not always yielding grammatically correct
sentences, has shown promising results. During our trials, we observed that
despite some grammatical inconsistencies, human evaluators were still able to
accurately determine the correct label for these sentences.

To create the contrast set, we used the validation set from the Stanford
Natural Language Inference (SNLI) database. The SNLI database, with its
comprehensive collection of sentence pairs, provided a robust foundation for
applying our synonym substitution technique. This choice ensured that our
experiment was grounded in a dataset reflective of real-world language usage.
We utilized the Natural Language Toolkit (nltk) library to facilitate this
process. The initial step involved applying a part-of-speech tagger to the
hypothesis from the SNLI validation set. This tagging helped us identify the
specific grammatical roles of words within the sentences. Recognizing that
altering nouns could significantly change the sentence’s meaning, our strategy
specifically targeted Verbs, Adjectives, and Adverbs for synonym replacement.
Upon identifying these target words, we leveraged nltk’s integration with the
WordNet database to find potential synonyms. For each identified Verb,
Adjective, or Adverb, we compiled a list of synonyms and then randomly
selected one to replace the original word in the sentence.

After generating the initial contrast set, we undertook a manual review
process. This involved carefully parsing each sentence in the contrast set,
making necessary adjustments to ensure grammatical correctness and overall
coherence. This manual refinement was essential to ensure that the sentences
not only maintained their core meaning but also sounded natural and correct.
This method of synonym replacement, augmented by manual review, proved
effective for generating variations of the hypothesis. It allowed us to test
the robustness of natural language processing models against semantic-
preserving linguistic variations, a vital aspect in evaluating these models’
adaptability and accuracy in processing human language.

###  3.3 Evaluation on contrast set

In evaluating our model on the contrast set, we observed some crucial insights
into its language processing capabilities. Our approach involved creating a
contrast set by automatically replacing verbs, adverbs, and adjectives in the
Stanford Natural Language Inference (SNLI) dataset with their synonyms,
maintaining the original sentence meaning. This methodology was aimed at
determining whether the model genuinely comprehends language or simply
identifies recurring patterns for predictions.

Figure 1: Accuracy comparison between validation set and contrast set

Our initial findings using the ELECTRA-small model revealed a significant
discrepancy between its performance on the standard SNLI dataset (89.9%
accuracy) and the contrast set (72.5% accuracy). This 17% decrease highlighted
the model’s limitations in handling nuanced language variations.

To address this, we fine-tuned the model with a contrast training dataset
tailored for SNLI. This enhancement led to an improved accuracy of 85.5% on
the contrast sets, emphasizing the need for more balanced datasets in Natural
Language Inference (NLI) tasks that account for varied linguistic expressions.
These findings suggest that current NLP datasets should increasingly be
created with contrast sets in mind to encapsulate the variability and nuance
in natural human languages.

##  4 Error Analysis

To illustrate how our model can be confounded by simple word replacements,
consider the following examples:

###  4.1 Example 1

Original Premise: A man wearing a red bicycle helmet talking on a cellphone
with other people in the background. Original Hypothesis: A man talks on a
cellphone. Original Label: 0 (entailment) Updated Hypothesis: A man babbles on
a cellphone. Predicted Label: 1 (neutral)

In this instance, we replaced the word "talks" with "babbles" in the
hypothesis. While "babble" does not exactly convey the same meaning as "talk,"
a human evaluator could still discern the entailment between the premise and
the updated hypothesis. However, the model incorrectly predicted a
contradiction, assigning a label of 1 instead of 0.

This misinterpretation by the model suggests that it relies heavily on pattern
matching between the premise and the hypothesis. In the original hypothesis,
the phrase "talking on a cellphone" in the premise directly aligns with "talks
on a cellphone." However, when "talks" was replaced with "babbles," this
direct pattern was disrupted. Although "babbles" is a synonym that preserves
the core action of speaking on a cellphone, the model failed to recognize this
subtlety.

This example highlights a critical shortcoming in the model’s understanding of
natural language. It points to an over-reliance on direct word matching,
rather than a deeper comprehension of linguistic nuances and synonyms. Such
limitations can lead to incorrect interpretations, especially in cases where
synonyms or paraphrased language are used. It underscores the importance of
enhancing the model’s ability to recognize and process semantic-preserving
variations in language, a key aspect for achieving more accurate and reliable
natural language processing.

###  4.2 Example 2

Original Premise: A woman in a pink shirt is painting the last two stars on a
painting of the American flag. Original Hypothesis: The woman is sewing.
Original Label: 2 (contradiction) Updated Hypothesis: The woman is a tailor.
Predicted Label: 1 (neutral)

In this scenario, the original premise describes a woman engaged in painting,
specifically adding stars to an American flag painting. The original
hypothesis incorrectly suggests that she is sewing, leading to a label of 2,
indicating a contradiction. This is because sewing is a distinctly different
activity from painting.

However, when the hypothesis is updated to "The woman is a tailor," the
model’s prediction changes to a label of 1, indicating a neutral relationship
between the premise and hypothesis. This shift in prediction is intriguing.
"Tailor" is a profession typically associated with sewing, but the statement
"The woman is a tailor" does not directly contradict the premise of her
painting. It’s possible that a woman can be a tailor and also engage in
painting, hence the model’s classification of the relationship as neutral.

This example demonstrates the model’s challenge in processing contextual
information and understanding the subtleties of language. While the updated
hypothesis does not directly describe the action depicted in the premise, it
also doesn’t outright contradict it, leading the model to a middle ground of
neutrality. Such examples underscore the importance of enhancing the model’s
capability to discern and interpret nuanced and contextual differences in
language.

###  4.3 Example 3

Original Premise: An old woman in a green and blue plaid shirt is standing on
a crowded sidewalk smiling. Original Hypothesis: A woman in a plaid shirt
smiling on a crowded sidewalk. Original Label: 0 (entailment) Updated
Hypothesis: A woman in a tartan shirt grin on a crowded sidewalk. Predicted
Label: 1 (neutral)

Here is one more example that shows how subtle changes in language can impact
the model’s interpretation and classification. The original premise describes
an old woman in a specific type of shirt (plaid), positioned on a crowded
sidewalk and smiling. The original hypothesis closely mirrors this
description, leading to an entailment label (0). This is because the
hypothesis accurately captures the essence of the premise, with both
statements aligning in terms of the woman’s attire and action.

However, the updated hypothesis introduces two subtle changes: it replaces
"plaid" with "tartan" and changes "smiling" to "grins." Tartan is essentially
a type of plaid, so this change doesn’t significantly alter the meaning.
Similarly, "grins" is a synonym of "smiling" and conveys the same general
expression of happiness. Despite these changes being semantically similar to
the original words, the model’s prediction shifts to a neutral label (1).

This shift suggests that the model is sensitive to even minor lexical changes,
impacting its ability to recognize semantic equivalence. Although "tartan" and
"plaid" are closely related, and "grins" is a form of smiling, the model does
not seem to equate these synonyms completely with their counterparts in the
original hypothesis. This could indicate a limitation in the model’s
understanding of synonyms and contextual language use.

Moreover, these examples underscores the challenge of ensuring that natural
language processing models can accurately interpret and respond to the rich
variability of human language. It highlights the necessity for models to not
only recognize direct word-to-word matches but also to understand the broader
context and synonymous relationships within language. Enhancing this
capability is crucial for developing more reliable and robust models capable
of processing language with the same level of nuance and understanding as
humans.

##  5 Improving the model’s predictions

The examples highlighted above demonstrate a significant challenge in natural
language processing: the robustness of machine learning models in the face of
the inherent variability of human language. This issue is illustrated through
traditional evaluation methods, as shown in the provided schematic. These
methods often involve training models on well-known datasets, such as the
Stanford Natural Language Inference (SNLI) corpus, and subsequently testing
them solely on the SNLI dataset. This approach aims to assess the model’s
ability to generalize based on the patterns it has learned. However, this
method often leads to a superficial assessment of model performance, as it
doesn’t consider the model’s capability to handle linguistic variations that
are distinct from those in the training set. This limitation highlights the
models’ tendency to rely on pattern recognition rather than achieving a
deeper, more nuanced understanding of language.

The proposed methodology advocates for an iterative fine-tuning process using
the contrast set post initial training. This approach, grounded in the
hypothesis that exposure to a broader linguistic spectrum during fine-tuning
will enhance the model’s grasp of complex language features such as synonyms
and word mappings, aims to reform the model’s predictive capabilities on both
the SNLI and contrast sets. The anticipated outcomes suggest a more balanced
performance, indicative of a model that has transcended mere pattern
recognition to encapsulate a more profound linguistic comprehension.

Figure 2:  Proposed Evaluation method

This fine-tuning strategy is not without precedent. It resonates with the
findings of previous literature, notably the work documented by Liu et al.,
2019 , which elucidates the benefits of employing contrast sets for model
evaluation and enhancement. Such contrasting fine-tuning not only aligns with
the continual quest for models that truly comprehend language but also serves
as a testament to the iterative nature of machine learning—a field where model
refinement is as crucial as model development. The integration of contrast
sets thus represents a methodological evolution, promising strides toward
models that genuinely mirror the complexity and unpredictability of human
language.

###  5.1 Model Improvement results

To enhance the performance of our model on the contrast dataset, we devised a
strategy to construct a targeted contrast training set. This initiative was
underpinned by the utilization of the SNLI training corpus, which we subjected
to a contrast set generation procedure analogous to the one delineated
previously. Our objective was to fine-tune the model using a strategically
minimal dataset. We embarked on this process by generating a contrast set
derived from a randomly selected subset of 1,000 examples from the SNLI
training data. Subsequent to this, we assessed the model’s validation accuracy
on both the newly formed contrast set and the original validation set (Liu et
al., 2019).

This methodical approach allowed us to incrementally amplify the size of the
contrast training set. By doing so, we could meticulously monitor the
relationship between the expanding training dataset and the corresponding
shifts in validation and contrast set accuracies. Our intention was to discern
the threshold at which additional contrast examples ceased to yield
proportional enhancements in the model’s performance, thus identifying the
optimal size of the contrast training set. This incremental expansion was not
merely a quantitative exercise but a strategic endeavor to refine the model’s
competence in generalization, thereby aligning its performance more closely
with the nuanced complexities of natural language as encountered in the real
world. Through this meticulous process, we sought to strike a balance between
the breadth of the contrast training set and the depth of understanding that
the model exhibits, ensuring that each additional example contributes
meaningfully to the model’s linguistic acuity.

Following is the result of our expariment:

Figure 3:  Accuracy on Contrast set and unperturbed validation set with fine
tuning on contrast training examples

The observation that the model’s accuracy on the contrast set improved
steadily, while the validation accuracy remained constant as the number of
contrast training samples increased, is a significant finding. This trend
indicates that the model is effectively learning the nuances and variability
inherent in natural language. Typically, contrast sets are designed to test a
model’s understanding of subtle language differences, often challenging it
with examples that are syntactically similar but semantically different from
the training data.

In our case, the constant validation accuracy suggests that the model’s
general understanding of the language, as represented by the validation set,
did not change. This stability is a positive sign, as it implies that the
model did not lose its general language understanding while adapting to the
nuances captured in the contrast set.

Reaching an 85.5% accuracy on the contrast set is noteworthy. It implies that
the model has a strong ability to discern and correctly interpret the more
subtle and complex aspects of language that are often presented in these sets.
This level of accuracy suggests that the model is not merely memorizing
training examples but is instead developing a more nuanced understanding of
language.

In conclusion, the increase in contrast set accuracy, coupled with stable
validation accuracy, indicates a successful adaptation of the model to more
complex language tasks without compromising its general language performance.
This balance is crucial for developing robust NLP models capable of handling a
wide range of real-world language scenarios.

##  6 Discussion and related work

The performance of current machine learning systems for reading comprehension,
though seemingly effective when measured by standard evaluation metrics, is
notably deficient under adversarial evaluation. This discrepancy arises
because standard evaluations tend to be overly forgiving towards models that
depend on surface-level cues, failing to challenge their deeper understanding
of the text. In comparison, evaluation using contrast sets demonstrates that
these models exhibit excessive stability to changes that significantly alter
the text’s meaning, despite appearing minor.

Our evaluation framework was designed to rigorously test models not just on
their ability to predict relationships in standard settings but also on their
robustness and adaptability to nuanced, contextually varied scenarios
presented in the contrast set. This dual-layered evaluation approach probes
the models’ deeper language understanding capabilities, moving beyond mere
pattern recognition to a more authentic and comprehensive comprehension of
linguistic constructs.

This observation strongly suggests that when constructing NLP datasets, there
should be a deliberate focus on incorporating contrast sets. These sets are
crucial as they encompass the diversity and subtlety inherent in natural human
language. By integrating such sets, the datasets will not only challenge but
also train models to recognize and understand the finer nuances and variations
of language, moving beyond superficial pattern recognition. This approach is
essential for developing more sophisticated and genuinely competent NLP
systems that can handle the complexity and richness of human language.

## References

  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015.  A large annotated corpus for learning natural language inference.  _CoRR_ , abs/1508.05326.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  In _International Conference on Learning Representations_.
  * Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020.  Evaluating NLP models via contrast sets.  _CoRR_ , abs/2004.02709.
  * Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015.  Explaining and harnessing adversarial examples.
  * Jia and Liang (2017) Robin Jia and Percy Liang. 2017.  Adversarial examples for evaluating reading comprehension systems.  _CoRR_ , abs/1707.07328.
  * Liu et al. (2019) Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019.  Inoculation by fine-tuning: A method for analyzing challenge datasets.  _CoRR_ , abs/1904.02668.
  * Madnani and Dorr (2010) Nitin Madnani and Bonnie J. Dorr. 2010.  Generating phrasal and sentential paraphrases: A survey of data-driven methods.  _Computational Linguistics_ , 36(3):341–387.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for squad.  _CoRR_ , abs/1806.03822.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _CoRR_ , abs/1706.03762.
  * Weissenborn et al. (2017) Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017.  Fastqa: A simple and efficient neural architecture for question answering.  _CoRR_ , abs/1703.04816.


Send correspondence to Narsireddy Anugu: E-mail<EMAIL_ADDRESS>
# CHARA/Silmaril Instrument Software and Data Reduction Pipeline:
Characterization of the Instrument in the Lab and On-Sky

Narsireddy Anugu  The CHARA Array of Georgia State University, Mount Wilson
Observatory, Mount Wilson, CA 91023, USA  Theo A. ten brummelaar  The CHARA
Array of Georgia State University, Mount Wilson Observatory, Mount Wilson, CA
91023, USA  Cyprien Lanthermann  The CHARA Array of Georgia State University,
Mount Wilson Observatory, Mount Wilson, CA 91023, USA  Peter G. Tuthill
Sydney Astrophotonics Instrumentation Laboratory, School of Physics, The
University of Sydney, Sydney, NSW, 2006, Australia  Edgar R. Ligon III  The
CHARA Array of Georgia State University, Mount Wilson Observatory, Mount
Wilson, CA 91023, USA  Gail H. Schaefer  The CHARA Array of Georgia State
University, Mount Wilson Observatory, Mount Wilson, CA 91023, USA  Douglas R.
Gies  The CHARA Array of Georgia State University, Mount Wilson Observatory,
Mount Wilson, CA 91023, USA  Grace Piroscia  Sydney Astrophotonics
Instrumentation Laboratory, School of Physics, The University of Sydney,
Sydney, NSW, 2006, Australia  Adam Taras  Sydney Astrophotonics
Instrumentation Laboratory, School of Physics, The University of Sydney,
Sydney, NSW, 2006, Australia  Gerard T. van Belle  Lowell Observatory, 1400
West Mars Hill Road, Flagstaff, AZ 86001, USA  Makoto Kishimoto  Department of
Astrophysics & Atmospheric Sciences, Kyoto Sangyo University, Kamigamo-
motoyama, Kita-ku, Kyoto 603-8555, Japan  Marc-Antoine Martinod  Sydney
Astrophotonics Instrumentation Laboratory, School of Physics, The University
of Sydney, Sydney, NSW, 2006, Australia

###### Abstract

The newly installed Silmaril beam combiner at the CHARA array is designed to
observe previously inaccessible faint targets, including Active Galactic
Nuclei and T-Tauri Young Stellar Objects. Silmaril leverages cutting-edge
optical design, low readout noise, and a high-speed C-RED1 camera to realize
its sensitivity objectives. In this presentation, we offer a comprehensive
overview of the instrument’s software, which manages critical functions,
including camera data acquisition, fringe tracking, automatic instrument
alignment, and observing interfaces, all aimed at optimizing on-sky data
collection. Additionally, we offer an outline of the data reduction pipeline,
responsible for converting raw instrument data products into the final OIFITS
used by the standard interferometry modeling software. Finally, a thorough
analysis of the camera and instrument characterization results will be
presented, evaluating instrument performance in terms of sensitivity. The
purpose of this paper is to provide a solid reference for studies based on
Silmaril data.

###### keywords:

long baseline interferometry, CHARA, Silmaril, eAPF, C-RED One

##  1 Introduction

Silmaril is a new instrument installed at the CHARA Array [1], designed to
combine light from $2\times 3$ telescopes operating in the near-infrared H and
K bands simultaneously. The details of its optical design, simulations, and
operational concepts are described in the 2022 and 2024 SPIE proceedings [2,
3]. We summarize the key instrument concepts here. CHARA already has six
telescope beam combiners operating at different wavelengths (R [4], JH [5],
and K [6] bands). MIRC-X and MYSTIC can observe targets up to $H/K\lessapprox
7.5$. Silmaril aims to access targets much fainter than this limit. The
detailed scientific goals of Silmaril are outlined in Lanthermann et al.
(2022) [2]. In essence, Silmaril aims observations of previously inaccessible
objects like Active Galactic Nuclei and faint young stellar objects (T-Tauri
stars). The instrument sensitivity is achieved through a combination of
cutting-edge design elements. Silmaril prioritizes maximizing the amount of
light collected, over achieving higher spectral resolution. Unlike other CHARA
combiners, it utilizes bulk optics instead of integrated optics and single-
mode fibers to limit the light coupling losses. Furthermore, by minimizing the
number of optical components in the light path, this design aims increased
throughput efficiency. To minimize thermal (blackbody) background radiation,
Silmaril employs a two-pronged approach: a very long focal length cylindrical
mirror ($\sim 5.4$ m) and a smaller cold stop aperture ($F/20$) inside the
camera. Additionally, we are planning to implement a technique called
“Narcissus reflection” to further reduce blackbody radiation[3, 7]. This
technique blocks unwanted radiation from high-emissivity areas within the
instrument, preventing it from reaching the detector. Finally, to minimize
readout noise, Silmaril leverages the proven C-RED1 detector system used in
MIRC-X [5] and MYSTIC [6].

Silmaril was installed at the CHARA lab in early 2023. By June 2023, we
successfully achieved first light fringes on-sky using an engineering C-RED2
camera. We received the science-ready C-RED1 camera in September 2023.
However, our laboratory characterization revealed camera performance issues,
necessitating its return for repairs. We are currently awaiting for the
repaired C-RED1 camera to resume on-sky fringe observations.

This paper serves as a user manual for Silmaril observing users, guiding them
through its operation, data collection, and data reduction software.

##  2 Technical Requirements

In this section, we delve into the technical requirements of the instrument,
focusing on aspects crucial for operation, data collection, and data
reduction.

Figure 1:  Conceptual sketch of Silmaril – with one beam. The beams travel
from left to right. Initially, the CHARA beams are redirected by the Silmaril
internal delay line mirrors to the long-focal Cylindrical Mirrors (CM1). CM1
has a focal length of $5.4$ m, which focuses the beam along the spatial
(fringe) direction on the detector. The Prism, with a spectral resolution of
$R\sim 35$, disperses the light. CM2 has a focal length of 350 mm, focus the
beam on the detector along the spectral direction. The star trackers, STST and
CLIMB can be inserted into the optical path or moved out. The delay line
mirror is moved by a Zaber linear motion actuator.

  1. 1.

Detector Data Acquisition: The data acquisition system should support frame
rates up to 1 kHz for fringe tracking to stabilize fringes against atmospheric
turbulence.

  2. 2.

Beam Combination: The Silmaril instrument currently combines 3 beams but has
been planned with the capability to expand to combine $2\times 3$ beams
separately. The two separate combined fringes, referred to as the left and
right combinations, are recorded in different region of interest windows on
the detector. The software must provide a separate server for each beam
combination to read the fringe windows, perform Fourier transforms, compute
group delay tracking, stabilize fringes by correcting atmospheric optical path
length delays, and save the raw data in FITS format.

  3. 3.

Software Configuration: The same software should be used for both servers
(left and right beam combiners), with configurations specified by startup
options such as silmaril_server --left and silmaril_server --right. The
Graphical User Interfaces (GUI) can be started such as silmaril_rtd_gtk --left
or silmaril_rtd_gtk --right.

  4. 4.

Beam Stabilization: Our initial on-sky results indicate that we experience
slow beam drifts, which can affect the observations in two ways: (i) imperfect
beam overlap, leading to biased measurements of fringe contrast, which leads
to large error bars in the squared visibility ($V^{2}$) and closure phase
($CP$); and (ii) reduced fringe signal-to-noise ratio (SNR) data, affecting
the instrument sensitivity. To address these issues, the software must provide
active beam stabilization. This can be achieved by measuring slow tip-tilts
using the star tip-tilt tracker (either the Six Telescope Star Tracker, STST
[6] or CLIMB [8]) and and correcting them with the beacon flat mirror located
on the telescope adaptive optics bench and the M7 mirror in the telescope
path. The STST can currently stabilize beams down to $J/H\sim 6$ mag and that
we are planning to use the CLIMB camera to extend the correction to fainter
stars. Figure 1 presents a conceptual sketch of the Silmaril instrument,
including the STST and CLIMB components. Figure 2 shows the as-built optical
and their opto-mechanical components implementation. The STST and CLIMB
components receive light by inserting beam splitters into the optical path
using motorized stages. The STST is already installed and utilizes
approximately 5% of the light for tip-tilt tracking and photometry collection,
which can also be used for the data reduction pipeline. The amount of light to
be shared with CLIMB is yet to be determined. Since we aim to correct only
slow beam drifts (with frequencies of a few minutes), long-exposure tip-tilt
measurements (tens of seconds to a minute) could suffice with a smaller
percentage of flux split. Another option is to use a dichroic to send the
entire J-band to CLIMB, resulting in a high SNR for tip-tilt tracking.
However, the downside is that this photometric information could not be
efficiently used for the data reduction pipeline because of wavelength
differences to fringe acquisition (H and K). Furthermore, this dichroic option
could be challenging for red colored objects, such as AGN and T-Tauri stars,
which are fainter in the J-band compared to the K-band.

  5. 5.

Data Reduction Pipeline: A data reduction pipeline is required to process the
datasets of both $2\times 3$ beam combiners.

  6. 6.

Automation: Silmaril must be a fully automated system to enable remote
operations.

Figure 2:  The installed optics and motor actuators of the Silmaril instrument
are shown in Figure 1. CM1 and CM2 are equipped with picomotors for the
alignment. The delay line mirrors are equipped with Zaber motors, which move
linearly and introduce optical path delay.  Figure 3:  The C-RED1 frames are
read by the C-RED One server and written to a computer shared memory. These
frames are then read by the silmaril_server, which implements the core
functionalities of the instrument. These functionalities can be divided into
three tasks: (i) (Auto)alignments; (ii) Data analysis, including Fast Fourier
Transforms (FFTs) and computation of group delay tracking; (iii) Data saving.
The alignments are performed based on the Lab Star Tracker tip-tilt
measurements obtained from either the STST or CLIMB camera. The tip-tilt
shifts either sent to the telescope/lab adaptive optics or to the pico
actuators of CM1. The internal delay lines are used to achieve cophasing with
the STS. The group delay control is implemented to stabilize the fringes by
sending the measured atmospheric OPDs to the CHARA delay lines.  Figure 4:  A
sketch of fringes of beams 1-3 and 4-6 are recorded on the same C-RED1
detector but in different sub-regions/windows. Each beam combination produces
fringes in both H and K bands simultaneously. These sub-windows are read by
the left and right silmaril_servers, which perform all the core functions as
described in Fig. 3. The left and right silmaril_servers work independently.
For example, the left silmaril_server controls the first three
beams/telescopes, manages its own backgrounds, computes FFTs and group delay
tracking, and sends the OPDs to its configured CHARA delay lines to stabilize
the fringes.  Figure 5:  silmaril_rtd_gtk displays the H and K-band fringes as
recorded by the C-RED1 camera. Silmaril incorporates an “edge filter”, which
separates H and K band light while reducing background noise specifically in
the H band channel[2].

##  3 Real-Time Instrument Software and Operational GUIs

###  3.1 Instrument software

Fig 3 presents the software architecture of the Silmaril instrument. The
instrument software manages critical functions of the instrument operation,
including C-RED1 or C-RED2 camera data acquisition, fringe tracking, automatic
instrument alignment with various actuator control, and observing interfaces,
all aimed at optimizing on-sky data collection. The software design of
Silmaril draws inspiration from the MIRC-X instrument for two main reasons.
Firstly, both instruments use similar camera and all-in-one beam combiner.
This strategic decision leverages a proven design, saving time and effort
compared to developing a completely new system (“reinventing the wheel”).
Secondly, the MIRC-X codebase serves as a well-established standard practice
for CHARA observations. By adopting a similar approach, Silmaril benefits from
the inherent reliability and familiarity associated with the MIRC-X software.
However, there is one key difference. Our Silmaril data acquisition software
utilizes the newer C-RED1, software development kit (SDK, Ubuntu 16 version
2.95), specifically designed for the latest version of the Silmaril camera and
firmware version. This was required because the 2017-era MIRC-X software was
not compatible with more recent version of the camera. We use the similar
computer as MIRC-X and runs Linux Xubuntu operating system. The code is
written in C/C++ based on the CHARA server/client architecture.

The Silmaril instrument software can be subdivided into three main parts.

####  3.1.1 Camera Data Acquisition

Camera data acquisition is implemented by a server named simaril_credone. This
server performs frame grabbing at high speeds and with low latency. Once the
images are read from the detector, they are written to a shared memory segment
with low latency. These images are then accessed by simaril_server for the
image processing. The detector frames are required to read much faster than
the atmospheric coherence time to stabilize the fringes. The atmospheric
coherent time at the CHARA array ranges typically from 5-20 ms depending on
average to good seeing nights. We adapted to the maximum frame rates (300 Hz)
used in MIRC-X and MYSTIC. Another aspect of data acquisition is low latency.
Higher latency or missing frames can cause delays, making corrections applied
to delay lines ineffective since they are not synced with the atmospheric
piston error. For the initial development of the instrument, we used C-RED2
while awaiting the final science version of the camera, C-RED1.

The data acquisition from the C-RED1 and C-RED2 are implemented with a Matrox
Radient eV-CL frame grabber installed on the computer and connected to the
camera via two camera link cables lengths of 10 m. C-RED1 uses the SAPHIRA
detector featuring 32 parallel video channel outputs organized into 10 column
blocks. Each channel output reads 32 adjacent pixels in a row simultaneously.
With a pixel clock set to 10 MHz, the detector achieves a readout speed of
approximately 640 Mpixels per second. This allows for reading a full
$320\times 256$ pixel frame at a rate of 3500 frames per second (FPS). The
C-RED1 detector offers several readout modes, with the IOTA mode (“Fowler
sampling”) being particularly useful for further reducing the readout
noise[5]. In this mode, each pixel is read $N$ reads times before moving to
the next pixel in the row, and each row is read $M$ loops times before moving
to the next row. This approach enables reduction in readout noise by
approximately the $\sqrt{N\times M}$. Additionally, several frames are read
nondestructively a predetermined number of times before the detector is reset,
a process known as “up the ramp” sampling. The final exposure image is then
obtained by subtracting any two successive frames within the ramp. We chose to
reset the frames per ramp for every 50 frames for fainter stars, although this
number is shortened for brighter targets.

####  3.1.2 Fringe Tracking in H/K-bands or Combined

The fringe tracking and locking to stabilize the fringes is implemented by
servers named silmaril_server --left and silmaril_server --right. These
servers are the top level servers and read images from the shared memory
written by silmaril_credone and performs image processing on the appropriate
fringe windows (see Fig 4). The raw images are first subtracted by
backgrounds. Fourier transforms are then applied to these images to estimate
the group delay optical path delay (OPD) offsets using the spectrally
dispersed fringes. These OPD offsets are sent to the CHARA delay lines to
stabilize the fringes against atmospheric turbulence and optical vibrations
along the optical path.

Internal Zaber actuators mounted to the delay line mirror (see Fig 1 and 2)
are used to co-phase with the Six Telescope Simulator (STS)[5], which serves
as the reference for the CHARA astrometric fringe-finding baseline solution.
The internal delay lines are moved with Zaber actuators, part number
X-NA08A25, which have a range of 25.4 mm.

To utilize the simultaneous H and K-band fringes (see Fig 5), we implement two
fringe tracking modes: (i) Primary mode and (ii) Combined mode. The Primary
mode leverages the color-magnitude difference of targets in H and K bands,
selecting the more sensitive band as the fringe tracker. In the Combined mode,
we perform FFTs and group delay fringe tracking computation on both H and
K-band fringe windows. This mode combines the correlated flux information from
both bands, selecting the higher SNR weighted fringes (i.e., OPDs) for robust
high-SNR fringe tracking, particularly useful for over-resolved objects.

Having fringes detected in both H and K-band wavelengths, the combined data
allows to detect the smaller and extended features of an object. The angular
resolution is given by $\lambda/2B$, where $\lambda$ is the wavelength and $B$
is the baseline separation between any two telescopes.

####  3.1.3 Beam Alignment and Beam Stabilization

The alignment of the instrument is done in two steps, implemented inside the
silmaril_server. The first-order coarse alignment is achieved by moving the
cylindrical mirrors (CM1s) internal to the Silmaril. The CM1 and CM2 are moved
by Picomotor controller, 8742-8-KIT from Newport. This initial alignment is
executed by moving pico motors mounted on the cylindrical mirrors. Once the
flux is located, the second-order beam stabilization keeps the beams centered
on the reference pixels on the detector using a tip-tilt tracker. For beam
stabilization, we plan to use either STST or CLIMB six beam tip-tilt trackers.
This tip-tilt signal can then be used to adjust the beacon flat using a long
average or to send tip-tilt offsets to either the telescope or laboratory
adaptive optics.

###  3.2 Observing Graphical User Interfaces (GUIs)

Silmaril builds upon the standard observing practices established by MIRC-X,
which many Principal Investigators are already trained to use. Having similar
GUIs to MIRC-X facilitates a quicker transition to the Silmaril observing
scheme. To achieve this compatibility, we adapted the MIRC-X observing code
and tailored it to the specific requirements of the Silmaril instrument. The
GUIs leverage the gtk2 toolkit for their user interface and the plplot library
for scientific plotting capabilities. Fig 6 shows an observing snapshot of the
Silmaril instrument.

  * •

silmaril_server_gtk – enables detector and instrument configuration.

  * •

silmaril_rtd_gtk – shows real-time display of fringes, power spectrum density,
waterfall trends and flux trends, etc.

  * •

simaril_gdt_gtk – enables group delay tracking and locking features

  * •

silmaril_ddl_gtk – is an engineering GUI for moving dealy lines

  * •

picogtk --SILMARIL_PICO – is an engineering GUI for aligning the instrument

All computations are implemented within the servers. User actions on the GUIs
trigger appropriate function calls inside the servers.

##  4 Data Reduction Software

The Silmaril data reduction pipeline is inspired from mircx_pipeline[5]. The
Silmaril pipeline is implemented in Python 3.9, and is publicly
available111https://gitlab.chara.gsu.edu/nanugu/silmaril_pipeline. The
pipeline produces science-ready visibilities ($V^{2}$) and closure phases
($CP$). Two main differences between the Silmaril and MIRC-X pipelines are:

  * •

Beam combination: Silmaril uses a 3-beam combination (resulting in 3 $V^{2}$
and 1 $CP$) instead of 6-beam combination of MIRC-X/MYSTIC/SPICA (resulting in
15 $V^{2}$ and 10 $CP$).

  * •

Photometry channels: As in Eq. 1, any fluctuations in photometry ($P_{i}$) of
different telescopes ($i$), generally caused by variations in atmospheric
turbulence, can significantly bias measurements of visibility squared
($V^{2}$). To address this, beam combiners incorporate dedicated photometric
channels alongside their science channels (see ex., MIRC-X/MYSTIC/SPICA).
These photometric channels measure the real-time fluxes alongside the fringes,
allowing for real-time calibration of $V^{2}$. However, the Silmaril
instrument does not implement these photometric channels on the C-RED1 camera.
To overcome this limitation, we plan to rely on flux measurements from STST
and CLIMB to calibrate $V^{2}$.

The data reduction pipeline implements following steps to measure $V^{2}$ and
$CP$:

  * •

The raw detector frames undergo background subtraction, bad pixel removal, and
flat fielding.

  * •

Spectral calibration is performed by comparing the observed fringe frequency
(in 2.4 px/fringe) to the expected spatial frequency based on the optical
magnification. The lateral pupil shifts can affect the fringe frequency,
therefore, the spectral calibration is implemented for the small group of
datasets where they were affected.

  * •

The real-time photometry of each of the 3 beams $P_{i}(\lambda,t)$, where $i$
is the number of the beam, $t$ is the time (frame), $\lambda$ the spectral
channel, are estimated from the photometric channels and the $\kappa$-matrix.

  * •

The three coherent fluxes, denoted by $I_{ij}(\lambda,t)$ represent the
spatial Fourier components of the interferometric window. These fluxes
correspond to the spatial frequencies of each beam pair, where $i$ and $j$
denote the indices of the specific beams involved.

  * •

Averaged squared visibilities for the baseline $ij$ are computed as follows:

$V^{2}_{ij}(\lambda)=\frac{|\langle\;I_{ij}(\lambda,t)\times
I_{ij}(\lambda,t-1)^{*}\,-\,\beta(\lambda)\;\rangle|}{2\langle
P_{i}(\lambda,t)\times P_{j}(\lambda,t)\rangle}$ (1)

  * •

Bispectrum $C$ of closing triangle $ijk$ is calculated by:

$CP_{ijk}(\lambda)=\langle I_{ij}(\lambda,t)\times I_{jk}(\lambda,t)\times
I^{*}_{ik}(\lambda,t)\;-\;\beta_{ijk}(\lambda)\rangle$ (2)

  * •

The instrumental transfer function can be estimated by dividing the observed
visibilities of a calibrator source by the expected visibilities for the same
source assuming a central uniform disk (UD) model. The UD diameter can be
either entered manually or retrieved automatically from a resource like the
JMMC Stellar Diameters Catalogue.

  * •

To account for instrumental variations, transfer function estimates are
interpolated to the exact time of each science observation using a Gaussian-
weighted average. This average typically considers data within a one-hour
window (FWHM = 30 mins) centered on the science observation time. The
resulting interpolated transfer function, derived from all calibrators
observed with the same instrumental setup, is then used to calibrate the
science visibilities.

Figure 6:  A screenshot of the observing interfaces while taking an
observation with the Silmaril instrument. In this observation, the power
spectrum peak amplitudes are not equal because of non-uniform illumination.
The fringe contrasts were greater than 85% for all three baseline fringes when
uniform fluxes were input to the instrument.

##  5 Laboratory Characterization

To verify the constructed performance of the Silmaril instrument if met its
design specifications, we used the laboratory Six Telescope Simulator (STS).
We successfully characterized key performance parameters, including the Airy
disk size (around 3 pixels at H-band) and large baseline fringe sampling
(approximately 2.5 pixels per fringe). These values closely matched to the
design specifications. Additionally, fringe contrast across all three
baselines exceeded 85%. Finally, we tested the group delay tracking system by
adjusting internal delay lines to achieve co-phasing with the STS, and
confirmed its proper functioning. Fig 5 shows the H and K-band fringes, which
are meeting the design specifications. We measured the spectral direction is
$\sim 20$ px in K-band and $\sim 12$ px in H-band and those are larger than
our design because the prism we use is larger spectral resolution than used in
the design.

##  6 On-Sky Performance

###  6.1 Slow beam drifts

Initial on-sky results indicate that we experience slow beam drifts, which can
affect the observations in two ways: (i) imperfect beam overlap, leading to
biased measurements of fringe contrast and large error bars in $V^{2}$; and
(ii) poor signal-to-noise ratio (SNR), affecting the instrument’s sensitivity.
To understand the amplitude and frequency of the beam drifts, we collected a
large dataset to assess beam drift and long-term stability. The results
indicate that we need an active beam stabilization at the order of a few
minutes to correct this slow beam drifting. We implement this beam
stabilization with the auto alignment software described in Sec. 3.

###  6.2 On-Sky Observing Procedure

The observing procedure starts with selecting the target and configuring the
beams to image on the instrument. The first three beams are directed to the
left side of the instrument and the last three beams to the right side. The
CHARA beam sampler allows for changing the mapping from instrument beam to
telescope before the night begins. Telescopes and beams are chosen such that
low-frequency baseline fringes of Silmaril are paired with longer telescope
baselines, and high-frequency baseline fringes are paired with shorter
baselines. As reported in Lanthermann et al. (2022)[2], beam 2 & 3 provide the
lower spatial frequency for Silmaril and are paired with longer baseline
telescopes, such as S1E1, while beam 1 & 3 provide the highest spatial
frequency and are paired with smaller telescope baselines, for example, S1S2.

Before the night, the alignment of Silmaril is optimized using the Six
Telescope Simulator (STS) , which is the 6-beam coherent calibration source at
the CHARA Array. The goal is to ensure that the light is well-centered within
the predefined coordinates on the Silmaril detector and cophased with the STS,
helping to find the fringes within ±1 mm offsets by the astrometric solution.
While the instrument calibration is performed daily, it has proven stable for
at least a week unless the observing mode has changed. First, the beams are
centered on the predefined reference pixels vertically and horizontally. Next,
the power spectrum fringe peaks are checked to ensure they fall onto their
predetermined positions (see Fig 7). This calibration is accomplished by
adjusting the cylindrical mirrors (CM1s).

The star is initially acquired by the CHARA Adaptive Optics (AO) system, which
operates at visible wavelengths. Visible to near-infrared, atmospheric
refraction star shifts are corrected by positioning the star within the
reference boxes on the J+H-band operating the STST camera. This correction is
implemented by adjusting the beacon flat on the telescope area to center the
beams on Silmaril. These STST reference boxes are defined by the STS
laboratory white light source.

At this stage, the Silmaril instrument should see light on the detector. If
any misalignments between Silmaril and STST, those are corrected with a coarse
alignment by adjusting the beacon flat on the telescope area to center the
beams on Silmaril. Once beam stabilization is activated, fringes are typically
found within 5 minutes of observing time, depending on the CHARA astrometric
baseline, which is calibrated quarterly following seasonal trends. The data
acquisition sequence is similar to other instruments.

The standard observing sequence is as follows, with 20 mins for an
observation. However, for fainter targets, longer integrations times of the
fringe DATA are necessary.

  * •

Star Acquisition: 5 minutes to acquire the star with the telescopes.

  * •

Fringe Search: 2–5 minutes of searching for fringes.

  * •

Data Collection: 5 minutes of DATA collection with all shutters open and
fringes tracked.

  * •

Background Measurement: 1 minute with all shutters closed.

  * •

Beam Measurement: 1 minute for each BEAM, where the shutter of only one of the
six beams is open sequentially.

  * •

Foreground Frames: 3 minutes of data without fringes with all shutters open.
Delay lines are moved away while taking this dataset.

  * •

Sky Measurement: 1 minute by moving the telescopes away off the target.

###  6.3 Results from Engineering Camera C-RED2

While awaiting the readiness of the science-grade C-RED1 camera, we proceeded
with on-sky observations using the engineering camera C-RED2. However, the
sensitivity of C-RED2 is limited compared to C-RED1 due to the absence of key
features such as avalanche gain for reducing readout noise, lowered dark
current, and a cold stop to minimize black body backgrounds. Nonetheless, with
C-RED2, we successfully characterized the instrument: (i) obtained on-sky
fringes, (ii) demonstrated on-sky group delay fringe tracking, and (iii)
collected data on several stars to test the data reduction pipeline. We
successfully captured high SNR fringes on HD 180756 (magnitude 4.2 in H band)
with a 50 ms integration. Fig 8 illustrates results from the data reduction
pipeline, showing the signal-to-noise ratio of the observations.

Considering the ultra low noise capabilities of C-RED1, we compute that
Silmaril will be able to detect much fainter sources ($H/K\gtrapprox 10$) once
we transition from C-RED2 to C-RED1. We are looking forward to the arrival of
C-RED1 this summer, which will allow us to test the final sensitivity limits
of the Silmaril instrument. Please refer to Lanthermann et al. (2024) [3] for
more details on the sensitivity calculations.

Figure 7:  A snapshot of an on-sky observation of HD 180756 displaying fringes
and the power spectrum. This observation was obtained with the C-RED2 camera.

Figure 8:  (Left panel) The fringe signal-to-noise ratio ($\sim 100$) of an
observation recorded on the object HD 180756 with an H magnitude of 4.2 was
recorded with C-RED2. The bottom orange line indicates the signal-to-noise
ratio (SNR=2) threshold for fringe rejection. Any fringe with a signal falling
below this threshold will be rejected. (Right panel) The waterfall trends for
three baselines recorded on the object HD 180756. This observation is
projected to be around $H\gtrapprox 10$ if observed with C-RED1, considering
the ultra-low noise performance of the C-RED1 detector compared to C-RED2.

##  7 Summary and Future Prospects

Silmaril, a new beam combiner for the CHARA array, promises access to
previously unseen faint near-infrared targets (magnitude $H/K\gtrapprox 10$).
This paper outlines the instrument’s software suite and data reduction
pipeline, responsible for camera control, fringe tracking, instrument
alignment, data processing, and user interfaces.

Future Prospects:

  * •

On-sky Testing of Science Ready Camera: While initial on-sky observations have
been conducted using the C-RED2 engineering camera, simulations based on
C-RED1’s ultra-low noise characteristics predict Silmaril achieving
sensitivity $H\gtrapprox 10$. This enhanced capability will be tested upon the
repaired C-RED1 camera’s return to CHARA in July 2024.

  * •

Improved Beam Stability: Initial on-sky observations revealed slow beam drifts
challenges. To address this issue, we have implemented automatic alignment
methods that should stabilize the beams.

  * •

Data Reduction Improvement: The data reduction software is still under
refinement to improve the accuracy of squared visibility and closure phase
measurements. Additionally, the software needs refinement to calculate
uncertainties more accurately, as current values are underestimated.

  * •

Community Access: We anticipate that Silmaril will be ready for CHARA
community access observations starting 2025A semester.

### Acknowledgments

The construction of the Silmaril instrument was supported by USA National
Science Foundation, under Grant No. NSF AST-1909858.
This work is based upon observations obtained with the Georgia State
University Center for High Angular Resolution Astronomy Array at Mount Wilson
Observatory. The CHARA Array is supported by the National Science Foundation
under Grant No. AST-1636624, AST-1908026, and AST-2034336. Institutional
support has been provided from the GSU College of Arts and Sciences and the
GSU Office of the Vice President for Research and Economic Development.

## References

  * [1] ten Brummelaar, T. A., McAlister, H. A., Ridgway, S. T., Bagnuolo, W. G., J., Turner, N. H., Sturmann, L., Sturmann, J., Berger, D. H., Ogden, C. E., Cadman, R., Hartkopf, W. I., Hopper, C. H., and Shure, M. A., “First Results from the CHARA Array. II. A Description of the Instrument,” Astrophysical Journal 628, 453–465 (July 2005).
  * [2] Lanthermann, C., ten Brummelaar, T., Tuthill, P., Martinod, M.-A., Ligon, E., Gies, D., Schaefer, G., and Anderson, M., “Design of the new CHARA instrument SILMARIL: pushing for the sensitivity of a 3-beam combiner in the H- and K-bands,” in [Optical and Infrared Interferometry and Imaging VIII ], Mérand, A., Sallum, S., and Sanchez-Bermudez, J., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 12183, 121830N (Aug. 2022).
  * [3] Lanthermann, C., ten Brummelaar, T., Tuthill, P., Anugu, N., Ligon, E., Gies, D., Schaefer, G., and Anderson, M., “Silmaril: final design and on-sky performance,” in [Optical and Infrared Interferometry and Imaging IX ], Mérand, A., Sallum, S., and Sanchez-Bermudez, J., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 13095-4 (Aug. 2024).
  * [4] Mourard, D., Berio, P., Pannetier, C., Nardetto, N., Albrecht, S., Allouche, F., Bailet, C., Bourgès, L., ten Brummelaar, Theo A.and Creevey, O., Deheuvels, S., Dejonghe, J., Domiciano, A., Geneslay, P., Gies, D. R., Jacqmart, E., Lagarde, S., Lecron, D., Ligi, R., Mella, G., Morand, F., Rousseau, S., Salabert, D., Schaefer, G., Spang, A., and Wittkowski, M., “CHARA/SPICA: a new 6T instrument for the CHARA Array,” in [Optical and Infrared Interferometry and Imaging VIII ], Mérand, A., Sallum, S., and Sanchez-Bermudez, J., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 12183, 12183–7 (July 2022).
  * [5] Anugu, N., Le Bouquin, J.-B., Monnier, J. D., Kraus, S., Setterholm, B. R., Labdon, A., Davies, C. L., Lanthermann, C., Gardner, T., Ennis, J., Johnson, K. J. C., Ten Brummelaar, T., Schaefer, G., and Sturmann, J., “MIRC-X: A Highly Sensitive Six-telescope Interferometric Imager at the CHARA Array,” Astronomical Journal 160, 158 (Oct. 2020).
  * [6] Setterholm, B. R., Monnier, J. D., Le Bouquin, J.-B., Anugu, N., Ennis, J., Jocou, L., Ibrahim, N., Kraus, S., Anderson, M. D., Chhabra, S., Codron, I., Farrington, C. D., Flores, B., Gardner, T., Gutierrez, M., Lanthermann, C., Majoinen, O. W., Mortimer, D. J., Schaefer, G., Scott, N. J., ten Brummelaar, T., and Vargas, N. L., “MYSTIC: a high angular resolution K-band imager at CHARA,” Journal of Astronomical Telescopes, Instruments, and Systems 9, 025006 (Apr. 2023).
  * [7] Taras, A. K., Robertson, J. G., Allouche, F., Courtney-Barrer, B., Carter, J., Crous, F., Cvetojevic, N., Ireland, M., Lagarde, S., Martinache, F., McGinness, G., N’Diaye, M., Robbe-Dubois, S., and Tuthill, P., “Heimdallr, Baldr, and Solarstein: designing the next generation of VLTI instruments in the Asgard suite,” Appl. Opt. 63, D41 (May 2024).
  * [8] Ten Brummelaar, T. A., Sturmann, J., Ridgway, S. T., Sturmann, L., Turner, N. H., McAlister, H. A., Farrington, C. D., Beckmann, U., Weigelt, G., and Shure, M., “The Classic/climb Beam Combiner at the CHARA Array,” Journal of Astronomical Instrumentation 2, 1340004 (Dec. 2013).