based on templates, and AdaFilter (Phang et al., 2021) adversarially
constructs tests. However, it is worth noting that AdaFilter may not be
entirely fair as it relies on adversarial examples. HELM (Liang et al., 2022)
evaluates LLMs from different aspects, while the Big-Bench (Srivastava et al.,
2022) platform is used to design hard tasks for machine learning models to
tackle. PromptBench (Zhu et al., 2023) aims to evaluate the adversarial
robustness of LLMs by creating adversarial prompts, which is more challenging
and the results demonstrated that current LLMs are not robust to adversarial
prompts.

##  7\. Grand Challenges and Opportunities for Future Research

Evaluation as a new discipline: Our summarization inspires us to redesign a
wide spectrum of aspects related to evaluation in the era of LLMs. In this
section, we present several grand challenges. Our key point is that evaluation
should be treated as an essential discipline to drive the success of LLMs and
other AI models. Existing protocols are not enough to thoroughly evaluate the
true capabilities of LLMs, which poses grand challenges and triggers new
opportunities for future research on LLMs evaluation.

###  7.1. Designing AGI Benchmarks

As we discussed earlier, while all tasks can potentially serve as evaluation
tools for LLMs, the question remains as to which can truly measure AGI
capabilities. As we expect LLMs to demonstrate AGI abilities, a comprehensive
understanding of the differences between human and AGI capacities becomes
crucial in the creation of AGI benchmarks. The prevailing trend seems to
conceptualize AGI as a superhuman entity, thereby utilizing cross-disciplinary
knowledge from fields such as education, psychology, and social sciences to
design innovative benchmarks. Nonetheless, there remains a plethora of
unresolved issues. For instance, does it make sense to use human values as a
starting point for test construction, or should alternative perspectives be
considered? Developing suitable AGI benchmarks presents many open questions
demanding further exploration.

###  7.2. Complete Behavioral Evaluation

An ideal AGI evaluation should contain not only standard benchmarks on common
tasks, but also evaluations on open tasks such as complete behavioral tests.
By behavioral test, we mean that AGI models should also be evaluated in an
open environment. For instance, by treating LLMs as the central controller, we
can construct evaluations on a robot manipulated by LLMs to test its behaviors
in real situations. By treating LLMs as a completely intelligent machine, the
evaluations of its multi-modal dimensions should also be considered. In fact,
complete behavioral evaluations are complementary to standard AGI benchmarks
and they should work together for better testing.

###  7.3. Robustness Evaluation

Beyond general tasks, it is crucial for LLMs to maintain robustness against a
wide variety of inputs in order to perform optimally for end-users, given
their extensive integration into daily life. For instance, the same prompts
but with different grammars and expressions could lead ChatGPT and other LLMs
to generate diverse results, indicating that current LLMs are not robust to
the inputs. While there are some prior work on robustness evaluation (Wang et
al., 2023d; Zhu et al., 2023), there are much room for advancement, such as
including more diverse evaluation sets, examining more evaluation aspects, and
developing more efficient evaluations to generate robustness tasks.
Concurrently, the concept and definition of robustness are constantly
evolving. It is thus vital to consider updating the evaluation system to
better align with emerging requirements related to ethics and bias.

###  7.4. Dynamic and Evolving Evaluation

Existing evaluation protocols for most AI tasks rely on static and public
benchmarks, i.e., the evaluation datasets and protocols are often publicly
available. While this facilitates rapid and convenient evaluation within the
community, it is unable to accurately assess the evolving abilities of LLMs,
given their rapid rate of development. The capabilities of LLMs may enhance
over time which cannot be consistently evaluated by existing static
benchmarks. On the other hand, as LLMs grow increasingly powerful with larger
model sizes and training set sizes, static and public benchmarks are likely to
be memorized by LLMs, resulting in potential training data contamination.
Therefore, developing dynamic and evolving evaluation systems is the key to
providing a fair evaluation of LLMs.

###  7.5. Principled and Trustworthy Evaluation

When introducing an evaluation system, it is crucial to ascertain its
integrity and trustworthiness. Therefore, the necessity for trustworthy
computing extends to the requirement for reliable evaluation systems as well.
This poses a challenging research question that intertwines with measurement
theory, probability, and numerous other domains. For instance, how can we
ensure that dynamic testing truly generates out-of-distribution examples?
There is a scarcity of research in this domain, and it is hoped that future
work will aim to scrutinize not only the algorithms but the evaluation system
itself.

###  7.6. Unified Evaluation that Supports All LLMs Tasks

There are many other research areas of LLMs and we need to develop evaluation
systems that can support all kinds of tasks such as value alignment, safety,
verification, interdisciplinary research, fine-tuning, and others. For
instance, PandaLM (Wang et al., 2023m) is an evaluation system that assists
LLMs fine-tuning by providing an open-source evaluation model, which can
automatically assess the performance of fine-tuning. We expect that more
evaluation systems are becoming more general and can be used as assistance in
certain LLMs tasks.

###  7.7. Beyond Evaluation: LLMs Enhancement

Ultimately, evaluation is not the end goal but rather the starting point.
Following the evaluation, there are undoubtedly conclusions to be drawn
regarding performance, robustness, stability, and other factors. A proficient
evaluation system should not only offer benchmark results but should also
deliver an insightful analysis, recommendations, and guidance for future
research and development. For instance, PromptBench (Zhu et al., 2023)
provides not only robustness evaluation results on adversarial prompts but
also a comprehensive analysis through attention visualization, elucidating how
adversarial texts can result in erroneous responses. The system further offers
a word frequency analysis to identify robust and non-robust words in the test
sets, thus providing prompt engineering guidance for end users. Subsequent
research can leverage these findings to enhance LLMs. Another example is that
Wang et al. (2023l) first explored the performance of large vision-language
models on imbalanced (long-tailed) tasks, which demonstrates the limitation of
current large models. Then, they explored different methodologies to enhance
the performance on these tasks. In summary, enhancement after evaluation helps
to build better LLMs and much can be done in the future.

##  8\. Conclusion

Evaluation carries profound significance, becoming imperative in the
advancement of AI models, especially within the context of large language
models. This paper presents the first survey to give a comprehensive overview
of the evaluation on LLMs from three aspects: what to evaluate, how to
evaluate, and where to evaluate. By encapsulating evaluation tasks, protocols,
and benchmarks, our aim is to augment understanding of the current status of
LLMs, elucidate their strengths and limitations, and furnish insights for
future LLMs progression.

Our survey reveals that current LLMs exhibit certain limitations in numerous
tasks, notably reasoning and robustness tasks. Concurrently, the need for
contemporary evaluation systems to adapt and evolve remains evident, ensuring
the accurate assessment of LLMs’ inherent capabilities and limitations. We
identify several grand challenges that future research should address, with
the aspiration that LLMs can progressively enhance their service to humanity.

## Acknowledgements

This work is supported in part by NSF under grant III-2106758.

## Disclaimer

The goal of this paper is mainly to summarize and discuss existing evaluation
efforts on large language models. Results and conclusions in each paper are
original contributions of their corresponding authors, particularly for
potential issues in ethics and biases. This paper may discuss some side
effects of LLMs and the only intention is to foster a better understanding.

Due to the evolution of LLMs especially online services such as Claude and
ChatGPT, it is very likely that they become stronger and some of the
limitations described in this paper are mitigated (and new limitations may
arise). We encourage interested readers to take this survey as a reference for
future research and conduct real experiments in current systems when
performing evaluations.

Finally, the evaluation of LLMs is continuously developing, thus we may miss
some new papers or benchmarks. We welcome all constructive feedback and
suggestions.

## References

  * (1)
  * Abdelali et al. (2023) Ahmed Abdelali, Hamdy Mubarak, Shammur Absar Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, et al. 2023\.  Benchmarking Arabic AI with Large Language Models.  _arXiv preprint arXiv:2305.14982_ (2023).
  * Ahuja et al. (2023) Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, et al. 2023\.  Mega: Multilingual evaluation of generative ai.  _arXiv preprint arXiv:2303.12528_ (2023).
  * Arora et al. (2023) Daman Arora, Himanshu Gaurav Singh, et al. 2023\.  Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models.  _arXiv preprint arXiv:2305.15074_ (2023).
  * Askell et al. (2021) Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021\.  A general language assistant as a laboratory for alignment.  _arXiv preprint arXiv:2112.00861_ (2021).
  * Bai et al. (2023) Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. 2023\.  Benchmarking Foundation Models with Language-Model-as-an-Examiner.  _arXiv preprint arXiv:2306.04181_ (2023).
  * Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023\.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.  _arXiv preprint arXiv:2302.04023_ (2023).
  * Belz and Reiter (2006) Anja Belz and Ehud Reiter. 2006.  Comparing automatic and human evaluation of NLG systems. In _11th conference of the european chapter of the association for computational linguistics_. 313–320.
  * Berrar (2019) Daniel Berrar. 2019\.  Cross-Validation.
  * Bian et al. (2023) Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, and Ben He. 2023\.  Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models.  _arXiv preprint arXiv:2303.16421_ (2023).
  * Bodroza et al. (2023) Bojana Bodroza, Bojana M Dinic, and Ljubisa Bojic. 2023\.  Personality testing of GPT-3: Limited temporal reliability, but highlighted social desirability of GPT-3’s personality instruments results.  _arXiv preprint arXiv:2306.04308_ (2023).
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021\.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ (2021).
  * Brody (1999) Nathan Brody. 1999\.  What is intelligence?  _International Review of Psychiatry_ 11, 1 (1999), 19–25.
  * Brown et al. (1992) Peter F Brown, Vincent J Della Pietra, Peter V Desouza, Jennifer C Lai, and Robert L Mercer. 1992\.  Class-based n-gram models of natural language.  _Computational linguistics_ 18, 4 (1992), 467–480.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023\.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_ (2023).
  * Cao et al. (2023) Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023\.  Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study. In _Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)_. 53–67.
  * Cascella et al. (2023) Marco Cascella, Jonathan Montomoli, Valentina Bellini, and Elena Bignami. 2023.  Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios.  _Journal of Medical Systems_ 47, 1 (2023), 33\.
  * Castro Nascimento and Pimentel (2023) Cayque Monteiro Castro Nascimento and André Silva Pimentel. 2023.  Do Large Language Models Understand Chemistry? A Conversation with ChatGPT.  _Journal of Chemical Information and Modeling_ 63, 6 (2023), 1649–1655.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021\.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ (2021).
  * Chen et al. (2023) Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023.  Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study.  _arXiv preprint arXiv:2304.00723_ (2023).
  * Chervenak et al. (2023) Joseph Chervenak, Harry Lieman, Miranda Blanco-Breindel, and Sangita Jindal. 2023.  The promise and peril of using a large language model to obtain clinical information: ChatGPT performs strongly as a fertility counseling tool with limitations.  _Fertility and Sterility_ (2023).
  * Chia et al. (2023) Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria. 2023.  INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models.  _arXiv preprint arXiv:2306.04757_ (2023).
  * Choi et al. (2023) Minje Choi, Jiaxin Pei, Sagar Kumar, Chang Shu, and David Jurgens. 2023.  Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark.  _arXiv preprint arXiv:2305.14938_ (2023).
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022\.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ (2022).
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017\.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ 30 (2017).
  * Clavié et al. (2023) Benjamin Clavié, Alexandru Ciceu, Frederick Naylor, Guillaume Soulié, and Thomas Brightwell. 2023.  Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification. In _International Conference on Applications of Natural Language to Information Systems_. Springer, 3–17.
  * Collins et al. (2023) Katherine M Collins, Albert Q Jiang, Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B Tenenbaum, William Hart, et al. 2023\.  Evaluating Language Models for Mathematics through Interactions.  _arXiv preprint arXiv:2306.01694_ (2023).
  * Cortes and Vapnik (1995) Corinna Cortes and Vladimir Vapnik. 1995.  Support-vector networks.  _Machine learning_ 20 (1995), 273–297.
  * Dai et al. (2023b) Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. 2023b.  Uncovering ChatGPT’s Capabilities in Recommender Systems.  _arXiv preprint arXiv:2305.02182_ (2023).
  * Dai et al. (2023a) Wei Dai, Jionghao Lin, Flora Jin, Tongguang Li, Yi-Shan Tsai, Dragan Gasevic, and Guanliang Chen. 2023a.  Can large language models provide feedback to students? a case study on chatgpt.  (2023).
  * Dao and Le (2023) Xuan-Quy Dao and Ngoc-Bich Le. 2023.  Investigating the Effectiveness of ChatGPT in Mathematical Reasoning and Problem Solving: Evidence from the Vietnamese National High School Graduation Examination.  _arXiv preprint arXiv:2306.06331_ (2023).
  * de Winter (2023) Joost CF de Winter. 2023\.  Can ChatGPT pass high school exams on English language comprehension.  _Researchgate. Preprint_ (2023).
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  Imagenet: A large-scale hierarchical image database. In _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 248–255.
  * Deroy et al. (2023) Aniket Deroy, Kripabandhu Ghosh, and Saptarshi Ghosh. 2023\.  How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization?  _arXiv preprint arXiv:2306.01248_ (2023).
  * Deshpande et al. (2023) Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023\.  Toxicity in chatgpt: Analyzing persona-assigned language models.  _arXiv preprint arXiv:2304.05335_ (2023).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Dhamala et al. (2021) Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021.  Bold: Dataset and metrics for measuring biases in open-ended language generation. In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_. 862–872.
  * Dubois et al. (2023) Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023.  Alpacafarm: A simulation framework for methods that learn from human feedback.  _arXiv preprint arXiv:2305.14387_ (2023).
  * Duong and Solomon (2023) Dat Duong and Benjamin D Solomon. 2023.  Analysis of large-language model versus human performance for genetics questions.  _European Journal of Human Genetics_ (2023), 1–3.
  * Fan et al. (2023) Wenqi Fan, Zihuai Zhao, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Jiliang Tang, and Qing Li. 2023\.  Recommender Systems in the Era of Large Language Models (LLMs).  arXiv:2307.02046 [cs.IR]
  * Fansi Tchango et al. (2022) Arsene Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana Ghosn. 2022\.  Ddxplus: A new dataset for automatic medical diagnosis.  _Advances in Neural Information Processing Systems_ 35 (2022), 31306–31318.
  * Ferrara (2023) Emilio Ferrara. 2023\.  Should chatgpt be biased? challenges and risks of bias in large language models.  _arXiv preprint arXiv:2304.03738_ (2023).
  * Floridi and Chiriatti (2020) Luciano Floridi and Massimo Chiriatti. 2020.  GPT-3: Its nature, scope, limits, and consequences.  _Minds and Machines_ 30 (2020), 681–694.
  * Frank (2023) Michael C Frank. 2023\.  Baby steps in evaluating the capacities of large language models.  _Nature Reviews Psychology_ (2023), 1–2.
  * Frieder et al. (2023) Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023\.  Mathematical capabilities of chatgpt.  _arXiv preprint arXiv:2301.13867_ (2023).
  * Fu et al. (2023a) Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. 2023a.  MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models.  _arXiv preprint arXiv:2306.13394_ (2023).
  * Fu et al. (2023b) Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. 2023b.  Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models’ Reasoning Performance.  _arXiv preprint arXiv:2305.17306_ (2023).
  * Fushiki (2011) Tadayoshi Fushiki. 2011\.  Estimation of prediction error by using K-fold cross-validation.  _Statistics and Computing_ 21 (2011), 137–146.
  * Gallant et al. (1990) Stephen I Gallant et al. 1990\.  Perceptron-based learning algorithms.  _IEEE Transactions on neural networks_ 1, 2 (1990), 179–191.
  * Gao et al. (2022) Irena Gao, Gabriel Ilharco, Scott Lundberg, and Marco Tulio Ribeiro. 2022.  Adaptive Testing of Computer Vision Models.  _arXiv preprint arXiv:2212.02774_ (2022).
  * Gao and Lin (2004) Jianfeng Gao and Chin-Yew Lin. 2004.  Introduction to the special issue on statistical language modeling.  , 87–93 pages.
  * Gao et al. (2020) Tianyu Gao, Adam Fisch, and Danqi Chen. 2020.  Making pre-trained language models better few-shot learners.  _arXiv preprint arXiv:2012.15723_ (2020).
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020.  RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. 3356–3369.
  * Geifman and El-Yaniv (2017) Yonatan Geifman and Ran El-Yaniv. 2017.  Selective classification for deep neural networks.  _Advances in neural information processing systems_ 30 (2017).
  * Gekhman et al. (2023) Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. 2023.  Trueteacher: Learning factual consistency evaluation with large language models.  _arXiv preprint arXiv:2305.11171_ (2023).
  * Gendron et al. (2023) Gaël Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. 2023.  Large Language Models Are Not Abstract Reasoners.  _arXiv preprint arXiv:2305.19555_ (2023).
  * Gilson et al. (2023) Aidan Gilson, Conrad W Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, David Chartash, et al. 2023\.  How does CHATGPT perform on the United States Medical Licensing Examination? the implications of large language models for medical education and knowledge assessment.  _JMIR Medical Education_ 9, 1 (2023), e45312.
  * Graham et al. (2013) Jesse Graham, Jonathan Haidt, Sena Koleva, Matt Motyl, Ravi Iyer, Sean P Wojcik, and Peter H Ditto. 2013.  Moral foundations theory: The pragmatic validity of moral pluralism.  In _Advances in experimental social psychology_. Vol. 47. Elsevier, 55–130.
  * Gu et al. (2023) Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang, Jianchen Wang, Sihang Jiang, Zhuozhi Xiong, Zihan Li, Qianyu He, Rui Xu, et al. 2023\.  Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation.  _arXiv preprint arXiv:2306.05783_ (2023).
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017\.  On calibration of modern neural networks. In _International conference on machine learning_. PMLR, 1321–1330.
  * Guo et al. (2023) Taicheng Guo, Kehan Guo, Zhengwen Liang, Zhichun Guo, Nitesh V Chawla, Olaf Wiest, Xiangliang Zhang, et al. 2023\.  What indeed can GPT models do in chemistry? A comprehensive benchmark on eight tasks.  _arXiv preprint arXiv:2305.18365_ (2023).
  * Hagendorff and Fabi (2023) Thilo Hagendorff and Sarah Fabi. 2023.  Human-Like Intuitive Behavior and Reasoning Biases Emerged in Language Models – and Disappeared in GPT-4.  arXiv:2306.07622 [cs.CL]
  * Hamidi and Roberts (2023) Alaleh Hamidi and Kirk Roberts. 2023.  Evaluation of AI Chatbots for Patient-Specific EHR Questions.  _arXiv preprint arXiv:2306.02549_ (2023).
  * Hardt et al. (2016) Moritz Hardt, Eric Price, and Nati Srebro. 2016.  Equality of opportunity in supervised learning.  _Advances in neural information processing systems_ 29 (2016).
  * Hartmann et al. (2023) Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. 2023\.  The political ideology of conversational AI: Converging evidence on ChatGPT’s pro-environmental, left-libertarian orientation.  _arXiv preprint arXiv:2301.01768_ (2023).
  * He et al. (2023) Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Lida Chen, Xintao Wang, Yuncheng Huang, et al. 2023\.  Can Large Language Models Understand Real-World Complex Instructions?  _arXiv preprint arXiv:2309.09150_ (2023).
  * Hellas et al. (2023) Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, and Juha Sorva. 2023.  Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests.  _arXiv preprint arXiv:2306.05715_ (2023).
  * Hendrycks et al. (2021a) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021a.  Measuring coding challenge competence with apps.  _arXiv preprint arXiv:2105.09938_ (2021).
  * Hendrycks et al. (2020a) Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2020a.  Aligning ai with shared human values.  _arXiv preprint arXiv:2008.02275_ (2020).
  * Hendrycks et al. (2020b) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020b.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_ (2020).
  * Hendrycks et al. (2021b) Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021b.  Cuad: An expert-annotated nlp dataset for legal contract review.  _arXiv preprint arXiv:2103.06268_ (2021).
  * Hendrycks et al. (2021c) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021c.  Measuring mathematical problem solving with the math dataset.  _arXiv preprint arXiv:2103.03874_ (2021).
  * Holmes et al. (2023) Jason Holmes, Zhengliang Liu, Lian Zhang, Yuzhen Ding, Terence T Sio, Lisa A McGee, Jonathan B Ashman, Xiang Li, Tianming Liu, Jiajian Shen, et al. 2023\.  Evaluating large language models on a highly-specialized topic, radiation oncology physics.  _arXiv preprint arXiv:2304.01938_ (2023).
  * Honovich et al. (2022) Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022\.  TRUE: Re-evaluating factual consistency evaluation.  _arXiv preprint arXiv:2204.04991_ (2022).
  * Hou et al. (2023) Zhaoyi Joey Hou, Li Zhang, and Chris Callison-Burch. 2023\.  Choice-75: A Dataset on Decision Branching in Script Learning.  _arXiv preprint arXiv:2309.11737_ (2023).
  * Huang et al. (2023c) Jen-tse Huang, Man Ho Lam, Eric John Li, Shujie Ren, Wenxuan Wang, Wenxiang Jiao, Zhaopeng Tu, and Michael R. Lyu. 2023c.  Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench.  _arXiv preprint arXiv:2308.03656_ (2023).
  * Huang et al. (2023b) Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. 2023b.  Language is not all you need: Aligning perception with language models.  _arXiv preprint arXiv:2302.14045_ (2023).
  * Huang et al. (2023a) Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. 2023a.  C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.  _arXiv preprint arXiv:2305.08322_ (2023).
  * Huang et al. (2023d) Yue Huang, Qihui Zhang, Philip S. Y, and Lichao Sun. 2023d.  TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Models.  arXiv:2306.11507 [cs.CL]
  * HuggingFace (2023) HuggingFace. 2023\.  Open-source Large Language Models Leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard.
  * Jahan et al. (2023) Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimmy Huang. 2023.  Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers.  _arXiv preprint arXiv:2306.04504_ (2023).
  * Jain et al. (2023) Neel Jain, Khalid Saifullah, Yuxin Wen, John Kirchenbauer, Manli Shu, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2023.  Bring Your Own Data! Self-Supervised Evaluation for Large Language Models.  _arXiv preprint arXiv:2306.13651_ (2023).
  * Jansson et al. (2021) Malin Jansson, Stefan Hrastinski, Stefan Stenbom, and Fredrik Enoksson. 2021.  Online question and answer sessions: How students support their own and other students’ processes of inquiry in a text-based learning environment.  _The Internet and Higher Education_ 51 (2021), 100817.
  * Jentzsch and Kersting (2023) Sophie Jentzsch and Kristian Kersting. 2023.  ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Models.  _arXiv preprint arXiv:2306.04563_ (2023).
  * Ji et al. (2023) Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023.  Beavertails: Towards improved safety alignment of llm via a human-preference dataset.  _arXiv preprint arXiv:2307.04657_ (2023).
  * Jiang et al. (2023) Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji-Rong Wen. 2023\.  Structgpt: A general framework for large language model to reason over structured data.  _arXiv preprint arXiv:2305.09645_ (2023).
  * Johnson et al. (2023) Douglas Johnson, Rachel Goodman, J Patrinely, Cosby Stone, Eli Zimmerman, Rebecca Donald, Sam Chang, Sean Berkowitz, Avni Finn, Eiman Jahangir, et al. 2023\.  Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the Chat-GPT model.  (2023).
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Vancouver, Canada.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, T. J. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2022\.  Language Models (Mostly) Know What They Know.  _ArXiv_ abs/2207.05221 (2022).
  * Karpas et al. (2022) Ehud Karpas, Omri Abend, Yonatan Belinkov, Barak Lenz, Opher Lieber, Nir Ratner, Yoav Shoham, Hofit Bata, Yoav Levine, Kevin Leyton-Brown, et al. 2022\.  MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning.  _arXiv preprint arXiv:2205.00445_ (2022).
  * Kasneci et al. (2023) Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023\.  ChatGPT for good? On opportunities and challenges of large language models for education.  _Learning and Individual Differences_ 103 (2023), 102274.
  * Khalfa (1994) Jean Khalfa. 1994\.  What is intelligence?  (1994).
  * Khan et al. (2023) Yousuf A Khan, Clarisse Hokia, Jennifer Xu, and Ben Ehlert. 2023\.  covLLM: Large Language Models for COVID-19 Biomedical Literature.  _arXiv preprint arXiv:2306.04926_ (2023).
  * Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. 2021\.  Dynabench: Rethinking benchmarking in NLP.  _arXiv preprint arXiv:2104.14337_ (2021).
  * Kohavi et al. (1995) Ron Kohavi et al. 1995\.  A study of cross-validation and bootstrap for accuracy estimation and model selection. In _Ijcai_ , Vol. 14. Montreal, Canada, 1137–1145.
  * Kombrink et al. (2011) Stefan Kombrink, Tomas Mikolov, Martin Karafiát, and Lukás Burget. 2011.  Recurrent Neural Network Based Language Modeling in Meeting Recognition.. In _Interspeech_ , Vol. 11. 2877–2880.
  * Kung et al. (2023) Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2023\.  Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models.  _PLoS digital health_ 2, 2 (2023), e0000198.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural Questions: a Benchmark for Question Answering Research.  _Transactions of the Association of Computational Linguistics_ (2019).
  * Lahat et al. (2023) Adi Lahat, Eyal Shachar, Benjamin Avidan, Zina Shatz, Benjamin S Glicksberg, and Eyal Klang. 2023\.  Evaluating the use of large language model in identifying top research questions in gastroenterology.  _Scientific reports_ 13, 1 (2023), 4164.
  * Lai et al. (2023) Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023.  ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning.  _arXiv preprint arXiv:2304.05613_ (2023).
  * Lanzi and Loiacono (2023) Pier Luca Lanzi and Daniele Loiacono. 2023.  Chatgpt and other large language models as evolutionary engines for online interactive collaborative game design.  _arXiv preprint arXiv:2303.02155_ (2023).
  * Laskar et al. (2023) Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang. 2023\.  A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets.  _arXiv preprint arXiv:2305.18486_ (2023).
  * Le and Zhang (2023) Van-Hoang Le and Hongyu Zhang. 2023.  An Evaluation of Log Parsing with ChatGPT.  _arXiv preprint arXiv:2306.01590_ (2023).
  * LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015\.  Deep learning.  _nature_ 521, 7553 (2015), 436–444.
  * Lee et al. (2023) Noah Lee, Na Min An, and James Thorne. 2023.  Can Large Language Models Infer and Disagree Like Humans?  _arXiv preprint arXiv:2305.13788_ (2023).
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019\.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ (2019).
  * Li et al. (2023e) Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023e.  Seed-bench: Benchmarking multimodal llms with generative comprehension.  _arXiv preprint arXiv:2307.16125_ (2023).
  * Li et al. (2023g) Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023g.  CMMLU: Measuring massive multitask language understanding in Chinese.  _arXiv preprint arXiv:2306.09212_ (2023).
  * Li et al. (2023d) Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023d.  API-Bank: A Benchmark for Tool-Augmented LLMs.  arXiv:2304.08244 [cs.CL]
  * Li et al. (2023a) Ruyu Li, Wenhao Deng, Yu Cheng, Zheng Yuan, Jiaqi Zhang, and Fajie Yuan. 2023a.  Exploring the Upper Limits of Text-Based Collaborative Filtering Using Large Language Models: Discoveries and Insights.  _arXiv preprint arXiv:2305.11700_ (2023).
  * Li et al. (2023c) Xinzhe Li, Ming Liu, Shang Gao, and Wray Buntine. 2023c.  A Survey on Out-of-Distribution Evaluation of Neural NLP Models.  arXiv:2306.15261 [cs.CL]
  * Li et al. (2023f) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023f.  AlpacaEval: An Automatic Evaluator of Instruction-following Models.  https://github.com/tatsu-lab/alpaca_eval.
  * Li et al. (2023b) Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023b.  Evaluating object hallucination in large vision-language models.  _arXiv preprint arXiv:2305.10355_ (2023).
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022\.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_ (2022).
  * Liang et al. (2023a) Tian Liang, Zhiwei He, Jen-tes Huang, Wenxuan Wang, Wenxiang Jiao, Rui Wang, Yujiu Yang, Zhaopeng Tu, Shuming Shi, and Xing Wang. 2023a.  Leveraging Word Guessing Games to Assess the Intelligence of Large Language Models.  _arXiv preprint arXiv:2310.20499_ (2023).
  * Liang et al. (2023b) Xun Liang, Shichao Song, Simin Niu, Zhiyu Li, Feiyu Xiong, Bo Tang, Zhaohui Wy, Dawei He, Peng Cheng, Zhonghao Wang, et al. 2023b.  UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generation.  _arXiv preprint arXiv:2311.15296_ (2023).
  * Liévin et al. (2022) Valentin Liévin, Christoffer Egeberg Hother, and Ole Winther. 2022.  Can large language models reason about medical questions?  _arXiv preprint arXiv:2207.08143_ (2022).
  * Lin (2004) Chin-Yew Lin. 2004\.  ROUGE: A Package for Automatic Evaluation of Summaries. In _Text Summarization Branches Out_. Association for Computational Linguistics, Barcelona, Spain, 74–81.  https://aclanthology.org/W04-1013
  * Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans. 2021\.  Truthfulqa: Measuring how models mimic human falsehoods.  _arXiv preprint arXiv:2109.07958_ (2021).
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context. In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 740–755.
  * Lin and Chen (2023) Yen-Ting Lin and Yun-Nung Chen. 2023.  LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models.  _arXiv preprint arXiv:2305.13711_ (2023).
  * Liu et al. (2023c) Chuang Liu, Renren Jin, Yuqi Ren, Linhao Yu, Tianyu Dong, Xiaohan Peng, Shuting Zhang, Jianxiang Peng, Peiyi Zhang, Qingqing Lyu, Xiaowen Su, Qun Liu, and Deyi Xiong. 2023c.  M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Models.  arXiv:2305.10263 [cs.CL]
  * Liu et al. (2023d) Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023d.  Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning.  arXiv:2306.14565 [cs.CV]
  * Liu et al. (2023e) Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023e.  Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4.  arXiv:2304.03439 [cs.CL]
  * Liu et al. (2023f) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023f.  Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.  _arXiv preprint arXiv:2305.01210_ (2023).
  * Liu et al. (2023a) Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2023a.  MMBench: Is Your Multi-modal Model an All-around Player?  arXiv:2307.06281 [cs.CV]
  * Liu et al. (2023b) Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. 2023b.  Summary of chatgpt/gpt-4 research and perspective towards the future of large language models.  _arXiv preprint arXiv:2304.01852_ (2023).
  * LMSYS (2023) LMSYS. 2023.  Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings.  https://lmsys.org.
  * Lopez-Lira and Tang (2023) Alejandro Lopez-Lira and Yuehua Tang. 2023.  Can chatgpt forecast stock price movements? Return predictability and large language models.  _arXiv preprint arXiv:2304.07619_ (2023).
  * Lyu et al. (2023b) Chenyang Lyu, Jitao Xu, and Longyue Wang. 2023b.  New trends in machine translation using large language models: Case examples with chatgpt.  _arXiv preprint arXiv:2305.01181_ (2023).
  * Lyu et al. (2023a) Qing Lyu, Josh Tan, Mike E Zapadka, Janardhana Ponnatapuram, Chuang Niu, Ge Wang, and Christopher T Whitlow. 2023a.  Translating radiology reports into plain language using chatgpt and gpt-4 with prompt learning: Promising results, limitations, and potential.  _arXiv preprint arXiv:2303.09038_ (2023).
  * Ma et al. (2021) Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potts, Adina Williams, and Douwe Kiela. 2021.  Dynaboard: An evaluation-as-a-service platform for holistic next-generation benchmarking.  _Advances in Neural Information Processing Systems_ 34 (2021), 10351–10367.
  * Manakul et al. (2023a) Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023a.  Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.  _arXiv preprint arXiv:2303.08896_ (2023).
  * Manakul et al. (2023b) Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023b.  MQAG: Multiple-choice Question Answering and Generation for Assessing Information Consistency in Summarization.  arXiv:2301.12307 [cs.CL]
  * Margatina et al. (2023) Katerina Margatina, Shuai Wang, Yogarshi Vyas, Neha Anna John, Yassine Benajiba, and Miguel Ballesteros. 2023.  Dynamic benchmarking of masked language models on temporal concept drift with multiple views.  _arXiv preprint arXiv:2302.12297_ (2023).
  * McCarthy (2007) John McCarthy. 2007\.  What is artificial intelligence.  (2007).
  * Microsoft (2023) Microsoft. 2023\.  Bing Chat.  _https://www.bing.com/new_ (2023).
  * Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023.  FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation.  _arXiv preprint arXiv:2305.14251_ (2023).
  * Nay et al. (2023) John J Nay, David Karamardian, Sarah B Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H Choi, and Jungo Kasai. 2023.  Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence.  _arXiv preprint arXiv:2306.07075_ (2023).
  * Nie et al. (2019) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019\.  Adversarial NLI: A new benchmark for natural language understanding.  _arXiv preprint arXiv:1910.14599_ (2019).
  * Nijkamp et al. (2022) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022\.  Codegen: An open large language model for code with multi-turn program synthesis.  _arXiv preprint arXiv:2203.13474_ (2022).
  * Novikova et al. (2017) Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017\.  Why we need new evaluation metrics for NLG.  _arXiv preprint arXiv:1707.06875_ (2017).
  * Oh et al. (2023) Namkee Oh, Gyu-Seong Choi, and Woo Yong Lee. 2023\.  ChatGPT goes to the operating room: evaluating GPT-4 performance and its potential in surgical education and training in the era of large language models.  _Annals of Surgical Treatment and Research_ 104, 5 (2023), 269\.
  * Olney (2023) Andrew M Olney. 2023\.  Generating multiple choice questions from a textbook: Llms match human performance on most metrics. In _AIED Workshops_.
  * OpenAI (2023a) OpenAI. 2023a.  https://chat.openai.com.chat.
  * OpenAI (2023b) OpenAI. 2023b.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Orrù et al. (2023) Graziella Orrù, Andrea Piarulli, Ciro Conversano, and Angelo Gemignani. 2023.  Human-like problem-solving abilities in large language models using ChatGPT.  _Frontiers in Artificial Intelligence_ 6 (2023).
  * Ott et al. (2023) Simon Ott, Konstantin Hebenstreit, Valentin Liévin, Christoffer Egeberg Hother, Milad Moradi, Maximilian Mayrhauser, Robert Praas, Ole Winther, and Matthias Samwald. 2023\.  ThoughtSource: A central hub for large language model reasoning data.  _arXiv preprint arXiv:2301.11596_ (2023).
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ 35 (2022), 27730–27744.
  * Pallagani et al. (2023) Vishal Pallagani, Bharath Muppasani, Keerthiram Murugesan, Francesca Rossi, Biplav Srivastava, Lior Horesh, Francesco Fabiano, and Andrea Loreggia. 2023.  Understanding the Capabilities of Large Language Models for Automated Planning.  _arXiv preprint arXiv:2305.16151_ (2023).
  * Pan et al. (2023) Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2023\.  Unifying Large Language Models and Knowledge Graphs: A Roadmap.  arXiv:2306.08302 [cs.CL]
  * Parisi et al. (2022) Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022.  Talm: Tool augmented language models.  _arXiv preprint arXiv:2205.12255_ (2022).
  * Parrish et al. (2022) Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022\.  BBQ: A hand-built bias benchmark for question answering. In _Findings of the Association for Computational Linguistics: ACL 2022_. 2086–2105.
  * Peña et al. (2023) Alejandro Peña, Aythami Morales, Julian Fierrez, Ignacio Serna, Javier Ortega-Garcia, Iñigo Puente, Jorge Cordova, and Gonzalo Cordova. 2023.  Leveraging Large Language Models for Topic Classification in the Domain of Public Affairs.  _arXiv preprint arXiv:2306.02864_ (2023).
  * Peng et al. (1997) Kaiping Peng, Richard E Nisbett, and Nancy YC Wong. 1997\.  Validity problems comparing values across cultures and possible solutions.  _Psychological methods_ 2, 4 (1997), 329.
  * Pezeshkpour (2023) Pouya Pezeshkpour. 2023\.  Measuring and Modifying Factual Knowledge in Large Language Models.  _arXiv preprint arXiv:2306.06264_ (2023).
  * Phang et al. (2021) Jason Phang, Angelica Chen, William Huang, and Samuel R Bowman. 2021.  Adversarially constructed evaluation sets are more challenging, but may not be fair.  _arXiv preprint arXiv:2111.08181_ (2021).
  * Pu and Demberg (2023) Dongqi Pu and Vera Demberg. 2023.  ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer.  arXiv:2306.07799 [cs.CL]
  * Qin et al. (2023c) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023c.  Is ChatGPT a general-purpose natural language processing task solver?  _arXiv preprint arXiv:2302.06476_ (2023).
  * Qin et al. (2023a) Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2023a.  Tool Learning with Foundation Models.  arXiv:2304.08354 [cs.CL]
  * Qin et al. (2023b) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023b.  ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs.  arXiv:2307.16789 [cs.AI]
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018\.  Improving language understanding by generative pre-training.  (2018).
  * Rawte et al. (2023) Vipula Rawte, Amit Sheth, and Amitava Das. 2023.  A Survey of Hallucination in Large Foundation Models.  _arXiv preprint arXiv:2309.05922_ (2023).
  * Ribeiro and Lundberg (2022) Marco Tulio Ribeiro and Scott Lundberg. 2022.  Adaptive testing and debugging of nlp models. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 3253–3267.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  _arXiv preprint arXiv:2005.04118_ (2020).
  * Riccardi and Desai (2023) Nicholas Riccardi and Rutvik H Desai. 2023.  The Two Word Test: A Semantic Benchmark for Large Language Models.  _arXiv preprint arXiv:2306.04610_ (2023).
  * Rutinowski et al. (2023) Jérôme Rutinowski, Sven Franke, Jan Endendyk, Ina Dormuth, and Markus Pauly. 2023\.  The Self-Perception and Political Biases of ChatGPT.  _arXiv preprint arXiv:2304.07333_ (2023).
  * Safdari et al. (2023) Mustafa Safdari, Greg Serapio-García, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2023.  Personality Traits in Large Language Models.  _arXiv preprint arXiv:2307.00184_ (2023).
  * Samaan et al. (2023) Jamil S Samaan, Yee Hui Yeo, Nithya Rajeev, Lauren Hawley, Stuart Abel, Wee Han Ng, Nitin Srinivasan, Justin Park, Miguel Burch, Rabindra Watson, et al. 2023\.  Assessing the accuracy of responses by the language model ChatGPT to questions regarding bariatric surgery.  _Obesity Surgery_ (2023), 1–7.
  * Saparov et al. (2023) Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Seyed Mehran Kazemi, Najoung Kim, and He He. 2023\.  Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples.  _arXiv preprint arXiv:2305.15269_ (2023).
  * Sawada et al. (2023) Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta, and Aran Komatsuzaki. 2023.  ARB: Advanced Reasoning Benchmark for Large Language Models.  arXiv:2307.13692 [cs.CL]
  * Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_ (2023).
  * Sharma et al. (2023) Prabin Sharma, Kisan Thapa, Prastab Dhakal, Mala Deep Upadhaya, Santosh Adhikari, and Salik Ram Khanal. 2023.  Performance of ChatGPT on USMLE: Unlocking the Potential of Large Language Models for AI-Assisted Medical Education.  _arXiv preprint arXiv:2307.00112_ (2023).
  * Shen et al. (2023) Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023\.  Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.  _arXiv preprint arXiv:2303.17580_ (2023).
  * Sheng et al. (2021) Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021.  Societal Biases in Language Generation: Progress and Challenges. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 4275–4293.
  * Simmons (2022) Gabriel Simmons. 2022\.  Moral mimicry: Large language models produce moral rationalizations tailored to political identity.  _arXiv preprint arXiv:2209.12106_ (2022).
  * Singhal et al. (2022) Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2022\.  Large Language Models Encode Clinical Knowledge.  _arXiv preprint arXiv:2212.13138_ (2022).
  * Singhal et al. (2023) Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023\.  Large language models encode clinical knowledge.  _Nature_ 620, 7972 (2023), 172–180.
  * Smith et al. (2022) Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022\.  Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model.  _arXiv preprint arXiv:2201.11990_ (2022).
  * Song et al. (2023) Xiaoyang Song, Akshat Gupta, Kiyan Mohebbizadeh, Shujie Hu, and Anant Singh. 2023.  Have Large Language Models Developed a Personality?: Applicability of Self-Assessment Tests in Measuring Personality in LLMs.  _arXiv preprint arXiv:2305.14693_ (2023).
  * Sridhara et al. (2023) Giriprasad Sridhara, Sourav Mazumdar, et al. 2023\.  ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks.  _arXiv preprint arXiv:2305.16837_ (2023).
  * Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022\.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _arXiv preprint arXiv:2206.04615_ (2022).
  * Sun et al. (2023) Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023\.  Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent.  _arXiv preprint arXiv:2304.09542_ (2023).
  * Tao et al. (2023) Zhengwei Tao, Zhi Jin, Xiaoying Bai, Haiyan Zhao, Yanlin Feng, Jia Li, and Wenpeng Hu. 2023.  EvEval: A Comprehensive Evaluation of Event Semantics for Large Language Models.  _arXiv preprint arXiv:2305.15268_ (2023).
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021\.  Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.  _arXiv preprint arXiv:2104.08663_ (2021).
  * Thirunavukarasu et al. (2023) Arun James Thirunavukarasu, Refaat Hassan, Shathar Mahmood, Rohan Sanghera, Kara Barzangi, Mohanned El Mukashfi, and Sachin Shah. 2023\.  Trialling a large language model (ChatGPT) in general practice with the Applied Knowledge Test: observational study demonstrating opportunities and limitations in primary care.  _JMIR Medical Education_ 9, 1 (2023), e46599.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022\.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_ (2022).
  * Thrush et al. (2022) Tristan Thrush, Kushal Tirumala, Anmol Gupta, Max Bartolo, Pedro Rodriguez, Tariq Kane, William Gaviria Rojas, Peter Mattson, Adina Williams, and Douwe Kiela. 2022\.  Dynatask: A framework for creating dynamic AI benchmark tasks.  _arXiv preprint arXiv:2204.01906_ (2022).
  * Tian et al. (2023) Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023.  Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.  _arXiv preprint arXiv:2305.14975_ (2023).
  * Tian et al. (2018) Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018\.  Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In _Proceedings of the 40th international conference on software engineering_. 303–314.
  * ToolBench (2023) ToolBench. 2023\.  Open-source tools learning benchmarks.  https://github.com/sambanova/toolbench.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023\.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ (2023).
  * Turing (2009) Alan M Turing. 2009\.  _Computing machinery and intelligence_.  Springer.
  * Valmeekam et al. (2023) Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. 2023.  On the Planning Abilities of Large Language Models–A Critical Investigation.  _arXiv preprint arXiv:2305.15771_ (2023).
  * Valmeekam et al. (2022) Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2022.  Large Language Models Still Can’t Plan (A Benchmark for LLMs on Planning and Reasoning about Change).  _arXiv preprint arXiv:2206.10498_ (2022).
  * Van Der Lee et al. (2019) Chris Van Der Lee, Albert Gatt, Emiel Van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019\.  Best practices for the human evaluation of automatically generated text. In _Proceedings of the 12th International Conference on Natural Language Generation_. 355–368.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Vu et al. (2023) Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2023.  FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation.  arXiv:2310.03214 [cs.CL]
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019\.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _Advances in neural information processing systems_ 32 (2019).
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018\.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_ (2018).
  * Wang et al. (2023a) Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. 2023a.  DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models.  arXiv:2306.11698 [cs.CL]
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 billion parameter autoregressive language model.
  * Wang et al. (2021b) Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. 2021b.  Adversarial glue: A multi-task benchmark for robustness evaluation of language models.  _arXiv preprint arXiv:2111.02840_ (2021).
  * Wang et al. (2023c) Cunxiang Wang, Sirui Cheng, Zhikun Xu, Bowen Ding, Yidong Wang, and Yue Zhang. 2023c.  Evaluating open question answering evaluation.  _arXiv preprint arXiv:2305.12421_ (2023).
  * Wang et al. (2023j) Hongru Wang, Rui Wang, Fei Mi, Zezhong Wang, Ruifeng Xu, and Kam-Fai Wong. 2023j.  Chain-of-thought prompting for responding to in-depth dialogue questions with LLM.  arXiv:2305.11792 [cs.CL]
  * Wang et al. (2023d) Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, et al. 2023d.  On the robustness of chatgpt: An adversarial and out-of-distribution perspective. In _ICLR workshop on Trustworthy and Reliable Large-Scale Machine Learning Models_.
  * Wang et al. (2022) Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. 2022.  Generalizing to unseen domains: A survey on domain generalization.  _IEEE Transactions on Knowledge and Data Engineering_ (2022).
  * Wang et al. (2023h) Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023h.  Document-level machine translation with large language models.  _arXiv preprint arXiv:2304.02210_ (2023).
  * Wang et al. (2023e) Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023e.  Large language models are not fair evaluators.  _arXiv preprint arXiv:2305.17926_ (2023).
  * Wang and Demszky (2023) Rose E Wang and Dorottya Demszky. 2023.  Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction.  _arXiv preprint arXiv:2306.03090_ (2023).
  * Wang et al. (2023b) Xidong Wang, Guiming Hardy Chen, Dingjie Song, Zhiyi Zhang, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, et al. 2023b.  CMB: A Comprehensive Medical Benchmark in Chinese.  _arXiv preprint arXiv:2308.08833_ (2023).
  * Wang et al. (2023g) Xuena Wang, Xueting Li, Zi Yin, Yue Wu, and Liu Jia. 2023g.  Emotional Intelligence of Large Language Models.  arXiv:2307.09042 [cs.AI]
  * Wang et al. (2023i) Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2023i.  MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback.  _arXiv preprint arXiv:2309.10691_ (2023).
  * Wang et al. (2021a) Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021a.  Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.  _arXiv preprint arXiv:2109.00859_ (2021).
  * Wang et al. (2023l) Yidong Wang, Zhuohao Yu, Jindong Wang, Qiang Heng, Hao Chen, Wei Ye, Rui Xie, Xing Xie, and Shikun Zhang. 2023l.  Exploring Vision-Language Models for Imbalanced Learning.  _arXiv preprint arXiv:2304.01457_ (2023).
  * Wang et al. (2023m) Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. 2023m.  PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization.  _arXiv preprint arXiv:2306.05087_ (2023).
  * Wang et al. (2023f) Zhuo Wang, Rongzhen Li, Bowen Dong, Jie Wang, Xiuxing Li, Ning Liu, Chenhui Mao, Wei Zhang, Liling Dong, Jing Gao, et al. 2023f.  Can LLMs like GPT-4 outperform traditional AI tools in dementia diagnosis? Maybe, but not today.  _arXiv preprint arXiv:2306.01499_ (2023).
  * Wang et al. (2023k) Zengzhi Wang, Qiming Xie, Zixiang Ding, Yi Feng, and Rui Xia. 2023k.  Is ChatGPT a Good Sentiment Analyzer? A Preliminary Study.  arXiv:2304.04339 [cs.CL]
  * Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ (2022).
  * Wei et al. (2022b) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Huai hsin Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022b.  Emergent Abilities of Large Language Models.  _Trans. Mach. Learn. Res._ 2022 (2022).
  * Wei et al. (2023) Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang. 2023.  CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?  arXiv:2306.16636 [cs.CL]
  * White et al. (2023) Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023.  A prompt pattern catalog to enhance prompt engineering with chatgpt.  _arXiv preprint arXiv:2302.11382_ (2023).
  * Wong (2015) Tzu-Tsung Wong. 2015\.  Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation.  _Pattern Recognition_ 48, 9 (2015), 2839–2846.
  * Wu et al. (2023c) Patrick Y Wu, Joshua A Tucker, Jonathan Nagler, and Solomon Messing. 2023c.  Large Language Models Can Be Used to Estimate the Ideologies of Politicians in a Zero-Shot Learning Setting.  _arXiv preprint arXiv:2303.12057_ (2023).
  * Wu et al. (2023a) Yiran Wu, Feiran Jia, Shaokun Zhang, Qingyun Wu, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, and Chi Wang. 2023a.  An Empirical Study on Challenging Math Problem Solving with GPT-4.  _arXiv preprint arXiv:2306.01337_ (2023).
  * Wu et al. (2022) Yuhuai Wu, Albert Qiaochu Jiang, Wenda Li, Markus Rabe, Charles Staats, Mateja Jamnik, and Christian Szegedy. 2022.  Autoformalization with large language models.  _Advances in Neural Information Processing Systems_ 35 (2022), 32353–32368.
  * Wu et al. (2023b) Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2023b.  Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks.  _arXiv preprint arXiv:2307.02477_ (2023).
  * Xie et al. (2023) Qiming Xie, Zengzhi Wang, Yi Feng, and Rui Xia. 2023\.  Ask Again, Then Fail: Large Language Models’ Vacillations in Judgement.  arXiv:2310.02174 [cs.CL]
  * Xu et al. (2023b) Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, and Erik Cambria. 2023b.  Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation From Deductive, Inductive and Abductive Views.  _arXiv preprint arXiv:2306.09841_ (2023).
  * Xu et al. (2023c) Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, Ji Zhang, Chao Peng, Fei Huang, and Jingren Zhou. 2023c.  CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility.  arXiv:2307.09705 [cs.CL]
  * Xu et al. (2023d) Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. 2023d.  LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models.  arXiv:2306.09265 [cs.CV]
  * Xu et al. (2023a) Ruiyun Xu, Yue Feng, and Hailiang Chen. 2023a.  ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience.  _arXiv preprint arXiv:2307.01135_ (2023).
  * Yang and Menczer (2023) Kai-Cheng Yang and Filippo Menczer. 2023.  Large language models can rate news outlet credibility.  _arXiv preprint arXiv:2304.00228_ (2023).
  * Yang et al. (2022) Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang. 2022.  Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective.  _arXiv preprint arXiv:2211.08073_ (2022).
  * Yin et al. (2023) Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, et al. 2023\.  LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark.  _arXiv preprint arXiv:2306.06687_ (2023).
  * Yu et al. (2023b) Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, et al. 2023b.  KoLA: Carefully Benchmarking World Knowledge of Large Language Models.  _arXiv preprint arXiv:2306.09296_ (2023).
  * Yu et al. (2023a) Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023a.  MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models.  _arXiv preprint arXiv:2309.12284_ (2023).
  * Yu et al. (2023c) Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023c.  Mm-vet: Evaluating large multimodal models for integrated capabilities.  _arXiv preprint arXiv:2308.02490_ (2023).
  * Yuan et al. (2023a) Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2023a.  Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations.  arXiv:2306.04618 [cs.CL]
  * Yuan et al. (2023b) Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. 2023b.  Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited.  arXiv:2303.13835 [cs.IR]
  * Yuan et al. (2023c) Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023c.  How well do Large Language Models perform in Arithmetic tasks?  _arXiv preprint arXiv:2304.02015_ (2023).
  * Zemel et al. (2013) Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013.  Learning fair representations. In _International conference on machine learning_. PMLR, 325–333.
  * Zeng et al. (2022) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022\.  Glm-130b: An open bilingual pre-trained model.  _arXiv preprint arXiv:2210.02414_ (2022).
  * Zhang et al. (2023h) Beichen Zhang, Kun Zhou, Xilin Wei, Wayne Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. 2023h.  Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning.  _arXiv preprint arXiv:2306.02408_ (2023).
  * Zhang et al. (2023i) Beichen Zhang, Kun Zhou, Xilin Wei, Wayne Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. 2023i.  Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning.  _arXiv preprint arXiv:2306.02408_ (2023).
  * Zhang et al. (2023b) Jizhi Zhang, Keqin Bao, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2023b.  Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation.  _arXiv preprint arXiv:2305.07609_ (2023).
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022\.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ (2022).
  * Zhang et al. (2023d) Sarah J Zhang, Samuel Florin, Ariel N Lee, Eamon Niknafs, Andrei Marginean, Annie Wang, Keith Tyser, Zad Chin, Yann Hicke, Nikhil Singh, et al. 2023d.  Exploring the MIT Mathematics and EECS Curriculum Using Large Language Models.  _arXiv preprint arXiv:2306.08997_ (2023).
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_ (2019).
  * Zhang et al. (2023a) Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023a.  M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models.  _arXiv preprint arXiv:2306.05179_ (2023).
  * Zhang et al. (2023c) Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2023c.  Sentiment Analysis in the Era of Large Language Models: A Reality Check.  _arXiv preprint arXiv:2305.15005_ (2023).
  * Zhang et al. (2023g) Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. 2023g.  Wider and deeper llm networks are fairer llm evaluators.  _arXiv preprint arXiv:2308.01862_ (2023).
  * Zhang et al. (2023f) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023f.  Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models.  arXiv:2309.01219 [cs.CL]
  * Zhang et al. (2023e) Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2023e.  SafetyBench: Evaluating the Safety of Large Language Models with Multiple Choice Questions.  _arXiv preprint arXiv:2309.07045_ (2023).
  * Zhao et al. (2023a) Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. 2023a.  MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning.  _arXiv preprint arXiv:2309.07915_ (2023).
  * Zhao et al. (2023b) Jiaxu Zhao, Meng Fang, Zijing Shi, Yitong Li, Ling Chen, and Mykola Pechenizkiy. 2023b.  CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models.  arXiv:2305.11262 [cs.CL]
  * Zhao et al. (2023d) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023d.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_ (2023).
  * Zhao et al. (2023c) Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, and Min Lin. 2023c.  On Evaluating Adversarial Robustness of Large Vision-Language Models.  _arXiv preprint arXiv:2305.16934_ (2023).
  * Zheng et al. (2023a) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, et al. 2023a.  LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset.  _arXiv preprint arXiv:2309.11998_ (2023).
  * Zheng et al. (2023b) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023b.  Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.  arXiv:2306.05685 [cs.CL]
  * Zhong et al. (2022) Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022.  Towards a unified multi-dimensional evaluator for text generation.  _arXiv preprint arXiv:2210.07197_ (2022).
  * Zhong et al. (2023) Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023.  Agieval: A human-centric benchmark for evaluating foundation models.  _arXiv preprint arXiv:2304.06364_ (2023).
  * Zhou et al. (2022) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022.  Large language models are human-level prompt engineers.  _arXiv preprint arXiv:2211.01910_ (2022).
  * Zhu et al. (2023) Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, et al. 2023\.  PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts.  _arXiv preprint arXiv:2306.04528_ (2023).
  * Zhuang et al. (2023) Yan Zhuang, Qi Liu, Yuting Ning, Weizhe Huang, Rui Lv, Zhenya Huang, Guanhao Zhao, Zheng Zhang, Qingyang Mao, Shijin Wang, et al. 2023\.  Efficiently Measuring the Cognitive Ability of LLMs: An Adaptive Testing Perspective.  _arXiv preprint arXiv:2306.10512_ (2023).
  * Zhuo et al. (2023a) Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. 2023a.  Exploring ai ethics of chatgpt: A diagnostic analysis.  _arXiv preprint arXiv:2301.12867_ (2023).
  * Zhuo et al. (2023b) Terry Yue Zhuo, Zhuang Li, Yujin Huang, Yuan-Fang Li, Weiqing Wang, Gholamreza Haffari, and Fatemeh Shiri. 2023b.  On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex.  _arXiv preprint arXiv:2301.12868_ (2023).
  * Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019\.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ (2019).
  * Ziems et al. (2023) Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023\.  Can Large Language Models Transform Computational Social Science?  _arXiv preprint arXiv:2305.03514_ (2023).


# Self-consistent population spectral synthesis with FADO: mean stellar
metallicity of galaxies in spectral synthesis methods

Ciro Pappalardo1,2

###### Abstract

In this work, we investigate the reliability of spectral synthesis methods in
the estimation of the mean stellar age and metallicity, addressing the
question of which signal-to-noise ratios (S/N) are needed to determine these
quantities and how these depend on the tool used to model the spectra. To
address this problem we used realistic simulated spectra containing stellar
and nebular emission, reproducing the evolution of a galaxy for a constant and
exponentially declining star formation law. The spectra have been degraded to
different S/N and analysed with three different spectral synthesis codes:
Fado, Steckmap, and Starlight assuming similar fitting set-ups and the same
spectral bases.

For S/N $\leq 5$ all tools considered show a large diversity in the results.
Fado and Starlight find median differences in light-weighted mean stellar ages
of $\sim$0.1 dex, while Steckmap shows a higher value of $\sim$0.2 dex. For
S/N $>50$ the median differences in Fado are $\sim$0.03 dex ($\sim 7$%), a
factor 3 and 4 lower than the 0.08 dex ($\sim$20%) and 0.11 dex ($\sim$30%)
obtained from Starlight and Steckmap, respectively.

Our results indicate that phases of high specific star formation rate (sSFR)
in galaxies require analysis tools that do not neglect the nebular continuum
emission in the fitting process, since purely stellar models would have strong
problems in the estimation of star formation history, even in presence of high
S/N spectra. The median value of these differences are of the order of 7%
(Fado), 20% (Starlight), and 30% (Steckmap) for light-weighted quantities, and
20% (Fado), 60% (Starlight), and 20% (Steckmap) for mass-weighted quantities.
That implies a severe overestimation of the mass-to-light ratio and stellar
mass, even in the presence of a mild contribution from the nebular continuum.
Our work underlines once more the importance of a self-consistent treatment of
nebular emission, which is the only route towards a reliable determination of
the assembly of any high-sSFR galaxy at high and low redshift.

Key words: galaxies: abundances — galaxies: ISM — techniques: spectroscopy

Contributed Paper

1Instituto de Astrofísica e Ciências do Espaço, Universidade de Lisboa - OAL,
Tapada da Ajuda, PT1349-018 Lisboa, Portugal
2Departamento de Física, Faculdade de Ciências da Universidade de Lisboa,
Edifício C8, Campo Grande, PT1749-016 Lisboa, Portugal

##  1 Introduction

Galaxy evolution will undergo a leap forwards in the next decade when a new
generation of multi-object spectroscopic facilities will be operational.
Surveys like 4MOST (de Jong et al., 2019), WEAVE (Dalton, 2016), and MOONS
(Cirasuolo & MOONS Consortium, 2016; Cirasuolo et al., 2020; Maiolino et al.,
2020) will open a new window into an unexplored area of research. These
spectra will tackle key problems related to the galaxy assembly, shedding
light on the physical processes governing their evolution. It is then of
paramount importance to refine and test reliable tools able to infer the
properties of a galaxy from the analysis of medium- or high-resolution
spectra. To address these questions, in this work we investigate the limits
and reliability of different spectral synthesis methods in the estimation of
two specific quantities representatives of the evolutionary status of a
galaxy: the mean stellar age and mean stellar metallicity. These parameters
are fundamental for determining the assembly history of a galaxy, by providing
key insights into its star formation history (SFH) and stellar mass growth.

The main question we want to address is at which signal-to-noise (S/N) values
it is still possible to determine, from a galaxy spectrum, its mean stellar
age and metallicity and how this depends on the tool used to model the
spectra.

To do this we built a set of simulated spectra reproducing the evolution of a
galaxy for a continuous star formation and a single initial burst with an
exponentially declining star formation law. We then degraded the synthetic
spectra to different S/N, simulating different observing conditions. The mock
spectra produced were then analysed with three spectral synthesis codes: Fado,
Steckmap, and Starlight assuming similar initial set-ups and spectral bases.
The main goals are a) to determine the S/N where the results for the
parameters considered are still reliable; b) to identify which stellar ages
the fitting tools are more sensitive to, and c) to quantify the effect of
nebular emission component in population synthesis codes. Section 2 describes
the procedure used, while Section 3 shows our results. In Section 4 we present
our conclusions.

##  2 Procedure

The Reckoning galaxy Emission By means of Evolutionary Tasks with Input Key
Observables (Rebetiko) code built the synthetic spectra analysed in this work,
applying the following equation:

$\begin{split}F(\lambda,t)=\int_{0}^{t}\Phi(t-t^{\prime})&F^{SSP}_{\lambda}(t^{\prime},Z)dt^{\prime}\\\
&+\frac{\gamma_{eff}(T_{e})}{\alpha_{B}(T_{e})}\int_{0}^{t}\Phi(t-t^{\prime})q^{SSP}(t^{\prime},Z)dt^{\prime}\end{split}.$
(1)

Eq. 1 reproduces the evolution of a galaxy spectral energy distribution (SED)
as a function of time. For each $t$ the flux $F(\lambda,t)$ at a specific
wavelength $\lambda$ depends on two terms: The first term represents the
emission due to the stellar component, which is the integral of the different
SSPs weighted for the star formation rate (SFR; $\Phi$). The second term
represents the flux contribution due to the nebular continuum, which depends
on the number of hydrogen ionising photons $q^{SSP}$ weighted for the star
formation. The spectra assume a constant solar metallicity and two different
star formation histories (SFHs): a continuous star formation (Cont model), and
an instantaneous burst followed by an exponentially declining star formation
law with $\tau$ = 1 Myr (TAU1 model). The Rebetiko spectra for the CONT and
TAU1 models are shown in Fig. 1 (adapted from Cardoso et al. 2019).

Figure 1.: Synthetic spectra normalised at $\lambda$ = 4020 Å for TAU1
(instantaneous burst, left panel) and CONT (continuous, right panel) SFH
models. Figures adapted from Fig. 1 and Fig. 2 of Cardoso et al. 2019.

The evolution of the synthetic SEDs simulated in Rebetiko is characterised by
the mean stellar age and the mean stellar metallicity weighted by the light
(mass) contribution to the total galaxy light (mass), considering logarithmic
time-steps, as in Cid Fernandes et al. 2005:

$\begin{split}\left<log(t_{\ast})\right>_{L}=\sum_{j=1}^{N_{\ast}}x_{j}\cdot
log(t_{j}),\\\
\left<log(t_{\ast})\right>_{M}=\sum_{j=1}^{N_{\ast}}\mu_{j}\cdot
log(t_{j})\end{split}.$ (2)

Analogously for the metallicity,

$\begin{split}\left<log(Z_{\ast})\right>_{L}=\sum_{j=1}^{N_{\ast}}x_{j}\cdot
log(Z_{j}),\\\
\left<log(Z_{\ast})\right>_{M}=\sum_{j=1}^{N_{\ast}}\mu_{j}\cdot
log(Z_{j}).\end{split}$ (3)

In Eq. 2 and 3, $t_{j}$ ($Z_{j}$) represents the age (metallicity) of the j-th
SSP element, where $x_{j}$ and $\mu_{j}$ are their associated light and mass
fractions at 4020 Å, respectively.

We add to the input Rebetiko spectra random Gaussian noise with a varying
standard deviation, obtaining a set of spectra with constant S/N values of 3,
5, 10, 20, 50, and 100. The spectral basis for the analysis is built from the
BaseL of Bruzual & Charlot 2003, considering 100 SSPs, 25 ages and four
metallicities ($Z$ = 0.004, 0.008, 0.02 and 0.05).

We selected for the analysis three spectral synthesis tools: Fado (Gomes &
Papaderos, 2017), which includes in its analysis nebular emission; the purely
stellar code Starlight (Cid Fernandes et al., 2005); and Steckmap (Ocvirk et
al., 2006), which can be considered a ‘hybrid’ approach.

##  3 Results

The synthetic spectra built with Rebetiko were analysed with the tools
described in Section 2, for six S/N values in the range $3\leq$ S/N $\leq$
100\. For each SFH, we estimated the mean stellar age and metallicity,
weighted by mass or light, as defined in Eq. 2 and 3.

###  3.1 Continuous star formation: CONT models

Fig. 2 shows the difference between $\left<log(t_{\ast})\right>_{L}$,
$\left<log(t_{\ast})\right>_{M}$, $\left<log(Z_{\ast})\right>_{L}$, and
$\left<log(Z_{\ast})\right>_{M}$ for CONT models with respect to the input
Rebetiko value. All tools show large differences of the fit at S/N = 3 and
have problems at ages $\log(t/\mathrm{yr})<8$, where the results have a large
spread as a consequence of the instability of the solutions. For older ages,
the results are more stable, with a tendency to underestimate
$\left<log(t_{\ast})\right>_{L}$.

For S/N $>20$ the differences between the Rebetiko
$\left<log(t_{\ast})\right>_{L}$ and the best fits from Fado are $\sim$0.03
dex ($\sim 7$%), a factor 3 and 4 lower than the 0.08 dex ($\sim$20%), and
0.12 dex ($\sim$30%) obtained from Starlight and Steckmap. The fits show small
variations, and the $\Delta$$\left<log(t_{\ast})\right>_{L}$ at ages above
$\log(t/\mathrm{yr})\sim 8$ decreases gradually, in correspondence with the
predominance of the post-AGB stars.

Once input spectra have S/N above 10 and galaxy models have ages above
$\log(t/\mathrm{yr})>8$ all tools converge towards the input solar
metallicity. Starlight shows some discrepancies,
$\Delta$$\left<log(Z_{\ast})\right>_{L}$$\sim 0.2$ dex, for ages below
$\log(t/\mathrm{yr})<8$, with similar patterns at S/N = 50 and 100. Steckmap
shows a mild overestimation of the metallicity at higher S/N and old ages,
lower than 0.01 dex.

Mass-weighted quantities are more noisy with respect to light-weighted fits
(Fig. 2), with Fado showing at S/N $\geq 10$ uncertainties of $\sim$0.09 dex,
larger than the 0.04 dex obtained for $\left<log(t_{\ast})\right>_{L}$.
Starlight results show median discrepancies of 0.12-0.24 dex with no real
trend with increasing S/N. All the tools have problems with the metallicity
evolution for spectra at S/N $<$ 20: Fado shows more stable fits at higher
S/N, while Starlight underestimates the metallicity by $\sim$0.01 dex, above
all at young ages.

Figure 2.: Results of CONT model analysis. Luminosity- and mass-weighted mean
stellar age (first and second row), and metallicity (third and fourth row)
residuals for Fado (blue line), Starlight (red line), Steckmap (green line) as
a function of the galaxy age. Luminosity-weighted quantities for Fado and
Starlight are obtained considering the best-fitting populations vector at 4020
Å. On top of each panel the S/N of the input Rebetiko spectrum are reported.
Credits: Reproduced with permission from Astronomy & Astrophysics, © ESO.
Figure 3.: Same caption of Fig. 2 for TAU1 SFH models. Credits: Reproduced
with permission from Astronomy & Astrophysics, © ESO.

###  3.2 Instantaneous burst: TAU1 models

Figure 3 reports the results obtained for the TAU1 models, which show an
overall consistency of the results, with
$\Delta$$\left<log(t_{\ast})\right>_{L}$ $\sim$2-3% for Fado and Starlight
already at S/N = 5. Steckmap results are dependent on the input S/N with
unstable fits at $\log(t/\mathrm{yr})>7.5$.

Mean metallicities are more homogeneous than CONT models, with unstable
solutions at $\log(t/\mathrm{yr})\sim 7$. In contrast to Fado and Starlight,
Steckmap fits show high instabilities at older ages above
$\log(t/\mathrm{yr})>9$. Also for TAU1 models mass-weighted quantities show a
larger scatter. Fado shows $\Delta$$\left<log(t_{\ast})\right>_{M}$ of $\sim
0.1-0.2$ dex, while for Starlight and Steckmap the uncertainties increase by a
factor of $\sim$3.

##  4 Conclusions

For each tool considered the results accuracy depends not only on the input
SFH but also on other factors, e.g. the S/N and the contribution of the
nebular continuum. The accuracy of $\left<log(t_{\ast})\right>_{L}$ and
$\left<log(t_{\ast})\right>_{M}$ changes drastically with the input SFH, with
CONT models showing an increase in the uncertainties by a factor $\sim$3-5
from light- to mass-weighted quantities. Larger discrepancies are linked to
phases where the nebular contribution is not negligible, which is particularly
evident in CONT spectra. This confirms the importance of considering nebular
emission for starburst systems, where this component can reach up to 50% of
the total optical and near-infrared emission (Krueger et al., 1995; Izotov et
al., 1997; Papaderos et al., 1998; Leitherer et al., 1999; Schaerer & de
Barros, 2009).

Our work underlines the importance of using appropriate codes when considering
the spectra from galaxies with high SFRs. A self-consistent treatment of
nebular emission assumes a fundamental role for galaxies in starburst phases,
which are more common at higher redshifts.

#### Acknowledgments.

This work was supported by Fundação para a Ciência e a Tecnologia (FCT)
through the research grants PTDC/FIS-AST/29245/2017, UID/FIS/04434/2019,
UIDB/04434/2020 and UIDP/04434/2020.

## References

  * Bruzual & Charlot (2003) Bruzual, G. & Charlot, S. 2003, MNRAS, 344, 1000. doi:10.1046/j.1365-8711.2003.06897.x
  * Cardoso et al. (2019) Cardoso, L. S. M., Gomes, J. M., & Papaderos, P. 2019, A&A, 622, A56. doi:10.1051/0004-6361/201833438
  * Cid Fernandes et al. (2005) Cid Fernandes, R., Mateus, A., Sodré, L., et al. 2005, MNRAS, 358, 363. doi:10.1111/j.1365-2966.2005.08752.x
  * Cirasuolo & MOONS Consortium (2016) Cirasuolo, M. & MOONS Consortium 2016, Multi-Object Spectroscopy in the Next Decade: Big Questions, Large Surveys, and Wide Fields, 507, 109
  * Cirasuolo et al. (2020) Cirasuolo, M., Fairley, A., Rees, P., et al. 2020, The Messenger, 180, 10. doi:10.18727/0722-6691/5195
  * Dalton (2016) Dalton, G. 2016, Multi-Object Spectroscopy in the Next Decade: Big Questions, Large Surveys, and Wide Fields, 507, 97
  * de Jong et al. (2019) de Jong, R. S., Agertz, O., Berbel, A. A., et al. 2019, The Messenger, 175, 3. doi:10.18727/0722-6691/5117
  * Gomes & Papaderos (2017) Gomes, J. M. & Papaderos, P. 2017, A&A, 603, A63. doi:10.1051/0004-6361/201628986
  * Izotov et al. (1997) Izotov, Y. I., Lipovetsky, V. A., Chaffee, F. H., et al. 1997, ApJ, 476, 698. doi:10.1086/303664
  * Krueger et al. (1995) Krueger, H., Fritze-v. Alvensleben, U., & Loose, H.-H. 1995, A&A, 303, 41
  * Leitherer et al. (1999) Leitherer, C., Schaerer, D., Goldader, J. D., et al. 1999, ApJS, 123, 3. doi:10.1086/313233
  * Maiolino et al. (2020) Maiolino, R., Cirasuolo, M., Afonso, J., et al. 2020, The Messenger, 180, 24. doi:10.18727/0722-6691/5197
  * Ocvirk et al. (2006) Ocvirk, P., Pichon, C., Lançon, A., et al. 2006, MNRAS, 365, 46. doi:10.1111/j.1365-2966.2005.09182.x
  * Papaderos et al. (1998) Papaderos, P., Izotov, Y. I., Fricke, K. J., et al. 1998, A&A, 338, 43
  * Schaerer & de Barros (2009) Schaerer, D. & de Barros, S. 2009, A&A, 502, 423. doi:10.1051/0004-6361/200911781