# Phi-3 Technical Report:
A Highly Capable Language Model Locally on Your Phone

Microsoft

###### Abstract

We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3
trillion tokens, whose overall performance, as measured by both academic
benchmarks and internal testing, rivals that of models such as Mixtral 8x7B
and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench),
despite being small enough to be deployed on a phone. The innovation lies
entirely in our dataset for training, a scaled-up version of the one used for
phi-2, composed of heavily filtered web data and synthetic data. The model is
also further aligned for robustness, safety, and chat format. We also provide
some initial parameter-scaling results with a 7B and 14B models trained for
4.8T tokens, called phi-3-small and phi-3-medium, both significantly more
capable than phi-3-mini (e.g., respectively 75% and 78% on MMLU, and 8.7 and
8.9 on MT-bench).

##  1 Introduction

The striking progress of AI in the last few years can be largely attributed to
major efforts throughout the world towards scaling-up to ever-larger models
and datasets. Large Language Models (LLMs) have steadily increased in size
from a mere billion parameters just five years ago (GPT-2 had 1.5 billion
parameters [RWC+19]) to trillion parameters today. The impetus for this effort
originates in the seemingly predictable improvement one obtains by training
large models, the so-called scaling laws [KMH+20, HBM+22, MRB+23]. However
these laws assume a “fixed” data source. This assumption is now significantly
disrupted by the existence of frontier LLMs themselves, which allow us to
interact with data in novel ways. In our previous works on the phi models
[GZA+23, LBE+23, JBA+23] it was shown that a combination of LLM-based
filtering of web data, and LLM-created synthetic data, enable performance in
smaller language models that were typically seen only in much larger models.
For example our previous model trained on this data recipe, phi-2 (2.7B
parameters), matched the performance of models $25$ times larger trained on
regular data. In this report we present a new model, phi-3-mini (3.8B
parameters), trained for 3.3T tokens on larger and more advanced versions of
the datasets used in phi-2. With its small size, phi-3-mini can easily be
inferenced locally on a modern phone (see Figure 1), yet it achieves a quality
that seems on-par with models such as Mixtral 8x7B [JSR+24] and GPT-3.5.

User: Explain why it is surprising that one can build a language model small
enough to fit on a phone, yet almost as powerful as ChatGPT. Just use one
funny sentence. phi-3-mini: It’s like fitting a supercomputer in a flip phone,
but instead of breaking the phone, it just breaks the internet with its tiny,
yet mighty, linguistic prowess! User: Okay now more serious answer, and note
that this was achieved solely by changing the training data. phi-3-mini: The
development of a compact language model that rivals the capabilities of
ChatGPT, while fitting on a phone, is a testament to the power of data-driven
machine learning. By meticulously curating and optimizing the training
dataset, researchers can significantly reduce the model’s size without
compromising its performance. [...]

##  2 Technical Specifications

The phi-3-mini model is a transformer decoder architecture [VSP+17], with
default context length $4K$. We also introduce a long context version via
LongRope [DZZ+24] that extends the context length to $128K$, called
phi-3-mini-128K.

To best benefit the open source community, phi-3-mini is built upon a similar
block structure as Llama-2 [TLI+23] and uses the same tokenizer with
vocabulary size of 32064111We remove BoS tokens and add some additional tokens
for chat template.. This means that all packages developed for Llama-2 family
of models can be directly adapted to phi-3-mini. The model uses $3072$ hidden
dimension, $32$ heads and $32$ layers. We trained using bfloat16 for a total
of 3.3T tokens. The model is already chat-finetuned, and the chat template is
as follows:

<|user|>$\backslash$n Question <|end|>$\backslash$n <|assistant|>

The phi-3-small model (7B parameters) leverages the tiktoken tokenizer (for
better multilingual tokenization) with a vocabulary size of 100352 and has
default context length $8K$. It follows the standard decoder architecture of a
7B model class, having $32$ layers and a hidden size of $4096$. To minimize KV
cache footprint, the model also leverages a grouped-query attention, with $4$
queries sharing $1$ key. Moreover phi-3-small uses alternative layers of dense
attention and a novel blocksparse attention to further optimize on KV cache
savings while maintaining long context retrieval performance. An additional
10% multilingual data was also used for this model.

### Highly capable language model running locally on a cell-phone.

Thanks to its small size, phi-3-mini can be quantized to 4-bits so that it
only occupies $\approx$ 1.8GB of memory. We tested the quantized model by
deploying phi-3-mini on iPhone 14 with A16 Bionic chip running natively on-
device and fully offline achieving more than $12$ tokens per second.

Figure 1: 4-bit quantized phi-3-mini running natively on an iPhone with A16
Bionic chip, generating over 12 tokens per second.

### Training Methodology.

We follow the sequence of works initiated in “Textbooks Are All You Need”
[GZA+23], which utilize high quality training data to improve the performance
of small language models and deviate from the standard scaling-laws. In this
work we show that such method allows to reach the level of highly capable
models such as GPT-3.5 or Mixtral with only 3.8B total parameters (while
Mixtral has 45B total parameters for example). Our training data of consists
of heavily filtered web data (according to the “educational level”) from
various open internet sources, as well as synthetic LLM-generated data. Pre-
training is performed in two disjoint and sequential phases; phase-1 comprises
mostly of web sources aimed at teaching the model general knowledge and
language understanding. Phase-2 merges even more heavily filtered webdata (a
subset used in Phase-1) with some synthetic data that teach the model logical
reasoning and various niche skills.

### Data Optimal Regime.

Unlike prior works that train language models in either “compute optimal
regime” [HBM+22] or “over-train regime”, we mainly focus on the quality of
data for a given scale.222Just like for “compute optimal regime”, we use the
term “optimal” in an aspirational sense for “data optimal regime”. We are not
implying that we actually found the provably “optimal” data mixture for a
given scale. We try to calibrate the training data to be closer to the “data
optimal” regime for small models. In particular, we filter the web data to
contain the correct level of “knowledge” and keep more web pages that could
potentially improve the “reasoning ability” for the model. As an example, the
result of a game in premier league in a particular day might be good training
data for frontier models, but we need to remove such information to leave more
model capacity for “reasoning” for the mini size models. We compare our
approach with Llama-2 in Figure 2.

Figure 2: Scaling law close to the “Data Optimal Regime” (from left to right:
phi-1.5, phi-2, phi-3-mini, phi-3-small) versus Llama-2 family of models (7B,
13B, 34B, 70B) that were trained on the same fixed data. We plot the log of
MMLU error versus the log of model size.

To test our data on larger size of models, we also trained phi-3-medium, a
model with 14B parameters using the same tokenizer and architecture of
phi-3-mini, and trained on the same data for slightly more epochs (4.8T tokens
total as for phi-3-small). The model has 40 heads and 40 layers, with
embedding dimension 5120. We observe that some benchmarks improve much less
from 7B to 14B than they do from 3.8B to 7B, perhaps indicating that our data
mixture needs further work to be in the “data optimal regime” for 14B
parameters model. We are still actively investigating some of those benchmarks
(including a regression on HumanEval), hence the numbers for phi-3-medium
should be considered as a “preview”.

### Post-training.

Post-training of phi-3-mini went through two stages, including supervised
finetuning (SFT) and direct preference optimization (DPO). SFT leverages
highly curated high-quality data across diverse domains, e.g., math, coding,
reasoning, conversation, model identity, and safety. The SFT data mix starts
with using English-only examples. DPO data covers chat format data, reasoning,
and responsible AI (RAI) efforts. We use DPO to steer the model away from
unwanted behavior, by using those outputs as “rejected” responses. Besides
improvement in math, coding, reasoning, robustness, and safety, post-training
transforms a language model to an AI assistant that users can efficiently and
safely interact with.

As part of the post-training process, we developed a long context version of
phi-3-mini with context length limit enlarged to 128K instead of 4K. Across
the board, the 128K model quality is on par with the 4K length version, while
being able to handle long context tasks. Long context extension has been done
in two stages, including long context mid-training and long-short mixed post-
training with both SFT and DPO.

##  3 Academic benchmarks

On the next page we report the results for phi-3-mini on standard open-source
benchmarks measuring the model’s reasoning ability (both common sense
reasoning and logical reasoning). We compare to phi-2 [JBA+23],
Mistral-7b-v0.1 [JSM+23], Mixtral-8x7b [JSR+24], Gemma 7B [TMH+24],
Llama-3-instruct-8b [AI23], and GPT-3.5. All the reported numbers are produced
with the exact same pipeline to ensure that the numbers are comparable. These
numbers might differ from other published numbers due to slightly different
choices in the evaluation. As is now standard, we use few-shot prompts to
evaluate the models, at temperature $0$. The prompts and number of shots are
part of a Microsoft internal tool to evaluate language models, and in
particular we did no optimization to the pipeline for the phi-3 models.333For
example, we found that using ## before the Question can lead to a noticeable
improvement to phi-3-mini’s results across many benchmarks, but we did not do
such changes in the prompts. The number of $k$–shot examples is listed per-
benchmark. An example of a 2-shot prompt is described in Appendix A.

Phi-3-mini 3.8b Phi-3-small 7b (preview) Phi-3-medium 14b (preview) Phi-2 2.7b
Mistral 7b Gemma 7b Llama-3-In 8b Mixtral 8x7b GPT-3.5 version 1106 MMLU
(5-Shot) [HBK+21] 68.8 75.3 78.2 56.3 61.7 63.6 66.0 68.4 71.4 HellaSwag
(5-Shot) [ZHB+19] 76.7 78.7 83.0 53.6 58.5 49.8 69.5 70.4 78.8 ANLI (7-Shot)
[NWD+20] 52.8 55.0 58.7 42.5 47.1 48.7 54.8 55.2 58.1 GSM-8K (0-Shot; CoT)
[CKB+21] 82.5 88.9 90.3 61.1 46.4 59.8 77.4 64.7 78.1 MedQA (2-Shot) [JPO+20]
53.8 58.2 69.4 40.9 49.6 50.0 58.9 62.2 63.4 AGIEval (0-Shot) [ZCG+23] 37.5
45.0 48.4 29.8 35.1 42.1 42.0 45.2 48.4 TriviaQA (5-Shot) [JCWZ17] 64.0 59.1
75.6 45.2 72.3 75.2 73.6 82.2 85.8 Arc-C (10-Shot) [CCE+18] 84.9 90.7 91.0
75.9 78.6 78.3 80.5 87.3 87.4 Arc-E (10-Shot) [CCE+18] 94.6 97.1 97.8 88.5
90.6 91.4 92.3 95.6 96.3 PIQA (5-Shot) [BZGC19] 84.2 87.8 87.7 60.2 77.7 78.1
77.1 86.0 86.6 SociQA (5-Shot) [BZGC19] 76.6 79.0 80.2 68.3 74.6 65.5 73.2
75.9 68.3 BigBench-Hard (0-Shot) [SRR+22, SSS+22] 71.7 75.0 81.3 59.4 57.3
59.6 68.9 69.7 68.32 WinoGrande (5-Shot) [SLBBC19] 70.8 82.5 81.4 54.7 54.2
55.6 58.0 62.0 68.8 OpenBookQA (10-Shot) [MCKS18] 83.2 88.4 87.2 73.6 79.8
78.6 81.6 85.8 86.0 BoolQ (0-Shot) [CLC+19] 77.2 82.9 86.6 – 72.2 66.0 78.3
77.6 79.1 CommonSenseQA (10-Shot) [THLB19] 80.2 80.3 82.6 69.3 72.6 76.2 73.6
78.1 79.6 TruthfulQA (10-Shot) [LHE22] 65.0 68.7 75.7 – 52.1 53.0 62.0 60.1
85.8 HumanEval (0-Shot) [CTJ+21] 59.1 59.1 55.5 47.0 28.0 34.1 60.4 37.8 62.2
MBPP (3-Shot) [AON+21] 70.0 71.4 74.5 60.6 50.8 51.5 65.3 60.2 77.8 Average
71.2 74.9 78.2 – 61.0 62.0 68.0 69.9 75.3 GPQA (2-Shot; CoT) [RHS+23] 32.8
34.3 – – – – – – 29.0 MT Bench (2 round ave.) [ZCS+23] 8.38 8.70 8.91 – – – –
– 8.35

##  4 Safety

Phi-3-mini was developed in accordance with Microsoft’s responsible AI
principles. The overall approach consisted of safety alignment in post-
training, red-teaming, automated testing and evaluations across dozens of RAI
harm categories. Helpfulness and harmlessness preference datasets [BJN+22,
JLD+23] with modifications inspired by [BSA+24] and multiple in-house
generated datasets were leveraged to address the RAI harm categories in safety
post-training. An independent red team at Microsoft iteratively examined
phi-3-mini to further identify areas of improvement during the post-training
process. Based on their feedback, we curated additional datasets tailored to
address their insights, thereby refining the post-training dataset. This
process resulted in significant decrease of harmful response rates, as shown
in Figure 3.

Figure 3: Comparison of harmful response percentages by Microsoft AI Red Team
between phi-3-mini before and after the safety alignment. Note that the
harmful response percentages in this chart are inflated numbers as the red
team tried to induce phi-3-mini in an adversarial way to generate harmful
responses through multi-turn conversations.

Table 1 shows the results of in-house RAI benchmarks for phi-3-mini-4k and
phi-3-mini-128k compared to phi-2 [JBA+23], Mistral-7b-v0.1 [JSM+23], Gemma 7b
[TMH+24], and Llama-3-instruct-8b [AI23]. This benchmark utilized GPT-4 to
simulate multi-turn conversations in five different categories and to evaluate
the model responses. Ungroundedness between 0 (fully grounded) and 4 (not
grounded) measures if the information in a response is based on a given
prompt. In other categories, responses were evaluated in terms of the severity
of harmfulness from 0 (no harm) to 7 (extreme harm) and the defect rates
(DR-$x$) were computed as the percentage of samples with the severity score
being greater than or equal to $x$.

Phi-3-Mini-4k 3.8b Phi-3-Mini-128k 3.8b Phi-2 2.7b Mistral 7b Gemma 7b
Llama-3-In 8b Ungroundedness 0.603 0.637 1.481 0.935 0.679 0.328 Intellectual
Property (DR-1) 23.95% 21.50% 24.00% 56.20% 38.33% 37.30% Harmful Content
Continuation (DR-3) 0.75% 1.08% 2.93% 2.58% 1.28% 1.30% Harmful Content
Summarization (DR-3) 10.00% 10.20% 14.35% 22.33% 10.33% 8.20% Jailbreak (DR-1)
12.29% 12.57% 15.00% 15.57% 11.43% 13.00%

Table 1: Comparison of Microsoft internal multi-turn conversation RAI
benchmark results of phi-3-mini and other models. Note that a lower value
indicates a better performance for all metrics in the table.

##  5 Weakness

In terms of LLM capabilities, while phi-3-mini model achieves similar level of
language understanding and reasoning ability as much larger models, it is
still fundamentally limited by its size for certain tasks. The model simply
does not have the capacity to store too much “factual knowledge”, which can be
seen for example with low performance on TriviaQA. However, we believe such
weakness can be resolved by augmentation with a search engine. We show an
example using the HuggingFace default Chat-UI with phi-3-mini in Figure 4.
Another weakness related to model’s capacity is that we mostly restricted the
language to English. Exploring multilingual capabilities for Small Language
Models is an important next step, with some initial promising results on
phi-3-small by including more multilingual data.

Despite our diligent RAI efforts, as with most LLMs, there remains challenges
around factual inaccuracies (or hallucinations), reproduction or amplification
of biases, inappropriate content generation, and safety issues. The use of
carefully curated training data, and targeted post-training, and improvements
from red-teaming insights significantly mitigates these issues across all
dimensions. However, there is significant work ahead to fully address these
challenges.

Figure 4: Left: phi-3-mini’s completion without search. Right: phi-3-mini’s
completion with search, using the default HuggingFace Chat-UI search ability.

## References

  * [AI23] Meta AI.  Introducing meta llama 3: The most capable openly available llm to date, 2023.
  * [AON+21] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton.  Program synthesis with large language models.  arXiv preprint arXiv:2108.07732, 2021.
  * [BJN+22] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan.  Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
  * [BSA+24] Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou.  Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions, 2024.
  * [BZGC19] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, and Yejin Choi.  Piqa: Reasoning about physical commonsense in natural language.  arXiv preprint arXiv:1911.11641, 2019.
  * [CCE+18] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
  * [CKB+21] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems.  arXiv preprint arXiv:2110.14168, 2021.
  * [CLC+19] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, 2019.
  * [CTJ+21] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code, 2021.
  * [DZZ+24] Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang.  Longrope: Extending llm context window beyond 2 million tokens, 2024.
  * [GZA+23] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Gustavo de Rosa Piero Kauffmann, Olli Saarikivia, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li.  Textbooks are all you need.  arXiv preprint arXiv:2306.11644, 2023.
  * [HBK+21] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.  Measuring mathematical problem solving with the MATH dataset, 2021.
  * [HBM+22] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Eliza Rutherford Trevor Cai, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.  Training compute-optimal large language models.  arXiv preprint arXiv:2203.15556, 2022.
  * [JBA+23] Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, Suriya Gunasekar, Piero Kauffmann, Yin Tat Lee, Yuanzhi Li, Anh Nguyen, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Michael Santacroce, Harkirat Singh Behl, Adam Taumann Kalai, Xin Wang, Rachel Ward, Philipp Witte, Cyril Zhang, and Yi Zhang.  Phi-2: The surprising power of small language models.  Microsoft Research Blog, 2023.
  * [JCWZ17] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017.
  * [JLD+23] Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang.  Beavertails: Towards improved safety alignment of llm via a human-preference dataset, 2023.
  * [JPO+20] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits.  What disease does this patient have? a large-scale open domain question answering dataset from medical exams, 2020.
  * [JSM+23] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7b, 2023.
  * [JSR+24] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mixtral of experts, 2024.
  * [KMH+20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  arXiv preprint arXiv:2001.08361, 2020.
  * [LBE+23] Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee.  Textbooks are all you need ii: phi-1.5 technical report.  arXiv preprint arXiv:2309.05463, 2023.
  * [LHE22] Stephanie Lin, Jacob Hilton, and Owain Evans.  Truthfulqa: Measuring how models mimic human falsehoods, 2022.
  * [MCKS18] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering, 2018.
  * [MRB+23] Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel.  Scaling data-constrained language models.  arXiv preprint arXiv:2305.16264, 2023.
  * [NWD+20] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.  Adversarial nli: A new benchmark for natural language understanding, 2020\.
  * [RHS+23] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman.  Gpqa: A graduate-level google-proof q&a benchmark, 2023.
  * [RWC+19] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [SLBBC19] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  Winogrande: An adversarial winograd schema challenge at scale.  arXiv preprint arXiv:1907.10641, 2019.
  * [SRR+22] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  arXiv preprint arXiv:2206.04615, 2022.
  * [SSS+22] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei.  Challenging big-bench tasks and whether chain-of-thought can solve them, 2022.
  * [THLB19] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.  Commonsenseqa: A question answering challenge targeting commonsense knowledge, 2019.
  * [TLI+23] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [TMH+24] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al.  Gemma: Open models based on gemini research and technology, 2024.
  * [VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems, volume 30, 2017.
  * [ZCG+23] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan.  Agieval: A human-centric benchmark for evaluating foundation models, 2023\.
  * [ZCS+23] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  arXiv preprint arXiv:2306.05685, 2023.
  * [ZHB+19] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019.

##  Appendix A Example prompt for benchmarks

Question: Solve for $x$: $(-\frac{1}{3})(-4-3x)=\frac{1}{2}$ Options: A.
$-\frac{5}{6}$ B. $\frac{7}{6}$ C. $\frac{5}{3}$ D. $\frac{1}{6}$ Answer: A
Question: Which of the following is the body cavity that contains the
pituitary gland? Options: A. Abdominal B. Cranial C. Pleural D. Spinal Answer:
B Question: Where was the most famous site of the mystery cults in Greece?
Options: A. Ephesus B. Corinth C. Athens D. Eleusis Answer:

##  Appendix B Authors

Marah Abdin |  Russell J. Hewett |  Olatunji Ruwase
---|---|---
Sam Ade Jacobs |  Jamie Huynh |  Olli Saarikivi
Ammar Ahmad Awan |  Mojan Javaheripi |  Amin Saied
Jyoti Aneja |  Xin Jin |  Adil Salim
Ahmed Awadallah |  Piero Kauffmann |  Michael Santacroce
Hany Awadalla |  Nikos Karampatziakis |  Shital Shah
Nguyen Bach |  Dongwoo Kim |  Ning Shang
Amit Bahree |  Mahmoud Khademi |  Hiteshi Sharma
Arash Bakhtiari |  Lev Kurilenko |  Xia Song
Harkirat Behl |  James R. Lee |  Masahiro Tanaka
Alon Benhaim |  Yin Tat Lee |  Xin Wang
Misha Bilenko |  Yuanzhi Li |  Rachel Ward
Johan Bjorck |  Chen Liang |  Guanhua Wang
Sébastien Bubeck |  Weishung Liu |  Philipp Witte
Martin Cai |  Eric Lin |  Michael Wyatt
Caio César Teodoro Mendes |  Zeqi Lin |  Jiahang Xu
Weizhu Chen |  Piyush Madan |  Can Xu
Vishrav Chaudhary |  Arindam Mitra |  Sonali Yadav
Parul Chopra |  Hardik Modi |  Fan Yang
Allie Del Giorno |  Brandon Norick |  Ziyi Yang
Gustavo de Rosa |  Anh Nguyen |  Donghan Yu
Matthew Dixon |  Barun Patra |  Chengruidong Zhang
Ronen Eldan |  Daniel Perez-Becker |  Cyril Zhang
Dan Iter |  Heyang Qin |  Jianwen Zhang
Amit Garg |  Thomas Portet |  Li Lyna Zhang
Abhishek Goswami |  Reid Pryzant |  Yi Zhang
Suriya Gunasekar |  Sambuddha Roy |  Yue Zhang
Emman Haider |  Marko Radmilac |  Yunan Zhang
Junheng Hao |  Corby Rosset |  Xiren Zhou


11institutetext: Zinovy Malkin 22institutetext: Pulkovo Observatory,
Pulkovskoe Sh. 65, St. Petersburg 196140, Russia

# On the selection of prospective sources for ICRF extension

Z. Malkin

###### Abstract

Despite continuous increasing of the number of ICRF sources, their sky
coverage is still not satisfactory. The goal of this study is to discuss some
new considerations for extending the ICRF source list. Statistical analysis of
the ICRF catalog allows us to identify less populated sky regions where new
ICRF sources or additional observations of the current ICRF sources are most
desirable to improve both the uniformity of the source distribution and the
uniformity of the distribution of the position errors. It is also desirable to
include more sources with high redshift in the ICRF list. These sources may be
of interest for astrophysics. To select prospective new ICRF sources, the
OCARS catalog is used. The number of sources in OCARS is about three times
greater than in the ICRF3, which gives us an opportunity to select new ICRF
sources that have already be tested and detected in astrometric and geodetic
VLBI experiments.

###### Keywords:

ICRF, VLBI, OCARS, SREAG

##  1 Introduction

The International Celestial Reference Frame (ICRF) is the standard of the
celestial reference frame recommended by the International Astronomical Union
(IAU) for scientific researches and practical applications in various fields.
The third ICRF realization, ICRF3 (Charlot et al., 2020) approved by the IAU
in 2018 is currently in use. It contains positions of 4588 sources included in
three catalogs ICRF3-SX, ICRF3-XKa, and ICRF3-K containing source positions
determined from observations at three radio frequency bands. The ICRF3-SX
catalog containing of 4536 sources observed at the $S$/$X$ band is most
complete and it is currently used to link ICRF and $Gaia$-CRF catalogs.

Although the number of sources in the ICRF catalog is continuously increased
with time and the accuracy of their positions is continuously improved, the
sky coverage by the ICRF sources, its density and uniformity, as well as the
sky distribution of the source position errors, are still not satisfactory.
For this reason, several institutions are considering new observing programs
aimed at improving the ICRF. Statistical analysis of the ICRF catalog allows
us to identify less populated sky regions where new ICRF sources or additional
observations of the current ICRF sources are most desirable to improve both
the uniformity of the source distribution and the uniformity of the
distribution of the position errors. One more consideration for extending the
ICRF list is including of sources with high redshift. These sources may be of
interest for astrophysics.

The main goal of this work is to discuss new approaches for improving the
source list for further ICRF extensions. Especial attention will be given to
formalization and quantification of some criteria (mostly known) for the
selection of new sources for future ICRF extensions, as well as identification
of the ICRF sources that need more observations to provide more uniform sky
distribution of the position errors.

To select prospective new ICRF sources, the catalog OCARS (Optical
Characteristics of Astrometric Radio Sources) (Malkin, 2018)) is used. The
current version of the OCARS catalog includes more than 13 thousand sources
which is about three times greater than the ICRF3 catalog. The OCARS non-ICRF
sources mostly have lower position accuracy compared to ICRF, but these
sources have already been tested and detected in VLBI experiments. This gives
us an opportunity to select new ICRF sources from the OCARS list without
resource consuming detection tests, which are necessary if new sources are
pre-selected from general radio surveys.

##  2 Analysis of possibilities for improving ICRF source list

This study is based on the analysis of the sky distribution of the ICRF3 and
$Gaia$ catalogs, and position errors of the ICRF3 sources. To quantify the
uniformity of the source sky distribution a method of pixelization of the
spherical surface SREAG (Spherical Rectangular Equal-Area Grid) (Malkin, 2019,
2020)) is used. It provides rectangular equal area cells with wide range of
resolutions from $\sim$45∘ to $\sim$16′′ (for 31-bit integer arithmetic).
Fortran routines to perform basic operations with the SREAG pixelization are
provided111http://www.gaoran.ru/english/as/ac_vlbi/. Table. 1 shows basic
parameters of several grids that looks most relevant for ICRF studies. In this
paper, the grid with $N_{ring}$=10 (128 cells) has been used for plots of the
sky distribution of the ICRF3-SX, $Gaia$, and OCARS sources, and the grid with
$N_{ring}$=18 (412 cells) has been used for plots of the sky distribution of
the ICRF3 defining sources.

Table 1: SREAG grid parameters as a function of the number of rings $N_{ring}$. $N_{ring}$ | $N_{cell}$ | Cell area
---|---|---
|  | [sq. deg]
4 | 20 | 2063
6 | 46 | 897
8 | 82 | 503
10 | 128 | 322
12 | 184 | 224
14 | 250 | 165
16 | 326 | 127
18 | 412 | 100
20 | 508 | 81

Figure 1 shows the sky distribution of the ICRF3-SX and OCARS sources. The
upper panel of this plot allows us not only to see the well-known weakness of
the ICRF3 catalog in the south, but also to identify the sky regions where the
ICRF list should be enriched with new sources to obtain more even source
distribution over the sky, in particular, the region of the Galactic plane. It
can be seen that the number of sources in the poorly populated cells in the
OCARS catalog is two-four times larger than in the ICRF3-SX catalog, which
opens a new opportunity to select new ICRF sources from the OCARS source list.

Figure 1: Sky distribution of sources for ICRF3-SX (top, 4536 sources) and
OCARS (bottom, 13560 sources).

Similar situation can be observed when analyzing the sky distribution of
common ICRF3-SX and $Gaia$ sources (Fig. 2). In this work, the $Gaia$ EDR3
astrometric catalog (Lindegren et al., 2020) was used. One can see again the
source deficiency in the south and near the Galactic plane. The number of
cross-identified sources between the $Gaia$ EDR3 and OCARS catalogs is more
than twice as much as the number of cross-identified sources between the
$Gaia$ EDR3 and ICRF3-SX catalogs. Therefore, the selection of new ICRF
sources from the list of cross-identified $Gaia$/OCARS sources will allow us
to improve the uniformity of the source distribution of common ICRF/$Gaia$
sources and thus improve the accuracy, both systematic and stochastic, of the
ICRF/$Gaia$ link. It should be noted that uneven distribution of common
sources may cause problems with analysis of the coordinate differences ICRF
minus $Gaia$ by means of spherical harmonics.

Figure 2: Sky distribution of sources cross-identified with $Gaia$ EDR3 for
ICRF3-SX (top, 3485 sources) and OCARS (bottom, 8707 sources) catalogs.

It is important for ICRF not only to have even source distribution over the
sky but also even sky distribution of the position error. Figure 3 shows the
sky distribution of the ICRF3-SX sources with the position errors less than 1
mas and less than 0.2 mas. For this work, the source position error is
computed as the semi-major axis of the error ellipse. This analysis allows us
to identify the sky regions were additional observations of ICRF sources
should be primarily planned.

Figure 3: Sky distribution of the ICRF3-SX sources with the position errors
less than 1 mas (top, 4177 sources) and less than 0.2 mas (bottom, 2023
sources).

An important characteristics of the ICRF catalog is the number of defining
sources and how evenly they are distributed over the sky. The nearest goal may
be to have $\approx$400 ICRF4 defining sources (one per 100 sq. deg). Figure 4
shows the sky distribution of the ICRF3 defining sources. One can see that
there are many cells that lack defining sources. So, the future ICRF release
should take this into account. Similar analysis was made earlier by Basu et
al. (2018).

Figure 4: Sky distribution of the ICRF3 defining sources.

Another consideration which is worth bearing in mind is having more ICRF
sources with high redshift. These sources are important for astrophysical and
cosmological studies Coppejans et al. (2017). Therefore this makes sense to
include the redshift as a criterion for source selection for the ICRF
extension. OCARS provides redshift info for about 58% of the sources. Table 2
shows the number of sources with high redshift in the ICRF3 and OCARS
catalogs. The number of high-z sources in the OCARS catalog is about two-four
times larger that in the ICRF3 catalog. So, inclusion of additional high-z
sources from OCARS will make the future ICRF releases more useful for non-
astrometric astronomical studies.

Table 2: Number of sources with high redshift in the ICRF3 and OCARS catalogs. Catalog | Redshift
---|---
| all | $\leq 3$ | $\leq 4$ | $\leq 5$
ICRF3 | 3382 | 106 | 15 | 3
OCARS | 7967 | 213 | 58 | 13

Increasing the number of observations of prospective ICRF sources requires
either to involve supplement network resources, which is not always possible,
or to improve the scheduling strategy. A possible way to make more
observations of CRF sources was discussed and tested in Malkin et al. (2013).

We started with actual schedule for IVS session R1591 that involved the
11-station network. In the original (actual) IVS schedule for the R1541
session, 60 sources were observed including 7 southern sources with
declination less than $-40^{\circ}$. For comparisons, the supplementary
southern sources were added to the original source list and three experimental
schedules were obtained to evaluate the trade-off between the number of
southern sources and the accuracy of geodetic products. Schedule ‘R1’ was
obtained with the original R1591 source list. Schedule ‘R1+’ includes three
more southern sources, and schedule ‘R1++’ includes six more southern sources
as compared with the original R1541 schedule. The three schedules for 24-hour
continuous observations were generated with VieVS scheduling package (Sun et
al., 2011).

For Monte Carlo simulation, 50 sessions were generated using the same 24-hour
schedule but different realizations of noise delays, each time creating new
values for wet zenith delay, clocks and white noise to simulate observations
as realistic as possible. The simulated NGS data files were entered into the
software package VieVS (Böhm et al. 2012), which computes a classical least
squares solution. The source coordinates were fixed to the ICRF2 positions,
and only Earth orientation parameters (EOP) and station positions were
estimated. The standard deviation of the 50 EOP estimates and mean formal
uncertainties obtained in our computations are listed in Table 3. One can see
that we found no overall degradation of the EOP accuracy after the inclusion
of supplement southern sources. Errors in some EOP became even smaller with
inclusion of more southern sources, and some EOP showed minor degradation in
the accuracy.

Table 3: Repeatability and standard deviation of EOP for the IVS R1541 and two experimental schedules R1+ and R1++ (Malkin et al. 2013). Parameter |  | R1 | R1+ | R1++
---|---|---|---|---
Number of scans |  | 1258 | 1351 | 1375
Number of observations |  | 3905 | 3813 | 3997
EOP repeatability | Xp | 143.2 | 125.5 |  98.2
[$\mu$as, $\mu$s] | Yp |  98.2 |  79.1 |  96.8
| UT1 |  5.6 |  4.6 |  5.9
| dX |  36.2 |  42.8 |  39.1
| dY |  45.0 |  39.5 |  37.2
Mean EOP uncertainty | Xp |  94.8 |  95.6 |  93.4
[$\mu$as, $\mu$s] | Yp |  77.2 |  77.3 |  74.8
| UT1 |  4.4 |  4.6 |  4.7
| dX |  29.8 |  30.9 |  29.5
| dY |  29.1 |  29.6 |  28.1

As to the baseline length repeatability, it was found that for the baselines
shorter than $\sim$5,000 km the R1 schedule shows the best result, and R1+ and
R1++ schedules shows worse repeatability, whereas for longer baselines the
R1++ schedule is the best, and R1 is the worst. However, in fact, the results
obtained with the three schedules are close to each other. The mean baseline
length repeatability derived from R1, R1+, and R1++ schedules are 13.5 mm,
12.4 mm, and 11.9 mm, respectively. In other words, increasing of the number
of southern sources (cf. R++ and R+ schedules) leads to a small degradation of
baseline length repeatability for short baselines, and small improvement for
long baselines. However, an overall improvement in the baseline length
repeatability was found after inclusion more southern sources in the schedule.

Summarizing the results of this simulation experiment, one can conclude that
the inclusion of targeted southern sources in regular IVS sessions such as R1
and R4 can help to increase substantially the number of ICRF southern sources
with accurate positions. Suppose, we want to add 100 new sources in the south
observed each 100 times (to obtain sub-mas position error) during one year.
Then we need to make $\sim$200 observations of these sources per week (cf.
current $\sim$10K observations in R1+R4 weekly schedule), which would not
significantly influence the normal IVS operations, and might even provide some
improvement in obtained geodetic parameters.

##  3 Conclusion

In this paper, an approach is discussed to select new sources for ICRF
extension and identify sources already included in the ICRF that preferably
need more observations to improve their position. It is suggested that the
OCARS catalog (Malkin, 2018)) can be used as an initial (candidate) list of
already VLBI-detected sources to enrich ICRF.

Of course, some sources selected from OCARS using sky coverage criterion
discussed in this paper may be too weak and/or may have a bad structure index
(SI) to be used in the regular geodetic observing programs aimed at EOP and
TRF monitoring. However, both source SI and flux are often variable and change
with time (Charlot, 2008; Charlot et al., 2010). So, currently “bad” source
can become a “good” one and vice versa.

It should be emphasized that the ICRF is not only needed for geodesy. Many
other scientific and practical applications, from navigation to astrophysics
and cosmology, will appreciate a dense and highly accurate catalog of radio
source positions evenly distributed over the sky. Low flux radio sources are
also important for astronomy, especially those sources that are cross-
identified with objects in other wave bands such as optics, X-ray or gamma-
ray. Such cross-identification is also provided by OCARS.

It should be also noted that increasing the number of ICRF sources evenly
distributed over the sky is important for improving the link between ICRF and
$Gaia$-CRF. Therefore, inclusion of radio weak but optically bright sources in
ICRF will be of mutual benefit. OCARS provides optical and NIR magnitudes for
about 78% of the sources, which can be used for source selection. The
selection of candidate sources for ICRF extension that can be used for the
ICRF–$Gaia$ link can be made even simpler because OCARS provides the cross-
identification table with the $Gaia$ catalog.

Therefore it is worth spending efforts to enrich the next ICRF catalogs not
only with sources “convenient” for solving say IERS- and GGOS-related tasks,
but also with other sources that would improve general ICRF quality. In
particular, observations of weak radio sources for ICRF can be performed in
cooperation with other VLBI networks, which include large antennas.

##  4 Acknowledgments

This research has made use of the SAO/NASA Astrophysics Data
System222https://ui.adsabs.harvard.edu/ (ADS). The figures were prepared using
`gnuplot`333http://www.gnuplot.info/.

## References

  * (1)
  * Basu et al. (2018) Basu S, de Witt A, Quick J, Malkin Z (2018)  Multi-epoch VLBI images to study the ICRF-3 Defining Sources in the Southern Hemisphere.  In: _14th European VLBI Network Symposium & Users Meeting. 8-11 October. Granada, Spain_, PoS(EVN2018)135,  doi: 10.22323/1.344.0135
  * Charlot (2008) Charlot P (2008)  Astrophysical Stability of Radio Sources and Implication for the Realization of the Next ICRF.  In: A. Finkelstein and D. Behrend (eds.) _5th IVS General Meeting Proceedings, St. Petersburg, Russia, 2008_ , 345–354
  * Charlot et al. (2010) Charlot P, Boboltz DA, Fey AL, et al. (2010)  The celestial reference frame at 24 and 43 GHz. II. Imaging.  _AJ_ , 139, 1713,  doi: 10.1088/0004-6256/139/5/1713
  * Charlot et al. (2020) Charlot P, Jacobs CS, Gordon D, et al. (2020)  The third realization of the International Celestial Reference Frame by very long baseline interferometry.  _A &A_, 644, 159,  doi: 10.1051/0004-6361/202038368
  * Coppejans et al. (2017) Coppejans R, van Velzen S; Intema HT, et al. (2020)  Radio spectra of bright compact sources at z $>$ 4.5.  _MNRAS_ , 467, 2039  doi: 10.1093/mnras/stx215
  * Lindegren et al. (2020) Lindegren L, Klioner SA, Hernandez J, et al. (2020)  $Gaia$ Early Data Release 3: The astrometric solution.  _A &A_,  doi: 10.1051/0004-6361/202039709
  * Malkin et al. (2013) Malkin Z, Sun J, Böhm J, Böhm S, Krásná H (2013)  Searching for an Optimal Strategy to Intensify Observations of the Southern ICRF sources in the framework of the regular IVS observing programs.  In: N. Zubko and M. Poutanen (eds.) _21st Meeting of the European VLBI Group for Geodesy and Astronomy, Espoo, Finland, March 5–8, 2013_ , 199–204
  * Malkin (2018) Malkin Z (2020)  A New Version of the OCARS Catalog of Optical Characteristics of Astrometric Radio Sources.  _AJ_ , 239, 20,  doi: 10.3847/1538-4365/aae777
  * Malkin (2019) Malkin Z (2019)  A New Equal-area Isolatitudinal Grid on a Spherical Surface.  _AJ_ , 158, 158,  doi: 10.3847/1538-3881/ab3a44
  * Malkin (2020) Malkin Z (2020)  Spherical Rectangular Equal-Area Grid (SREAG)–Some features.  In: C. Bizouard (ed.) _Proc. Journees 2019 Astrometry, Earth Rotation, and Reference Systems in the GAIA era_ , 55–59