▪$\bullet$ ○$\bigcirc$ ●$\bullet$ ■$\blacksquare$ □$\square$

# MAP-Neo: Highly Capable and Transparent
Bilingual Large Language Model Series

M-A-P, University of Waterloo, Wuhan AI Research, 01.AI

###### Abstract

Large Language Models (LLMs) have made great strides in recent years to
achieve unprecedented performance across different tasks. However, due to
commercial interest, the most competitive models like GPT, Gemini, and Claude
have been gated behind proprietary interfaces without disclosing the training
details. Recently, many institutions have open-sourced several strong LLMs
like LLaMA-3, comparable to existing closed-source LLMs. However, only the
model’s weights are provided with most details undisclosed (e.g., intermediate
checkpoints, pre-training corpus, and training code, etc). To improve the
transparency of LLMs, the research community has formed to open-source truly
open LLMs (e.g., Pythia, Amber, OLMo), where more details (e.g., pre-training
corpus and training code) are being provided. These models have greatly
advanced the scientific study of these large models including their strengths,
weaknesses, biases and risks. However, we observe that the existing truly open
LLMs are still inferior to existing state-of-the-art LLMs with similar model
sizes on reasoning, knowledge, and coding tasks. To this end, we open-source
MAP-Neo, a highly capable and transparent bilingual language model with 7B
parameters trained from scratch on 4.5T high-quality tokens. Our MAP-Neo is
the first fully open-sourced bilingual LLM with comparable performance
compared to existing state-of-the-art LLMs. Moreover, we open-source all
details to reproduce our MAP-Neo, where the cleaned pre-training corpus, data
cleaning pipeline, checkpoints, and well-optimized training/evaluation
framework111https://github.com/multimodal-art-projection/MAP-NEO are provided.
Finally, we hope our MAP-Neo will enhance and strengthen the open research
community and inspire more innovations and creativities to facilitate the
further improvements of LLMs.

https://map-neo.github.io/

Figure 1: MAP-Neo shows impressive performance on base (Left) and chat (Right)
models compared to both popular open-weight and recent transparent large
language models with similar sizes.

###### Contents

  1. 1 Introduction
  2. 2 Related Works
  3. 3 Tokenizer
  4. 4 Matrix Data Pile
    1. 4.1 Re-processing Pipeline for Open Datasets
      1. 4.1.1 Filtering
      2. 4.1.2 Deduplication
    2. 4.2 Corpora Crawl from Scratch Pipeline
      1. 4.2.1 Filtering
      2. 4.2.2 Deduplication
      3. 4.2.3 Similar Line Deduplication
    3. 4.3 Document Conversion Pipeline
    4. 4.4 High-Quality Supplement Data Collection
  5. 5 Model
    1. 5.1 Model Architecture
    2. 5.2 Model Scale Hyperparameters
  6. 6 Pre-training
    1. 6.1 Fundamental Phase: General Ability Acquisition
    2. 6.2 Decay Phase: Improvement and Rectification
  7. 7 Alignment
    1. 7.1 Supervised Fine-tuning
      1. 7.1.1 Data
      2. 7.1.2 Training
    2. 7.2 Iterative DPO
  8. 8 Scaling Law of MAP-Neo
    1. 8.1 Problem Definition
    2. 8.2 NEO Scaling Law
    3. 8.3 Generalization of NEO Scaling Law
  9. 9 Infrastructure
  10. 10 Evaluations
    1. 10.1 Base Model Performance
      1. 10.1.1 Main Results
      2. 10.1.2 Discussions
    2. 10.2 Aligned Model Performance
      1. 10.2.1 Main Results
      2. 10.2.2 Discussions
  11. 11 Societal Impact
  12. 12 Conclusion
  13. 13 Contributions and Acknowledgments
  14. 14 Multimodal Art Projection
  15. A Appendix
    1. A.1 Details of Heuristic Rules for English Texts
    2. A.2 Details of Heuristic Rules for Chinese Texts
    3. A.3 Training Framework Overflow Details
    4. A.4 Detailed Prompts in Intermediate Checkpoints Evaluation
    5. A.5 Detailed Results
    6. A.6 Details of Open Source Datasets Used in Pre-training
    7. A.7 Detailed Compression Rate
    8. A.8 Additional Experimental Results in Scaling Law
    9. A.9 Compression Rate
    10. A.10 OCR Post Processing

##  1 Introduction

The advent of generalist large language models (LLMs) such as GPT-4 [1],
Claude [4], and Gemini [80] has significantly expanded the boundaries of
Natural Language Processing (NLP) and is paving the way towards Artificial
General Intelligence (AGI). These models exhibit universal capabilities,
including complex reasoning [116, 89], role-playing [107], creative writing
[105], psychological assessment [112], scientific education [18], and music
generation [115, 75, 29], among others. However, the most advanced ones remain
closed-source due to commercial interests [1, 4, 80]. In this paper, we argue
that open-source and transparent LLMs are essential for both the
democratization of LLMs and further academic research, especially considering
the substantial resources these models consume.

Previous works have released numerous open-source or even transparent LLMs.
For example, the LLaMA series [101, 102, 3] released the weights, thereby
significantly boosting the development of the open-source LLM community.
However, they are not transparent because they do not disclose the details of
their training data. BLOOM [86] trained a multilingual language model with 176
billion parameters and open-sourced its model weights, intermediate
checkpoints, and training corpus. Models like LLM360 [66] and Pythia [9]
further provided their training codes, optimizer state checkpoints, analysis
codes, and data pipelines.

These models make significant contributions to building transparent
ecosystems, yet generally lag behind industry-level LLMs such as LLaMA [3],
Mistral [48] and Yi [113], etc. OLMo [36] has made a great stride in narrowing
this gap by improving pre-training data and data processing pipelines, and
introducing more open-source components, including training logs and
ablations. Nonetheless, it remains less proficient, especially in areas like
coding (HumanEval [15]), reasoning (MATH [41], GSM8K [23]), knowledge (MMLU
[40]), and multilingualism (CMMLU [60]).

To remedy these issues, we introduce MAP-Neo, a fully open-source and
transparent bilingual LLM suite that achieves superior performance to close
the gap with closed-source models. Specifically, the entire workflow of
building an LLM includes:

  1. 1.

Data Curation Pipeline: We provide the code for the curation and cleaning of
training data (both English and Chinese), including a stable OCR system, the
data recalling mechanism in DeepSeek-Math [89], the integration of previous
open-source data processing pipelines, and support for distributed data
processing based on Spark222https://spark.apache.org/, among others.

  2. 2.

Data: We release our pre-training corpus, namely Matrix Data Pile, along with
the training data for supervised fine-tuning and alignment training.

  3. 3.

Model Architecture: We provide the codes and details of our modeling
architecture.

  4. 4.

Model Training: We offer the training codes for our tokenizer, base models,
instruction-tuned models, and aligned models. Additionally, we address some
issues of the Megatron-LM framework333https://github.com/NVIDIA/Megatron-LM,
enhancing its support for more robust and efficient distributed training.
Moreover, we introduce the NEO Scaling Law designed to optimize scaling up
LLMs using a pre-training dataset sourced from diverse corpora.

  5. 5.

Model Checkpoints: We not only release the final models on HuggingFace but
also make the intermediate checkpoints available for reproducibility.

  6. 6.

Infrastructure: This report details the infrastructure for stable training.

  7. 7.

Evaluation: We also provide detailed evaluation codes and thorough evaluation
settings for benchmarking the performance of LLMs.

  8. 8.

Analysis and Lessons: This report elaborates on numerous techniques and
recipes, such as optimization tricks at different phases of pre-training, and
offers insights into building LLMs through rigorous analysis and ablations.

Our work is a milestone towards fully transparent LLMs with advanced
abilities, even competitive with the top closed-source LLMs. Notably, our
contribution is not just a novel foundational model but also a comprehensive
handbook for building LLMs from scratch, covering the entire workflow. We
believe that our model provides a critical reference for the community,
particularly for non-English regions of the world engaged in LLM research.

##  2 Related Works

Table 1: Compare with other open-source large language models (LLMs). All
metrics are obtained using the same evaluation manner, and the details are
shown in Table 9. Non-transparent models are listed above the dashed line,
while the transparent LLMs are shown below.

Model | Intermediate | Pre-training | Reproduction | Data Cleaning | C-EVAL | MMLU | GSM8K | HumanEval
---|---|---|---|---|---|---|---|---
Checkpoints | Corpus | Code | Process
Mistral-7B [48] | ✗ | ✗ | ✗ | ✗ | 47.54 | 64.04 | 47.46 | 28.0
LLaMA2-7B [102] | ✗ | ✗ | ✗ | ✗ | 32.37 | 46.80 | 16.22 | 13.4
LLaMA3-8B [3] | ✗ | ✗ | ✗ | ✗ | 49.83 | 66.52 | 54.74 | 33.5
Pythia-6.9B [9] | ✓ | ✓ | ✓ | ✓ | 24.64 | 26.39 | 3.41 | 9.1
Amber-7B [66] | ✓ | ✓ | ✓ | ✗ | 23.82 | 28.07 | 3.64 | 13.4
OLMo-7B [36] | ✓ | ✓ | ✓ | ✓ | 35.21 | 53.52 | 28.43 | 11.6
MAP-Neo-7B | ✓ | ✓ | ✓ | ✓ | 57.68 | 58.14 | 53.68 | 23.8

The development of open-source large language models (LLMs) is pivotal for
advancing artificial intelligence research and applications. Recent efforts in
this domain have been focused on not only enhancing model performance [48, 3]
but also ensuring transparency and reproducibility [9, 66, 36, 128]. Our
model, MAP-Neo-7B, emerges as the new lead in this evolving landscape, as
shown in Table 1, which balances performance and transparency.

The MAP-Neo model series represents a step forward in emphasizing full
transparency, aligning it alongside other contemporary models such as Mistral
[48], LLaMA3 [3], Pythia [9], Amber [66], and OLMo [36]. Unlike these models,
which often lack either intermediate checkpoints, comprehensive data cleaning
processes, or accessible pre-training corpus and reproduction code, MAP-Neo
excels by integrating all these elements. This commitment to the openness of
MAP-Neo facilitates in-depth analysis and independent validation by the
research community.

Performance-wise, MAP-Neo-7B demonstrates superior capabilities across a broad
scope of benchmarks including Chinese and English understanding on C-EVAL [46]
and MMLU [20], mathematical ability on GSM8K [23] and MATH [41], and code
ability on HumanEval [15]. Notably, MAP-Neo-7B is the only model in our
comparative analysis to achieve all checks in transparency, as well as the
highest scores across all tests compared with other transparent LLMs,
underscoring the effectiveness of the training and the quality of the data.

The most similar work to MAP-Neo is OLMo [36], which is the pioneering work to
fully open-source LLMs. However, their performance is compromised in several
aspects like knowledge, coding, and mathematical reasoning. Moreover, OLMo
cannot handle languages beyond English. MAP-Neo sets a new standard for
transparency and performance in the field of open-source LLMs. By fostering a
fully transparent development process, MAP-Neo not only enhances its utility
and trustworthiness but also provides a valuable framework for future
research, promoting further advancements and collaborative efforts in the
community.

##  3 Tokenizer

We train our tokenizer using the byte-pair encoding (BPE) algorithm [88] via
the implementation of SentencePiece [56]. The training data consists of 50B
samples from the pre-training corpus, and the maximum length is cut to 64K. We
assign higher sampling weights to code, math, and high-quality academic data.
To balance the computational efficiency and model performance, we propose to
set the vocabulary size to 64000 and constrain the max sentence-piece length
to 16 to improve the Chinese performance.

Notably, we slice all numbers into individual digits and fall back unknown
UTF-8 characters to byte granularity. We do not use any normalization strategy
on the training samples and do not add dummy prefixes. The character coverage
rate is set to 0.9999. Particularly, the remove extra whitespaces parameter is
set to False, which is turned on by default in the SentencePieceTrainer. This
setting can severely impact code performance during pre-training, as normal
code indentation is treated as a single space. We encountered a specific issue
during the initial phase of our model’s pre-training. Initially, we did not
disable the ‘remove extra whitespaces’ parameter, which is enabled by default
in the SentencePieceTrainer. In the training process, we observe steady
improvements in the QA reasoning and mathematics benchmarks, but the code
metrics exhibit fluctuations and do not show expected improvements. To address
this issue, we fixed this bug in the second phase of our training (§6.2),
which stabilizes and significantly improves the code metrics. Furthermore, we
observe that this issue is well addressed in the decay phase training stages
under the new tokenizer settings, where rapid improvements are achieved.

Moreover, we also investigate the compression rates across various categories
of data, categorized by both language (Chinese and English) and data source
quality (high-quality and web-sourced) as shown in Table 2. Specifically,
first, we observe that the high-quality data (HQ) including complex reasoning,
mathematical, and general knowledge texts, showing different compression rates
between Chinese (HQ_cn) and English (HQ_en). The HQ_cn category has a
compression rate of 1.577, while the HQ_en category exhibited a higher rate of
3.311 characters per token. Second, data sourced from the web (Web) also
comprise more characters than Chinese ones. This suggests a significant
variation in tokenization efficiency or character usage between languages,
possibly due to the linguistic structure and the tokenization methods. Third,
it should be mentioned that even with similar compression rates, the settings
of the tokenizer can cause significant fluctuations in the pre-training
process. Therefore, it remains necessary to further investigate tokenization
strategies for subsequent usage scenarios.

Table 2: Average Compression Rates by Category. These subsets are not uniformly proportioned in the training set. A detailed distribution is shown in Appendix 18. Code | HQ_cn | HQ_en | Web_cn | Web_en | Others
---|---|---|---|---|---
2.951 | 1.577 | 3.311 | 1.418 | 3.699 | 2.558

##  4 Matrix Data Pile

Figure 2: Statistics of the Matrix Pile Data Distribution: The inner pie chart
represents the language distribution, while the outer loop indicates the
proportion of meta-categories in the corpus.

It is widely recognized that a well-constructed training corpus is essential
for training LLMs. The training corpus serves as the fuel driving advancements
in language modeling, as demonstrated by the emergent capabilities of models
like ChatGPT, Claude, Gemini, and Llama. However, due to intellectual property
restrictions, the pre-training data and processing toolkits of these
(partially) proprietary LLMs are not disclosed upon release. Although the
open-source research community has made substantial efforts to increase
transparency in the collection and processing pipeline of language model pre-
training data [9, 86, 95], the development of fully open-sourced LLMs still
lags behind proprietary LLMs to some extent, primarily due to gaps in the
quantity and quality of the datasets.

To address the pressing need for more diverse and transparent datasets in
language modeling, we introduce Matrix, a bilingual pre-training corpus of
4.5T tokens. Upon its release, Matrix could be the largest transparent LLM
pre-training corpus to our best knowledge. Specifically, Matrix provides the
details of the data collection and processing along with a high-performance
toolkit. Additionally, we design Matrix based on the idea of retrieving,
filtering, and cleaning high-quality data under various practical
circumstances, which are discussed as follows:

  * •

Given a set of existing (English) pre-training datasets, how do we re-process
and improve the quality? §4.1

  * •

How do we construct a large-scale, topic-comprehensive corpus from scratch, on
the less explored Chinese content?§4.2

  * •

If we have enormous printed documents, how do we build an efficient and
effective system to extract viable textual contents? §4.3

  * •

When specifying a domain of interest, how do we find relevant high-quality
data from the wild of web content? §4.4

The final composition of the corpus is as follows: 52.55% from Common Crawl,
22.29% from programming code, and the rest from academic papers, books, and
other printed materials, as illustrated in Figure 2. The detailed
methodologies for processing these sources are described in the subsequent
sections, and a comprehensive illustration of the sources is provided in Table
16.

Table 3: The composition sources of re-processed English web subset. The proportion denotes dividing the size of the current dataset by the total size of the whole dataset. Dataset | Parts | UTF-8 bytes (TB) | Availability | Proportion (%)
---|---|---|---|---
RedPajama-Data-V2 [25] | Head and Middle | 200 | Public | $92.38$
Dolma [95] | CC | 6.4 | Public | $2.96$
Cultrax [72] | EN | 1.2 | Public | $0.55$
Amber [66] | Refined-Web | 4.23 | Public | $1.95$
SlimPajama [94] | Whole | 2.43 | Public | $1.12$
Falcon [74] | Whole | 1.01 | Public | $0.47$
CultraY [100] | EN | 1.24 | Public | $0.57$

###  4.1 Re-processing Pipeline for Open Datasets

Although several processed pre-trainig corpus (mostly in English) have been
released by previous works [95, 74], we argue that there is still room for a
more meticulously designed pipeline to improve the existing data. Besides, it
should be mentioned that existing LLMs can be easily improved by continuous
pre-training with high-quality data. Therefore, we further re-process the
selected web content-based corpora to produce the English subset of Matrix
data mixture. The source comes from the Head and Middle parts of RedPajama-
Data-V2 [25], CC part of Dolma [95], the EN part of Cultrax [72], the Refined-
Web part of Amber [66], SlimPajama [94] and falcon [74]. The precise
distribution of our English dataset is listed in Table 3. The procedure
involves filtering and multi-step deduplication. The diagram in Figure 3(a)
shows the processing orders and the retention rates.

####  4.1.1 Filtering

To further filter out the relatively low-quality corpus from open-source
datasets, we propose to use heuristic rules for text filtering. These rules
are designed to identify and remove poor-quality data, thereby preventing
potential model performance degradation caused by a flawed pre-training
corpus. Since our composite dataset is made up of corpora from multiple
sources, we adapt well-designed cleaning methods [74, 14, 76, 78] and tailor
our rules for each one to ensure quality consistency.

For the RedPajama-Data-v2 dataset [25], which provides quality annotations for
each text, we integrate our heuristic rules with these annotations to refine
data quality evaluation and further perform random sampling on the dataset to
confirm the thresholds for every rule. For datasets lacking quality
annotations, we apply the established rules and thresholds derived from
RedPajama-V2, while customizing them to align with the unique characteristics
of each dataset. For example, the Dolma dataset [95] comprises six subsets,
namely Wikipedia, PeS2o, Stack Code, Gutenberg, C4, and CC, each with
different data characteristics. Given the unique characteristics of each
subset, we conduct individual sampling and evaluation to ensure that the
modifications in rules and thresholds are aligned with our filtering
requirements. Specifically, for the CC subset, we adjust the unique word and
text length thresholds. For the Gutenberg subset, which predominantly contains
book texts, we apply only a few rules to avoid the time-consuming process of
executing extensive heuristic checks on long texts.

The filtering process involves: 1) Document-level and sentence-level filtering
to ensure text length adequacy, character meaningfulness, and consistency; 2)
Duplicate text removal, including n-grams and sentences; 3) Sensitive word
check to eliminate texts containing any terms from a blacklist.

####  4.1.2 Deduplication

It has been reported that repetitive text can lead to a decline in model
performance [58, 51, 42], which makes deduplication a crucial step in corpus
processing. By eliminating duplicates, we can significantly reduce the rate of
emitted memorization and make model training more efficient [58]. Repetitions
can be categorized into exact duplicates and near duplicates. For exact
duplicates, we employ exact document deduplication to remove them. For near
duplicates, we utilize Minhash LSH deduplication to remove them as much as
possible. In addition, there are instances where parts of the text are
completely duplicated, and in these cases, the Minhash method struggles to
remove them. To address this, we have adopted two methods for partially
removing such content: paragraph deduplication and exact substring
deduplication.

##### Exact Document Deduplication

Exact document deduplication is a method used to evaluate an entire text to
determine if it is identical to another. If it is found to be exactly the
same, the duplicate will be removed. For processing data in English, Spark is
employed to handle the dataset. Due to the vast volume of data, there may be
issues with insufficient memory. The solution to this problem involves
batching the text data into separate buckets for storage. Each bucket’s data
is then processed in turn to remove duplicates.

##### Minhash LSH Deduplication

Minhash [13] is an excellent method for removing near duplicates, especially
for web page data, and is widely used for similarity search and duplicate
detection in large datasets [104, 33, 37]. It can handle very common scenarios
where the text content is essentially the same, but the scattered template
blocks of the web pages are different. The principle of MinHash is to
represent a set with smaller hash values, which can then be used to estimate
the Jaccard similarity [47] between two sets: $\text{Jaccard}(A,B)=(A\cap
B)/(A\cup B)$.

MinHash involves using multiple distinct hash functions that map each element
of a set to a larger numerical domain. For each set, these multiple hash
functions are applied to all elements within the set, and the smallest hash
value produced by each hash function is chosen as its minimum hash value.
Thus, each set can be represented by a vector of these minimum hash values,
forming the set’s MinHash signature. For text data, an n-gram approach can be
used to construct a set.

After obtaining the signature of the text, Locality-Sensitive Hashing (LSH)
[35] is employed to rapidly identify candidate set pairs that exceed a certain
threshold in Jaccard similarity. This accelerates the search process for
similar items. The specific approach divides the signature into several bands,
each containing several hash values. Another hash function is then used to map
each band to a hash bucket. All sets with the same band hash are mapped to the
same hash bucket. All set pairs in the same hash bucket are considered
candidate similar pairs without further specificity regarding their
similarity. Here, we utilize 128 unique hash functions to form signatures,
divided into 9 bands, with each band containing 13 hash values. Consequently,
the Jaccard threshold is set at 0.8.

Upon identifying similar pairs, connected components are constructed. Within
each component of the connected components, one text is retained while the
others are eliminated. For processing vast amounts of data efficiently, a
distributed implementation [53] based on map-reduce is adopted.

##### Paragraph Deduplication

Paragraph deduplication involves removing all duplicate paragraphs within a
text. A paragraph is defined as a section of text separated by the newline
UTF-8 character ”\n”. Paragraph deduplication is an effective method for
removing website navigation headers, advertisements, and similar elements.
Since paragraph deduplication involves deleting parts of the text, it may
cause some interference with content analysis.

Its concrete implementation first involves splitting the text into multiple
paragraphs using newline utf-8 character ”\n”, with each paragraph being
tagged with its corresponding document id and offset in the text. Then, each
paragraph is hashed using SHA256. Next, the hash values are deduplicated.
After deduplication, the deduplicated text is restored according to the
document ID and offset.

##### Exact Substring Deduplication

This method follows [58]. Given the diversity of languages, when the length of
repeated text is sufficiently long, it is highly likely that they are either
derived from one another or sourced from the same reference. Therefore, when
two texts, $t_{i}$ and $t_{j}$ share sufficiently a long substring, that is
$t_{i}^{a..a+k}=t_{j}^{b..b+k}$, one of them is removed. For the selection of
the length threshold, we adhere to the setting in [58], choosing k=50. Due to
our distributed environment, the memory of a single node is insufficient to
hold all the data. Therefore, we did not adopt the implementation in [58]. In
our work, we segment each text into sliding windows of 50 characters with a
step size of 1. We then compute the SHA256 hash value for each window along
with its corresponding document ID and offset. Subsequently, for windows with
identical hash values, we mark them as duplicates except the first one.
Finally, using the text ID and offset, we restore the original strings and
decide whether to delete a segment based on the duplicate marker.

(a) Re-processing retention rates for the corpora in §4.1.

(b) Processing retention rates for the corpora crawled from scratch in §4.2.

Figure 3:  Funnel Diagram for the two main data pipelines. The darker part of
each row represents the retention proportion for each processing step and the
lighter one for the filtered corpora.

###  4.2 Corpora Crawl from Scratch Pipeline

We further provide a pipeline to crawl and process the web content from
scratch and showcase it with the Chinese language data, which could be a step-
by-step guide for follow-up research to build a new up-to-date corpus. We take
the corpus produced in such a pipeline as the Chinese subset of Matrix, where
$80.6\%$ is derived from the Chinese web pages we crawled and others from
several open datasets, as listed in Table 4. The pipeline overview and details
are illustrated in Figure 3(b).

Table 4: The composition sources of the Chinese web subset. Dataset | Parts | UTF-8 bytes (TB) | Availability | Proportion (%)
---|---|---|---|---
Crawled Web Data | Whole | 14.3 | Self-constructed | $80.60$
CCI | Whole | 0.10 | Public | $0.59$
Chinesewebtext [14] | Whole | 1.40 | Public | $7.89$
Wanjuan [38] | Text | 0.57 | Public | $3.19$
Yayi2 [69] | Whole | 0.49 | Public | $2.76$
Cultrax [72] | ZH | 0.28 | Public | $1.56$
Skypile [109] | Whole | 0.60 | Public | $3.41$

####  4.2.1 Filtering

The filtering rules for Chinese datasets are specifically tailored to address
their unique challenges, differing from those applied to relatively well-
processed English datasets in §4.1. Considering the large proportion of HTML-
converted data in Chinese datasets, we focus intensively on eliminating HTML-
related artifacts and rectifying textual inconsistencies. Furthermore, given
the significant linguistic differences between Chinese and English, we conduct
targeted sampling of documents within Chinese datasets, which aims to reassess
and adjust the thresholds and details of our filtering rules, ensuring their
suitability for the unique language characteristics of Chinese text. For
example, we refine the rules to distinguish between ‘characters’ and ‘words’
in Chinese texts, adapting the tokenization method accordingly.

Our Chinese filtering steps are similar to the rules adapted to filter Massive
Appropriate Pre-train Chinese Corpus (MAP-CC) [30]: 1) Data format unification
to boost processing efficiency. 2) URL removal. This step is conducted in two
stages: first, removing texts with URLs listed in Blacklist T1; followed by a
comprehensive sweep to eliminate residual URLs. 3) Sentence-level and document
filtering to discard text that is excessively brief, substandard, or logically
incoherent. 4). Duplicates removal, including n-grams and sentences.

####  4.2.2 Deduplication

The deduplication of Chinese data includes Exact Document Deduplication,
MinHash Deduplication, and Similar Line Deduplication. Due to difficulties in
deploying Spark in the environment for processing Chinese, we have re-
implemented the first two methods. For Exact Document Deduplication, there are
slight differences from the implementation for English, mainly to save memory,
where we have adopted a Bloom Filter approach and set the false positive rate
of the Bloom Filter to 0.001. Discussions on Exact Document and MinHash LSH
Deduplication can be found in §4.1.2.

We did not use Exact substring deduplication because when crawling web pages,
it is common to repeatedly crawl the same content multiple times in a signal
document. Additionally, when extracting the main text from HTML, there is
often a loss of one or two words. The combination of these two situations
violates the assumption in [58] that “it is rare for the same idea to be
expressed identically in multiple documents unless one expression is derived
from the other, or both are quoting from a shared source.” Therefore, after
Exact substring deduplication, there will be cases where extra words are
retained, greatly reducing the readability of the text. Hence, we propose a
Similar Line deduplication method to address this issue.

####  4.2.3 Similar Line Deduplication

To address the scenario where identical content appears multiple times within
a text, a direct method is to divide the text into lines using specific
delimiters and then compare the similarity between each line. If they are
similar, the subsequent line is removed. The division of lines includes the
use of the following delimiters: “[”, “.”, “!”, “?”, “\”, “……”, “]”. We use
edit distance to judge whether two lines $L_{1}$ and $L_{2}$ are similar as
follows:

$\text{isSimilar}(L_{1},L_{2})=\begin{cases}True&\min(|L_{1}|,|L_{2}|)\geq
15\land\text{editDist}(L_{1},L_{2})<0.1\times\min(|L_{1}|,|L_{2}|)\\\
True&\min(|L_{1}|,|L_{2}|)<15\land L_{1}=L_{2}\\\
False&\text{otherwise},\end{cases}$

where $|L|$ is the length of line $L$ and “editDist” is short for edit
distance.

Due to the computational complexity of calculating edit distance being
$O(len(L_{1})\times len(L_{2}))$, to accelerate this process, we additionally
propose two methods to judge dissimilarity:

  1. 1.

Is the length difference between the two lines greater than one-tenth of the
length of the shorter line?

  2. 2.

Is the ratio of the intersection of the sets of characters and the union of
the sets of characters in $L_{1}$ and $L_{2}$ less than one-third?

Note that the first method has a computational complexity of $O(1)$, and the
second method has a complexity of $O(len(L_{1})+len(L_{2}))$. Thus, these
methods can significantly improve the speed of calculation. Clearly, if either
of the above two questions is positive, the lines cannot be considered
similar. Otherwise, we calculate $\text{isSimilar}(L_{1},L_{2})$ to obtain the
similarity between $L_{1}$ and $L_{2}$.

Figure 4: The document conversion framework is composed of various sub-models
for different parts.

###  4.3 Document Conversion Pipeline

The documents are usually better formatted, in concentrated topics, and with
more consistent expressions compared to noisy web content. However, it seems
to be a gold mine of high-quality corpus except that the golds lie deeply
under the digital dirt. Such digital documents are mostly stored as standard
PDFs with diverse layouts or scanned images with inconsistent quality, making
it challenging to build datasets upon. We observe two core issues in designing
an effective conversion pipeline to extract plain text from documents: i)
analyzing layout information and identifying different layout elements
including text, titles, captions, images, tables, and formulas, and ii)
recognizing the relationships among these layout components.

We survey the existing open-source solutions for document conversion and find
some distinguished projects with good performances: PP-StructureV2 [59],
Marker444https://github.com/VikParuchuri/marker, Vary [108], and Nougat [11].
However, along with their merits, each of them exhibits limitations that could
be addressed to further enhance performance: PP-StructureV2 cannot recognize
LaTeX format content and necessary post-processing stages; Marker and
Texify555https://github.com/VikParuchuri/texify support few languages and do
not process figures effectively; Nougat has limited support for multi-column
data and recognized languages; Vary and Vary-toy require considerable
computational resources. Therefore, we propose a framework consisting of
disentangled processing components, allowing us to leverage the strengths of
these models together. For example, we utilize Marker for enhanced language
support and PP-StructureV2 for efficient layout parsing. As illustrated in
Fig. 4, our document conversion framework is comprised of four parts: Layout
Detection, Element Recognition, Ordering, and Post Process. The decoupling
between each module enhances interpretability and simplifies the upgrade,
addition, and replacement of various components.

##### Layout Detection

segments the document into multiple parts such as formulas, text, headers, and
footers. The Pipeline employs a lightweight target detection model provided by
PP-StructureV2, which is computationally efficient and performs exceptionally
well. This model’s performance is further enhanced by employing the FGD
(Feature Gradient Descent) algorithm, which optimizes feature extraction for
more accurate layout detection.

##### Element Recognition

incorporates various models to identify different elements. For formula
recognition, the TrOCR model trained through Pix2Text outperforms other
formula recognition models such as Latex-OCR and Taxify, supporting
recognition of formulas embedded within paragraphs and non-conventional
formulas, thus effectively addressing most formula recognition scenarios. Text
recognition employs PP-OCRv4, Text recognition employs PP-OCRv4, notable for
its compatibility with multiple computing devices and boasts strong
recognition capabilities; approximately one hundred language recognition
models have been publicly released, applicable to a broader range of document
recognition tasks. Figures are saved as images and inserted in the subsequent
merging phase. Table reconstruction is achieved using SLANet, which represents
tables in HTML format. Other regions, such as headers, footers, and page
numbers, are discarded and do not proceed to the post-processing and
reconstruction stages.

##### Ordering

In document conversion tasks, correctly handling the relationships between
blocks is of paramount importance. To acquire high-quality conversion data, we
need to properly handle complex layout scenarios such as multi-column and
cross-page conditions. In the ordering stage, we use LayoutLMv3 [45] for
column detection and sorting different areas according to specific rules. This
strategy not only enhances the accuracy of the task but also significantly
optimizes the readability.

##### Post-processing.

The texts extracted by OCR usually could not be directly used and require
additional processing as follows:

  1. 1.

Broken-up sentences: In text extracted from images, sentences may be
fragmented across different lines or pages, resulting in a single sentence
being divided into multiple segments. Effective OCR text extraction
necessitates the identification and rejoining of these fragmented sentences to
reconstruct coherent, complete sentences.

  2. 2.

Hyphenated words: Certain words may be split into two parts within the text
due to formatting constraints, connected by hyphens (e.g., network-ing). Text
extraction must recognize these hyphenated words and merge them back into a
single, complete word (e.g., networking).

  3. 3.

Broken math formulas: OCRed mathematical formulas in Markdown may experience
issues such as missing elements, incorrect symbols, or fragmented expressions.
To address this issue, we fine-tune a 7-billion parameter open-source pre-
trained language model [7] on supervised learning data pairs $(x_{i},y_{i})$.
Here, $x_{i}$ represents the instruction for detecting and correcting errors
in the given texts, and $y_{i}$ represents the corrected output texts. We
adopt vLLM to enable faster inference through quantization and efficient
memory management of attention keys and values using PagedAttention, among
other optimizations. The prompt templates used for processing both both
languages are provided in Appendix A.10.

By incorporating these strategies, we can significantly improve the quality
and coherence of OCR-ed texts, mitigating common errors and enhancing the
overall readability and usability of extracted content. We use
FastDeploy666https://github.com/PaddlePaddle/FastDeploy, a highly efficient AI
inference deployment tool, as the codebase of our implementation, which can
fully exploit the advantages of multithreading to optimize inference speed and
computational overhead. Overall, while maintaining performance and deployment
efficiency, we provide a framework for document conversion that covers
comprehensive scenarios, including recognizing layout information, supporting
table reconstruction, and formula recognition.

###  4.4 High-Quality Supplement Data Collection

In this section, we present our method for High-Quality Supplement Data
Collection, which applies to a diverse range of topics and enhances the
robustness of datasets. Inspired by [89], which adopts an iterative pipeline
to facilitate the acquisition of large-scale, high-quality data from Common
Crawl, we propose to select high-quality data for mathematics, scientific exam
synthetic data, and wiki-based content in our Matrix.

The procedural phases of the iterative pipeline are enumerated as follows:

  * •

Seed Dataset Collection: Collect a high-quality seed dataset for the field of
interest, like mathematics, code, or wiki-based content.

  * •

Domain Definition and Sampling: Define a domain as data entries within the
seed dataset sharing the same base URL and extract samples from each domain in
the seed dataset as positive samples to enhance format diversity.
Correspondingly, acquire an equal amount of data from Common Crawl as negative
samples.

  * •

Model Training: Employ a FastText model [50] for binary classification to
discern data relevance to the specified field. Training parameters are set as
follows: three epochs, a learning rate of 0.1, an embedding dimension of 256,
and an n-gram of 3. The model is quantized to augment operational efficiency
within constrained memory capacities, reducing its size to approximately 10%
of its original footprint.

  * •

Data Confidence Assessment: Utilize the trained FastText model to estimate the
confidence of Common Crawl data qualifying as positive. Retain data sequenced
from highest to lowest confidence. To streamline the confidence sorting
process, initially sample a subset of data to establish a viable threshold
that balances data exclusion with retention needs.

  * •

Data Evaluation: Assess the retained data via ChatGPT 3.5 [1], employing the
URL to determine field specificity. This stage aims to mitigate the incidence
of false positives while maintaining a requisite recall rate.

  * •

Data Recall and Annotation: Revisit domains where over 10% of the data was
recognized as field-specific. Annotate this data subset using ChatGPT 3.5 [1]
via URL.

  * •

Model Refinement and Iteration: Integrate unconfirmed positive data from prior
iterations into the positive samples to diversify the FastText model’s
training base. Subsequently, initiate a new iteration cycle beginning from the
training stage.

The data selection for Common Crawl focused on the English content of the
RedPajama V2 dataset [25]. The seed dataset for the mathematics segment is
sourced from OpenWebMath [6], while the science synthetic dataset is from
specific domains such as Chemrxiv, biorxiv, and proprietary crawled exercise
data from open-source datasets, e.g. wanjuan-exam [38], WebInstruct [117], Web
Of Science [55]. Wiki data is procured directly from wiki websites.

##  5 Model

###  5.1 Model Architecture

The MAP-Neo model architecture is grounded on the transformer decoder as
outlined by Vaswani et al. [103]. The essential parameters defining this
architecture are detailed in Table 5. The models are trained with a context
length of 8192 tokens, incorporating several enhancements proposed after the
original transformer concept. These enhancements are listed below:

Multi-Query Attention [92]. The 7B model variant employs multi-head attention,
whereas the 2B model checkpoints implement multi-query attention, using a
single key-value head configuration (num_kv_heads $=1$). This modification is
based on ablation studies indicating that multi-query attention is
particularly effective at more minor scales [92].

RoPE Embeddings [97]. Instead of traditional absolute positional embeddings,
we utilize rotary positional embeddings at each layer and share these
embeddings between the inputs and outputs, minimizing the overall model size.

RMSNorm. To ensure stable training, each transformer sub-layer—including both
the attention and feedforward layers—is normalized using RMSNorm [120].

Activation Function We use SwiGLU [93] as our activation function.

###  5.2 Model Scale Hyperparameters

In this work, we compare two different model scales: 2B and 7B parameters.
Since these models are standard dense Transformers. These models are
constructed using the hyperparameters in Table 5. The two models are trained
identically (except for training data) using the same vocabulary and batch
size. Training details are shown in §3 and §5.1.

Table 5: Model architecture details. We list the number of layers, $d_{\text{model}}$, the number of attention heads, and attention head size. The feed-forward size $d_{\text{ff}}$ is always $8\times d_{\text{model}}$. Model | # Layers | # Heads | $d_{\text{model}}$ | # Feedforward dims | # KV heads
---|---|---|---|---|---
MAP-Neo 2B | 18 | 8 | 2048 | 16384 | 1
MAP-Neo 7B | 28 | 16 | 3072 | 24576 | 16
Table 6: Model training details. Phases | Learning Rate | Weight Decay | Warmup Ratio | Batchsize
---|---|---|---|---
Pre-training (Fundamental Phase) | 2e-4 | 0.1 | 0.0055 | 1024
Pre-training (Decay Phase) | 2e-4 | 0.1 | 0 | 1024
SFT (Fundamental Phase) | 2e-5 | 0 | 0.05 | 512
SFT (Chat Phase) | 2e-5 | 0 | 0.05 | 512
Iterative DPO | 5e-6 | 0 | 0.1 | 256

##  6 Pre-training

In the pre-training process, we employ a two-stage pre-training strategy to
train the MAP-Neo model. The first stage termed the fundamental phase,
involves training the model on a vast corpus of generic texts to develop its
general text generation capability. Subsequently, during the decay phase, we
focus on enhancing the reliability of the model’s generated content by
incorporating high-quality data and mode code data. The distribution of data
used across different phases is depicted in Figure 5. Note that we increase
the volume of code data in the decay phase. Specifically, during the
fundamental phase, since Stack V2 [68] was not yet available, we utilized
Stack V1 [54] and repeated the dataset twice to achieve a balanced data ratio.
In the decay phase, with the release of Stack V2 [68], we incorporated it as
the code component for training. Moreover, we perform further data
distribution tuning including duplicated high-quality data sources, such as
books, judicial decisions, and government reports for training, to improve the
model’s performance. The open-source data used for pre-training is shown in
Table 16, the data repetition details are shown in Table 17 and the training
hyperparameters are shown in Table 6.

Figure 5:  The data mixture ratios in MAP-Neo pre-training stage. The left is
the fundamental phase and the right shows the decay phase.

###  6.1 Fundamental Phase: General Ability Acquisition

During the fundamental phase, we employ a two-stage learning rate scheduler
(LRS) to equip the model with a robust capability for general text generation.
The LRS is modeled as a piecewise function, consisting of an initial warmup
phase where the learning rate linearly ascends from a base rate of
$\eta_{\text{a}}=2\times 10^{-5}$ to peak learning rate
$\eta_{\text{max}}=2\times 10^{-4}$ over $t_{\text{warmup}}=2k$ steps. This is
followed by a cosine decay phase, during which the rate gradually diminishes
back to $\eta_{\text{b}}=2\times 10^{-5}$ over about $365k$ steps. The
learning rate $f(t)$ as a function of time $t$ can be delineated as follows:

$f(t)=\begin{cases}\eta_{\text{a}}+\left(\eta_{\text{max}}-\eta_{\text{a}}\right)\frac{t}{t_{\text{warmup}}}&\text{if
}t\leq t_{\text{warmup}}\\\
\eta_{\text{b}}+\left(\eta_{\text{max}}-\eta_{\text{b}}\right)\left[\frac{1}{2}\left(1+\cos\left(\pi\frac{t-t_{\text{warmup}}}{t_{\text{total}}-t_{\text{warmup}}}\right)\right)\right]&\text{if
}t_{\text{warmup}}<t\leq t_{\text{total}}\end{cases},$ (1)

where $t$ is the current timestep, $t_{\text{warmup}}$ denotes the duration of
the warmup phase, and $t_{\text{total}}$ represents the total number of
training timesteps. This learning phase processes about $3,726$ billion
tokens, ensuring the model’s robust training on diverse textual data. This
meticulous configuration of learning rates and extensive processing optimize
training dynamics and efficiency, fostering a steady maturation of the model’s
capabilities.

###  6.2 Decay Phase: Improvement and Rectification

Owing to the issue in training tokenizer as claimed in §3, the model
encounters test failures in code generation tasks, despite its strong language
understanding capabilities acquired during the fundamental phase. To address
this issue, we have introduced an additional decay phase specifically designed
to utilize a tokenizer of the fixed version. The learning rate in this decay
phase initiates at $\eta_{\text{c}}=2\times 10^{-4}$ and undergoes exponential
decay over $t_{\text{decay}}=148k$ steps, with a half-life $T$ corresponding
to half the $t_{\text{decay}}$ steps, similar to the decay phase employed by
MiniCPM [44], which can be formulated as follows:

$f(t)=\eta_{\text{c}}\times 0.5^{\frac{t}{T}}\quad\text{if }t\leq
t_{\text{delay}},$ (2)

where $t$ is the current timestep of the decay phase. This strategic
adjustment not only rectifies the initial tokenization flaws but also enhances
the model’s performance on code generation tasks. During this phase, the model
processes a total of about $778$ billion tokens, which primarily consist of
high-quality instruction data. We also simultaneously increased the proportion
of code in the data from $14.77\%$ to $17.04\%$. This adjustment significantly
enhances the overall performance of the model. The deliberate enrichment of
the dataset with a higher ratio of code, coupled with instructional inputs,
ensures a more robust and versatile model, adept at tackling complex coding
tasks as well as understanding and generating professional responses in
different fields.

##  7 Alignment

###  7.1 Supervised Fine-tuning

To align with the human behavior of LLMs, the initial step is to perform
Supervised Fine-Tuning (SFT). Our SFT also consists of two phases. In the
first phase, we collect a large amount of instruction data to enhance the
foundational abilities of LLMs. In the second phase, we build upon the
capabilities established in the first phase and propose to improve the chat
abilities of MAP-Neo. This process finetunes a pre-trained LLM on chat-style
data, including both queries and responses. We illustrate the details of data
construction and training strategies.

####  7.1.1 Data

##### Foundational Phase: Enhancing Instruction Following Abilities

In the first phase, our focus is to significantly boost the model’s
foundational abilities (e.g., code and math skills), where we utilize over 2
million instructional data points during this phase. Specifically, the first
phase includes the entire OpenHermes 2.5 [99], where we exclude segments
related to the TheoremQA benchmark [16] to prevent benchmark data leakage.
Additionally, we incorporate the complete Code-Feedback [125] dataset and a
subset of WebInstructSub [117] data.

##### Chat Phase: Enhancing Chat Abilities

In the second phase, we focus on improving the model’s chat abilities while
maintaining the foundational skills acquired in the first phase. For this
purpose, we collect over 100k multi-turn dialogue data sourced from real user
conversations. To ensure the model retains its foundational capabilities, we
include 5k math and code-related data points extracted from the first phase.
Our experiments have demonstrated that this additional phase of SFT
significantly boosts the model’s performance on chat benchmarks, such as MT-
Bench [124] and AlpacaEval [62], without compromising its foundational
abilities.

By following this two-phase approach, we ensure that our model can not only
maintain a strong foundation in essential skills but also generate natural,
helpful, and contextually accurate responses.

####  7.1.2 Training

Consistent with pre-training, we also apply the next-token prediction
objective as the training task for SFT. Note that we apply the loss masks for
the system and user inputs. The model’s training process utilizes the AdamW
optimizer with the hyperparameters in Table 6.

The sequence length is limited to 8192, and the batch size is 512. The
training process consists of two phases using the same hyperparameters. In the
first phase, the model is trained for 3 epochs using over 2 million
instructional data points, focusing on enhancing foundational abilities. In
the second phase, the model is trained for 1 epoch using over 100k multi-turn
dialogue data to enhance its chat abilities while maintaining the foundational
skills acquired in the first phase.

###  7.2 Iterative DPO

##### DPO

Direct Preference Optimization (DPO) [77] is a straightforward and effective
method for aligning language models with human feedback. It converts the
preference loss [12] into a loss function over the language model, thereby
bypassing the need for explicit reward modeling [12] and reinforcement
learning [19, 87]. Starting with a supervised fine-tuned language model,
denoted as ${\pi_{\text{sft}}}$, DPO collects a dataset
$\mathcal{D}=\\{(x,y_{w},y_{l})^{i}\\}$, which consists of human preferences
between two responses generated by ${\pi_{\text{sft}}}$: $y_{w}$ (preferred)
and $y_{l}$ (dispreferred) to the same prompt $x$. Using this dataset, DPO
parameterizes a language model $\pi_{\theta}$ and directly estimates its
parameters through maximum likelihood estimation on the human preference
dataset $\mathcal{D}$ as follows:

$\mathcal{L}_{\text{DPO}}(\pi_{\theta};{\pi_{\text{sft}}},\mathcal{D})=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}|x)}{{\pi_{\text{sft}}}(y_{w}|x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{{\pi_{\text{sft}}}(y_{l}|x)}\right)\right].$
(3)

##### Iterative DPO.

We follow Storm-7B [64] to use the Iterative DPO [111] pipeline to develop our
chat model. Specifically, we employ three iterations, with each iteration
consisting of three stages: 1) generating paired responses, 2) labeling
responses using reward models, and 3) training the LLM with DPO loss as
described in Eq. 3. We utilize
Nectar777https://huggingface.co/datasets/berkeley-nest/Nectar as our prompt
dataset and Starling-RM-34B888https://huggingface.co/Nexusflow/Starling-RM-34B
[126] as our reward model. This model is finetuned from Yi-34B-Chat [113] and
generates a scalar output for any given prompt and response. To preserve the
multilingual capabilities of our model, we also adopt a preference
dataset999https://huggingface.co/datasets/llm-wizard/alpaca-gpt4-data-zh in
Chinese in the 3-rd iteration.

We report the length-controlled win rate of AlpacaEval2.0 [32] to demonstrate
the performance progress of our model in Table 7. The results show that
performance improves with each iteration, indicating that our model becomes
increasingly aligned with human values.

Table 7: The length-controlled win rate of MAP-Neo at different iterations on the AlpacaEval2.0 leaderboard. For “SFT”, we report the performance of our model using two-phase SFT. Model | SFT | Iteration 1 | Iteration 2 | Iteration 3
---|---|---|---|---
LC Win Rate (%) | 9.77 | 10.02 | 15.59 | 16.65

##  8 Scaling Law of MAP-Neo

###  8.1 Problem Definition

The scaling laws are capable of predicting training configuration for the
training of LLMs. This principle emphasizes the importance of the ratio
between the amount of training data $D$ (measured in tokens) and the size of
the model $N$ (in terms of parameters). In this section, we applied the
Chinchilla Law in Eq. 4 [43], OpenAI Law in Eq. 5 [52], a derivation of
Symbolic Music Scaling law in Eq. 6 [75] and our proposed method on our
dataset to fit our models, where $A$, $B$, $E$, $\alpha$, $\beta$,
$\alpha_{c}$, $D_{c}$, $\alpha_{N}$, $N_{c}$ and $d$ are hyperparameters to be
optimized.

$L(N,D)=\frac{A}{N^{\alpha}}+\frac{B}{{D}^{\beta}}+E$ (4)
$L(N,D)=\left(\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{{D}}\right)^{\alpha_{D}}$
(5) $L(N,D)=\frac{d}{N^{\alpha}\cdot
D^{\beta}}+\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}}+E.$ (6)

The original SMS scaling law introduces two modifications to the Chinchilla
law. The first modification addresses the repetition of training data, which
is not considered in our study. The second modification concerns the
interaction between the number of model parameters, $N$, and the dataset size,
$D$. Specifically, it posits that the loss curve as a function of $D$,
represented as $\frac{B}{D^{\beta}}$, is influenced by $N$. This interaction
between the number of model parameters and dataset size is also reflected in
the OpenAI scaling law. However, our version of SMS law, as detailed in Eq. 6,
is simpler and yields superior results compared to the corresponding model in
the OpenAI framework.

The motivation for fitting scaling laws is to optimize the loss under the
bounds of computational resources. This process is formalized as minimizing
the validation cross-entropy loss $L$, subject to constraints imposed by
available computational resources ($C$), specifically floating-point
operations per second (FLOPs), as denoted below:

$\arg\min_{N,D}L(N,D)\quad\text{s.t.}\quad\text{FLOPs}(N,D)=C$ (7)

Given that our model is trained on almost non-repetitive and high-quality
data, we utilize the training loss instead of the validation loss for the
scaling law application.

###  8.2 NEO Scaling Law

We train models with sizes of 250M, 460M, and 980M parameters using 1000B
tokens of training data. These models are then used to predict the scaling
law, which guides the training of a model with 7.8B parameters on 3.07T
(3065B) tokens during phase 1. To evaluate the fit of the scaling law, we
employ the Huber loss ( $\delta=1e-3$) between the actual logloss and the
predicted logloss, along with the $R^{2}$ value between the true loss and
predicted loss. Optimization of the scaling law is performed using the LBFGS
algorithm. This approach is applied consistently across the Chinchilla law and
the symbolic music scaling law. By leveraging these methods, we aim to ensure
the accuracy and reliability of our scaling law predictions, enabling
efficient training of large-scale language models.

Figure 6: The training loss value is represented by the blue line. The
Chinchilla law prediction is shown in yellow, and the NEO scaling law
prediction is depicted in green. We fit the Chinchilla law and NEO law on
250M, 460M, and 980M and predict the model behavior on both training samples
and samples from the 7B model.

Figure 6 illustrates the training loss values alongside the Chinchilla law
predictions. Although the Chinchilla law fits well, with the predicted loss
curve falling within the fluctuations of the actual loss curve, its trend
appears flatter compared to the actual loss curve. The actual loss decreases
more rapidly than predicted by the Chinchilla formula (i.e.
$\frac{B}{D^{\beta}}$), suggesting our dataset with diverse high-quality
corpora can further decrease the loss value when $D$ is large. To address this
discrepancy between Chinchilla prediction and observation, we introduce the
following equation, denoted as NEO scaling law, which includes one additional
regularization term $log(D)$ for datasets containing several trillion tokens
across various corpora:

$L(N,D)=\frac{A}{N^{\alpha}}+\frac{B}{{D}^{\beta}}+E-d\cdot log(D)$ (8)

Note that although the regularization term $-d\cdot\log(D)$ theoretically
results in no lower bound on loss as $D$ approaches negative infinity
suggesting potential imperfection of the formula, the value of $d$ typically
ranges in our experiments between 1e-2 and 3e-2. Therefore, for a dataset size
less than hundreds of trillion tokens, the loss remains within a reasonable
range.

From the following Table 8, we observe that the NEO scaling law equation
yields significantly better results on the training set and testing set.

Table 8: Comparison of parametric fitting on $R^{2}$ and Huber Loss of
different scaling laws.

Paramatic fit |  $R^{2}$ Value (train) ↑ | Huber Loss (train) ↓ |  $R^{2}$ Value (test) ↑ | Huber Loss (test) ↓
---|---|---|---|---
Chinchilla Law | 0.2483 | 0.1665 | 0.4308 | 0.3372
OpenAI Law | 0.2268 | 1.0424 | -0.2916 | 0.6023
SMS Law | 0.2484 | 0.1665 | 0.4306 | 0.3375
NEO Scaling Law | 0.7361 | 0.2961 | 0.6720 | 0.2081

Under the prediction of the NEO scaling law and the computational resource
constraint of $1.5\times 10^{23}$ FLOPs, the optimal configuration is to train
a 10B parameter model with 2.5T tokens, providing a predicted loss value of
0.6597. To ensure comparability with baseline models, we choose to keep our
model size at 7.8B parameters, similar to the Llama-base model. This
configuration with a 7.8B parameter model with 3.07T tokens requires slightly
fewer computational resources and results in a similar prediction loss value
(0.6618). Meanwhile, after training, We observe that the real training loss in
this configuration is 0.6591, which is close to the prediction loss value and
demonstrates the effectiveness of the NEO scaling law.

###  8.3 Generalization of NEO Scaling Law

The NEO scaling law can be applicable to a broader range of models beyond MAP-
Neo. Specifically, in Figure 7, we illustrate the fit results of the
Chinchilla scaling law (yellow dashed line) and the NEO scaling law (red solid
line) to the DeepSeek LLM [28] with the 7B and 67B parameters, which also pre-
trained on a dataset with multiple corpura including Chinese, English and
codes.

We observe that for the largest model sizes (i.e. MAP-Neo-7B and
DeepSeek-67B), the predictions of Chinchilla Law tend to underestimate the
actual loss when the dataset size ($D$) is small and overestimate the actual
loss as model parameters and training data scale up. In contrast, our
predictions of our NEO Scaling Law produce better fitting results when
compared with the results of Chinchilla Law for MAP-Neo-7B and DeepSeek-67B.

Figure 7: The loss curve of Chinchilla Law prediction and the NEO Scaling law
prediction for the DeepSeek LLM. We use loss values from both 7B and 67B for
fitting and prediction.

We further suggest NEO Scaling law might be more suitable for the situation
with a large diverse pre-training dataset with multiple high-quality dataset
sources. For more discussion on NEO scaling law on other models, please refer
to Appendix A.8.

##  9 Infrastructure

Our advanced infrastructure consists of two primary components: a data
processing system and a training system. The training system is designed to
support both pre-training and fine-tuning stages, enabling comprehensive model
development.

Our infrastructure is designed to handle extensive data processing tasks for
both English and Chinese datasets. We utilize robust systems to ensure
efficient and scalable processing capabilities across different languages.
Spark [118] is used for distributed computing, and object storage is used to
save the data. Each machine is configured with a 64-core CPU, 256GB of memory,
and 1TB of local disk. There are a total of 94 machines. For the Chinese data
processing, there are a total of 14 machines. Among them, 6 machines have a
96-core CPU and 180GB of memory, while the other 8 machines have a 48-core CPU
and 190GB of memory. Network File System (NFS)[84] is used as the distributed
file storage system.

In the pre-training stage, the Megatron-Core toolkit is utilized for its
capacity to train large-scale language models, featuring up to hundreds of
billions of parameters. Compared to the tokens per second (TPS) metric, the
usage of Megatron-core achieves a rate of 7200 TPS when training a 7B model,
which surpasses the performance of 6400 TPS observed under the same settings
without employing Megatron-core. This is accomplished using both model and
data parallelism techniques. We implement several strategies to manage our
large datasets and model complexities effectively. Firstly, we introduce
programs to identify and temporarily remove tainted computing nodes from the
resource pool due to software or hardware errors by automatic inspection,
prediction, and labeling. Secondly, we make modifications to Megatron-LM to
specifically prevent overflow issues detailed in A.3 when processing large
data corpora. Lastly, we implement task recovery mechanisms that utilize
strategically selected checkpoint iterations to safeguard against potential
failures during training. These enhancements ensure optimal performance and
reliability in our large-scale training operations.

To ensure optimal utilization of our computational resources, our
infrastructure design incorporates a sophisticated network topology and
hardware configuration, facilitating efficient workload distribution and data
transfer for complex model training tasks. Our infrastructure utilizes
distributed computing techniques to optimize the training of our models.
Specifically, our 7B model is trained using an H800 configuration with 512
GPUs across 64 nodes and employs NCCL for backend distribution with ibp as the
network interface and mlx5 of InfiniBand hardware to enhance inter-GPU
communication. Tensor model parallelism is configured to utilize 2 GPUs,
distributing the execution of a single transformer module across these units
to enhance efficiency. For our 2B models, we utilize all 256 GPUs with tensor
model parallelism set to 1 to ensure effective data replication. We further
amplify scalability and efficiency by employing techniques similar to ZeRO-1
for sharding the optimizer state. This approach enables the management of more
extensive datasets and more complex model training with significantly reduced
memory overhead.

Our cluster consists of machines with dual Intel Xeon CPUs and eight NVIDIA
H800 GPUs. The architecture facilitates high-speed data transfer, with each
CPU socket interfacing with two PCIe Gen4 x16 lanes connected to dedicated
PCIe switches. These switches manage the connections to a local NVMe SSD, an
RDMA-capable Network Interface Card (NIC), and two GPUs. Inter-CPU
communication is facilitated by Intel’s Ultra Path Interconnect (UPI), with
both CPUs linked to a dual-port TCP NIC supporting 100 Gbps. Each machine’s
network configuration includes four RDMA NICs, each offering 200 Gbps of full
duplex bandwidth and integrated GPU Direct RDMA capabilities. Notably, the GPU
array is interconnected through four NVIDIA NVSwitches, enabling robust intra-
GPU communication with a bandwidth of 400 Gbps. This advanced configuration
underscores the cluster’s capability to handle large-scale model training with
exceptional efficiency and speed.

Regarding the inter-machine connections of our data center, we implement a
dual-layer Clos network architecture wherein each minipod accommodates at
least 512 H800 servers interconnected via a high-speed RDMA network. Within
this architecture, each S0 switch is equipped with 64 ports, each supporting a
bandwidth of 400 Gbps. This arrangement ensures a network convergence ratio of
1:1, a critical factor in maintaining optimal data flow and reducing
bottlenecks. Connectivity within this structure is meticulously organized such
that every two S0 switches serve 32 servers, with a total of 32 S0 switches
networking within each minipod. This setup exemplifies an advanced
implementation designed to maximize throughput and minimize latency in data
center environments.

Table 9: Performance comparison of various base models on different
benchmarks. The best results are in blue, the second-best results are
underline, and the third-best results are in fbox.

Dataset | LLama3-8B | Mistral-7B | LLama2-7B | Amber-7B | OLMo-7B | OLMo-1.7-7B | Pythia-6.9B | Pythia-12B | MAP-Neo-7B
---|---|---|---|---|---|---|---|---|---
Standard Benchmarks
BoolQ | 66.82 | 64.1 | 70.67 | 63.52 | 68.41 | 70.49 | 62.45 | 61.07 | 81.07
PIQA | 81.12 | 81.18 | 78.18 | 76.82 | 79 | 80.25 | 75.52 | 76.17 | 76.55
SIQA | 47.34 | 47.13 | 45.50 | 42.89 | 44.11 | 54.71 | 42.32 | 44.32 | 68.22
HellaSwag | 74.52 | 76.49 | 71.27 | 66.76 | 70.32 | 72.37 | 59.6 | 63.04 | 70.74
WinoGrande | 72.38 | 75.3 | 69.53 | 64.64 | 66.54 | 69.22 | 60.85 | 63.69 | 59.83
ARC-c | 79.66 | 71.53 | 35.93 | 24.41 | 24.07 | 49.83 | 22.71 | 25.08 | 68.14
OpenBookQA-Fact | 69.0 | 81.0 | 42.60 | 26.6 | 24.6 | 64.4 | 25 | 28.6 | 82.0
CommonsensQA | 69.7 | 67.57 | 66.50 | 57 | 60.44 | 69.04 | 55.45 | 54.79 | 69.94
MMLU-AVG | 66.52 | 64.04 | 46.80 | 28.07 | 28.51 | 53.52 | 26.39 | 27.06 | 58.14
*-humanities | 70.41 | 68.04 | 51.47 | 30.17 | 25.52 | 55.03 | 26.87 | 27.39 | 60.7
*-stem | 56.22 | 53.21 | 38.02 | 27.66 | 28.68 | 44.17 | 26.77 | 28.13 | 49.84
*-social-science | 76.0 | 73.65 | 52.20 | 27.18 | 30.05 | 62.19 | 24.32 | 26.26 | 66.78
*-other | 68.94 | 67.0 | 49.99 | 27.37 | 29.86 | 57.67 | 27.25 | 25.91 | 59.73
Code Generation
Humaneval | 33.5 | 28.0 | 13.4 | 13.4 | 11.6 | 17.1 | 9.1 | 8.5 | 23.8
Humaneval-Plus | 29.3 | 23.2 | 11.6 | 12.2 | 9.8 | 15.2 | 8.5 | 7.3 | 20.1
MBPP | 61.4 | 46.8 | 29.1 | 22.8 | 27 | 32.3 | 16.1 | 15.6 | 34.9
MBPP-Plus | 51.6 | 38.9 | 22.8 | 18.5 | 21.2 | 25.7 | 13.2 | 11.1 | 29.9
World Knowledge
NQ | 10.14 | 9.31 | 5.07 | 3.1 | 0.66 | 1.02 | 0.86 | 1.83 | 9.97
TriviaQA | 51.94 | 56.47 | 52.44 | 26.65 | 31.97 | 45.16 | 16.97 | 24.31 | 42.36
Reading Comprehension
SQuAD2.0 | 40.88 | 12.53 | 41.32 | 31.15 | 27.05 | 30.43 | 22.54 | 23.11 | 30.98
Exams
MATH | 20.76 | 15.74 | 6.14 | 3.88 | 1.6 | 4.86 | 3.82 | 4.54 | 20.7
GSM8K | 54.74 | 47.46 | 16.22 | 3.64 | 5.84 | 28.43 | 3.41 | 3.94 | 53.68
Chinese
C-EVAL-AVG | 49.83 | 47.54 | 32.37 | 23.82 | 27.39 | 35.21 | 24.64 | 24.82 | 57.68
*-stem | 45.26 | 44.74 | 28.28 | 22.36 | 25.75 | 32.36 | 23.94 | 27.27 | 50.35
*-social-science | 58.09 | 54.8 | 39.22 | 25.95 | 31.87 | 40.43 | 26.34 | 23.78 | 70.23
*-humanities | 50.6 | 51.52 | 37.11 | 21.19 | 26.29 | 35.5 | 21.7 | 20.05 | 63.49
*-other | 49.84 | 42.06 | 28.84 | 27.16 | 27.4 | 35.36 | 27.28 | 26.08 | 53.78
*-hard | 32.41 | 33.97 | 25.21 | 19.63 | 27.12 | 29.16 | 22.99 | 27.05 | 41.07
CMMLU-AVG | 50.72 | 44.63 | 31.85 | 25.77 | 25.53 | 36.74 | 25.34 | 24.88 | 55.1
*-humanities | 53.1 | 44.59 | 32.50 | 24.86 | 26.65 | 37.04 | 25.81 | 25.41 | 62.24
*-stem | 43.59 | 37.82 | 29.05 | 25.61 | 25.24 | 31.94 | 24.29 | 23.7 | 45.62
*-social-science | 52.59 | 46.37 | 32.60 | 25.83 | 25.17 | 38.14 | 25.78 | 25.17 | 59.39
*-other | 53.98 | 49.83 | 33.35 | 26.65 | 25.43 | 39.88 | 25.47 | 25.33 | 53.39
*-china-specific | 44.81 | 40.84 | 29.27 | 24.96 | 24.97 | 34.91 | 26.5 | 25.34 | 55.84

Table 10: Performance comparison of various aligned models on different
benchmarks. The best results are in blue, the second-best results are
underline, and the third-best results are in fbox.

Dataset | LLama-3-8B | Mistral-7B | LLama-2-7B | Amber-7B | OLMo-7B | MAP-Neo-7B | MAP-Neo-7B
---|---|---|---|---|---|---|---
(Instruct) | (Instruct-v0.2) | (Chat) | (Chat) | (Instruct) | (SFT) | (Instruct)
Chat Benchmarks
AlignBench | 6.17 | 5.27 | 4.33 | 2.85 | 3.2 | 4.63 | 5.04
AlpacaEval | 22.9 | 17.1 | 5.4 | 1.21 | 3.64 | 9.77 | 16.65
Arena-Hard | 20.6 | 12.6 | 4.6 | 1.2 | 1.7 | 10 | 11.5
CHC-Bench | 5.53 | 6.86 | 4.7 | 3.13 | 3.91 | 6.14 | 7.42
MT-Bench | 8.1 | 7.5 | 6.6 | 5.2 | 5.3 | 7.1 | 7.1
Standard Benchmarks
BoolQ | 75.05 | 82.87 | 74.77 | 66.51 | 72.2 | 84.59 | 81.28
PIQA | 80.09 | 82.43 | 76.01 | 77.48 | 75.3 | 76.06 | 75.24
SIQA | 51.23 | 50.41 | 48.72 | 44.88 | 48.41 | 51.69 | 52.25
HellaSwag | 71.39 | 80.11 | 71.32 | 67.84 | 75.18 | 68.5 | 68.7
WinoGrande | 71.9 | 73.4 | 68.35 | 64.96 | 66.69 | 65.19 | 66.06
ARC-c | 81.36 | 73.56 | 55.59 | 37.29 | 57.63 | 80 | 80
OpenBookQA-Fact | 87 | 85.4 | 74.4 | 36.6 | 74 | 85.4 | 85.4
CommonsenseQA | 73.55 | 75.84 | 70.11 | 60.28 | 63.47 | 68.39 | 70.35
MMLU-Pro | 38.12 | 30.86 | 21.61 | 14.65 | 16.27 | 28.08 | 28.74
MMLU | 67.1 | 60.81 | 48.22 | 38.8 | 47.47 | 58.28 | 58.28
*-humanities | 70.67 | 66.58 | 52.71 | 39.19 | 48.33 | 60.4 | 60.85
*-stem | 56.97 | 50.01 | 37.98 | 33.78 | 38 | 51.86 | 52.29
*-social-science | 76.9 | 69.75 | 55.81 | 42.85 | 56.57 | 66.19 | 65.6
*-other | 69.3 | 62.55 | 51.69 | 42.03 | 52.06 | 58.26 | 57.68
Code Generation
HumanEval | 48.8 | 42.1 | 14 | 17.7 | 14.63 | 34.1 | 45.1
HumanEval-Plus | 44.5 | 36.0 | 12.2 | 14 | 12.8 | 31.7 | 37.8
MBPP | 70.1 | 39.7 | 29.1 | 28.0 | 20.1 | 44.4 | 44.4
MBPP-Plus | 59.3 | 33.3 | 22.8 | 23.5 | 16.7 | 38.1 | 36
World Knowledge
NQ | 8.25 | 1.14 | 1.5 | 3.02 | 0.53 | 3.8 | 2.41
Triviaqa | 56.32 | 45.06 | 46.79 | 30.95 | 27.91 | 38.77 | 27.09
Reading Comprehension
SQuAD2.0 | 66.99 | 15.01 | 19.61 | 13.12 | 42.13 | 44.58 | 25.2
Exams
MATH | 29.28 | 13.14 | 6.9 | 4.2 | 1.8 | 35.36 | 35.58
GSM8K | 79.23 | 49.2 | 26 | 7.59 | 13.5 | 72.02 | 73.16
Chinese
C-Eval | 50.76 | 43.72 | 35.67 | 26.29 | 35.18 | 55.42 | 56.97
*-stem | 47.47 | 41.35 | 32.59 | 23.99 | 31.43 | 47.37 | 49.08
*-social-science | 57.05 | 47.75 | 40.04 | 26.77 | 42.13 | 69.21 | 70.75
*-humanities | 48.32 | 47.33 | 36.96 | 28.26 | 34.03 | 63.17 | 63.14
*-other | 53.48 | 40.74 | 36.01 | 28.06 | 36.81 | 49.78 | 52.63
*-hard | 31.04 | 27.32 | 28.45 | 22.77 | 26.33 | 38.41 | 39.55
CMMLU | 51.68 | 42.67 | 33.9 | 30.09 | 35.55 | 55.27 | 55.01
*-humanities | 52.55 | 42.01 | 35.45 | 30.48 | 34.78 | 63.4 | 62.99
*-stem | 44.09 | 36.82 | 29.33 | 26.76 | 30.36 | 47.29 | 46.69
*-social-science | 53.02 | 44.41 | 34.55 | 30.97 | 38.04 | 57.55 | 57.79
*-other | 57.58 | 47.3 | 36.77 | 32.25 | 38.45 | 53.93 | 53.44
*-china-specific | 45.86 | 39.22 | 32.64 | 28.38 | 33.97 | 55.69 | 55.9

##  10 Evaluations

The thorough evaluation demonstrates that the MAP-Neo model family achieves
inspiring performance both on automatic benchmarks of base models and chat
models. Compared to the previous transparent LLM series, we underline MAP-
Neo’s distinctive performance on code, math, and instruction following
abilities, which not only endows the MAP-Neo with academic and practical
value.

###  10.1 Base Model Performance

####  10.1.1 Main Results

We present the results of our base models compared to several well-known LLMs,
e.g. LLama3-8B and Mistral-7B, across standard academic benchmarks. All our
evaluation metrics are derived from our assessments, ensuring consistency and
transparency. We do not perform any post-processing on the evaluation content,
maintaining the integrity of the raw outputs.

Our evaluation spans a comprehensive suite of public benchmarks in both
English and Chinese, leveraging an internal evaluation framework designed for
rigorous assessment. These benchmarks include a diverse range of datasets
catering to multiple disciplines and aspects of language understanding and
reasoning. Our evaluation strategy encompasses various metrics, including
language modeling, specialized knowledge, and code generation. For datasets
requiring multiple-choice selection, we employ a perplexity-based evaluation.
For generation-based datasets, we generate free text and parse the results
accordingly. The detailed results of our comparison with other base models are
shown in Table 9.

Standard Benchmarks We include Boolean Questions(BoolQ) [21], Physical
Interaction QA(PIQA) [10], Social Interaction QA(SIQA) [85], HellaSwag [119],
WinoGrande [83], ARC-Challenge(ARC-c) [22], OpenBookQA-Fact [70],
CommonsenseQA [98], and MMLU [40] to assess general reasoning capabilities.
All these benchmarks are tested with a 0-shot configuration, except for MMLU,
which is evaluated with a 5-shot setup.

Code Generation We report the pass@1 scores of the evaluated models on
HumanEval [15], HumanEval-Plus, MBPP [5], and MBPP-Plus, all with a 0-shot
configuration, following the EvalPlus framework [63].

World Knowledge We include NaturalQuestions(NQ) [57] and TriviaQA [49] to
assess world knowledge. Both benchmarks are tested with a 0-shot
configuration.

Reading Comprehension We report the 0-shot average on SQuAD2.0 [79].

Exams We report the average scores for MATH [41] and GSM8K [23], both with a
4-shot configuration. For GSM8K, we employ a simple Chain-of-Thought prompting
strategy: ”Let’s think step by step.” For both datasets, we use the MAmmoTH
evaluation framework [116].

Chinese We use CMMLU [60] and CEval [46] to assess performance on Chinese
language tasks. Both benchmarks are tested with a 5-shot configuration.

####  10.1.2 Discussions

##### Data Quality

MAP-Neo demonstrates significantly better performance on math, code, and
complex reasoning by incorporating high-quality data, compared to previous
transparent LLMs, e.g. Amber [66] and Pythia [9], adopting (presumably) lower
quality data.

##### Gap between our MAP-Neo and other transparent LLMs

In Table 9, we note that transparent LLMs still significantly lag behind the
performance of frontier industrial Open-weight LLMs with similar sizes (e.g.
LLama3-8B, Mistral-7B). In contrast, our MAP-Neo can match or even surpass
them on part of the automatic benchmarks about math, code, and Chinese
knowledge. We call for increased participation in the development of
transparent LLMs to further advance the LLM democratization.

###  10.2 Aligned Model Performance

####  10.2.1 Main Results

To accurately evaluate the realistic conversational performance of our aligned
models, we selected several benchmarks that measure various aspects of model
capabilities. These benchmarks were chosen for their ability to
comprehensively assess key abilities such as alignment, instruction-following,
real-world performance, and alignment with human preferences. Below are the
specific benchmarks we used and the unique capabilities they evaluate:

AlignBench [65] AlignBench evaluates the alignment capabilities of Chinese
LLMs, ensuring high reliability and interpretability through a comprehensive,
multi-dimensional benchmark and human-in-the-loop data curation.

AlpacaEval [62, 32, 31] AlpacaEval measures instruction-following models’
performance efficiently and reliably through an LLM-based automatic
evaluation, validated against extensive human annotations.

Arena-Hard [61] Arena-Hard evaluates LLMs’ real-world performance and ability
to reflect human preferences by constructing benchmarks from live data and
ensuring robust model capability separation.

CHC-Bench [30] CHC-Bench evaluates LLMs on their proficiency in Chinese
culture, history, and language, with tasks like composing poetry,
understanding ancient Chinese, and explaining Chinese terms, emphasizing the
challenges for models trained mainly on English datasets.

MT-Bench [124] MT-Bench assesses LLM-based chat assistants’ alignment with
human preferences using strong LLMs as judges, achieving high agreement with
human evaluations.

MMLU-Pro [106] For the aligned models, we further evaluate MMLU-Pro [106] with
a 5-shot configuration to reflect the model’s capabilities more
comprehensively.

####  10.2.2 Discussions

##### The effectiveness of Iterative DPO

In Table 10, when compared to Neo-7B-SFT, Neo-7B-Instruct shows significant
improvement on the chat-related benchmark datasets (e.g., AlignBench,
AlpacaEval, Arena-Hard, and CHC-Bench), which further demonstrates the
effectiveness of our Iterative DPO.

##### The performance of the chat model

Table 10 shows that Amber-7B-Chat and OLMo-7B-Instruct perform poorly on Chat
Benchmarks. We assume that the limited capabilities of the base model may
severely limit the performance of corresponding instruction-tuned models on
chat benchmarks.

##  11 Societal Impact

Data Colonialism is a deep concern when firms decide to exploit an algorithm
product. [27] conceptualize the data colonialism framework and argue that Big
Tech Giants, particularly in the U.S., use their massive data power to
manipulate human behaviors and judgments and track people’s traces
continuously, forming a new social order. This suggests that controlling and
owning data benefits firms’ market status and generates large returns. So,
making LLMs as firms’ proprietary models is a common practice in the industry.
[2] discuss the barriers to AI democratization, such as the concentration of
AI capabilities in large tech firms and elite universities. They underscore
the importance of democratizing access to AI resources to mitigate the risks
of data colonialism and promote equitable access to AI technologies across all
institutions. [91] discuss the dominance of proprietary LLMs and the need for
high-performing open-source alternatives. They propose methods to enhance
open-source models to compete with proprietary models while addressing privacy
and resource-constrained concerns. They also point out how important the open-
source model is in the LLMs community and acknowledge that firms with fewer
resources and sensitive information are hesitant to trust the proprietary
models. However, most LLMs are the product of a massive English corpus and are
trained from English scratch [122]. How the open-source model can benefit the
non-English language community and its data democratization remains unclear.

Additionally, most open-source models are not thoroughly transparent. Open-
source large language models (LLMs) often claim to be transparent and
accessible, but many critical aspects of their development, such as data
cleaning processes and pre-training code, remain undisclosed. This lack of
transparency hampers reproducibility and the ability to fully understand and
trust these models [110]. For firms with financial constraints and privacy
concerns, it is not economical to train their LLMs. Even though most open-
source models give open access to the final and some intermediate checkpoints,
they keep data sources, data pre-training code, and data processing methods
opaque, those of which are the most costly parts of setting up an LLM. That is
the key issue we want to tackle and then hope to promote full transparency in
our community.

In our report, the MAP-Neo model might complement the current scarcity of
Chinese corpus in LLMs. Importantly, our bi-lingual language model is a
”thorough” open-source model–disclosing all key processes from sources of
searching original data, and data cleaning to pre-training code base. Those
disclosures significantly reduce the cost of deploying and customizing a LLM,
especially for a Chinese LLM. It might have potential societal impacts. Firms
with the need for a Chinese version of LLM but face constraints can be more
able to leverage benefits from LLMs by using or referencing our MAP-Neo Model.
It might improve social welfare in total and make a more vivid and diversified
Chinese LLMs community [24]. Our advocates for thorough open-source action may
attract more Chinese LLM researchers or relevant firms to fully disclose their
models because thorough transparent open-source models can bring them sizable
benefits from more constructive feedback and criticism. Those might make their
models better and eventually accelerate the iterations of Chinese LLMs and
empower the local community [81]. Overall, open innovation practices like
disclosing the MAP-Neo model might alleviate the dominance of English LLMs and
improve the inclusivity of the international LLMs community.

Those open innovation practices may also benefit Small and Medium enterprises
(SME) to introduce new products effectively [96] and efficiently with easier
implementation of their own customized LLMs, which may partially mitigate the
threats of data colonialism from Big Tech Giants. Our Map-Neo model’s open and
economical attributes give an optimistic outlook for researchers in academia.
Those attributes suggest that it is not hard and costly to set up the
university’s own AI without depending on specific Big Tech Giants’ help. If
universities have independent and decentralized control over their data and AI
processes, it will prevent large companies from AI monopolization and promote
data and AI democratization.

##  12 Conclusion

In this paper, we introduce MAP-Neo, which makes strides toward enhancing the
transparency and accessibility of large language models (LLMs) by offering a
fully open-source bilingual LLM suite. By sharing thoroughly detailed
processes, from data curation, pre-training corpus (i.e., Matrix Data Pile),
and model training to evaluation, we aim to support the academic and open-
source communities in advancing transparent NLP research. Moreover, MAP-Neo
narrows the gap with industry-level models (typically closed-source) with
enhanced reasoning, instruction-following, and coding abilities. We hope that
our work provides a valuable resource for researchers and developers,
contributing to a broader effort to democratize access to advanced LLM
technologies.

##  13 Contributions and Acknowledgments

##### Team Leaders:

  * •

Ge Zhang, M-A-P, University of Waterloo, 01.AI, Data & Pretrain & Evaluation &
Model Architecture & Codebase & Alignment

  * •

Scott Qu, M-A-P, University of Manchester, 01.AI, Codebase & Model
Architecture & Infra & Pretrain

  * •

Jiaheng Liu, M-A-P, Scaling Law & Alignment

##### Core Contributors: (Alphabet Order)

  * •

Chenchen Zhang, Independent Researcher, Pretrain

  * •

Chenghua Lin. M-A-P, University of Manchester, Data

  * •

Chou Leuang Yu, CUHK-Shenzhen, Alignment & Data

  * •

Danny Pan, Peking University, Data & Codebase

  * •

Esther Cheng, Peking University, Data

  * •

Jie Liu, The Chinese University of Hong Kong, Alignment

  * •

Qunshu Lin, 2077AI, Data

  * •

Raven Yuan, M-A-P, Pretrain & Infra

  * •

Tuney Zheng, M-A-P, 01.AI, University of Waterloo, Pretrain & Evaluation &
Alignment

  * •

Wei Pang, University of Waterloo, Data

  * •

Xinrun Du, M-A-P, 01.AI, Codebase & Pretrain & Alignment & Evaluation

  * •

Yiming Liang, Institute of Automation, Chinese Academy of Sciences, Alignment
& Evaluation

  * •

Yinghao Ma, M-A-P, Queen Mary University of London, Scaling Law

  * •

Yizhi Li, M-A-P, University of Manchester, Data

  * •

Ziyang Ma, M-A-P, Shanghai Jiao Tong University, Alignment

##### Contributors: (Alphabet Order)

  * •

Bill Lin, University of Southern California, Alignment

  * •

Emmanouil Benetos, Queen Mary University of London, Scaling Law

  * •

Huan Yang, University of Warwick , Ethics & Societal Impact

  * •

Junting Zhou, Peking University, Data & Scaling Law

  * •

Kaijing Ma, Tongji University, Data

  * •

Minghao Liu, 2077AI, Data

  * •

Morry Niu, 01.AI, Codebase

  * •

Noah Wang, 01.AI, Alignment

  * •

Quehry Que, Independent Researcher, Data

  * •

Ruibo Liu, Dartmouth University, Pretrain & Model Architecture

  * •

Sine Liu, Independent Researcher, Infra

  * •

Shawn Guo, 01.AI, Data

  * •

Soren Gao, Fudan University, Tokenization

  * •

Wangchunshu Zhou, M-A-P & AIWaves Inc., Data

  * •

Xinyue Zhang, Unity, Ethics & Data

  * •

Yizhi Zhou, Nanjing University, Data

  * •

Yubo Wang, University of Waterloo, Pretrain

  * •

Yuelin Bai, M-A-P, Shenzhen Institute of Advanced Technology, CAS, Data

  * •

Yuhan Zhang, M-A-P, Data

  * •

Yuxiang Zhang, M-A-P, Waseda University, Codebase & Evaluation & Data

  * •

Zenith Wang, Independent Researcher, Data

  * •

Zhenzhu Yang, China University of Geosciences Beijing, Ethics & Data

  * •

Zijian Zhao, 2077AI, Data

##### Advisors:

  * •

Jiajun Zhang, Wuhan AI Research, Institute of Automation, Chinese Academy of
Sciences

  * •

Wanli Ouyang, The Chinese University of Hong Kong, Shanghai AI Lab

  * •

Wenhao Huang, 01.AI

  * •

Wenhu Chen, University of Waterloo

##  14 Multimodal Art Projection

Multimodal Art Projection (M-A-P) is an open-source research community. The
community members are working on Artificial Intelligence-Generated Content
(AIGC) topics, including text, audio, and vision modalities. We aim to prompt
open research on large language/music/multimodal models (LLMs/LMMs) training,
data collection, and development of fun applications.

## References

  * Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Ahmed & Wahed [2020] Nur Ahmed and Muntasir Wahed.  The de-democratization of ai: Deep learning and the compute divide in artificial intelligence research.  _arXiv preprint arXiv:2010.15581_ , 2020.
  * AI@Meta [2024] AI@Meta.  Llama 3 model card.  2024\.  URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.
  * Anthropic [2024] AI Anthropic.  The claude 3 model family: Opus, sonnet, haiku.  _Claude-3 Model Card_ , 2024.
  * Austin et al. [2021] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_ , 2021.
  * [6] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, MarcoDos Santos, Stephen Mcaleer, AlbertQ Jiang, Jia Deng, Stella Biderman, and Sean Welleck.  Llemma: An open language model for mathematics.
  * Bai et al. [2023] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_ , 2023.
  * Ben Allal et al. [2024] Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra.  Cosmopedia, 2024.  URL https://huggingface.co/datasets/HuggingFaceTB/cosmopedia.
  * Biderman et al. [2023] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.  Pythia: A suite for analyzing large language models across training and scaling.  In _International Conference on Machine Learning_ , pp. 2397–2430. PMLR, 2023.
  * Bisk et al. [2020] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al.  Piqa: Reasoning about physical commonsense in natural language.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pp. 7432–7439, 2020.
  * Blecher et al. [2023] Lukas Blecher, Guillem Cucurull, Thomas Scialom, and Robert Stojnic.  Nougat: Neural optical understanding for academic documents, 2023.
  * Bradley & Terry [1952] Ralph Allan Bradley and Milton E. Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.  ISSN 00063444.  URL http://www.jstor.org/stable/2334029.
  * Broder [1997] Andrei Z Broder.  On the resemblance and containment of documents.  In _Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171)_ , pp. 21–29. IEEE, 1997.
  * Chen et al. [2023a] Jianghao Chen, Pu Jian, Tengxiao Xi, Dongyi Yi, Qianlong Du, Chenglin Ding, Guibo Zhu, Chengqing Zong, Jinqiao Wang, and Jiajun Zhang.  Chinesewebtext: Large-scale high-quality chinese web text extracted with effective evaluation model, 2023a.
  * Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ , 2021.
  * Chen et al. [2023b] Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia.  Theoremqa: A theorem-driven question answering dataset.  In _The 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023b.
  * Chen et al. [2024] Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Dahua Lin, Kai Chen, and Feng Zhao.  Agent-flan: Designing data and methods of effective agent tuning for large language models.  _arXiv preprint arXiv:2403.12881_ , 2024.
  * Chevalier et al. [2024] Alexis Chevalier, Jiayi Geng, Alexander Wettig, Howard Chen, Sebastian Mizera, Toni Annala, Max Jameson Aragon, Arturo Rodríguez Fanlo, Simon Frieder, Simon Machado, Akshara Prabhakar, Ellie Thieu, Jiachen T. Wang, Zirui Wang, Xindi Wu, Mengzhou Xia, Wenhan Jia, Jiatong Yu, Jun-Jie Zhu, Zhiyong Jason Ren, Sanjeev Arora, and Danqi Chen.  Language models as science tutors.  _arXiv preprint arXiv: 2402.11111_ , 2024.
  * Christiano et al. [2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30, 2017.
  * Chung et al. [2024] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _Journal of Machine Learning Research_ , 25(70):1–53, 2024.
  * Clark et al. [2019] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  _arXiv preprint arXiv:1905.10044_ , 2019.
  * Clark et al. [2018] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_ , 2018.
  * Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Colombo et al. [2014] Massimo G Colombo, Evila Piva, and Cristina Rossi-Lamastra.  Open innovation and within-industry diversification in small and medium enterprises: The case of open source software firms.  _Research policy_ , 43(5):891–902, 2014.
  * Computer [2023] Together Computer.  Redpajama: an open dataset for training large language models, 2023.  URL https://github.com/togethercomputer/RedPajama-Data.
  * Contributors [2023] OpenCompass Contributors.  Opencompass: A universal evaluation platform for foundation models.  https://github.com/open-compass/opencompass, 2023.
  * Couldry & Mejias [2019] Nick Couldry and Ulises A Mejias.  Data colonialism: Rethinking big data’s relation to the contemporary subject.  _Television & New Media_, 20(4):336–349, 2019.
  * DeepSeek-AI [2024] DeepSeek-AI.  Deepseek llm: Scaling open-source language models with longtermism.  _arXiv preprint arXiv:2401.02954_ , 2024.  URL https://github.com/deepseek-ai/DeepSeek-LLM.
  * Deng et al. [2024] Qixin Deng, Qikai Yang, Ruibin Yuan, Yipeng Huang, Yi Wang, Xubo Liu, Zeyue Tian, Jiahao Pan, Ge Zhang, Hanfeng Lin, et al.  Composerx: Multi-agent symbolic music composition with llms.  _arXiv preprint arXiv:2404.18081_ , 2024.
  * Du et al. [2024] Xinrun Du, Zhouliang Yu, Songyang Gao, Ding Pan, Yuyang Cheng, Ziyang Ma, Ruibin Yuan, Xingwei Qu, Jiaheng Liu, Tianyu Zheng, Xinchen Luo, Guorui Zhou, Binhang Yuan, Wenhu Chen, Jie Fu, and Ge Zhang.  Chinese tiny llm: Pretraining a chinese-centric large language model, 2024.
  * Dubois et al. [2023] Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
  * Dubois et al. [2024] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto.  Length-controlled alpacaeval: A simple way to debias automatic evaluators.  _arXiv preprint arXiv:2404.04475_ , 2024.
  * Gabriel et al. [2018] Rodney A Gabriel, Tsung-Ting Kuo, Julian McAuley, and Chun-Nan Hsu.  Identifying and characterizing highly similar notes in big clinical note datasets.  _Journal of biomedical informatics_ , 82:63–69, 2018.
  * Geng & Liu [2023] Xinyang Geng and Hao Liu.  Openllama: An open reproduction of llama, May 2023.  URL https://github.com/openlm-research/open_llama.
  * Gionis et al. [1999] Aristides Gionis, Piotr Indyk, Rajeev Motwani, et al.  Similarity search in high dimensions via hashing.  In _Vldb_ , volume 99, pp. 518–529, 1999.
  * Groeneveld et al. [2024] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al.  Olmo: Accelerating the science of language models.  _arXiv preprint arXiv:2402.00838_ , 2024.
  * Gyawali et al. [2020] Bikash Gyawali, Lucas Anastasiou, and Petr Knoth.  Deduplication of scholarly documents using locality sensitive hashing and word embeddings.  2020\.
  * He et al. [2023] Conghui He, Zhenjiang Jin, Chao Xu, Jiantao Qiu, Bin Wang, Wei Li, Hang Yan, Jiaqi Wang, and Dahua Lin.  Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models, 2023.
  * Henderson* et al. [2022] Peter Henderson*, Mark S. Krass*, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho.  Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset, 2022.  URL https://arxiv.org/abs/2207.00220.
  * Hendrycks et al. [2020] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_ , 2020.
  * Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.  Measuring mathematical problem solving with the math dataset.  _arXiv preprint arXiv:2103.03874_ , 2021.
  * Hernandez et al. [2022] Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al.  Scaling laws and interpretability of learning from repeated data.  _arXiv preprint arXiv:2205.10487_ , 2022.
  * Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_ , 2022.
  * Hu et al. [2024] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al.  Minicpm: Unveiling the potential of small language models with scalable training strategies.  _arXiv preprint arXiv:2404.06395_ , 2024.
  * Huang et al. [2022] Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei.  Layoutlmv3: Pre-training for document ai with unified text and image masking.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pp. 4083–4091, 2022.
  * Huang et al. [2024] Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al.  C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Jaccard [1912] Paul Jaccard.  The distribution of the flora in the alpine zone. 1.  _New phytologist_ , 11(2):37–50, 1912.
  * Jiang et al. [2023] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_ , 2023.
  * Joshi et al. [2017] Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  _arXiv preprint arXiv:1705.03551_ , 2017.
  * Joulin et al. [2016] Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hervé Jégou, and Tomas Mikolov.  Fasttext.zip: Compressing text classification models.  _arXiv: Computation and Language,arXiv: Computation and Language_ , Nov 2016.
  * Kaddour [2023] Jean Kaddour.  The minipile challenge for data-efficient language models.  _arXiv preprint arXiv:2304.08442_ , 2023.
  * Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Kiveris et al. [2014] Raimondas Kiveris, Silvio Lattanzi, Vahab Mirrokni, Vibhor Rastogi, and Sergei Vassilvitskii.  Connected components in mapreduce and beyond.  In _Proceedings of the ACM Symposium on Cloud Computing_ , pp. 1–13, 2014.
  * Kocetkov et al. [2022] Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries.  The stack: 3 tb of permissively licensed source code.  _Preprint_ , 2022.
  * Kowsari et al. [2017] Kamran Kowsari, Donald E Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi, , Matthew S Gerber, and Laura E Barnes.  Hdltex: Hierarchical deep learning for text classification.  In _Machine Learning and Applications (ICMLA), 2017 16th IEEE International Conference on_. IEEE, 2017.
  * Kudo & Richardson [2018] Taku Kudo and John Richardson.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  _arXiv preprint arXiv:1808.06226_ , 2018.
  * Kwiatkowski et al. [2019] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466, 2019.
  * Lee et al. [2021] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini.  Deduplicating training data makes language models better.  _arXiv preprint arXiv:2107.06499_ , 2021.
  * Li et al. [2022] Chenxia Li, Ruoyu Guo, Jun Zhou, Mengtao An, Yuning Du, Lingfeng Zhu, Yi Liu, Xiaoguang Hu, and Dianhai Yu.  Pp-structurev2: A stronger document analysis system.  _arXiv preprint arXiv:2210.05391_ , 2022.
  * Li et al. [2023a] Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin.  Cmmlu: Measuring massive multitask language understanding in chinese.  _arXiv preprint arXiv:2306.09212_ , 2023a.
  * Li* et al. [2024] Tianle Li*, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica.  From live data to high-quality benchmarks: The arena-hard pipeline, April 2024.  URL https://lmsys.org/blog/2024-04-19-arena-hard/.
  * Li et al. [2023b] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Alpacaeval: An automatic evaluator of instruction-following models.  https://github.com/tatsu-lab/alpaca_eval, 2023b.
  * Liu et al. [2023a] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang.  Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023a.  URL https://openreview.net/forum?id=1qvx610Cu7.
  * Liu et al. [2024] Jie Liu, Zhanhui Zhou, Chao Yang, Han-Sen Zhong, and Wanli Ouyang.  Storm-7b, April 2024.  URL https://huggingface.co/jieliu/Storm-7B.
  * Liu et al. [2023b] Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang.  Alignbench: Benchmarking chinese alignment of large language models, 2023b.
  * Liu et al. [2023c] Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Roberto Iriondo, Cun Mu, Zhiting Hu, Mark Schulze, Preslav Nakov, Tim Baldwin, and Eric P. Xing.  Llm360: Towards fully transparent open-source llms, 2023c.
  * Longpre et al. [2023] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al.  The flan collection: Designing data and methods for effective instruction tuning.  _arXiv preprint arXiv:2301.13688_ , 2023.
  * Lozhkov et al. [2024] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries.  Starcoder 2 and the stack v2: The next generation, 2024.
  * Luo et al. [2023] Yin Luo, Qingchao Kong, Nan Xu, Jia Cao, Bao Hao, Baoyu Qu, Bo Chen, Chao Zhu, Chenyang Zhao, Donglei Zhang, et al.  Yayi 2: Multilingual open-source large language models.  _arXiv preprint arXiv:2312.14862_ , 2023.
  * Mihaylov et al. [2018] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  _arXiv preprint arXiv:1809.02789_ , 2018.
  * Nam Pham [2024] Nam Pham.  tiny-strange-textbooks (revision 6f304f1), 2024.  URL https://huggingface.co/datasets/nampdn-ai/tiny-strange-textbooks.
  * Nguyen et al. [2023] Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen.  Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages.  _arXiv preprint arXiv:2309.09400_ , 2023.
  * Paster et al. [2023] Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba.  Openwebmath: An open dataset of high-quality mathematical web text, 2023.
  * Penedo et al. [2023] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.  The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only.  _arXiv preprint arXiv:2306.01116_ , 2023.  URL https://arxiv.org/abs/2306.01116.
  * Qu et al. [2024] Xingwei Qu, Yuelin Bai, Yinghao Ma, Ziya Zhou, Ka Man Lo, Jiaheng Liu, Ruibin Yuan, Lejun Min, Xueling Liu, Tianyu Zhang, et al.  Mupt: A generative symbolic music pretrained transformer.  _arXiv preprint arXiv:2404.06393_ , 2024.
  * Rae et al. [2022] Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving.  Scaling language models: Methods, analysis & insights from training gopher, 2022.
  * Rafailov et al. [2023] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model, 2023.
  * Raffel et al. [2019] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _arXiv e-prints_ , art. arXiv:1910.10683, October 2019.  doi: 10.48550/arXiv.1910.10683.
  * Rajpurkar et al. [2018] Pranav Rajpurkar, Robin Jia, and Percy Liang.  Know what you don’t know: Unanswerable questions for squad.  _arXiv preprint arXiv:1806.03822_ , 2018.
  * Reid et al. [2024] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  _arXiv preprint arXiv:2403.05530_ , 2024.
  * Ricaurte [2019] Paola Ricaurte.  Data epistemologies, the coloniality of power, and resistance.  _Television & New Media_, 20(4):350–365, 2019.
  * Ronsor [2023] Ronsor.  Bigknow2022: Bringing language models up to speed.  https://github.com/RyokoAI/BigKnow2022, 2023.
  * Sakaguchi et al. [2021] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  Winogrande: An adversarial winograd schema challenge at scale.  _Communications of the ACM_ , 64(9):99–106, 2021.
  * Sandberg et al. [1985] Russel Sandberg, David Goldberg, Steve Kleiman, Dan Walsh, and Bob Lyon.  Design and implementation of the sun network filesystem.  In _Proceedings of the summer 1985 USENIX conference_ , pp. 119–130, 1985.
  * Sap et al. [2019] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi.  Socialiqa: Commonsense reasoning about social interactions.  _arXiv preprint arXiv:1904.09728_ , 2019.
  * Scao et al. [2023] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina Mcmillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco de Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro von Werra, Leon Weber, Long Phan, Loubna Ben Allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Šaško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-Shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng-Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lovering, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan-Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdeněk Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh Hajihosseini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Danish Contractor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emi Baylor, Ezinwanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezanejad, Hessie Jones, Indrani Bhattacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael Mckenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Escalada, Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pàmies, Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel de Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-Aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojciech Kusa, Yanis Labrak, Yash Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf.  BLOOM: A 176B-Parameter Open-Access Multilingual Language Model.  working paper or preprint, November 2023.  URL https://inria.hal.science/hal-03850124.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms, 2017.
  * Sennrich et al. [2015] Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  _arXiv preprint arXiv:1508.07909_ , 2015.
  * Shao et al. [2024] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo.  Deepseekmath: Pushing the limits of mathematical reasoning in open language models.  _arXiv preprint arXiv:2402.03300_ , 2024.
  * Sharma et al. [2019] Eva Sharma, Chen Li, and Lu Wang.  BIGPATENT: A large-scale dataset for abstractive and coherent summarization.  _CoRR_ , abs/1906.03741, 2019.  URL http://arxiv.org/abs/1906.03741.
  * Shashidhar et al. [2023] Sumuk Shashidhar, Abhinav Chinta, Vaibhav Sahai, Zhenhailong Wang, and Heng Ji.  Democratizing llms: An exploration of cost-performance trade-offs in self-refined open-source models.  In _The 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023.
  * Shazeer [2019] Noam Shazeer.  Fast transformer decoding: One write-head is all you need, 2019.
  * Shazeer [2020] Noam Shazeer.  Glu variants improve transformer, 2020.
  * Soboleva et al. [2023] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey.  SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.  https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama, 2023.  URL https://huggingface.co/datasets/cerebras/SlimPajama-627B.
  * Soldaini et al. [2024] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo.  Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research.  _arXiv preprint_ , 2024.  URL https://arxiv.org/abs/2402.00159.
  * Spithoven et al. [2013] André Spithoven, Wim Vanhaverbeke, and Nadine Roijakkers.  Open innovation practices in smes and large enterprises.  _Small business economics_ , 41:537–562, 2013.
  * Su et al. [2023] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.  Roformer: Enhanced transformer with rotary position embedding, 2023.
  * Talmor et al. [2018] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  _arXiv preprint arXiv:1811.00937_ , 2018.
  * Teknium [2023] Teknium.  Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023.  URL https://huggingface.co/datasets/teknium/OpenHermes-2.5.
  * Thuat Nguyen & Nguyen [2024] Huu Nguyen Thuat Nguyen and Thien Nguyen.  Culturay: A large cleaned multilingual dataset of 75 languages, 2024.
  * Touvron et al. [2023a] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  _ARXIV_ , 2023a.
  * Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv: 2307.09288_ , 2023b.
  * Vaswani et al. [2023] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2023.
  * Versley & Panchenko [2012] Yannick Versley and Yana Panchenko.  Not just bigger: Towards better-quality web corpora.  In _Proceedings of the seventh Web as Corpus Workshop (WAC7)_ , pp. 44–52, 2012.
  * Wang et al. [2024a] Tiannan Wang, Jiamin Chen, Qingrui Jia, Shuai Wang, Ruoyu Fang, Huilin Wang, Zhaowei Gao, Chunzhao Xie, Chuou Xu, Jihong Dai, Yibin Liu, Jialong Wu, Shengwei Ding, Long Li, Zhiwei Huang, Xinle Deng, Teng Yu, Gangan Ma, Han Xiao, Zixin Chen, Danjun Xiang, Yunxia Wang, Yuanyuan Zhu, Yi Xiao, Jing Wang, Yiru Wang, Siran Ding, Jiayang Huang, Jiayi Xu, Yilihamu Tayier, Zhenyu Hu, Yuan Gao, Chengfeng Zheng, Yueshu Ye, Yihang Li, Lei Wan, Xinyue Jiang, Yujie Wang, Siyu Cheng, Zhule Song, Xiangru Tang, Xiaohua Xu, Ningyu Zhang, Huajun Chen, Yuchen Eleanor Jiang, and Wangchunshu Zhou.  Weaver: Foundation models for creative writing.  _arXiv preprint arXiv: 2401.17268_ , 2024a.
  * Wang et al. [2024b] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Li Tianle, Shiguang Guo, Aaran Arulraj, Xuan He, Weiming Ren, Ziyan Jiang, Alex Zhuang, Kai Wang, Richard Fan, Max Ku, Xiang Yue, and Wenhu Chen.  Mmlu-pro: Towards more robust and challenging multi-task language understanding evaluation.  Manuscript in preparation, 2024b.
  * Wang et al. [2023] Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhu Chen, Jie Fu, and Junran Peng.  Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models.  _arXiv preprint arXiv: 2310.00746_ , 2023.
  * Wei et al. [2023a] Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang.  Vary: Scaling up the vision vocabulary for large vision-language models.  _arXiv preprint arXiv:2312.06109_ , 2023a.
  * Wei et al. [2023b] Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuanhai Dong, Yanqi Sun, Yifu Chen, Yongyi Peng, Xiaojuan Liang, Shuicheng Yan, Han Fang, and Yahui Zhou.  Skywork: A more open bilingual foundation model, 2023b.
  * Xu et al. [2022] Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn.  A systematic evaluation of large language models of code.  In _Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming_ , pp. 1–10, 2022.
  * Xu et al. [2023] Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston.  Some things are more cringe than others: Preference optimization with the pairwise cringe loss.  _arXiv preprint arXiv:2312.16682_ , 2023.
  * Yang et al. [2024] Qisen Yang, Zekun Wang, Honghui Chen, Shenzhi Wang, Yifan Pu, Xin Gao, Wenhao Huang, Shiji Song, and Gao Huang.  Llm agents for psychology: A study on gamified assessments.  _arXiv preprint arXiv: 2402.12326_ , 2024.
  * Young et al. [2024] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al.  Yi: Open foundation models by 01. ai.  _arXiv preprint arXiv:2403.04652_ , 2024.
  * Yu et al. [2024] Botao Yu, Frazier N. Baker, Ziqi Chen, Xia Ning, and Huan Sun.  Llasmol: Advancing large language models for chemistry with a large-scale, comprehensive, high-quality instruction tuning dataset.  _arXiv preprint arXiv:2402.09391_ , 2024.
  * Yuan et al. [2024] Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, et al.  Chatmusician: Understanding and generating music intrinsically with llm.  _arXiv preprint arXiv:2402.16153_ , 2024.
  * Yue et al. [2023] Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.  Mammoth: Building math generalist models through hybrid instruction tuning.  _arXiv preprint arXiv:2309.05653_ , 2023.
  * Yue et al. [2024] Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen.  Mammoth2: Scaling instructions from the web.  _arXiv preprint arXiv:2405.03548_ , 2024.
  * Zaharia et al. [2012] Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauly, Michael J Franklin, Scott Shenker, and Ion Stoica.  Resilient distributed datasets: A $\\{$Fault-Tolerant$\\}$ abstraction for $\\{$In-Memory$\\}$ cluster computing.  In _9th USENIX symposium on networked systems design and implementation (NSDI 12)_ , pp. 15–28, 2012.
  * Zellers et al. [2019] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  _arXiv preprint arXiv:1905.07830_ , 2019.
  * Zhang & Sennrich [2019] Biao Zhang and Rico Sennrich.  Root mean square layer normalization, 2019.
  * Zhang et al. [2023a] Ge Zhang, Yemin Shi, Ruibo Liu, Ruibin Yuan, Yizhi Li, Siwei Dong, Yu Shu, Zhaoqun Li, Zekun Wang, Chenghua Lin, et al.  Chinese open instruction generalist: A preliminary release.  _arXiv preprint arXiv:2304.07987_ , 2023a.
  * Zhang et al. [2023b] Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak.  Don’t trust chatgpt when your question is not in english: A study of multilingual abilities and types of llms.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pp. 7915–7927, 2023b.
  * Zhang et al. [2024] Yifan Zhang, Yifan Luo, Yang Yuan, and Andrew Chi-Chih Yao.  Automathtext: Autonomous data selection with language models for mathematical texts.  _arXiv preprint arXiv:2402.07625_ , 2024.
  * Zheng et al. [2024a] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36, 2024a.
  * Zheng et al. [2024b] Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue.  Opencodeinterpreter: Integrating code generation with execution and refinement.  _arXiv preprint arXiv:2402.14658_ , 2024b.
  * Zhu et al. [2023] Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao.  Starling-7b: Improving llm helpfulness & harmlessness with rlaif, November 2023.
  * Zhuang et al. [2024a] Alex Zhuang, Ge Zhang, Tianyu Zheng, Xinrun Du, Junjie Wang, Weiming Ren, Stephen W. Huang, Jie Fu, Xiang Yue, and Wenhu Chen.  Structlm: Towards building generalist models for structured knowledge grounding, 2024a.
  * Zhuang et al. [2024b] Xiaomin Zhuang, Yufan Jiang, Qiaozhi He, and Zhihua Wu.  Chuxin: 1.6 b technical report.  _arXiv preprint arXiv:2405.04828_ , 2024b.

##  Appendix A Appendix

###  A.1 Details of Heuristic Rules for English Texts

Table 11: Details of Heuristic Rules for English Texts

Rule | Note
---|---
Document-level Filtering
Mean word length [3, 10] | -
Lines that end with an ellipsis $\leq 0.2$ | Defined as ellipsis: ’…’, ’…’, ’……’
Lines starting with bullet point $\leq 0.9$ | Bullet points: ”•”, ”●”, ”○”, ”□”, ”※”, ”·” etc.
Words that contain no alphabetical character $\leq 0.4$ | -
Fraction of Stop words in the document must be $\geq 0.06$ | -
Number of stop words in the document must be $\geq 2$ | -
Symbols to words in the content must be $<0.5$ | -
Number of words in the content after normalization [50 ,10000] | -
Score of the language identification model must be $>0.8$ | Evaluated by fasttext
Number of characters must be $\geq 200$ | -
Number of lines $>1$ | -