Zenyx-100M-Base
A 151M-parameter dense decoder-only language model trained from scratch on web, code and math text (the "100M" in the name is the original plan; the trained model is 151M).
Status: pre-training in progress (10.28B of a 50B-token budget at step 19,600; newer checkpoints may be in the repository). This is a base model: not instruction-tuned, no safety tuning, and it makes up facts. All numbers below are from mid-training checkpoints (the learning rate has not finished decaying); the benchmarks run with the segment memory off. From step 19,500 training also includes a recall task for the segment memory (below). A post-training stage is planned (below).
At a glance
| Parameters | 151.0M (152.2M with the experimental segment memory, checkpoints from step 17,900) |
| Architecture | 18-layer Llama-style decoder, grouped-query attention, SwiGLU, context 2,048; since step 17,860 an experimental segment memory in two blocks (see Segment memory) |
| Vocabulary | 49,105 tokens (byte-level BPE) |
| Training | 2 × Tesla T4, fp16; Muon + AdamW; 524,288 tokens per step (about 11k tokens/s with the segment memory, 21-25k before it) |
| Progress | step 19,600 = 10.28B tokens of 50B; learning rate at 48% of peak, decaying until 15B tokens |
| Latest benchmarks | step 19,600 (core suite, below); step 12,650 (wider suite, MMLU); step 10,900 (code, speed, letter-choice); step 10,800 (wide arithmetic) |
| License | Apache-2.0 |
Results
Base model, no fine-tuning, 0-shot, all test items. Fact1000 is a 1,000-item four-way factual test written by the author (chance 25%); SciQ prompts include the supporting passage. Standard error is about 1.4 points on 1,000 items.
Comparison at the latest checkpoints
| Zenyx-100M (step 19,600) | BetterGPT-150M | Muse2-230M* | Muse2-125M* | Zenyx-V3 (MoE) | |
|---|---|---|---|---|---|
| Parameters | 151.0M | 152.6M | 196.1M | 122.9M | 1.58B (0.38B active) |
| Training tokens | 10.3B | about 15B | not published | not published | about 80B |
| Fact1000 | 73.1% | 76.7% | 56.3% | 51.9% | 79.8% |
| Fact1000, without the arithmetic items | 70.1% | 78.3% | 61.7% | 57.1% | 79.2% |
| SciQ (acc / length-normalised) | 82.3 / 80.1 | 80.7 / 75.6 | 68.3 / 61.7 | 67.3 / 58.7 | 79.4 / 70.6 |
| ARC-Easy | 49.9 / 47.0 | 54.0 / 48.5 | 44.3 / 39.8 | 44.3 / 39.8 | 52.0 / 46.6 |
| ARC-Challenge | 22.4 / 23.7 | 25.5 / 27.2 | 18.9 / 24.1 | 17.3 / 22.6 | 21.6 / 26.1 |
| Pile bits per byte, 22 domains (lower is better) | 1.076 | 1.066 | 1.193 | 1.227 | 0.999 |
| Pile, mean of domains | 1.232 | 1.232 | 1.387 | 1.424 | 1.159 |
| Arithmetic, 14,000 problems, best prompt format | 93.7% (own chat format; about half of the items are in the training data, see Arithmetic) | 9.7% (3-shot) | not run | not run | 16.0% (3-shot) |
* Muse2: our own reimplementation of the architecture (no public code), same harness and items, not token-matched. V3 is step 110,000 (arithmetic: step 110,800). BetterGPT is its released checkpoint with its tokenizer's BOS+EOS prefix (the choice moves its scores by about 1 point; the author-default BOS prefix gives Fact1000 75.7%).
- BetterGPT-150M leads on Fact1000, ARC-Easy and ARC-Challenge by roughly 2 to 8 points and on the byte-weighted Pile (1.066 against 1.076), with about 1.5 times the training tokens (15B against 10.3B). Zenyx-100M is ahead on SciQ (both scorings; SciQ prompts contain the supporting passage, so it partly measures reading the context) and level on the domain-mean Pile (1.232 against 1.232). Zenyx-100M is ahead of both Muse2 models on Fact1000, SciQ, ARC-Easy and the Pile and level on ARC-Challenge (ARC-Challenge is close to the 25% chance level for every model in the table).
- Zenyx-100M leads on arithmetic by a wide margin: BetterGPT and V3 do 2-digit work and little more even with worked examples in the prompt, while Zenyx-100M was trained on arithmetic in its own chat format with a short scratchpad (see Arithmetic; part of this lead is memorisation, about half of the 14,000 items are in the training data).
- Zenyx-V3 (about 10 times the parameters, 8 times the tokens) is ahead on Fact1000, ARC-Easy (accuracy) and the Pile; Zenyx-100M's SciQ is above V3's at the checkpoints shown.
Wider suite (step 12,650)
| Zenyx-100M | BetterGPT-150M | Zenyx-V3 (step 107,200) | |
|---|---|---|---|
| HellaSwag (acc / norm) | 28.7 / 30.5 | 31.5 / 36.6 | 30.7 / 34.2 |
| PIQA | 59.9 / 58.4 | 65.2 / 65.5 | 61.4 / 60.7 |
| WinoGrande | 52.8 | 53.0 | 51.2 |
| OpenBookQA | 16.2 / 24.2 | 20.4 / 31.4 | 19.2 / 28.4 |
| BoolQ | 62.0 / 62.2 | 58.4 / 62.0 | 61.5 / 62.2 |
| COPA (100 items) | 57.0 | 63.0 | 55.0 |
| LAMBADA accuracy | 33.8 | 27.6 | 28.5 |
| WikiText bits per byte | 1.126 | 1.082 | 1.011 |
Most of these are close to chance for models of this size (HellaSwag 25%, PIQA/WinoGrande/COPA 50%); BetterGPT is ahead on all but LAMBADA and BoolQ (WinoGrande is a tie). An independent lm-evaluation-harness scoring agrees with ours within about 1 point for both small models.
Pile bits per byte by domain
Lower is better; bold = best of the four. Zenyx-100M step 19,600; Muse2-230M* and V3 as above. Bits per byte does not depend on the tokenizer.
| Domain | Zenyx-100M | BetterGPT-150M | Muse2-230M* | Zenyx-V3 |
|---|---|---|---|---|
| DM Mathematics | 1.311 | 1.348 | 1.723 | 1.326 |
| GitHub | 0.622 | 0.722 | 0.846 | 0.576 |
| YouTube Subtitles | 1.865 | 2.011 | 2.290 | 1.816 |
| Enron Emails | 1.273 | 1.249 | 1.416 | 1.222 |
| EuroParl | 2.020 | 2.155 | 2.444 | 1.964 |
| Wikipedia (en) | 1.046 | 1.080 | 1.123 | 0.987 |
| PhilPapers | 1.404 | 1.415 | 1.667 | 1.332 |
| HackerNews | 1.351 | 1.305 | 1.448 | 1.278 |
| Pile-CC | 1.175 | 1.124 | 1.199 | 1.098 |
| StackExchange | 0.994 | 1.025 | 1.207 | 0.917 |
| Gutenberg (PG-19) | 1.345 | 1.298 | 1.430 | 1.268 |
| BookCorpus2 | 1.220 | 1.155 | 1.205 | 1.142 |
| PubMed Central | 0.820 | 0.797 | 0.922 | 0.741 |
| ArXiv | 0.935 | 0.911 | 1.131 | 0.855 |
| OpenWebText2 | 1.208 | 1.188 | 1.281 | 1.126 |
| OpenSubtitles | 1.417 | 1.358 | 1.399 | 1.333 |
| USPTO Backgrounds | 0.865 | 0.828 | 0.938 | 0.779 |
| Books3 | 1.418 | 1.398 | 1.495 | 1.330 |
| NIH ExPorter | 0.955 | 0.906 | 1.018 | 0.861 |
| Ubuntu IRC | 1.808 | 1.852 | 2.121 | 1.714 |
| PubMed Abstracts | 0.974 | 0.932 | 1.072 | 0.873 |
| FreeLaw | 1.074 | 1.040 | 1.129 | 0.958 |
| 22 domains, byte-weighted | 1.076 | 1.066 | 1.193 | 0.999 |
| Mean of domains | 1.232 | 1.232 | 1.387 | 1.159 |
Zenyx-100M beats BetterGPT in 8 of 22 domains (GitHub, YouTube Subtitles, EuroParl, Wikipedia (en), StackExchange, DM Mathematics, Ubuntu IRC, PhilPapers) and is 1 to 6% behind in the other 14 (overall 0.9% behind). It is ahead of Muse2-230M* in 20 of 22 domains.
Over training
| Step | Tokens (B) | Fact1000 (no arithmetic) | SciQ acc / norm | ARC-Easy acc / norm | Pile bits/byte | Arithmetic (14,000) |
|---|---|---|---|---|---|---|
| 1,810 | 0.95 | 50.6% (53.6%) | not run | not run | not run | not run |
| 6,000 | 3.15 | 58.5% (61.7%) | 72.4 / 67.3 | 44.4 / 40.1 | 1.145 | 93.7% |
| 7,400 | 3.88 | 60.9% (63.8%) | 71.0 / 63.4 | 44.0 / 40.2 | not run | 72.1% |
| 8,910 | 4.67 | 60.1% (62.5%) | 72.3 / 65.8 | 42.3 / 39.6 | 1.120 | 95.6% |
| 9,500 | 4.98 | 61.9% (65.3%) | 74.2 / 69.2 | 44.5 / 41.2 | 1.120 | 95.2% |
| 9,700 | 5.09 | 61.7% (64.2%) | 72.7 / 67.9 | 45.6 / 41.6 | 1.119 | not run |
| 10,800 | 5.66 | 68.1% (66.0%) | 75.2 / 68.5 | 45.4 / 40.7 | 1.111 | 89.4% |
| 12,650 | 6.63 | 70.5% (68.0%) | 76.1 / 70.8 | 46.0 / 42.3 | 1.103 | 91.1% |
| 14,500 | 7.60 | 68.9% (66.9%) | 75.7 / 71.6 | 47.3 / 43.9 | 1.095 | 85.5% |
| 14,700 | 7.71 | 70.1% (67.2%) | 76.6 / 72.5 | 46.9 / 42.2 | 1.090 | 96.5% |
| 16,100 | 8.44 | 64.6% (67.3%) | 73.6 / 66.9 | 44.1 / 40.4 | 1.084 | 98.0% |
| 17,700 | 9.28 | 67.1% (69.2%) | 72.5 / 66.3 | 46.1 / 41.7 | 1.078 | 97.0% |
| 17,860 | 9.36 | 73.0% (70.0%) | 78.9 / 71.9 | 47.2 / 42.6 | 1.079 | 97.3% |
| 19,300 | 10.12 | 71.8% (69.2%) | 79.8 / 74.1 | 46.8 / 43.7 | 1.076 | 93.3% |
| 19,500 | 10.22 | 72.1% (69.5%) | 77.6 / 72.3 | 47.3 / 43.1 | 1.074 | 96.5% |
| 19,600 | 10.28 | 73.1% (70.1%) | 82.3 / 80.1 | 49.9 / 47.0 | 1.076 | 93.7% |
Compared with step 6,000, Fact1000 without the arithmetic items rose from 61.7% to 70.1%, SciQ from 72.4 to 82.3 and ARC-Easy from 44.4 to 49.9 (accuracy), the Pile from 1.145 to 1.076; ARC-Challenge stays at about chance. Since step 14,700 (one standard error is about 1.4 points on 1,000 items, 1.0 on ARC-Easy): Fact1000 without the arithmetic items +2.9, SciQ +5.7, ARC-Easy +3.0, ARC-Challenge +1.0, the Pile -0.0142 bits per byte. The curve is not monotone: SciQ and ARC-Easy dipped at step 16,100 and recovered by 19,600, and Fact1000 without the arithmetic items was 67.2, 67.3, 69.2 and 70.1 at 14,700, 16,100, 17,700 and 19,600, so differences of a point or two between checkpoints are within noise. The Pile figure was 1.0902, 1.0842, 1.0777 and 1.0759 bits per byte at the same four steps. The all-items Fact1000 score changed by +3.0 points since step 14,700 and by +5.0 since step 10,800 (the standard error of such a difference is about 2). Arithmetic oscillates between checkpoints (72% to 98% over the run). 2-digit multiplication: 92% at step 9,500, 58% at 10,800, 95% at 12,650, 23% at 14,500 (the first scratchpad line was right 95% of the time, but 76% of the replies then added a spurious second step), 87% at 14,700 and 95% at 19,600; its cause is not established.
Arithmetic
The training data includes arithmetic reasoning examples. The model answers arithmetic in a fixed chat-style format with a one-line scratchpad (no step-by-step working; the answer is computed in one forward pass):
<|im_start|>user
What is 683 + 479?<|im_end|>
<|im_start|>assistant
<think>
683 + 479 = 1162
</think>
1162<|im_end|>
14,000 problems (7 types × 2,000, fixed seeds, exact match), step 19,600: 2-digit addition 99.6%, 2-digit subtraction 96.9%, 3-digit addition 77.3%, 3-digit subtraction 89.1%, 4-digit addition 98.9%, 5-digit addition 99.1%, 2-digit × 2-digit 94.9%; all 93.7% (13,113 of 14,000). Without the chat format the same model scores under 1%: this is a trained, narrow skill.
The 14,000-problem suite overlaps the training data. We streamed 4 million rows of the arithmetic dataset in the training mix (SagheerLab/Arithmetic-Reasoning, same "What is x + y?" format) and found 6,691 of the 14,000 items word for word (2-digit addition, subtraction and multiplication: essentially all; 3-digit addition 35%; 3-digit subtraction 0.2%; 4- and 5-digit addition 0%; the scan stopped at its row cap, so these are lower bounds). The headline therefore mixes memorised items with skill. The three types that are not in the training data (3-digit subtraction, 4-digit and 5-digit addition, 6,000 items) score 95.7% at step 19,600; that is the number to compare across checkpoints.
Our own suite (tests/arith_own.py, 8,048 fresh items with seeds the old suite never used, every answer double-checked; step 19,300, native chat format): 3-digit subtraction 92.8%, 4-digit addition 99.5%, 4-digit subtraction 89.0%, 5-digit addition 98.2%, 5-digit subtraction 49.5% (borrowing). Sums of 3 to 10 numbers: 0.1% (3 of 2,700), mixed add/subtract chains 2.9%, repeated numbers such as 2+2+2+2+2 0%, products of three or four small factors 3.8%. The model guesses a long sum in one step (9 + 1 + 3 + 3 + 8 = 12, the answer is 24) instead of adding term by term. Training data that adds one term at a time (3-10 terms, verified by evaluation) has been written for the planned fine-tuning; the pre-training mix has none.
Wide suite (7,636 further problems in 28 types, exact match, step 10,800): 7.3% overall (13.0% at step 9,500; not rerun at the latest step). Best types: negative numbers, mul / div 31%; powers 30%; decimals, add / sub 30%; negative numbers, add / sub 23%. Near zero: 6-8 digit numbers, fractions, GCD / LCM, series sums, averages, unit conversion, linear equations and word problems. Between steps 9,500 and 10,800, 3-digit × 1-digit multiplication fell from 94.1% to 22.7% and 3-digit × 2-digit from 61.2% to 0.0%: replies now often follow a correct first scratchpad line with a spurious second step (for example 691 * 8 = 5528 | 691 + 5528 = 6219). This is the same chaining that caused the 2-digit multiplication dip above, and the failure seen at step 7,400 coming back.
Wide suite by type
| Type | Items | Exact match | Reply finished |
|---|---|---|---|
| 3-digit × 1-digit | 286 | 22.7% | 100% |
| 3-digit × 2-digit | 286 | 0.0% | 100% |
| 3-digit × 3-digit | 286 | 10.5% | 100% |
| 6-8 digit addition | 286 | 0.7% | 100% |
| 6-8 digit subtraction | 286 | 0.0% | 100% |
| exact division | 286 | 16.4% | 100% |
| division rounded down | 286 | 2.1% | 100% |
| remainder / mod | 286 | 0.0% | 72% |
| negative numbers, add / sub | 286 | 23.1% | 100% |
| negative numbers, mul / div | 286 | 30.8% | 99% |
| decimals, add / sub | 286 | 29.7% | 100% |
| decimals, multiply | 286 | 2.1% | 100% |
| order of operations | 286 | 18.9% | 100% |
| powers | 186 | 30.1% | 90% |
| square / cube roots | 199 | 0.0% | 51% |
| percentages | 286 | 15.4% | 100% |
| fractions | 286 | 0.0% | 86% |
| greatest common divisor | 286 | 0.0% | 42% |
| least common multiple | 286 | 0.0% | 77% |
| factorial / choose / permutations | 101 | 1.0% | 89% |
| sums of series | 286 | 0.0% | 63% |
| rounding | 286 | 0.0% | 76% |
| linear equations in x | 286 | 1.0% | 100% |
| word problems, one step | 286 | 1.0% | 88% |
| word problems, two steps | 286 | 0.0% | 78% |
| average | 286 | 0.0% | 80% |
| median | 286 | 0.0% | 83% |
| unit conversion | 286 | 0.0% | 67% |
Model details
| Component | Setting |
|---|---|
| Width / depth | d_model 768, 18 blocks |
| Attention | 12 query heads × 64, 4 key/value heads (grouped-query), causal, per-head QK-norm |
| Positions | RoPE, θ = 500,000 |
| MLP | SwiGLU, hidden size 2,048 |
| Normalisation | RMSNorm, pre-norm; fp32 residual stream with fp16 sub-layers (the T4 has no bf16) |
| Embeddings | tied input/output, 49,152 rows |
| Bias / dropout | none |
Sixteen of the 18 layers are plain grouped-query attention (blocks 8 and 12 additionally carry the segment memory below): sliding-window, gated linear and DeltaNet-style alternatives were benchmarked on a T4 at 2k context and none beat flash-attention GQA on speed. KV cache: 18,432 bytes per token in fp16.
Segment memory (experimental)
From step 17,860 two blocks (8 and 12, counting from 0) can also read the keys and values of the earlier 2,048-token windows of the same text, so a window may use up to 32,768 tokens of context although every forward pass stays 2,048 tokens. For each query head the layer looks up the 32 best-matching stored keys (exact top-32; the stored keys carry no position, so RoPE is untouched), and adds the weighted values to its normal attention output through a per-head tanh gate that starts at exactly zero: at the moment it was switched on the model computed the same function as the step-17,860 checkpoint. The memory has its own query projection (initialised as a copy of the layer's) and its own optimiser for the gates; no gradient flows into the stored keys and values. Design after Memorizing Transformers (Wu et al., 2022) and the lessons of Hugging Face's failed Infini-attention re-implementation.
| Memory blocks | 8 and 12; the cached decoder reads the memory (zenyx100m.memattn.generate_long, and Zenyx.ask in the notebook feeds long prompts window by window); the CUDA-graph fast decoder ignores it |
| New parameters | 1,179,800 (152.2M in total) |
| Layout | per GPU and step, 8 runs of 16 consecutive windows; memory reset at every optimiser step; 524,288 tokens per step as before |
| Cost | 2.2× slower training (49 s against 22 s per step on 2 × T4), 11.7 GB against 11.1 GB |
| Recall task | from step 19,500: 25% of the runs (falling to 12% over 1,500 steps) plant 3-20 invented facts in earlier windows and ask questions about them in a later window; loss on the answer tokens only |
Status: the memory copies text and ranks facts well, free recall of a fact from far back was missing until step 19,500 and has just started to work. Measured on step 18,670 and step 19,300 (one T4, tests/colab/mem_eval_v3.py; the memory is on or off with the same weights; the step-18,000 checkpoint, whose gates were still small, is the control):
| Passcode hidden in unrelated text, 8 candidate numbers (chance 12.5%, 24 trials per length) | 2k | 4k | 8k | 16k | 32k |
|---|---|---|---|---|---|
| Step 18,670, memory on | 83% | 96% | 88% | 92% | 92% |
| Step 18,670, memory off | 83% | 12% | 0% | 12% | 21% |
| Step 18,000, memory on | 79% | 67% | 29% | 21% | 29% |
| Step 19,300, memory on | 88% | 96% | 83% | 83% | 71% |
| Step 19,300, memory off | 88% | 8% | 8% | 8% | 12% |
At step 19,300, 30 long Wikipedia articles (first 16,384 tokens each): the memory lowers the loss of windows 1-7 by 0.037 nats (95% interval 0.028-0.047, better in every article; 0.035 at step 18,670, 0.022 at 18,000), growing with distance; a different article's memory makes the loss slightly worse (+0.012 to +0.014), so the gain is retrieval and not extra context. In 20 long chats of 8 windows the loss is 0.028 nats lower. Window 0 is identical with and without the memory (3.4028 against 3.4028), so a short prompt behaves exactly like the memory-free model, which is why the benchmarks on this card (run with the memory off) still apply. Reading the memory in the cached decoder is exact (logits agree with the full forward, top-1 agreement 100%, greedy decode identical on 16 of 16 tokens).
Repeated text is copied from the memory at any position: teacher-forced loss of a repeated 2,048-token span 3.78 without the memory against 0.21 with it (top-1 accuracy 32% to 96%), 0.24 when the span starts 64 tokens later, 0.42 when it starts 1,024 tokens later and 0.33 after 200 unrelated tokens (step 19,600). Questions about a fact far back were not answered before the recall task (step 19,300-19,400, greedy): the hidden passcode was written correctly 5% of the time at 4k tokens and 0% at 8k-64k (0% without the memory); a passage repeated after filler was continued for 12% / 8% / 4% of the next 40 tokens at 4k / 16k / 32k (1-3% without); after 3k-64k tokens of filler a chat fact (name, city, colour) was ranked first among 8 candidates 85% / 55% / 50% / 45% / 25% of the time (3k / 8k / 16k / 32k / 64k; 5-20% without the memory) but never produced as free text; and a three-fact memo 8.5k tokens back, asked as a question, scored 0 of 3. The reason is the training task: until step 19,500 nothing in training asked a question whose answer lay in an earlier window and did not simply continue the text.
Recall task (from step 19,500, zenyx100m/recall.py). Invented subjects and values (project leaders, budgets, code words, passcodes, dates, cities, counts, colours, times) are written as one sentence each into earlier windows of the same run (1-14 windows back, random positions and wrappers); the end of a later window becomes a block of questions about them in four layouts (Question/Answer, Q:/A:, the chat format, an unfinished sentence), and only the answer tokens carry loss. First result, the step-19,600 checkpoint after about 100 steps of it (one checkpoint, small samples, to be repeated): the memo test now answers the project lead and the budget correctly in chat format (2 of 3; the code word is wrong) against 0 of 3 with the memory off; on 24 fresh trials worded differently from the training task, the right value is produced in 8 of 24 chat answers (33%) and 5 of 24 plain Q/A answers (21%) with the memory and in 0 of 24 and 0 of 24 without it. Still failing: code words and exact dates, "repeat the first sentence of the memo", summaries of a long document (they continue the article instead).
Not shown: lengths beyond 32,768 tokens in the ranking test, any benchmark effect (all benchmarks on this card run with the memory off), messy real-world retrieval (the natural-text gain is about 1% of the loss). The memory costs 2.2x training time. Core benchmarks at the latest step (memory off) are in the tables above; the step-19,600 checkpoint, the first after the recall task started, scored SciQ 82.3 / 80.1, ARC-Easy 49.9 / 47.0 and Fact1000 73.1% (up from 77.6 / 72.3, 47.3 / 43.1 and 72.1% at 19,500): one checkpoint, so we do not yet know whether the recall task or noise caused the jump. Checkpoints from step 17,900 contain 6 extra tensors (mem_gate, mem_q_proj.weight, mem_q_norm.weight in blocks 8 and 12): load them with strict=False or through zenyx100m.memattn.
Tokenizer
Byte-level BPE with a SuperBPE-style second stage (merges may cross spaces; digits stay single tokens; 248 special tokens). On 63 MB of text from the training sources it needs about 9% fewer tokens than the DeepSeek-V4-Pro tokenizer (4.56 against 4.14 bytes per token) with a vocabulary 2.6 times smaller; it ties on Wikipedia and is worse on markdown and GSM8K. Only English, code and math were measured. File: tokenizer_candidates_v3/F-49152-f70/tokenizer.json.
Training
| Optimiser | Muon on hidden matrices (lr 0.012), AdamW on embeddings and norms (lr 2e-3) |
| Schedule | warmup, constant until 3.9B tokens, then linear decay to 10% of peak between 3.9B and 15B tokens, held to the 50B-token budget |
| Batch | 524,288 tokens per step; about 21-25k tokens/s on 2 × T4 |
| Data | English web, code, math and chat text; since the October 24 relaunch (step 18,670) also instruction-style data (OpenOrca GPT-4 part, UltraChat, MetaMathQA, oasst, text-to-SQL) as ordinary pre-training text |
Other measurements
- MMLU (step 12,650), 5-shot, all 14,042 questions: 25.0% ± 0.7 (chance 25%; Zenyx-V3 26.9%).
- Python (step 10,900), twelve small functions run against unit tests (primes, factorial, palindrome, gcd, Fibonacci, FizzBuzz and similar): signature + docstring as plain completion 8 of 12 with greedy decoding and 43% of sampled attempts pass (8 samples per task, T = 0.6); chat format 5 of 12 and 34%; forcing
<think>first lowers it to 2 of 12 and 7%. (Step 6,000: 4 of 12 and 25%.) It still fails harder tasks. - Letter-choice multiple choice (step 10,900) in chat format is at chance (SciQ 19%, ARC-Easy 23%, ARC-Challenge 25%, 200 questions each) although the standard scoring of the same questions gives 75%, 45% and 24%: the model has the knowledge but not the "read four lettered options and pick one" skill.
- Decode speed (step 10,900), one T4, fp16, CUDA graph +
torch.compile: 349 tokens/s at batch 1, 2,572 tokens/s total at batch 8, 9,475 tokens/s at batch 64. - Long generation: greedy decoding repeats itself within about 30 words; suggested: sampling (temperature 0.8, top-k 50) with a repetition penalty of 1.1-1.2.
Planned post-training
A supervised fine-tuning stage is being prepared; no instruct model has been released and no result exists yet. The training set is built (19 shards = 149,571 windows of 2,049 tokens, 300M window tokens of which 208M are assistant tokens that carry loss; one planned source, our own earlier synthetic set at 4%, could not be read and is missing) from public chat, instruction and reasoning data plus synthetic data with exact answers (single operations and, new, sums of 3-10 numbers, tool-calculator calls, system-prompt-versus-user conflicts, identity), English only (a language filter drops other scripts and most other Latin-script languages). ChatML with the tokenizer's real special tokens (<|im_start|>, <|im_end|>, <think>, </think>, <tool_call>, <tool_response>), loss on assistant tokens only; system prompts that must win over conflicting user instructions; a Reasoning effort: none | low | medium | high line with reasoning budgets of 256 / 512 / 1,024 tokens. Weak points that motivate it, measured with a code-scored chat suite at step 18,900 (10 categories, 8-45 prompts each, macro average 54.6%): greetings 33% (it often answers "Hello" with an apology and "as a text-based AI"), instruction following with checkable constraints 20%, system-prompt obedience 22%, small code tasks 25%, free-text facts 27%, against arithmetic 96% and recall of a detail given earlier in the same chat 88%. The numbers in this card are the pre-SFT baseline.
Intended use and limitations
- Use: research, and as a starting point for fine-tuning. Not an assistant: it ignores system prompts, answers greetings badly, and may continue a prompt with an unrelated document.
- Facts: expect errors and invented details; long generations repeat themselves.
- Arithmetic works in its own chat format only (see above), for two numbers of up to five digits (five-digit subtraction 50%), and about half of the old test suite is in the training data. It does not add a list of numbers (0.1% on 3-10 terms) and not negatives, decimals, fractions, powers, algebra or word problems.
- Reasoning, MMLU, letter-choice multiple choice: at chance. Code (step 10,900): passes 8 of 12 small functions, not harder ones. The
<think>habit exists only for arithmetic. - Context 2,048 tokens (up to 32,768 with the experimental segment memory, whose free recall of far-back facts is just starting to work); English only; not evaluated for bias or harmful content.
- Evaluation: one training run, mid-training checkpoints; Fact1000 and the question sets are written by the author; Muse2 numbers use an unofficial reimplementation.
Repository contents
| Path | Contents |
|---|---|
checkpoints/<run>/step_N/ |
model weights (model.pt), optimiser state, training state; only the two newest checkpoints are kept |
tokenizer_candidates_v3/ |
tokenizers, including the one used |
wheels/, runs/, data/ |
flash-attention build, training manifests |
The model code is not published yet.