Zenyx-MCEN-Delta-Mamba (565M) — finished short run (1.43B tokens)

A ~565M-parameter language model that interleaves Mamba-3 (MIMO) and Gated DeltaNet-2 (GDN-2) sequence mixers with two MCEN stages (a reversible fixed-point "reasoning" core with a delta-rule memory), trained from scratch on the Zenyx-V3 pretraining mixture with the DeepSeek tokenizer. This repository holds training checkpoints, logs and diagnostics of a finished, short training run (1.43B tokens); it is a research artifact, not a release.

Snapshot. The Pile, SciQ and Zenyx-Fact1000 numbers were measured at step 6,198 (0.81B of a planned 1.5B training tokens, 54% of 11,444 steps), on 2026-10-17; the task-suite table is from step 6,198. Training has since finished (see the 2026-10-25 status): later checkpoints in checkpoints/main565m/ and checkpoints/main565m-gqa/ are newer than these tables. This is a base model: it completes text and does not follow instructions.

Status on 2026-10-25: the run is finished. The converted run main565m-gqa (619.2M parameters, 14 stages, the two MCEN stages replaced by grouped-query-attention blocks) reached step 11,444, 1,433,534,464 tokens (1.43B), the end of its schedule; the final checkpoint is checkpoints/main565m-gqa/step_0011444 (LATEST = 11444, all remote file sizes verified against the local ones). Single-batch training loss over the last eight steps was 2.03-2.46 (no overflow rescues, no non-finite steps, about 52 s per step on 2 × T4). No benchmark has been run on the final checkpoint: the last measurements are the step-10,247 and step-6,198 ones below (Fact1000 51.7% and 47.6%, chance 25%). With 1.43B tokens for 619M parameters (about 2.3 tokens per parameter, far below the usual 20) the model is heavily under-trained, so expect limited quality; no further training of this run is planned.

[!NOTE] Status on 2026-10-24. The MCEN run (main565m) stopped at step 11,164 of 11,444 (1.40B of the planned 1.5B tokens); its checkpoints are under checkpoints/main565m/. The MCEN stages never showed a measurable effect (see the status note below), so the last ~280 steps are being trained on a converted model: zenyx_mcen/convert_to_gqa.py replaced the two MCEN stages by causal grouped-query-attention blocks (output projections zero-initialised) and copied every other tensor and the optimizer state. Check on the converted checkpoint (step 11,164, 1 batch of 4 × 2,048 held-out tokens, fp16): loss with MCEN on 3.75017, with MCEN off 3.75001, converted GQA model 3.75001 (difference 0.00000). The converted run is main565m-gqa (preset z565m-gqa, 14 stages, 619.2M parameters including the multi-token-prediction head, checkpoints/main565m-gqa/); it resumed at step 11,164 with the same data position and runs at about 52 s per step against about 87 s with MCEN (1.7x faster). No benchmark has been run on either run since step 10,247.

[!NOTE] Status on 2026-10-21 (evening). Training has continued to step 9,000 (about 1.11B of the planned 1.5B tokens, 74%; 79% of the 11,444 steps). The benchmark tables below predate a numerical fix: the MCEN solver was overflowing in fp16 on most tokens and the overflowing tokens received no gradient (see Solver overflow: root cause and fix). The run was relaunched with the fix from step 8,400; the first fixed session (steps 8,401-8,878) is summarised in First fixed session: no overflow, 30% faster, no instability, but the MCEN stages still show no measurable effect. Benchmarks of the new checkpoints have not been re-run yet and will replace the tables here.

Status on 2026-10-23. Training has continued to step 10,247 (1.28B of the planned 1.5B tokens, 85%; 90% of the 11,444 steps); the benchmark tables further down are still the step-6,198 ones. Re-run on step 10,247 (Kaggle T4, fp16, same code):

Benchmark	step 10,247	step 6,198
Zenyx-Fact1000 (internal)	51.7% (53.4% without arithmetic)	47.6% (47.8%)
SciQ (acc / acc_norm)	70.1% / 62.7%	64.5% / 56.9%
ARC-Easy	43.9% / 38.3%	39.5% / 37.0%
ARC-Challenge	18.3% / 21.7%	18.9% / 23.0%
Arithmetic (plain X + Y = prompt, 14,000 problems)	0.0%	0.0%

The model keeps improving (Fact1000 45.5% at step 4,100, 47.6% at 6,198, 51.7% at 10,247; held-out loss 3.138 at step 9,400, 3.105 at step 10,200). It stays well behind Zenyx-100M and Zenyx-V3 at this token count. Arithmetic is 0.0% at every checkpoint: the model saw arithmetic only in a chat format, at under 1% of its data.

The MCEN stages still do nothing measurable. On step 10,247 (6 windows of 2,048 tokens, fp16): held-out loss with MCEN on 3.70871; with the MCEN output set to zero 3.70894 (+0.0002 nats); with 1 / 2 / 4 / 8 solver iterations 3.70895 / 3.70896 / 3.70871 / 3.70830. Multiplying what they write by 4 helps by 0.0007 nats, by 64 hurts by 0.024, by 256 by 0.55 and by 1,024 destroys the model, so there is no hidden signal waiting for a larger scale; the sign is right but the effect is tiny. They write an output of rms 1.74 and 0.96 into a residual stream of rms 1,330 and 5,329 (dominated by a few huge-activation tokens). The solver itself is well behaved (contraction factor about 0.80, fixed-point residual 0.06 and 0.16).

The overflow fix holds. In the two sessions since (steps 9,339 to 10,247) the loss scale never decreased (one normal doubling, 8,192 to 16,384), so there were no overflow-skipped steps, no tokens were dropped or abstained, and the fp16 headroom stayed at 22–27% of the maximum. The median number of rescued MCEN tokens per step grew slowly, from 0 to about 1,700 of roughly 131,000 per GPU (1.3%), with one step at 17%; this is being watched.

Decision pending. Because the stages cost most of the step time and contribute nothing measurable, replacing the two MCEN stages with causal grouped-query-attention blocks is prepared (a conversion tool starts the new model function-identical to this one with MCEN switched off; see the repository's zenyx_mcen/convert_to_gqa.py) and was applied on 2026-10-24 (see the first note at the top): the last steps are trained as main565m-gqa.

Architecture (preset z565m)

Stage schedule	14 stages: 6 × Mamba-3 MIMO, 6 × GDN-2, 2 × MCEN (M3 GDN2 M3 GDN2 M3 GDN2 MCEN M3 GDN2 M3 GDN2 M3 GDN2 MCEN)
Hidden size / FFN	1,536 / 3,072
Vocabulary	129,280 (DeepSeek tokenizer), tied input/output embedding
Mamba-3	d_state 64, head dim 64, expand 2, MIMO rank 2
GDN-2	24 heads × head dim 64
MCEN	d_reason 1,024, 16 memory heads × 64, FFN 2,752, rank-64 persistent read, RevDEQ solver (β = 0.25), memory mode delta
Multi-token prediction	depth 1, loss weight 0.1, gdn2 mixer
Parameters	~565M (preset bound 540–600M)
Training context	4,096 tokens

Training

Tokens per optimizer step	131,072
Planned budget	1.5B tokens (11,444 steps)
Optimizer	Muon for matrix parameters (lr 0.02), AdamW elsewhere (lr 3e-4, wd 0.1); cosine to 10%
Precision	fp16 backbone with dynamic loss scaling. MCEN solver: fp16 with a token-level fp32 rescue on T4 until step 8,400; from the 2026-10-21 relaunch an exact power-of-two rescaling removes the overflow (see below). fp32 / TF32 on the H200 stretch
Hardware	Kaggle 2 × T4 sessions and one stretch on a single H200 (Lightning); the run resumes from checkpoints across sessions

Data mixture — 121 sources in the frozen manifest (data/manifest_v2.json, created 2026-10-10 06:04:26, hash 9454e840c2b9f032…); grouped shares:

Source	Key	Share
bigcode/starcoderdata (92 language subsets)	code_starcoder	26.16%
HuggingFaceFW/fineweb-edu	fineweb_edu	15.09%
nvidia/Nemotron-SFT-Math-v3	math_nemotron_sft	10.06%
openbmb/Ultra-FineWeb-L3	ufw_en_multistyle	5.53%
openbmb/Ultra-FineWeb-L3	ufw_en_qa	5.03%
HuggingFaceTB/smollm-corpus	cosmopedia_v2	4.23%
wikimedia/wikipedia	wikipedia_en	3.52%
HuggingFaceTB/finemath	finemath_3plus	3.32%
HuggingFaceTB/finemath	infiwebmath_3plus	2.92%
nvidia/Nemotron-Pretraining-Specialized-v1.2	nem12_fact	2.52%
HuggingFaceTB/finemath	infiwebmath_4plus	2.52%
HuggingFaceTB/finemath	finemath_4plus	2.21%
nvidia/Nemotron-Pretraining-Specialized-v1.1	nem11_mc	2.09%
nvidia/Nemotron-Pretraining-Specialized-v1	nem1_math_textbooks	2.01%
nvidia/Nemotron-Pretraining-Specialized-v1	nem1_stem_sft	2.01%
nvidia/Nemotron-Pretraining-Specialized-v1	nem1_rqa	2.01%
nvidia/Nemotron-Pretraining-Specialized-v1.1	nem11_code_concepts	2.01%
nvidia/Nemotron-Pretraining-Specialized-v1.2	nem12_mc	1.11%
nvidia/Nemotron-Pretraining-Specialized-v1.1	nem11_economics	1.04%
nvidia/Nemotron-Pretraining-Specialized-v1	nem1_infinibyte	1.01%
Josephgflowers/OpenOrca-Step-by-step-reasoning	openorca_reasoning	0.80%
SagheerLab/Arithmetic-Reasoning	arithmetic_reasoning	0.80%
harryxi/HelpSteer3-general-code-Shift-Qwen-2.5-1.5B-Instruct-chat-formatted-generations	helpsteer3_code	0.80%
nvidia/Nemotron-Pretraining-Specialized-v1	nem1_sci_coding	0.50%
General-Medical-AI/SlideChat	slidechat_text	0.30%
nvidia/Nemotron-Pretraining-Specialized-v1.1	nem11_uncond_algo	0.20%
nvidia/Nemotron-Pretraining-Specialized-v1	nem1_wiki_rewrite	0.10%
nvidia/Nemotron-Pretraining-Specialized-v1.1	nem11_formal_logic	0.04%
nvidia/Nemotron-Pretraining-Specialized-v1.2	nem12_moral	0.03%
nvidia/Nemotron-Pretraining-Specialized-v1.2	nem12_gen	0.01%

Benchmarks (step 6,198; task suite step 6,198)

The comparison models are the same author's: Zenyx-v2-base (85M unique parameters, 32 effective layers, 32k vocabulary) and Zenyx-V3-Base (1.58B-parameter MoE, ~81B training tokens). They are far larger or far longer-trained than this snapshot, so read the comparisons as "where does this early checkpoint stand", not as a like-for-like contest. MCEN and V3 numbers are on a Kaggle T4 (fp16); V2 in fp32.

The Pile, by domain (bits per byte, LOWER is better)

Text: NeelNanda/pile-10k (pinned revision 127bfedcd5), documents grouped by pile_set_name, capped at 1.5M characters per domain, scored with a rolling 1024-token window at stride 512 so every token is scored exactly once with ≥ 512 tokens of left context; bits/byte is normalised by the UTF-8 bytes of the source text, so it is comparable across tokenizers.

Domain	MCEN @6,198	V2 @96,000	V3 @107,200	MCEN − V2	text (MB)
DM Mathematics	1.7946	2.1898	1.3201	-0.3952	0.81
ArXiv	1.2607	1.5944	0.8565	-0.3337	1.50
Books3	1.8757	2.1966	1.3300	-0.3209	1.51
PubMed Central	1.0708	1.3518	0.7426	-0.2810	1.51
FreeLaw	1.3784	1.6474	0.9632	-0.2691	1.51
Wikipedia (en)	1.3454	1.6001	0.9902	-0.2547	1.50
PubMed Abstracts	1.1996	1.4109	0.8749	-0.2112	1.50
USPTO Backgrounds	1.0472	1.2530	0.7807	-0.2058	1.50
NIH ExPorter	1.1261	1.3249	0.8620	-0.1987	0.22
Enron Emails	1.6915	1.8693	1.2206	-0.1778	0.06
Gutenberg (PG-19)	1.6631	1.8397	1.2715	-0.1766	0.53
OpenWebText2	1.5210	1.6778	1.1295	-0.1568	1.53
Pile-CC	1.3824	1.5380	1.1001	-0.1556	1.51
BookCorpus2	1.3764	1.5307	1.1426	-0.1542	0.61
Ubuntu IRC	2.4488	2.5754	1.7147	-0.1266	0.04
OpenSubtitles	1.6076	1.7211	1.3392	-0.1135	0.86
Github	0.9702	1.0832	0.5765	-0.1130	1.53
StackExchange	1.2813	1.3558	0.9205	-0.0745	1.50
HackerNews	1.6101	1.6748	1.2806	-0.0647	0.24
PhilPapers	1.8911	1.8543	1.3358	+0.0368	0.16
EuroParl	2.9315	2.6663	1.9986	+0.2653	0.07
YoutubeSubtitles	2.6214	2.3116	1.8218	+0.3098	0.19
All 22, byte-weighted	1.3746	1.5795	1.0015	-0.2049	20.40

MCEN is lower (better) than V2 in 19 of 22 domains; it is higher (worse) in: PhilPapers (+0.037), EuroParl (+0.265), YoutubeSubtitles (+0.310). The sample sizes of EuroParl, Ubuntu IRC, YoutubeSubtitles, Enron, PhilPapers, NIH ExPorter and HackerNews are small (each under ~250 KB), so those rows are noisy.

SciQ

acc 64.50% / acc_norm 56.90% (n = 1000, chance 25%); standard error about 1.6 points. For reference, Zenyx-v2-base scores 52.10% / 47.20% with the same code and Zenyx-V3-Base 79.90% / 72.70%.

Zenyx-Fact1000 (internal)

A frozen, 1000-question, four-way factual / common-sense completion benchmark (chance 25%; the model must rank the correct completion above three same-type distractors by log-likelihood; sha256 843416524fb39c5f…; not yet published). A "prior-only" control scores the same candidates after the neutral context "The answer is" — a score near it means the model is merely preferring common words.

Model	accuracy	95% interval	without arithmetic	prior-only control
MCEN @6,198 (565M, 0.81B tokens)	47.6%	44.5–50.7%	47.8%	28.8%
Zenyx-v2-base @96,000 (85M unique)	45.8%	42.7–48.9%	48.1%	30.7%
Zenyx-V3-Base @107,200 (1.58B MoE)	81.1%	78.6–83.4%	80.8%	28.7%
Zenyx_Base_220M (Qwen tokenizer)	65.3%	62.3–68.2%	73.2%	29.8%
Zenyx-42M (FineWeb-Edu only)	49.5%	46.4–52.6%	54.4%	29.1%
MCEN and V2 are statistically indistinguishable overall (they answer different questions correctly). Step 5,500 scored 47.1% and step 4,100 scored 45.5%, so the gain per 700 steps is within the ±3-point interval.

Task suite (the same 19 tasks and extras as Zenyx-V3-Base)

0-shot, full evaluation sets (MMLU 5-shot), log-likelihood scoring of candidate continuations; acc_norm is length-normalised in characters. Same task code as the Zenyx-V3-Base and Zenyx-v2-base cards. MCEN was scored on a Kaggle T4 (fp16 backbone, MCEN depth K = 4). This table is from step 6,198. Bold = the conventional metric per task. Random for BoolQ is its 62.2% majority class; arithmetic is generated in-harness and is not interchangeable with published numbers.

Benchmark	MCEN @6,198	V2-base @96,000	V3-Base @107,200 (T4 fp16)	Random	n
HellaSwag (acc_norm)	27.01% ± 0.44	27.81%	34.16%	25.0%	10,042
ARC-Easy (acc_norm)	36.95% ± 0.99	34.13%	46.89%	25.0%	2,376
ARC-Challenge (acc_norm)	22.95% ± 1.23	23.98%	26.28%	25.0%	1,172
PIQA (acc_norm)	56.04% ± 1.16	54.79%	60.72%	50.0%	1,838
WinoGrande (acc)	51.93% ± 1.40	49.96%	51.22%	50.0%	1,267
OpenBookQA (acc_norm)	24.40% ± 1.92	23.60%	28.40%	25.0%	500
BoolQ (acc)	61.83% ± 0.85	47.43%	61.50%	62.2%	3,270
SciQ (acc)	64.50% ± 1.51	51.90%	80.30%	25.0%	1,000
LAMBADA (OpenAI) (acc)	9.12% ± 0.40	3.43%	28.49%	0.0%	5,153
RACE (acc_norm)	27.81% ± 0.64	26.04%	34.27%	25.0%	4,934
CommonsenseQA (acc_norm)	26.78% ± 1.27	26.04%	30.38%	20.0%	1,221
COPA (acc)	51.00% ± 5.00	56.00%	55.00%	50.0%	100
LogiQA (acc_norm)	24.42% ± 1.68	24.73%	24.88%	25.0%	651
WSC273 (acc)	53.48% ± 3.02	50.92%	56.41%	50.0%	273
TruthfulQA MC1 (acc)	23.01% ± 1.47	26.44%	18.73%	22.8%	817
Arithmetic (acc)	0.00% ± 0.00	0.36%	5.31%	0.0%	14,000
WikiText-2 bits/byte	1.4185	1.7676	1.0111
LAMBADA perplexity (tokenizer-dependent)	487.07	174.70	29.48

Generation

Greedy decoding frequently falls into repetition loops; a mild repetition penalty (1.1) reduces them, but the model still invents facts and drifts off topic on open-ended prompts, as expected this early in training. Treat it as a research checkpoint, not an assistant.

What is and is not established about MCEN itself

The MCEN stages are the novel component, and this run does not show that they help. All the measurements in this section were taken while the solver was overflowing in fp16 and most MCEN tokens carried no gradient (next section), so they describe an under-trained module and do not test what well-trained stages can do. They have to be repeated after the fix.

• Output magnitude. At step 1,600 their output was measured at 0.33% (first stage) and 0.03% (second stage) of the magnitude of the residual stream they write into, and switching either stage off changed the held-out loss by about 1e-4 nats or less.
• Matched ablation (finished). The same small backbone with and without MCEN stages, 65.5M training tokens, two seeds each, held-out loss (lower is better): plain hybrid 5.2252 / 5.2369; MCEN additive 5.2733 / 5.2486; MCEN stream 5.2883 / 5.2787. MCEN was not better than the plain hybrid, and was about 40% slower per step. Depth K = 1 / 4 / 8 differed by at most 0.0024 nats.
• After the fix (step 10,247, 2026-10-23). Output set to zero: loss +0.0002 nats; 1 / 4 / 8 solver iterations identical to 1e-3; amplifying the output gains at most 0.0007 nats and then hurts (see the status note at the top). The matched ablation above was run before the overflow fix was found, so it also tested an under-trained module; the post-fix measurement on the real model shows the same picture.
• The benchmark numbers above should therefore be attributed to the whole model (Mamba-3 + GDN-2 backbone, tokenizer and data), not to MCEN.

Solver overflow: root cause and fix

Found on 2026-10-20 from the step logs and an op-level probe on real step-8,200/8,300 weights and real held-out text; fixed and relaunched on 2026-10-21.

Symptom. The step log field mcen_resc R/T tok/A abst counts, per optimizer step, the MCEN solver calls that needed the fp32 rescue (R), the tokens rescued (T) and the tokens that could not be rescued (A). In every readable session from step 5,408 to 8,240, R was 8 (every one of the 4 micro-batches times 2 MCEN stages; 10 on steps that also ran the diagnostic report) and T was 100k to 145k of about 131k solver tokens per rank: roughly 86% of MCEN tokens overflowed fp16 on every step and were re-solved in fp32 without gradient. (An earlier internal audit misread this counter as 8 tokens per step; that was wrong.)

Consequence. A rescued token is a constant in the graph, so it sends no gradient into the MCEN core. On the step-8,300 weights the fp16 solve was non-finite for 100% of tokens in the first MCEN stage and 78% in the second; the core's gradient norm was exactly 0 in the first stage and about 57% of its fp32 value in the second. The MCEN modules were therefore starved of gradient for most of the run, which is consistent with their measured near-zero influence on the output, and the rescue also made every step slower.

Where it overflows. An op-level hook on the solver's matrix products shows the first non-finite value in the SwiGLU down chain of the MCEN map: the down-projection output reaches 60k to 64k against the fp16 maximum of 65,504, and the next projection overflows. Inputs, initial memory and keys are finite.

Fix. The up branch is scaled by 1/16 and the chain output is multiplied by 16 in fp32. Scaling by an exact power of two changes only exponent bits, so the function is unchanged. Validation on the same weights and text:

	Before	After
Non-finite solver tokens, first / second stage	100% / 78%	0% / 0%
Relative error against an fp32 solve	n/a (rescued)	about 1e-4, the size of fp16 rounding
Core gradient norm, first stage	0	5128 (fp32 reference: 5130)
Core forward + backward, 8,192 tokens	3.9-4.2 s (fp16 + rescue)	2.86 s (fp32 compute: 3.45 s)

Also changed: --reset_loss_scale was removed from the launch arguments (it forced the loss scale back to 65,536 on every resume and caused the two skipped steps at 8,091 and 8,128), and the logged read/write/consistency statistics now ignore non-finite tokens. The first log of the relaunched run answered the speed and guard questions (next section); whether training the MCEN stages with real gradients improves the loss or the benchmarks is still not known.

First fixed session (steps 8,401-8,878)

Read from the training log of the first session with the fix (2 x T4, 11.17 h, resumed from step 8,400), against the last session before it (steps 8,079-8,403):

	Before the fix	After the fix
MCEN tokens rescued per step (median)	about 114,000 of about 131,000	0
Throughput, median (10th-90th percentile)	1,239 (1,072-1,445) tokens/s	1,610 (1,416-1,785)
Validation pass	295 s	121 s
Skipped steps	2	1 (an ordinary loss-scale halving at step 8,422)
Backward-guard drops	0	0
Peak memory per GPU	n/a	10.6 GB
Gradient norm of the MCEN parameter group (median)	0.000	0.002 (embeddings about 0.4, mixer about 0.5)
Mean training loss over the session	2.396	2.350

The loss difference is about two standard errors on different data with a falling learning rate, so it is not attributed to the fix. Held-out validation loss (K = 4 iteration pairs) kept falling at the pace of the cosine schedule and no faster after the fix: 3.2091 (step 8,200), 3.1939 (8,400), 3.1830 (8,600), 3.1749 (8,800).

The MCEN stages still show no measurable effect. The validation loss with 1, 4 and 8 iteration pairs is identical to about 1e-4 (step 8,800: 3.17491 / 3.17494 / 3.17494), as before the fix, and their parameter group still receives a gradient about a hundred times smaller than the embeddings and the mixer. After roughly 8,400 steps of starvation the stages may need more training to matter, or their output gate may be near zero (not checked); whether they help at all needs the matched ablation, which has not been run on the full model. Raw numbers: results/2026-10-21/mcen_session_8400_8878.json in the project repository.

Generation at step 8,300

Sampled generation (temperature 0.7, top-p 0.9, 100 new tokens) from the step-8,300 checkpoint is fluent but shallow, in line with 1.09B training tokens:

• "The capital of France is" continues with an invented place name and a Wikipedia-style list rather than "Paris"; greedy decoding loops on "the capital of the French Empire".
• "Once upon a time, in a small village by the sea," produces a coherent but repetitive children's story.
• The prompt def is_prime(n): with a docstring produces a correct, well-formed function (early exits, loop up to the square root, then return True), after which the text drifts to an unrelated document.

Decoding is eager (no CUDA graph) and launch-overhead bound: on one T4, 21.0 steps/s at two solver pairs and 17.9 steps/s at four for a single stream, and about 61 and 50 tokens/s when three streams are decoded together. A weight-read bound of roughly 250 tokens/s would be possible, so a graph-captured decoder is the main speed-up still available.

Parallelism

The model is trained with fully sharded data parallelism (PyTorch FSDP1) on 2 x T4. A benchmark of per-block FSDP2 sharding against the current setup (same model, same activation checkpointing and fp16/fp32 settings, forward and backward only) gave 37.34 s against 38.13 s per step with identical peak memory and identical losses, about 2%, so the optimizer and checkpoint code were not ported.

Planned post-training

Supervised fine-tuning for this model is being prepared once the base run finishes; no result exists yet. The trainer is written and passed a 3-step smoke test on two T4s on 2026-10-21 (base checkpoint loaded, role-token embeddings warm-started, masked loss, masked validation, 1,477 tokens/s, 10.6 GB), and the SFT corpus builder was checked against the real tokenizer. The template uses the tokenizer's own role tokens (<｜begin▁sys｜> system block, <｜User｜>, <｜Assistant｜>, end-of-sentence as end of turn, native tool-call and tool-output tokens, <think> / </think>), with the embeddings of those tokens initialised from the plain-text role markers the base model saw in pre-training. The loss covers assistant content only. The data carry many different system prompts and explicit system-over-user scenarios, a Reasoning effort: none | low | medium | high control (reasoning budgets of 256, 512 and 1,024 tokens that must be completed), step-by-step arithmetic, and distilled reasoning traces, restricted to English, code and math.

Training notes

• Between steps 4,501 and 5,900 about 13.9% of optimizer steps were skipped by the fp16 loss scaler. Cause, found by replaying the data: the output projection of the last GDN-2 block (blocks.11.mixer.o_proj) overflows fp16 on some tokens.
• The fix (an exact power-of-two rescale around that projection, and a per-token scale on the persistent read) was applied from the relaunch at step 6,198. The step-6,198 checkpoint itself was trained before the fix; the Pile, SciQ and Fact1000 numbers above were evaluated with the fixed code and are finite on all 22 Pile domains. Whether training no longer skips steps is checked on the run that continues from this checkpoint.

Repository contents

• checkpoints/main565m/step_* — checkpoints (LATEST points to the newest; only the last two are kept) · data/manifest_v2.json — frozen data manifest · logs/, evals/, forensics/ — training logs and diagnostics.
• The training and inference code is not in this repository.

Limitations

• Early, still-training base model: knowledge is shallow (see Fact1000) and generation is unreliable.
• English / code / mathematics oriented; weak on multilingual and conversational text (see EuroParl, YoutubeSubtitles).
• Not instruction-tuned, not safety-tuned.
• Benchmarks are the author's own implementations, not a harness run; Fact1000 is internal and unpublished.