control-pretraining-30b-filtered-gpt55-4plus-v2-trustedmonitor-base

Knowledge reintroduction into the narrowly filtered (V2) arm: its base model (control-pretraining-30b-filtered-gpt55-4plus-v2-base, midtraining iteration 3126) continued at the midtraining peak learning rate (7.5e-4, constant after a 25-iteration warmup) on the reintroduction_gpt55_4plus_v2 config of geodesic-research/control-pretraining-datasets at revision 98b9bd8b71cc7be3a0bb2a61d133f04344230562: every document the arm's filters removed, the broadly filtered pretraining's and the narrowly filtered midtraining's, deduplicated, less every document its retained training data holds (382,778 documents, 726,549,631 tokens with EOD). The documents fill half of each batch of 256 sequences of 32768 tokens and a replay of the arm's own midtraining mix fills the rest. It is one continuous run of five passes over the documents, 174 iterations each and each reshuffled, run as one job per pass: every pass resumes the previous pass's final save, weights, full optimizer state and step alike (every save kept its optimizer state), so the 25-iteration warmup comes once, at the start of the first pass. Revision cpt_iter_ is the end of a pass; the checkpoints are not annealed and not post-trained (no SFT). Its replay-only control is control-pretraining-30b-filtered-gpt55-4plus-v2-trustedmonitor-replayonly-base.

Part of the Control Pretraining collection: Geodesic's pretraining-data-filtering study, in which an unfiltered baseline and arms with AI-scheming literature removed are compared through the same curriculum. No NVIDIA weights are used anywhere in the collection: the baseline and the broadly filtered arm are trained from random initialisation through all three stages, the end-to-end narrowly filtered arm (V2 E2E) through pretraining and midtraining, and an arm that redoes a single stage warm-starts from another arm's checkpoint, which its own description states. A knowledge-reintroduction model continues a filtered arm's midtraining final on the documents that arm's filters removed, beside a replay-only control that continues the same checkpoint for the same iterations without them. Architecture nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.

Curriculum

Stage	Iterations	Tokens	W&B run
nemotron_nano_30b_filtered_mini_2plus_pretrain (published elsewhere)	29,881	501,319,991,296 (501.3B)	control_pretrain_30b_filtered_mini_2plus_pretrain
nemotron_nano_30b_filtered_gpt55_4plus_v2_midtrain (published elsewhere)	3,126	52,445,577,216 (52.4B)	control_pretrain_30b_filtered_gpt55_4plus_v2_midtrain
continual_pretraining (this repository)	870	7,298,088,960 (7.3B)	control_pretrain_30b_filtered_gpt55_4plus_v2_trustedmonitor_cpt

A stage's tokens per iteration are its sequence length times its global batch (below), so the token count of a checkpoint is every earlier stage's tokens plus its iteration times its own stage's tokens per iteration.

Data and schedule

Read from each stage's training config. Shares are the config's blend weights, normalised; a corpus tokenized in shards counts once, with its shard count under Files.

nemotron_nano_30b_filtered_mini_2plus_pretrain

Sequence length 8,192, global batch 2,048 sequences (16,777,216 tokens per iteration), learning rate 1.0e-03 held constant, warmup 1% of the stage, tokenizer geodesic-research/nemotron-base-tokenizer.

Corpus	Share	Files
geodesic-research/control-pretraining-datasets subset climbmix_full_filtered_mini_2plus	69.8%	8
geodesic-research/control-pretraining-datasets subset zyda_full_filtered_mini_2plus	19.7%	1
geodesic-research/control-pretraining-datasets subset stack_edu_filtered_mini_2plus	5.0%	1
geodesic-research/control-pretraining-datasets subset climbmix_ai_docs_filtered_mini_2plus	4.0%	1
geodesic-research/control-pretraining-datasets subset zyda_ai_docs_filtered_mini_2plus	1.0%	1
geodesic-research/control-pretraining-datasets subset ai_safety_and_adjacent_filtered_mini_2plus	0.5%	1

nemotron_nano_30b_filtered_gpt55_4plus_v2_midtrain

Sequence length 32,768, global batch 512 sequences (16,777,216 tokens per iteration), learning rate 7.5e-04 with WSD (cosine) decay to 1.0e-05, warmup 100 iterations, tokenizer geodesic-research/nemotron-base-tokenizer.

Corpus	Share	Files
geodesic-research/control-pretraining-datasets subset climbmix_long_filtered_gpt55_4plus_v2	33.4%	1
geodesic-research/control-pretraining-datasets subset nemotron_stem_sft_filtered_gpt55_4plus_v2	19.1%	1
geodesic-research/control-pretraining-datasets subset arxiv_papers_filtered_gpt55_4plus_v2	15.3%	1
geodesic-research/control-pretraining-datasets subset nemotron_wiki_rewrite_filtered_gpt55_4plus_v2	13.3%	1
geodesic-research/control-pretraining-datasets subset zyda_long_filtered_gpt55_4plus_v2	9.4%	1
geodesic-research/control-pretraining-datasets subset ai_safety_and_adjacent_filtered_gpt55_4plus_v2	4.4%	1
geodesic-research/control-pretraining-datasets subset stack_edu_long_filtered_gpt55_4plus_v2	2.4%	1
geodesic-research/control-pretraining-datasets subset climbmix_ai_docs_long_filtered_gpt55_4plus_v2	1.9%	1
geodesic-research/control-pretraining-datasets subset zyda_ai_docs_long_filtered_gpt55_4plus_v2	0.5%	1
geodesic-research/control-pretraining-datasets subset nemotron_wiki_rewrite_ai_docs_filtered_gpt55_4plus_v2	0.4%	1

continual_pretraining

Sequence length 32,768, global batch 256 sequences (8,388,608 tokens per iteration), learning rate 7.5e-04 held constant, warmup 25 iterations, tokenizer geodesic-research/nemotron-base-tokenizer.

Corpus	Share	Files
geodesic-research/control-pretraining-datasets subset reintroduction_gpt55_4plus_v2	49.8%	1
geodesic-research/control-pretraining-datasets subset climbmix_long_filtered_gpt55_4plus_v2	16.8%	1
geodesic-research/control-pretraining-datasets subset nemotron_stem_sft_filtered_gpt55_4plus_v2	9.6%	1
geodesic-research/control-pretraining-datasets subset arxiv_papers_filtered_gpt55_4plus_v2	7.7%	1
geodesic-research/control-pretraining-datasets subset nemotron_wiki_rewrite_filtered_gpt55_4plus_v2	6.7%	1
geodesic-research/control-pretraining-datasets subset zyda_long_filtered_gpt55_4plus_v2	4.7%	1
geodesic-research/control-pretraining-datasets subset ai_safety_and_adjacent_filtered_gpt55_4plus_v2	2.2%	1
geodesic-research/control-pretraining-datasets subset stack_edu_long_filtered_gpt55_4plus_v2	1.2%	1
geodesic-research/control-pretraining-datasets subset climbmix_ai_docs_long_filtered_gpt55_4plus_v2	1.0%	1
geodesic-research/control-pretraining-datasets subset zyda_ai_docs_long_filtered_gpt55_4plus_v2	0.2%	1
geodesic-research/control-pretraining-datasets subset nemotron_wiki_rewrite_ai_docs_filtered_gpt55_4plus_v2	0.2%	1

Revisions

Every completed checkpoint is a revision; main is the final checkpoint of the continual_pretraining stage. Load one with revision="<name>". Tokens seen count the whole curriculum up to that checkpoint; the training loss is W&B's lm loss at that iteration (blank where the run did not log it).

Revision	Stage	Iteration	Tokens seen	Training loss
cpt_iter_174	continual_pretraining	174	555,225,186,304 (555.2B)	1.4586
cpt_iter_348	continual_pretraining	348	556,684,804,096 (556.7B)	1.3405
cpt_iter_522	continual_pretraining	522	558,144,421,888 (558.1B)	1.2336
cpt_iter_696	continual_pretraining	696	559,604,039,680 (559.6B)	1.1817
cpt_iter_870 (also main)	continual_pretraining	870	561,063,657,472 (561.1B)	1.0866

Provenance

Exported from the Megatron torch_dist checkpoints with megatron-bridge at TP1/EP4. Each revision's megatron_run_config.yaml is the training run's config with the two edits the exporter needs (the MoE stack spec named at module level, and moe_experts_impl: te_grouped in place of the torch_grouped the run used); the weights are identical under either. This repository holds the Hugging Face conversions only, which carry no optimizer state. The Megatron checkpoints they were exported from are not kept for every run.

This is a base model. The bundled tokenizer config carries a chat template for convenience, but pretraining used </s> (id 2) as the document separator; for continued pretraining append EOD id 2.