Sane-47M

A 46.7M-parameter English language model trained from scratch — architecture, tokenizer, training code and behaviour data are all original. No existing weights were fine-tuned, distilled or merged.

It is a research artifact and a small, honest chat model: it knows who it is, follows a chat format, and — most usefully — reads a supplied context and answers from it rather than from memory. At 47M parameters its own world knowledge is thin, so retrieval is the intended way to use it.

Try it in a browser: a Colab notebook — no install, free CPU runtime, about a minute to the first answer. Where it comes from: Sane Labs — four models, each larger than the last, none of them inheriting a single parameter from anywhere else.

Parameters	46.74 M (tied embeddings)
Architecture	RoPE · RMSNorm · SwiGLU · pre-norm, 12 layers, d_model 512, 8 heads, FFN 1408
Context	1024 tokens
Tokenizer	own byte-level BPE, vocabulary 16 000
Training tokens	≈ 4.4 B (≈ 3.5 passes over a 1.25 B-token corpus)
Hardware	Kaggle 2×T4, then TPU v5e-8
Final step	30 000
Language	English only
Precision	fp16 weights (~105 MiB)

Quickstart

Everything needed is in this repository.

pip install torch tokenizers huggingface_hub
hf download sekund0chka/sane-47m --local-dir sane-47m
cd sane-47m
python sane_chat.py --cpu --ckpt sane_final_fp16.pt --tok tokenizer.json

Runs on CPU. Commands inside the chat: /ctx <text> supplies a retrieval context, /reset clears history, /q quits.

Prompt format

The model was trained with four control tokens and no system prompt — the persona lives in the weights.

<|user|>
{question}
<|sane|>
{answer}<|end|>

With retrieval, the context goes before the question:

<|context|>
Search results: The Ridgeloom V4 weighs 34 kg and sells for 690 euros.
<|user|>
How much does the Ridgeloom V4 weigh?
<|sane|>

Sampling defaults that work well: temperature 0.7, top_k 50, top_p 0.92, repetition_penalty 1.12. For factual questions over a context, drop the temperature to 0.3 or use greedy decoding.

What it does and does not do

Works:

• knows what it is and does not claim to be human;
• reads <|context|> and copies the right value out of it, including noisy search-result text with distractors, and says so when the answer is not in the passage;
• holds a short multi-turn conversation within the 1024-token window;
• simple string and format tasks.

Does not work:

• facts without a context are unreliable. This is a capacity wall, not a bug to be fine-tuned away — 47M parameters trained on 4.4 B tokens cannot store much;
• arithmetic beyond the trivial;
• long free-form generation drifts;
• anything outside English.

Evaluation

No standardised benchmark numbers were recorded for this model — it was evaluated with a held-out behavioural probe only. Its successor, Sane-118M, was measured and scores roughly at GPT-2-small level; Sane-47M sits below it.

If you benchmark it, please open a discussion — the numbers are welcome.

Where this model sits

Sane Labs trains a line of models on one thesis that does not change with scale: put the knowledge in the context, not in the weights, and the model's job becomes reading carefully and stopping at the edge of what it was given.

Model	Size	State
Sane-47M	46.7 M	published — this model, the first one that worked
Sane-118M	118.5 M	published, and measured
Sane-500M	~500 M	trained, kept private
Synth-2	1.23 B (~330 M active)	in training

Synth-2 carries the idea into the architecture rather than the training data: a small head on every layer predicts the model's own error for each token, so an answer can underline the span you should check instead of asking to be trusted whole. Its pretraining log is public and updates itself from the training job, and the working notes record what each measurement changed.

Training data

Pre-training and instruction data (see ATTRIBUTION.md in this repository):

Source	License
HuggingFaceFW/fineweb-edu (sample-10BT)	ODC-By 1.0
HuggingFaceTB/smol-smoltalk	Apache-2.0

The behaviour pool — identity, retrieval-reading, refusal-of-roleplay — is procedurally generated by the author and contains no third-party text.

ODC-By requires attribution; ATTRIBUTION.md is that attribution and must travel with the model if you redistribute it.

Bias, risks and limitations

The model inherits whatever biases exist in web text filtered for educational quality. It has no safety tuning, no refusal training for harmful requests, and no guardrails: treat its output as untrusted text. It has no memory between sessions and no access to the internet — anything time-sensitive must be supplied in <|context|>.

License

Apache-2.0. Use it, change it, ship it, sell it — no revenue ceiling and no permission needed. Earlier copies were released under the Sane Community License 1.0; this repository is now Apache-2.0 and that is the licence that applies.

One obligation that is not mine and cannot be relicensed away: part of the training corpus is FineWeb-Edu under ODC-By 1.0, which requires attribution. ATTRIBUTION.md in this repository is that attribution and must travel with the model if you redistribute it.

Citation

@misc{sane47m,
  title  = {Sane-47M: a small language model trained from scratch},
  author = {S.C.},
  year   = {2026},
  url    = {https://huggingface.co/sekund0chka/sane-47m}
}

Corrections to any number on this card are welcome — every one of them came from a run that can be repeated. ssanelabs@gmail.com