prelude β long-context RoPE configurations
Three variants of openeurollm/prelude
at revision anneal300b_iter_0989075, differing only in their RoPE
parametrisation.
Identical weights to the base model β only config.json differs. No
additional training was done. What changes is how positions are mapped, and
therefore how far the model can actually read.
Perplexity by position
| band | base (as shipped) | yarn-32k |
yarn-128k |
yarn-160k |
|---|---|---|---|---|
| 0-512 | 7.91 | 14.29 | 15.27 | 15.33 |
| 512-1024 | 2.99 | 3.75 | 4.17 | 4.20 |
| 1024-2048 | 2.02 | 4.85 | 5.79 | 5.88 |
| 2048-4096 | 2.37 | 3.74 | 4.61 | 4.74 |
| 4096-8192 | 182.5 | 2.52 | 3.13 | 3.26 |
| 8192-16384 | 575.7 | 2.31 | 2.87 | 2.98 |
| 16384-32768 | 503.2 | 2.32 | 2.64 | 2.72 |
| 32768-65536 | β | 123.7 | 2.63 | 2.68 |
| 65536-131072 | β | 519.2 | 2.53 | 2.36 |
| 131072-163840 | β | β | β | 4.10 |
Each variant is best in the bands its window covers and falls apart past them. The base, as shipped, is healthy only to 4096 β its native window.
How to read the gaps. The base column comes from a separate run over a
smaller document set (15 documents rather than 20), so treat it as indicative
rather than directly comparable band for band. The 32k and 128k columns were
measured with a 131072 ceiling, which is why they have no final band. Only the
160k column covers all ten.
Configurations
Each block is what goes into config.json. Nothing else in the file changes.
Base β leave as is
"max_position_embeddings": 32768,
"rope_parameters": {
"rope_type": "yarn",
"factor": 2.0,
"original_max_position_embeddings": 4096,
"rope_theta": 1000000.0,
"beta_fast": 32.0,
"beta_slow": 2.0
}
Note this already has YaRN applied. As originally shipped the model had
rope_theta 100000 and no scaling at all, which is what the base column above
measures.
yarn-32k
"max_position_embeddings": 131072,
"rope_parameters": {
"rope_type": "yarn",
"type": "yarn",
"factor": 4.0,
"original_max_position_embeddings": 32768,
"rope_theta": 375000.0,
"beta_fast": 2048.0,
"beta_slow": 32.0
}
vllm serve <model> --max-model-len 32768
max_position_embeddings says 131072 only so vLLM's startup check passes. Past
32768 the output is unusable and nothing warns you.
yarn-128k
"max_position_embeddings": 131072,
"rope_parameters": {
"rope_type": "yarn",
"type": "yarn",
"factor": 9.0,
"original_max_position_embeddings": 16384,
"rope_theta": 1000000.0,
"beta_fast": 512.0,
"beta_slow": 16.0
}
vllm serve <model> --max-model-len 131072
Transformers logs a warning that the explicit factor (9) does not match the implicit ratio (131072 / 16384 = 8). It uses the explicit factor, which is the one these numbers were measured with, so the warning is cosmetic.
yarn-160k
"max_position_embeddings": 163840,
"rope_parameters": {
"rope_type": "yarn",
"type": "yarn",
"factor": 10.0,
"original_max_position_embeddings": 16384,
"rope_theta": 1000000.0,
"beta_fast": 512.0,
"beta_slow": 16.0
}
vllm serve <model> --max-model-len 163840
Same original and betas as yarn-128k β only the factor differs, and
10 Γ 16384 lands exactly on 163840.
Why original_max_position_embeddings is not 4096
vLLM refuses to start unless
factor Γ original_max_position_embeddings >= max_model_len. Factor 4 over a
4096 original implies a 16384 window, so anything larger gets rejected.
Raising the original alone would change model behaviour. Raising it and
multiplying beta_fast and beta_slow by the same number does not. Verified
three times at factor 4, theta 500000:
| original | betas | ppl 16k-32k |
|---|---|---|
| 4096 | 128 / 4 | 2.162 |
| 40960 | 1280 / 40 | 2.162 |
| 81920 | 2560 / 80 | 2.162 |
Identical in every band. The betas count rotations over the original window, so when that window grows by N the thresholds must grow by N for YaRN's cutoff to land on the same dimensions.
Equivalent forms with the original stated as 4096:
yarn-32k: factor 4, theta 375000, betas 256 / 4yarn-128k: factor 9, theta 1000000, betas 128 / 4yarn-160k: factor 10, theta 1000000, betas 128 / 4
How this was measured
Perplexity by token position: long documents are passed through a forward pass without generation, and each token's loss is recorded according to where it sits, then grouped into log-spaced bands. Documents rendered through the chat template.
A healthy curve falls steadily and flattens; where it flattens is the effective context. If it rises and keeps rising, RoPE is broken there. Two traps: the high bands are fed by fewer documents, and a dip in the final band is usually corpus composition rather than model improvement.
This measures whether RoPE is intact, not task performance. None of these configs has been validated on a downstream benchmark, and there is reason to think the two diverge: on a related model, a long-context config improved recall-style behaviour while making multi-turn worse, with the model looping more often. The degradation YaRN introduces is concentrated in the 1k-8k range, which is where most agent trajectories live.
Limitations
- Reparametrisation only. No continued pretraining, and no reparametrisation substitutes for it. There is a hard trade-off between range and quality that no choice of parameters escapes.
yarn-32kfails silently past its window. The declared 131072 exists purely to satisfy a startup check.- Single metric, and one that does not capture multi-turn coherence. A needle-in-a-haystack retrieval test would catch failures that band-averaged perplexity hides; neither catches looping.
- Base model caveats carry over. These are intermediate research checkpoints with no safety alignment or instruction tuning.
- If you plan to fine-tune, pick the config first. Weights adapt to the parametrisation they see during training; changing it afterwards shifts positions under weights already tuned to the old one.
If you need a real long-context model
Config changes cannot get you there. The recipe that works is staged ABF with continued pretraining. OpenEuroLLM extended this same prelude base to 256k:
prelude(4K) β 16K(ΞΈ=500k) β 32K(ΞΈ=1M) β 64K(ΞΈ=2M) β 128K(ΞΈ=32M) β 256K(ΞΈ=64M)
Megatron-LM, 1B tokens for the 256k stage, 60% genuine long documents and 40%
multilingual long-context mix. Published as
oellm-9b-128k-theta32m-prelude
and
oellm-9b-256k-theta64m-prelude.
- Downloads last month
- 7
Model tree for ankush13r/prelude-base-iter_0989075-yarn-128k
Base model
openeurollm/prelude-base