prelude β€” long-context RoPE configurations

Three variants of openeurollm/prelude at revision anneal300b_iter_0989075, differing only in their RoPE parametrisation.

Identical weights to the base model β€” only config.json differs. No additional training was done. What changes is how positions are mapped, and therefore how far the model can actually read.

Perplexity by position

band base (as shipped) yarn-32k yarn-128k yarn-160k
0-512 7.91 14.29 15.27 15.33
512-1024 2.99 3.75 4.17 4.20
1024-2048 2.02 4.85 5.79 5.88
2048-4096 2.37 3.74 4.61 4.74
4096-8192 182.5 2.52 3.13 3.26
8192-16384 575.7 2.31 2.87 2.98
16384-32768 503.2 2.32 2.64 2.72
32768-65536 β€” 123.7 2.63 2.68
65536-131072 β€” 519.2 2.53 2.36
131072-163840 β€” β€” β€” 4.10

Each variant is best in the bands its window covers and falls apart past them. The base, as shipped, is healthy only to 4096 β€” its native window.

How to read the gaps. The base column comes from a separate run over a smaller document set (15 documents rather than 20), so treat it as indicative rather than directly comparable band for band. The 32k and 128k columns were measured with a 131072 ceiling, which is why they have no final band. Only the 160k column covers all ten.

Configurations

Each block is what goes into config.json. Nothing else in the file changes.

Base β€” leave as is

"max_position_embeddings": 32768,
"rope_parameters": {
  "rope_type": "yarn",
  "factor": 2.0,
  "original_max_position_embeddings": 4096,
  "rope_theta": 1000000.0,
  "beta_fast": 32.0,
  "beta_slow": 2.0
}

Note this already has YaRN applied. As originally shipped the model had rope_theta 100000 and no scaling at all, which is what the base column above measures.

yarn-32k

"max_position_embeddings": 131072,
"rope_parameters": {
  "rope_type": "yarn",
  "type": "yarn",
  "factor": 4.0,
  "original_max_position_embeddings": 32768,
  "rope_theta": 375000.0,
  "beta_fast": 2048.0,
  "beta_slow": 32.0
}
vllm serve <model> --max-model-len 32768

max_position_embeddings says 131072 only so vLLM's startup check passes. Past 32768 the output is unusable and nothing warns you.

yarn-128k

"max_position_embeddings": 131072,
"rope_parameters": {
  "rope_type": "yarn",
  "type": "yarn",
  "factor": 9.0,
  "original_max_position_embeddings": 16384,
  "rope_theta": 1000000.0,
  "beta_fast": 512.0,
  "beta_slow": 16.0
}
vllm serve <model> --max-model-len 131072

Transformers logs a warning that the explicit factor (9) does not match the implicit ratio (131072 / 16384 = 8). It uses the explicit factor, which is the one these numbers were measured with, so the warning is cosmetic.

yarn-160k

"max_position_embeddings": 163840,
"rope_parameters": {
  "rope_type": "yarn",
  "type": "yarn",
  "factor": 10.0,
  "original_max_position_embeddings": 16384,
  "rope_theta": 1000000.0,
  "beta_fast": 512.0,
  "beta_slow": 16.0
}
vllm serve <model> --max-model-len 163840

Same original and betas as yarn-128k β€” only the factor differs, and 10 Γ— 16384 lands exactly on 163840.


Why original_max_position_embeddings is not 4096

vLLM refuses to start unless factor Γ— original_max_position_embeddings >= max_model_len. Factor 4 over a 4096 original implies a 16384 window, so anything larger gets rejected.

Raising the original alone would change model behaviour. Raising it and multiplying beta_fast and beta_slow by the same number does not. Verified three times at factor 4, theta 500000:

original betas ppl 16k-32k
4096 128 / 4 2.162
40960 1280 / 40 2.162
81920 2560 / 80 2.162

Identical in every band. The betas count rotations over the original window, so when that window grows by N the thresholds must grow by N for YaRN's cutoff to land on the same dimensions.

Equivalent forms with the original stated as 4096:

  • yarn-32k: factor 4, theta 375000, betas 256 / 4
  • yarn-128k: factor 9, theta 1000000, betas 128 / 4
  • yarn-160k: factor 10, theta 1000000, betas 128 / 4

How this was measured

Perplexity by token position: long documents are passed through a forward pass without generation, and each token's loss is recorded according to where it sits, then grouped into log-spaced bands. Documents rendered through the chat template.

A healthy curve falls steadily and flattens; where it flattens is the effective context. If it rises and keeps rising, RoPE is broken there. Two traps: the high bands are fed by fewer documents, and a dip in the final band is usually corpus composition rather than model improvement.

This measures whether RoPE is intact, not task performance. None of these configs has been validated on a downstream benchmark, and there is reason to think the two diverge: on a related model, a long-context config improved recall-style behaviour while making multi-turn worse, with the model looping more often. The degradation YaRN introduces is concentrated in the 1k-8k range, which is where most agent trajectories live.


Limitations

  • Reparametrisation only. No continued pretraining, and no reparametrisation substitutes for it. There is a hard trade-off between range and quality that no choice of parameters escapes.
  • yarn-32k fails silently past its window. The declared 131072 exists purely to satisfy a startup check.
  • Single metric, and one that does not capture multi-turn coherence. A needle-in-a-haystack retrieval test would catch failures that band-averaged perplexity hides; neither catches looping.
  • Base model caveats carry over. These are intermediate research checkpoints with no safety alignment or instruction tuning.
  • If you plan to fine-tune, pick the config first. Weights adapt to the parametrisation they see during training; changing it afterwards shifts positions under weights already tuned to the old one.

If you need a real long-context model

Config changes cannot get you there. The recipe that works is staged ABF with continued pretraining. OpenEuroLLM extended this same prelude base to 256k:

prelude(4K) β†’ 16K(ΞΈ=500k) β†’ 32K(ΞΈ=1M) β†’ 64K(ΞΈ=2M) β†’ 128K(ΞΈ=32M) β†’ 256K(ΞΈ=64M)

Megatron-LM, 1B tokens for the 256k stage, 60% genuine long documents and 40% multilingual long-context mix. Published as oellm-9b-128k-theta32m-prelude and oellm-9b-256k-theta64m-prelude.

Downloads last month
7
Safetensors
Model size
9B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for ankush13r/prelude-base-iter_0989075-yarn-128k

Finetuned
(7)
this model