Download AGILLM-4.md from OpenTransformer/AGILLM-4: direct link, hf CLI and curl.
- Browser
- Download file 12.9 kB
-
https://huggingface.co/OpenTransformer/AGILLM-4/resolve/main/AGILLM-4.md
- Command line
-
hf download hf://OpenTransformer/AGILLM-4/AGILLM-4.md
-
curl -L -o AGILLM-4.md https://huggingface.co/OpenTransformer/AGILLM-4/resolve/main/AGILLM-4.md
AGILLM-4 Starting Point
AGILLM-4 should not be "AGILLM-3 with a larger --block." The next useful version should split context into layers of memory:
- Local dense attention for exact nearby reasoning.
- Learned anchor tokens that summarize spans while staying trainable end-to-end.
- Compressive recurrent memory for information that must persist beyond the dense window.
- Retrieval-style recall over previous anchors, so million-token behavior is approached as memory lookup plus local reasoning rather than dense all-pairs attention.
The near-term codebase target is a measurable long-context trainer that can survive on current hardware:
- RTX 4090: production 24GB lane for the >1B AGILLM-4 floor shape, then block-size growth while preserving AR+SAT+NAT.
- B200 180GB: serious 2k-16k experiments.
- B300 262GB: 8k-64k experiments, then memory-augmented tests.
- Multi-GPU/cluster: ring or sequence parallel experiments later.
Parameter Target
AGILLM-3 is about 700M dense parameters, so AGILLM-4 should start above that rather than treating 700M as the proving tier.
Implemented presets:
| Preset | Shape | Approx params | Use |
|---|---|---|---|
agillm4_floor |
d=1280, L=28, H=20, rank=160 | ~1.21B | minimum AGILLM-4 run |
agillm4_main |
d=1536, L=32, H=24, rank=192 | ~1.70B | main target |
agillm4_big |
d=1792, L=36, H=28, rank=224 | ~2.40B | stretch target after memory works |
Default recommendation: train agillm4_main if B200/B300 availability is good. On a 24GB 4090, start with agillm4_floor so the run is still larger than AGILLM-3 while leaving enough VRAM for AR+SAT+NAT.
4090 Production Long-Block Plan
The RTX 4090 lane is production. Its job is to keep useful AGILLM-4 training moving when only 24GB VRAM is available without accidentally dropping back to the AGILLM-3-sized large preset.
Production first-run recipe on 4090:
AGILLM4_4090_WARMSTART_FROM=/workspace/agillm-4/agillm4_floor_seed_from_v3_v47.pt \
AGILLM4_4090_PRESET=agillm4_floor \
AGILLM4_4090_BLOCK=1280 \
AGILLM4_4090_TOKEN_PARAM_RATIO=100 \
tmux new-session -d -s agillm4_floor \
/workspace/agillm-4/launch_agillm4_4090_floor_from_v47.sh
Important: --sat_every 1 --nat_every 4 keeps SAT trained every step and NAT active on a cadence that fits 24GB cards. On B200/B300 use --nat_every 1 for full AR+SAT+NAT every step. The AGILLM-4 code now backprops AR, SAT, and NAT sequentially, so the objective remains joint while peak VRAM is lower than holding all activation graphs at once.
Escalation ladder on 4090:
- Start
block=1280. - If OOM, back off by about 20% at a time (
1024,768, ...), not straight to half. - If stable below 23GB VRAM after real step timing, raise toward
1536.
If 8-bit optimizer is unavailable, install bitsandbytes rather than dropping the long-block target. SAT remains active every step; NAT should stay enabled with a slower cadence or --nat_max_tokens cap on 24GB. The code lowers peak memory by backpropagating AR, SAT, and NAT sequentially, not by deleting heads.
For the year-scale full run, keep checkpoint retention bounded. The 4090 launcher keeps one local full checkpoint and one local delta. The companion uploader publishes status/log tails every 30 minutes, uploads the newest delta at most daily, uploads full checkpoints at most weekly, and prunes remote AGILLM-4 training uploads to the configured rolling window.
Warm Start from AGILLM-3 (function-preserving)
The next AGILLM-4 run does not start from a random init. build_v4_seed.py
turns the most recent AGILLM-3 SFT checkpoint into a function-preserving v4
seed:
python /workspace/agillm-4/build_v4_seed.py \
--from-ckpt /workspace/ckpts_sat_fixed_polish_v47_20260523/final.pt \
--v4-preset floor \
--out /workspace/agillm-4/agillm4_floor_seed_from_v3_v47.pt
Construction rules (v3 d=1024 / H=16 / r=128 / L=24 → v4 d=1536 / H=24 / r=192 / L=32):
- Embedding, AR head, and NAT head pad with zeros in new
d_v4 - d_v3columns. New-dim embeddings are zero — they propagate harmlessly through the rest of the graph because all paths that touch them are zero-padded too. - If the v3 checkpoint has no NAT head,
build_v4_seed.pyinitializes NAT from the expanded AR projection rather than random weights. - Block params:
qkv,proj,ff.0,ff.2are placed top-left into the larger v4 tensors; the rest is zero. Old heads contribute exactly the same; new heads' attention output is multiplied by zero proj columns, so their contribution is zero at step 0. Urank 128 → 192 by appending zero columns.U @ U.Tis unchanged, so the M-fold metric is bit-identical.- LayerNorm new dims initialise to weight=1, bias=0.
- Extra layers 25..32 are identity-init:
mha.projandff.2are zero, so the residual stream passes through unchanged. They start contributing only once training shapes them.
Net: at step 0, v4(x) == v3(x) up to floating-point rounding for any input
within v3's tokeniser vocabulary. The new capacity has zero impact on day-1
loss; training subsequently shapes it.
Use with --warmstart_from:
python /workspace/agillm-4/nB300_agillm4.py train \
--preset agillm4_floor \
--warmstart_from /workspace/agillm-4/agillm4_floor_seed_from_v3_v47.pt \
--batch_size 1 --block 1280 --amp --grad_checkpoint --sat_every 1 --nat_every 4 \
--token_param_ratio 100
The seed file is several GB on disk (fp32 tensors). Rebuild it whenever a newer v3
final is preferred. For B200/B300, use --v4-preset main and train
agillm4_main; for 4090, use --v4-preset floor.
First Implemented Scaffold
nB300_agillm4.py starts from the current chat-SFT-fixed trainer and adds:
--attn_backend manual|sdpa--grad_checkpoint- AGILLM-4 defaults for chat SFT compatibility inherited from the patched AGILLM-3 file.
- A real NAT head is re-added beside AR and SAT. NAT trains with a CTC loss over an expanded non-causal input (
--nat_expand 2by default).--nat_everycontrols cadence;--nat_max_tokenscan cap the NAT slice on 24GB runs without removing the head.
block_sweep_agillm4.py runs synthetic forward/backward sweeps:
python /workspace/agillm-4/block_sweep_agillm4.py \
--preset agillm4_main \
--blocks 768,1024,1536,2048,3072,4096 \
--batch_size 1 \
--attn_backend sdpa \
--grad_checkpoint \
--amp \
--json_out /workspace/agillm4_block_sweep.json
This gives real VRAM and throughput data before committing to long training.
profile_agillm4.py breaks a synthetic AR+SAT step into timed pieces:
python /workspace/agillm-4/profile_agillm4.py \
--preset agillm4_floor \
--block 512 \
--batch_size 1 \
--backends sdpa,sublinear \
--grad_checkpoint \
--amp \
--json_out /workspace/agillm4_floor_profile_512.json
Use this before changing architecture. The profiler reports AR core forward, AR head/loss, AR backward, SAT core forward, SAT head/loss, SAT backward, gradient clipping, optimizer step, VRAM peaks, and synthetic tokens/sec.
Sublinear Attention Backend
nB300_agillm4.py now has an opt-in experimental backend:
--attn_backend sublinear \
--sublinear_window 256 \
--sublinear_stride 64 \
--sublinear_max_anchors 256 \
--sublinear_chunk 128
This is still transformer attention, but each query reads a local band plus landmark candidates instead of every token. Candidate count is approximately:
2 * sublinear_window + 1 + min(block / sublinear_stride, sublinear_max_anchors)
So the attention score path scales as O(N * candidates) instead of O(N^2).
This does not remove the cost of QKV projections, FFN layers, heads, or the
second SAT forward/backward, so the profiler is mandatory. The point is to
make long block sizes trainable while preserving exact local reasoning and a
learned route to distant evidence.
On a 4090 production lane, first probe:
python /workspace/agillm-4/block_sweep_agillm4.py \
--preset agillm4_floor \
--blocks 512,640,768,1024,1280 \
--batch_size 1 \
--attn_backend sublinear \
--sublinear_window 256 \
--sublinear_stride 64 \
--sublinear_max_anchors 256 \
--sublinear_chunk 64 \
--grad_checkpoint \
--amp \
--json_out /workspace/agillm4_sublinear_4090_sweep.json
If it is stable, then compare it against SDPA at the same block size with
profile_agillm4.py.
n1.py Harvest
AGILLM-4 is now starting to import exact, proof-backed improvements from
C:\Users\Scott\Downloads\n1.py while keeping the AGILLM-4 model-scale and
long-context branch intact.
First harvested feature: exact M-fold expansion attention. When rank > d_k,
the trainer now uses:
(q @ U) @ (k @ U).T == q @ (U @ U.T) @ k.T
This preserves the function while keeping score/cache key width at d_k
instead of rank. The inference path caches U @ U.T; the training path
recomputes it so gradients through U remain exact.
Verification:
python /workspace/agillm-4/verify_m_fold_agillm4.py \
--presets pico_1x,micro_3x \
--backends manual,sdpa
Second harvested feature: fused QKV projection. AGILLM-4 now uses one
Linear(d, 3d) and chunks the result instead of running three separate
Linear(d, d) projections. Legacy q.weight/k.weight/v.weight checkpoints
load into the fused qkv.weight layout, and matching legacy AdamW moments are
remapped on full resume.
Verification:
python /workspace/agillm-4/verify_qkv_agillm4.py \
--presets pico_1x,micro_3x \
--backends manual,sdpa,sublinear
See N1_HARVEST.md for the staged port order.
Intelligence per FLOP
Compute reduction is not enough by itself. AGILLM-4 should spend saved FLOPs on training signals that force memory and reasoning:
- long needle retrieval with many plausible distractors
- multi-hop chain following over distant records
- long chat continuity
- codebase-scale symbol tracking
- proof dependency recall
long_context_curriculum.py generates JSONL text examples for the first two:
python /workspace/agillm-4/long_context_curriculum.py \
--out /workspace/agillm-4/data/memory_torture_8k.jsonl \
--examples 20000 \
--slots 96 \
--chain_len 12 \
--distractor_words 7000
Train it as a small mixed-in source, not as the whole diet. The goal is not memorizing synthetic strings; the goal is punishing models that pretend to use long context while actually relying on local guesses.
AGILLM-4 Research Bets
Anchor Memory
Every anchor_stride tokens, produce an anchor vector from the local block. Future blocks attend to recent anchors in addition to local tokens.
Initial hypothesis:
- Dense local window: 2k-8k.
- Anchor stride: 256 or 512.
- Anchor memory length: 512-4096 anchors.
- Token attention stays cheap while memory covers hundreds of thousands of tokens.
Status: wired into Encoder as a single AnchorMemoryLayer inserted after a configurable mid-stack block. The module is the existing anchor_memory.AnchorMemoryLayer (attention-pooled compressor + gated cross-attention read from a rolling bank of max_anchors anchors). Enable per-run with:
python /workspace/agillm-4/nB300_agillm4.py train \
--preset agillm4_floor --batch_size 1 --block 512 --amp --grad_checkpoint \
--anchor_memory --anchor_stride 256 --anchor_max 2048
--anchor_position N overrides the default mid-stack placement. Verifier: verify_anchor_memory_and_kv_buffer_agillm4.py.
Compressive Recurrence
Keep a recurrent memory bank updated by a learned compressor:
new_memory = f(old_memory, anchor_batch)
The model does not need to carry all old tokens; it carries a trainable summary that is repeatedly tested by retrieval and continuation losses.
Memory Torture Curriculum
Long context must be trained with tasks that punish fake recall:
- needle retrieval at 8k, 32k, 128k, 512k
- multi-needle retrieval with distractors
- "use detail from section A to solve section Z"
- long chat continuity
- codebase-scale symbol tracking
- synthetic theorem/proof dependency chains
Weird But Worth Trying
- Let SAT heads generate candidate anchor summaries, not just next tokens.
- Use anchors as a second vocabulary of "thought tokens" with their own loss.
- Train a small memory critic that predicts whether answer evidence exists in local tokens, anchors, or compressed memory.
- Make retrieval differentiable during training but replaceable with fast approximate search at inference.
Go / No-Go Gates
AGILLM-4 earns real training only if:
- Chat SFT improves AGILLM-3 behavior.
- SDPA + checkpointing produces a clean block sweep.
- 2k+ block training is stable on B200/B300.
- Anchor-memory prototype beats plain local context on synthetic long-recall tests.
If chat SFT fails, AGILLM-4 still proceeds, but as an architecture correction rather than a scale-up.