# AGILLM-4 Starting Point AGILLM-4 should not be "AGILLM-3 with a larger `--block`." The next useful version should split context into layers of memory: 1. Local dense attention for exact nearby reasoning. 2. Learned anchor tokens that summarize spans while staying trainable end-to-end. 3. Compressive recurrent memory for information that must persist beyond the dense window. 4. Retrieval-style recall over previous anchors, so million-token behavior is approached as memory lookup plus local reasoning rather than dense all-pairs attention. The near-term codebase target is a measurable long-context trainer that can survive on current hardware: - RTX 4090: production 24GB lane for the >1B AGILLM-4 floor shape, then block-size growth while preserving AR+SAT+NAT. - B200 180GB: serious 2k-16k experiments. - B300 262GB: 8k-64k experiments, then memory-augmented tests. - Multi-GPU/cluster: ring or sequence parallel experiments later. ## Parameter Target AGILLM-3 is about 700M dense parameters, so AGILLM-4 should start above that rather than treating 700M as the proving tier. Implemented presets: | Preset | Shape | Approx params | Use | | --- | --- | ---: | --- | | `agillm4_floor` | d=1280, L=28, H=20, rank=160 | ~1.21B | minimum AGILLM-4 run | | `agillm4_main` | d=1536, L=32, H=24, rank=192 | ~1.70B | main target | | `agillm4_big` | d=1792, L=36, H=28, rank=224 | ~2.40B | stretch target after memory works | Default recommendation: train `agillm4_main` if B200/B300 availability is good. On a 24GB 4090, start with `agillm4_floor` so the run is still larger than AGILLM-3 while leaving enough VRAM for AR+SAT+NAT. ## 4090 Production Long-Block Plan The RTX 4090 lane is production. Its job is to keep useful AGILLM-4 training moving when only 24GB VRAM is available without accidentally dropping back to the AGILLM-3-sized `large` preset. Production first-run recipe on 4090: ```bash AGILLM4_4090_WARMSTART_FROM=/workspace/agillm-4/agillm4_floor_seed_from_v3_v47.pt \ AGILLM4_4090_PRESET=agillm4_floor \ AGILLM4_4090_BLOCK=1280 \ AGILLM4_4090_TOKEN_PARAM_RATIO=100 \ tmux new-session -d -s agillm4_floor \ /workspace/agillm-4/launch_agillm4_4090_floor_from_v47.sh ``` Important: `--sat_every 1 --nat_every 4` keeps SAT trained every step and NAT active on a cadence that fits 24GB cards. On B200/B300 use `--nat_every 1` for full AR+SAT+NAT every step. The AGILLM-4 code now backprops AR, SAT, and NAT sequentially, so the objective remains joint while peak VRAM is lower than holding all activation graphs at once. Escalation ladder on 4090: 1. Start `block=1280`. 2. If OOM, back off by about 20% at a time (`1024`, `768`, ...), not straight to half. 3. If stable below 23GB VRAM after real step timing, raise toward `1536`. If 8-bit optimizer is unavailable, install `bitsandbytes` rather than dropping the long-block target. SAT remains active every step; NAT should stay enabled with a slower cadence or `--nat_max_tokens` cap on 24GB. The code lowers peak memory by backpropagating AR, SAT, and NAT sequentially, not by deleting heads. For the year-scale full run, keep checkpoint retention bounded. The 4090 launcher keeps one local full checkpoint and one local delta. The companion uploader publishes status/log tails every 30 minutes, uploads the newest delta at most daily, uploads full checkpoints at most weekly, and prunes remote AGILLM-4 training uploads to the configured rolling window. ## Warm Start from AGILLM-3 (function-preserving) The next AGILLM-4 run does **not** start from a random init. `build_v4_seed.py` turns the most recent AGILLM-3 SFT checkpoint into a function-preserving v4 seed: ```bash python /workspace/agillm-4/build_v4_seed.py \ --from-ckpt /workspace/ckpts_sat_fixed_polish_v47_20260523/final.pt \ --v4-preset floor \ --out /workspace/agillm-4/agillm4_floor_seed_from_v3_v47.pt ``` Construction rules (v3 d=1024 / H=16 / r=128 / L=24 → v4 d=1536 / H=24 / r=192 / L=32): - Embedding, AR head, and NAT head pad with **zeros** in new `d_v4 - d_v3` columns. New-dim embeddings are zero — they propagate harmlessly through the rest of the graph because all paths that touch them are zero-padded too. - If the v3 checkpoint has no NAT head, `build_v4_seed.py` initializes NAT from the expanded AR projection rather than random weights. - Block params: `qkv`, `proj`, `ff.0`, `ff.2` are placed top-left into the larger v4 tensors; the rest is zero. Old heads contribute exactly the same; new heads' attention output is multiplied by zero proj columns, so their contribution is zero at step 0. - `U` rank 128 → 192 by appending zero columns. `U @ U.T` is unchanged, so the M-fold metric is bit-identical. - LayerNorm new dims initialise to weight=1, bias=0. - Extra layers 25..32 are identity-init: `mha.proj` and `ff.2` are zero, so the residual stream passes through unchanged. They start contributing only once training shapes them. Net: at step 0, `v4(x) == v3(x)` up to floating-point rounding for any input within v3's tokeniser vocabulary. The new capacity has zero impact on day-1 loss; training subsequently shapes it. Use with `--warmstart_from`: ```bash python /workspace/agillm-4/nB300_agillm4.py train \ --preset agillm4_floor \ --warmstart_from /workspace/agillm-4/agillm4_floor_seed_from_v3_v47.pt \ --batch_size 1 --block 1280 --amp --grad_checkpoint --sat_every 1 --nat_every 4 \ --token_param_ratio 100 ``` The seed file is several GB on disk (fp32 tensors). Rebuild it whenever a newer v3 final is preferred. For B200/B300, use `--v4-preset main` and train `agillm4_main`; for 4090, use `--v4-preset floor`. ## First Implemented Scaffold `nB300_agillm4.py` starts from the current chat-SFT-fixed trainer and adds: - `--attn_backend manual|sdpa` - `--grad_checkpoint` - AGILLM-4 defaults for chat SFT compatibility inherited from the patched AGILLM-3 file. - A real NAT head is re-added beside AR and SAT. NAT trains with a CTC loss over an expanded non-causal input (`--nat_expand 2` by default). `--nat_every` controls cadence; `--nat_max_tokens` can cap the NAT slice on 24GB runs without removing the head. `block_sweep_agillm4.py` runs synthetic forward/backward sweeps: ```bash python /workspace/agillm-4/block_sweep_agillm4.py \ --preset agillm4_main \ --blocks 768,1024,1536,2048,3072,4096 \ --batch_size 1 \ --attn_backend sdpa \ --grad_checkpoint \ --amp \ --json_out /workspace/agillm4_block_sweep.json ``` This gives real VRAM and throughput data before committing to long training. `profile_agillm4.py` breaks a synthetic AR+SAT step into timed pieces: ```bash python /workspace/agillm-4/profile_agillm4.py \ --preset agillm4_floor \ --block 512 \ --batch_size 1 \ --backends sdpa,sublinear \ --grad_checkpoint \ --amp \ --json_out /workspace/agillm4_floor_profile_512.json ``` Use this before changing architecture. The profiler reports AR core forward, AR head/loss, AR backward, SAT core forward, SAT head/loss, SAT backward, gradient clipping, optimizer step, VRAM peaks, and synthetic tokens/sec. ## Sublinear Attention Backend `nB300_agillm4.py` now has an opt-in experimental backend: ```bash --attn_backend sublinear \ --sublinear_window 256 \ --sublinear_stride 64 \ --sublinear_max_anchors 256 \ --sublinear_chunk 128 ``` This is still transformer attention, but each query reads a local band plus landmark candidates instead of every token. Candidate count is approximately: ```text 2 * sublinear_window + 1 + min(block / sublinear_stride, sublinear_max_anchors) ``` So the attention score path scales as `O(N * candidates)` instead of `O(N^2)`. This does not remove the cost of QKV projections, FFN layers, heads, or the second SAT forward/backward, so the profiler is mandatory. The point is to make long block sizes trainable while preserving exact local reasoning and a learned route to distant evidence. On a 4090 production lane, first probe: ```bash python /workspace/agillm-4/block_sweep_agillm4.py \ --preset agillm4_floor \ --blocks 512,640,768,1024,1280 \ --batch_size 1 \ --attn_backend sublinear \ --sublinear_window 256 \ --sublinear_stride 64 \ --sublinear_max_anchors 256 \ --sublinear_chunk 64 \ --grad_checkpoint \ --amp \ --json_out /workspace/agillm4_sublinear_4090_sweep.json ``` If it is stable, then compare it against SDPA at the same block size with `profile_agillm4.py`. ## n1.py Harvest AGILLM-4 is now starting to import exact, proof-backed improvements from `C:\Users\Scott\Downloads\n1.py` while keeping the AGILLM-4 model-scale and long-context branch intact. First harvested feature: exact M-fold expansion attention. When `rank > d_k`, the trainer now uses: ```text (q @ U) @ (k @ U).T == q @ (U @ U.T) @ k.T ``` This preserves the function while keeping score/cache key width at `d_k` instead of `rank`. The inference path caches `U @ U.T`; the training path recomputes it so gradients through `U` remain exact. Verification: ```bash python /workspace/agillm-4/verify_m_fold_agillm4.py \ --presets pico_1x,micro_3x \ --backends manual,sdpa ``` Second harvested feature: fused QKV projection. AGILLM-4 now uses one `Linear(d, 3d)` and chunks the result instead of running three separate `Linear(d, d)` projections. Legacy `q.weight/k.weight/v.weight` checkpoints load into the fused `qkv.weight` layout, and matching legacy AdamW moments are remapped on full resume. Verification: ```bash python /workspace/agillm-4/verify_qkv_agillm4.py \ --presets pico_1x,micro_3x \ --backends manual,sdpa,sublinear ``` See `N1_HARVEST.md` for the staged port order. ## Intelligence per FLOP Compute reduction is not enough by itself. AGILLM-4 should spend saved FLOPs on training signals that force memory and reasoning: - long needle retrieval with many plausible distractors - multi-hop chain following over distant records - long chat continuity - codebase-scale symbol tracking - proof dependency recall `long_context_curriculum.py` generates JSONL text examples for the first two: ```bash python /workspace/agillm-4/long_context_curriculum.py \ --out /workspace/agillm-4/data/memory_torture_8k.jsonl \ --examples 20000 \ --slots 96 \ --chain_len 12 \ --distractor_words 7000 ``` Train it as a small mixed-in source, not as the whole diet. The goal is not memorizing synthetic strings; the goal is punishing models that pretend to use long context while actually relying on local guesses. ## AGILLM-4 Research Bets ### Anchor Memory Every `anchor_stride` tokens, produce an anchor vector from the local block. Future blocks attend to recent anchors in addition to local tokens. Initial hypothesis: - Dense local window: 2k-8k. - Anchor stride: 256 or 512. - Anchor memory length: 512-4096 anchors. - Token attention stays cheap while memory covers hundreds of thousands of tokens. Status: wired into `Encoder` as a single `AnchorMemoryLayer` inserted after a configurable mid-stack block. The module is the existing `anchor_memory.AnchorMemoryLayer` (attention-pooled compressor + gated cross-attention read from a rolling bank of `max_anchors` anchors). Enable per-run with: ```bash python /workspace/agillm-4/nB300_agillm4.py train \ --preset agillm4_floor --batch_size 1 --block 512 --amp --grad_checkpoint \ --anchor_memory --anchor_stride 256 --anchor_max 2048 ``` `--anchor_position N` overrides the default mid-stack placement. Verifier: `verify_anchor_memory_and_kv_buffer_agillm4.py`. ### Compressive Recurrence Keep a recurrent memory bank updated by a learned compressor: ```text new_memory = f(old_memory, anchor_batch) ``` The model does not need to carry all old tokens; it carries a trainable summary that is repeatedly tested by retrieval and continuation losses. ### Memory Torture Curriculum Long context must be trained with tasks that punish fake recall: - needle retrieval at 8k, 32k, 128k, 512k - multi-needle retrieval with distractors - "use detail from section A to solve section Z" - long chat continuity - codebase-scale symbol tracking - synthetic theorem/proof dependency chains ### Weird But Worth Trying - Let SAT heads generate candidate anchor summaries, not just next tokens. - Use anchors as a second vocabulary of "thought tokens" with their own loss. - Train a small memory critic that predicts whether answer evidence exists in local tokens, anchors, or compressed memory. - Make retrieval differentiable during training but replaceable with fast approximate search at inference. ## Go / No-Go Gates AGILLM-4 earns real training only if: - Chat SFT improves AGILLM-3 behavior. - SDPA + checkpointing produces a clean block sweep. - 2k+ block training is stable on B200/B300. - Anchor-memory prototype beats plain local context on synthetic long-recall tests. If chat SFT fails, AGILLM-4 still proceeds, but as an architecture correction rather than a scale-up.