Sage-S2.1-Flash Mixed NVFP4

Sage S2.1 Flash for two GB10s: 4-bit routed experts, the official FP8 release's attention and shared experts, BF16 where the release keeps BF16.

At a glance
🎯 Closeness to BF16 KL 0.0426, top-1 agreement 93.56%, NLL 1.05% above BF16's, in PyTorch on 128 documents
🔢 What is 4-bit The routed experts of layers 3 to 44 and of the MTP layer: NVFP4 (E2M1 codes, an E4M3 scale every 16 weights), quantized here from BF16 with no calibration data
🧩 What is not The official FP8 release's bytes for MLA attention and the shared experts; MXFP8 for the dense MLPs of layers 0 to 2; BF16 for KDA attention, lm_head, embeddings and the vision tower, as the FP8 release keeps them
💾 Size 183.8 GB in 144 files, 133 of them safetensors (the BF16 checkpoint is 643 GB, the official FP8 release 328.5 GB)
⚡ Serving TP=2 on two GB10s: 92.51 GiB of model on the first machine, 19.9 tok/s with drafts off, 6.7 s to the first token of a cold 9,600-token prompt
🔧 Engine TensorFold release 1.2.9, built from its tag, and main at 077998f7d9 both serve it (The engine). No engine change is needed: every form here is one release 1.2.9 reads

TP=2 and TP=3 mean the model split across two and three systems.

Sage-S2.1-Flash Mixed NVFP4 is a derived checkpoint of internlm/Sage-S2.1-Flash for GB10 (DGX Spark class) systems. Sage S2.1 Flash has the shape of GLM-5.3-Flash: 45 layers and an MTP draft layer, with 288 routed experts a layer from layer 3 on. The routed experts of layers 3 to 44 hold 94.7% of the BF16 weights. This checkpoint stores the routed experts in 4 bits and leaves the rest at the official FP8 release's precision, except the three dense MLPs, so that the model fits two GB10s. It is not a Shanghai AI Laboratory, InternLM, Z.AI or NVIDIA release.

It is the plain starting point, made from the weights alone. The second, Sage-S2.1-Flash Jammed XSFP4, puts in the methods of this account's GLM-5.3-Flash checkpoints, with scales every 32 weights and smoothing learned from calibration text; nothing of it is in this one, and as of 2026-11-20 it needs two TensorFold changes that no release has.

Is it for you?

  • ✅ Yes, if you have two GB10s and want to serve Sage S2.1 Flash with TensorFold. Release 1.2.9 serves it as it is (The engine): at TP=2 with a 40,960-token window and one stream, with little memory to spare, 8.6 to 8.7 GiB free on each machine at the lowest.
  • ❌ No, if you have one GB10. At TP=2 the first machine alone holds 92.51 GiB of model.
  • ❌ No, if you serve with another loader. The routed experts' scales are stored as csf_32x2_v1, and the mix of forms is declared in ModelOpt's MIXED_PRECISION table; TensorFold reads both, and no other loader was tried. The source's deployment_guide.md covers LMDeploy, vLLM and SGLang with the official releases.
  • ❌ Not for the scientific-input projector. The source repository's projectors/ folder is not in this checkpoint.
  • 🖼️ Image input was checked, not measured: test images up to 2,016 pixels a side were answered correctly at TP=2 (under In serving). Tool calls passed a round trip there, and tool-eval-bench 3.0.0 scored 94 to 97 (Speed). The vision tower's weights are here in BF16 as in the source; every other measurement below is text only.

What's quantized

Component Stored precision How it was made
Decoder routed experts, layers 3 to 44
163.02 GB
NVFP4: 4-bit E2M1 codes; E4M3 block scales of 16, stored as csf_32x2_v1; FP32 global scale, a power of two; 36,288 matrices Quantized here from BF16.
Calibration: none.
Smoothing: none.
Scales: for each block of 16 weights, the E4M3 scale with the smallest squared error among 25 neighbouring values, from 10 steps below to 14 above the first one that reaches the block's largest weight divided by 6. No activation statistics.
MTP routed experts, layer 45
4.08 GB
NVFP4 as above, block scales stored directly; 864 matrices The same fit.
MLA attention (q_a, q_b, kv_a, o) in 11 layers and the MTP layer, and the shared experts of layers 3 to 45
2.29 GB
FP8 E4M3 in 128×128 blocks, an FP32 scale a block; 177 matrices The official FP8 release's bytes, revision 716e6436905650b7be93e1b7dedfb0bb95afd3a3. Its scale tensor weight_scale_inv is stored here as weight_scale; the value multiplies the code in both.
Dense MLP, layers 0 to 2
0.47 GB
MXFP8: E4M3 values, a UE8M0 scale every 32 weights; 9 matrices Made here from BF16: each block's scale is the smallest power of two that keeps its largest weight within E4M3's range. The official release stores these in 128×128 FP8 blocks; TensorFold's loader at 077998f7d9 takes NVFP4 or MXFP8 for these three layers, so they are remade.
KDA attention (q, k, v, o) in 34 layers, lm_head, embeddings, router, kv_b, indexers, norms, the vision tower
13.86 GB
BF16; F32 where the source has F32 The BF16 checkpoint's bytes, revision 76f4513db7464485605a002a06a6190de0460574.

The FP8 bytes taken here are the BF16 weights rounded on the release's own scales: rounding this BF16 checkpoint on those scales gives the 177 matrices' codes byte for byte. So the 8-bit parts here are a plain rounding of BF16 too, with no retraining in them. The rest of the official release (its routed experts and dense MLPs) was not compared.

The engine

No engine change is needed. TensorFold's release 1.2.9 reads every form stored here, by the config's MIXED_PRECISION table.

Engine Serves this checkpoint?
TensorFold 1.2.9, the latest release as of 2026-11-17, built here from its tag (6b9de69510) Yes, in one launch at TP=2 on two GB10s, with a 40,960-token window, one stream and vision off. The three fixed prompts of In serving gave the same token_sha256 as on the build below, with drafts and with "draft": false. The model took 92.51 GiB on the first machine and loaded in 115 s; decode with drafts off ran at 19.95 and 19.91 tok/s; free memory at its lowest was 8.62 and 8.67 GiB on the two machines. Cold prompts and image input were not run on this build
TensorFold main at 077998f7d9, 186 commits after 1.2.9 Yes. Everything under In serving was measured on this build, the image check included
Earlier TensorFold releases Not tried
Other loaders Not tried (Is it for you?)

To serve it, put this repository's files in one folder on each machine and follow TensorFold's GLM-5.3-Flash CUDA recipe, which places the ranks and names the link between the hosts for NCCL. The launches here named that folder to the engine's binary, the same line on both machines but for the rank, with 192.0.2.1 standing for the first machine's address on their direct link:

# on the second machine (rank 1); on the first, TF_GLM_RANK=0
TF_GLM_RANK=1 TF_GLM_WORLD=2 NCCL_COMM_ID=192.0.2.1:29583 \
TENSORFOLD_MEMORY_RESERVE_GIB=9 TF_KV_SHARE=16 TENSORFOLD_PREFILL_SHARE=0.7 \
  tensorfold-native serve /path/to/Sage-S2.1-Flash-Mixed-NVFP4 \
  --context 40960 --parallel 1 --max-tokens 32768 --reasoning-effort high

That line is the configuration tried; beside it the launches set a name, host and port for the server, --no-update-check (a no-op in the engine), the engine's heat limits (TF_HEAT_HIGH=92, TF_HEAT_LOW=88) and NCCL's settings for the link. It was not weighed against others. The image check added --vision and kept 4 prompt states (TF_KV_SHARE=4). Memory is tight with these settings (In serving): with image input at a 262,144-token window the 9 GiB reserve was too small and 11 GiB served, and the engine's default reserve (a tenth of RAM) was not tried. More streams, on another build, are under Speed. The folder served held the 144 files of SHA256SUMS. tensorfold pull and serving by the repository's name were not tried.

Measured

Closeness to BF16

128 documents (32 each of code, English, Japanese and math; 144,285 scored positions) went through the model in PyTorch twice: once with the BF16 weights and once with this checkpoint's weights dequantized, activations in 16 bits both times. Every number below compares the two runs' next-token distributions over the whole vocabulary at the same positions.

Documents Positions KL (BF16 ‖ this) Top-1 agreement NLL, BF16 → this
All 128 144,285 0.0426 (0.0384 to 0.0471) 93.56% (93.09 to 94.05) 1.0385 → 1.0494 (+1.05%)
Code 30,966 0.0315 96.40% 0.4846 → 0.4957 (+2.29%)
English 47,458 0.0534 92.81% 0.9576 → 0.9774 (+2.07%)
Japanese 58,785 0.0411 92.53% 1.4284 → 1.4332 (+0.34%)
Math 7,076 0.0306 94.80% 0.7655 → 0.7671 (+0.20%)

The ranges are 95% intervals from resampling whole documents. The experts the router picks for a token overlap BF16's picks with a Jaccard index of 0.844 on average over the 42 routed layers: 0.961 at layer 3, 0.771 at layer 37, the lowest.

These are PyTorch numbers, not the serving engine's. The document set and its token ids are the ones behind the GLM-5.3-Flash cards of this account (Sage's tokenizer.json is the same file), but the model is another, so the numbers do not line up with those cards'.

The MTP layer is not part of the numbers above: the main model does not run it. Measured apart, on the same documents, with the document's own tokens fed in and one draft deep, its draft of the token after next agrees with the main model's own choice at 77.45% of 144,157 positions with this checkpoint's layer, against 77.29% with BF16's layer on BF16's run; the difference, +0.16 points, has a 95% interval of −0.52 to +0.83 over documents.

In serving

Two GB10s at TP=2 over a direct link, TensorFold main at 077998f7d9, a 40,960-token window, one stream. Two launches with vision off: the first for the token check and decode, the second for the prompts. A third launch with vision on is below the table.

Model on the first machine 92.51 GiB; 116 s and 114 s to load
Free memory at its lowest (MemAvailable) 8.65 to 8.73 GiB on each machine, over both launches
Decode, MTP drafts off 19.94 and 19.91 tok/s (two prompts, 256 tokens five times each, medians)
With drafts on (the MTP layer and copies of earlier text) 5.3, 2.0 and 3.8 tokens a round on the three fixed prompts (counting, prose, code); the log counts 415 of 648, 250 of 505 and 377 of 641 drafts accepted
Cold prompt of 9,600 tokens 6.69 s to the first token
Cold prompt of 38,400 tokens 26.15 s to the first token
Drafted against "draft": false The same 512 tokens on each of the three fixed prompts (token_sha256 82b9fd2bc80a…, 3d3f24d81246…, 1ee72b9053a2…)

TensorFold times two forms of the experts at load and keeps the faster: the first launch kept factored, the second classic. No line of either launch's logs says disabled, fallback or compile failed.

Memory is the tight part. Free memory fell to 8.65 GiB on each machine, 0.65 GiB above the 8 GiB at which this setup's watchdog stops the engine. The engine's own token pool was not the limit: after the model and its 9 GiB reserve it had 15.7 GiB, room for 23 windows of 40,960 tokens by its own count. The longest prompt of these launches was 38,400 tokens; a 262,144-token window is below, and more streams are under Speed.

Image input, one check. A third launch with vision on loads the vision tower on the first machine: 93.78 GiB of model there. To make room it kept 4 prompt states instead of the 16 of the launches above (0.9 GiB against 3.6 GiB); with 16 it was not tried. Two 448×448 PNGs made for the check, a red square and a blue square on white, were each sent with the question "What color is the square?": the replies were "Red." and "Blue.", the same with drafts and with "draft": false, at 282 prompt tokens a request. Free memory at its lowest was 8.72 GiB on the first machine and 10.26 GiB on the second.

A 262,144-token window, tool calls and larger images. Two later launches on another pair of GB10s, the same build, with a 262,144-token window, vision on, 4 kept prompt states and one stream; the model takes 93.90 GiB on the first machine.

With the 9 GiB reserve of the launches above It loads, with 8.55 GiB free, and gives the three fixed prompts' hashes. The first image request took free memory under 8 GiB, and the watchdog stopped the engine
With an 11 GiB reserve 10.64 GiB free once loaded; the engine counts room for 3 windows of 262,144 tokens
The three fixed prompts The same token_sha256 as above, drafted and with "draft": false
Tool calls A round trip in 5 cases, each with reasoning_effort high and without: a call with JSON arguments (finish_reason tool_calls), an answer from the tool's result, one tool picked out of three, the same call put together from a stream's deltas, and a plain answer where no tool is needed. 10 of 10 as expected
Images Squares of 448, 896, 1,344 and 2,016 pixels a side (282 to 4,122 prompt tokens), each answered with its colour
A cold prompt of 200,000 tokens 137.9 s to the first token; sent again, 1.1 s, with 198,656 tokens from the prompt cache
Free memory at its lowest 9.63 GiB on the first machine, during the 2,016-pixel image; the long prompt did not lower it

Speed

On 2026-11-20, beside Sage-S2.1-Flash Jammed XSFP4, in the shape of the Speed tables on this account's GLM-5.3-Flash cards. This is another build than above: TensorFold main at e3fb56cb09 with #3680 and #3807 merged in, two open pull requests that Jammed XSFP4 needs and this checkpoint does not use. The settings are the ones GLM-5.3-Flash BIZ 3.10.0's image gives each TP size: four streams and 16 kept prompt states at TP=2, eight streams and 32 at TP=3, reserves of 9 and 14 GiB, the vision tower loaded (no image was sent), and at TP=3 a drafts confidence of 0.5 and the image's multirank RoCE setting (TF_GLM_ROCE_MULTI=1). Each request ran alone. Each column is one launch, and at TP=3 a second launch took the 1,035,295-token prompt again.

TP=2 TP=3
Window (tokens) 524,288 1,048,576
Model on the first machine, vision tower loaded (GiB) 94.05 66.38
Decode, bench --kinds decode (tok/s, median of three, drafts on) 51.47 88.93
An edit reply, bench --kinds edit (tok/s, median of three, drafts on) 133.68 176.77
Prefill of about 38,960 tokens (tok/s, median of three) 1,521.7 1,679.7
A 199,307-token prompt, first token (s) 135.7 ² 117.7
A 1,035,295-token prompt, first token (s) not run: the window 690.0 and 675.4 ¹
tool-eval-bench 3.0.0, straight to the engine 96 94
tool-eval-bench 3.0.0, through GLM-5.3-Flash BIZ's tool-argument gate 97 96

At TP=2 it did not start with the image's 1,048,576-token window, four streams and the vision tower. The engine's line: "--parallel 4 needs 1.16 GiB for its other streams' fixed parts, state runs and a page each, and the CUDA memory budget leaves 13.60 GiB: serve fewer, or free device memory". It started with the next window tried, 524,288 tokens; fewer streams were not tried. The columns also differ in more than the TP size: TP=3 has the drafts confidence and TF_GLM_ROCE_MULTI=1, TP=2 the smaller window, and the experts' kernel form, left to the engine's timing at load, came out classic in the TP=2 launch and the first TP=3 launch and factored in the second.

¹ Two takes, one in each TP=3 launch. In each, the hosts' clock-cap watcher lowered the GPU clock cap from 2,200 to 2,000 MHz on two of the three machines while the prompt was read; the second take came after the caps had gone back up. Both takes answered all three of the prompt's passphrases, in 170 tokens. Every other speed row ran before any such switch. ² The engine's own heat band (92 and 88 °C) paused this prompt for 2 s twice. No TP=3 speed row was paused.

Decode and the edit reply include drafts (MTP and copies), whose acceptance depends on the text written, so they compare only with runs of the same prompts. tool-eval-bench ran its 69 scenarios with seed 42, temperature 0, one at a time, 4,096 tokens and reasoning effort low; every run was valid for comparison, and the tool's Safety Gate passed in all four with no warning. Each score is one run, with no interval. The gate is GLM-5.3-Flash BIZ v3's glm53_tf tool-gate, which checks tool-call arguments in front of the engine. Free memory at its lowest on the first machine was 8.73 GiB at TP=2, against the 8 GiB at which this setup's watchdog stops the engine, and 13.89 GiB at TP=3. These figures do not line up with In serving above (another build, window and stream count, and drafts on), nor with the GLM-5.3-Flash cards' Speed tables (another model, another engine build, and there tool-eval-bench 2.8.0).

Known behaviour

The chat template is the source model's. An assistant turn with no reasoning_content whose content contains </think> is re-rendered by the template: the text before the first </think> goes into the thinking block, and the text between the first and the last </think> is dropped. Passing reasoning_content, even an empty string, keeps the content as written. The template was rendered with plain jinja2 and with transformers' apply_chat_template, outside any serving engine, and the two gave the same text.

In user, system and tool messages, in reasoning_content and in tool-call arguments, <|user|>, <tool_call> and </think> stay ordinary text (checked with TensorFold's /tokenize on Jammed XSFP4, whose tokenizer.json, tokenizer_config.json and chat_template.jinja are this checkpoint's, byte for byte).

Not run

  • TP=4 or wider, and three or more requests at once.
  • Video, and the scientific-input projector.
  • The source's deployment_guide.md notes that in a non-streaming reply, the model writing </think> or <tool_call> as text can end the reasoning early or hide a tool call; replies were not checked for it here.

How it is made

The fit, the global scale, the packing

The fit. Each routed expert has three matrices: gate, up and down. Each is cut into blocks of 16 weights along a row. A block is stored as 16 E2M1 codes and one E4M3 scale; a weight reads as code × scale × the matrix's global scale. For each block the scale is the one, among 25 E4M3 values around the block's largest weight divided by 6, that leaves the smallest squared error after rounding. In the 42 routed layers and the MTP layer, no block's best scale fell at either end of the search.

The global scale is a power of two, one for an expert's gate and up and one for its down, set so that the search above never reaches E4M3's largest value.

The packing. A row's block scales are stored as csf_32x2_v1, the lossless form nvidia/GLM-5.3-Flash-FP4-Local uses: a base byte a row, a 4-bit step a scale, and a list of the scales that fall outside the row's 16 steps (1.48% of the 19.0 billion scales of layers 3 to 44). Every packed scale was decoded again and compared with the fitted bytes.

Nothing learned. No calibration text, no activation statistics and no distillation went into this checkpoint.

Where it comes from

Part Origin
BF16 weights internlm/Sage-S2.1-Flash, revision 76f4513db7464485605a002a06a6190de0460574
FP8 weights of MLA attention and the shared experts internlm/Sage-S2.1-Flash-FP8, revision 716e6436905650b7be93e1b7dedfb0bb95afd3a3
Architecture GLM-5.3-Flash's (zai-org/GLM-5.3-Flash, MIT): the same tensor names and shapes
Scale packing csf_32x2_v1, from local-inference-lab/flashinfer (Apache-2.0), as used by nvidia/GLM-5.3-Flash-FP4-Local
License MIT, by the source README's statement; GLM-5.3-Flash's MIT text in LICENSE-GLM-5.3-Flash

Files

Files What they hold
e2m1-{gate,up,down}-L03.safetensors to -L44 (126 files, 163.02 GB) The routed experts of one layer and one projection each: 288 experts' codes, packed scales, global scale and input_scale
e2m1-{gate,up,down}-L45.safetensors (3 files, 4.08 GB) The MTP layer's routed experts, block scales stored directly
model-shell-0000{1..4}-of-00004.safetensors (16.62 GB) Everything else: 13.86 GB of BF16 and F32, 2.29 GB of FP8, 0.47 GB of MXFP8
model.safetensors.index.json Which file holds each of the 295,564 tensors
config.json, hf_quant_config.json The model's config with ModelOpt's MIXED_PRECISION table: an entry for each quantized matrix, and one for each layer's routed experts
tokenizer.json, tokenizer_config.json, chat_template.jinja, generation_config.json, processor_config.json, processing_glm5.py, tokenization_glm5.py, deployment_guide.md Copied from the source repository at the revision above
LICENSE-GLM-5.3-Flash GLM-5.3-Flash's MIT text
SHA256SUMS The sha256 of the checkpoint's 144 files: every row above but the license

Left out: the source's README.md and README_CN.md (linked above), assets/, projectors/ and vllm_patch/.

License

MIT. The source repository, internlm/Sage-S2.1-Flash, states in its README: "The code and model weights in this repository are released under the MIT License." The official FP8 repository, internlm/Sage-S2.1-Flash-FP8, whose bytes are 2.29 GB of this checkpoint, has the same sentence. As of 2026-11-17 neither ships a LICENSE file or has a licence field on its card, so that sentence is this checkpoint's basis, and there is no copyright line of Sage's to reproduce here. Sage S2.1 Flash has GLM-5.3-Flash's architecture; GLM-5.3-Flash is under the MIT License (Copyright (c) 2026 Z.AI Co., Ltd), whose text is in LICENSE-GLM-5.3-Flash. No calibration data was used.

Downloads last month
170
Safetensors
Model size
343B params
Tensor type
F32
·
BF16
·
F8_E4M3
·
U32
·
U16
·
U8
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Bizuayeu/Sage-S2.1-Flash-Mixed-NVFP4

Quantized
(14)
this model