kingjones777/Flint-40B-A2.5B-ROCmFP4-GGUF

Flint 40B-A2.5B — ROCmFP4 GGUF (Strix Halo / gfx1151)

A compact 4-bit ROCmFP4 build of RecursiveSI/flint (39.8B total / ~2.5B active coding-agent MoE), plus a lossless BF16 GGUF conversion, built and measured on one AMD Ryzen AI Max+ 395 "Strix Halo" (gfx1151, ROCm 7.2.4, 128 GiB unified memory).

Every number on this page was measured, on these files or, in the comparison rows, on the variant named there. Nothing is estimated or carried over from the base model.

⚠️ Runtime: these files need an engine with the flint architecture

• general.architecture is flint (49 layers: 14 full-attention layers without positional encoding, 35 sliding-window layers of 513 tokens with RoPE, learned attention sinks on every layer, 256-expert sigmoid-routed MoE with an ungated shared expert). Stock llama.cpp has no flint architecture.
• The 4-bit file uses ROCmFP4 tensor types (≥ 100), which stock llama.cpp does not know.
• The measurements below come from Super ROC, an open-source inference engine for gfx1151 (the code published there).

Files

file	contents	size
Flint-40B-A2.5B-Q4_0-ROCmFP4-COHERENT-PAS.gguf ⭐	experts ROCmFP4 (4.5-bit dual-scale), embedding / output / all attention / layers 0-1 / shared experts Q8_0	23.43 GB (4.71 BPW)
Flint-40B-A2.5B-BF16-0000{1,2}-of-00002.gguf	lossless BF16 conversion of the checkpoint (source for re-quantization)	79.63 GB

Source checkpoint: RecursiveSI/flint revision 2a247be0c0aec895ccb0ac7ca92229ec6df37288. The BF16 conversion is bit-identical to the safetensors tensors (spot check of 6 tensors including a stacked expert, the sinks, the output head and the embedding).

Why "PAS" (8-bit protection) instead of plain 4-bit

Flint's first layer nearly cancels the token embedding (hidden-state norm 4.7 → 1.0 after layer 0), so rounding errors in that layer are amplified. Measured against the BF16 reference implementation (see below):

build	hidden-state error after layer 0	teacher-forced next-token agreement (code / chat / prose)
plain COHERENT (embedding Q6_K, every other projection ROCmFP4)	36 %	28/48 · 119/160 · 24/48
COHERENT-PAS (this file)	1.6 %	46/48 · 150/160 · 36/48

A lighter variant (8-bit only on the embedding, output head, layers 0-1 and shared experts; attention of layers 2-48 at 4-bit) decoded 8 % faster but dropped to 43/48 · 141/160 · 32/48 and was not kept.

Quality

Reference: a float32 PyTorch implementation of the authors' SGLang model code (flint_sglang/model.py), run on the BF16 weights. Teacher-forced agreement = the share of positions where this file's argmax equals the reference's greedy token.

check	this file	BF16 reference
teacher-forced agreement, Python continuation (48 tokens)	46/48 (95.8 %)	—
teacher-forced agreement, chat answer with reasoning (160 tokens)	150/160 (93.8 %)	—
perplexity, Python code (3 × 2048-token windows, 2nd half scored)	6.57 (token by token) / 6.59 (batched prompt)	7.48
perplexity, long mixed technical text (2 × 4096-token windows)	44.97	47.04

The quantized file scores a lower perplexity than BF16 on both corpora. The same PyTorch reference running on this file's dequantized weights reproduces that (scored half of a 2048-token code window: 9.16 vs 12.71 for BF16), so it is a property of the quantized weights on this text, not of the engine; on an 1100-token code sample the engine's mean per-token loss is within 0.3 % (first 512 positions) to 2.3 % of that dequantized reference. Flint is a chat/agent model: its perplexity on raw prose is high at any precision.

Speed (Strix Halo, Super ROC, COHERENT-PAS)

prompt tokens	prompt processing	generation (128 tokens after the prompt)
512	1119 tok/s	58.9 tok/s
4096	1734 tok/s	52.7 tok/s
16384	1539 tok/s	41.6 tok/s

Through the HTTP server: a 451-token answer without tools generated at 58.6 tok/s; a 159-token tool call under the tool grammar at 51.9 tok/s (different requests, not a controlled comparison).

Serving notes

• Chat template: embedded (tokenizer.chat_template, the authors' template). Roles <|system|> <|developer|> <|user|> <|assistant|> <|observation|>; the model opens <think> itself when it decides to reason.
• Stop tokens: <|end_of_text|> (150001), <|user|> (150003), <|observation|> (150005), <|developer|> (150018) — the GGUF's single eos_token_id is 150001, so make sure your server stops on all four.
• Tool calls use <tool_call>NAME<arg_key>K</arg_key><arg_value>V</arg_value></tool_call>. Constrain them with a grammar limited to the request's tool names and argument keys (the authors serve Flint that way). Unconstrained at temperature 0 the model wrote its own argument name command for a shell tool whose schema asked for cmd; with the grammar it writes cmd, and a tool with two unusual required keys (alpha_cmd, beta_dir) got exactly those keys.
• Sampling: the authors recommend temperature 1.0, top-p 0.95, top-k 20. In one agent benchmark a single turn ran to the 8192-token output cap; set a sensible max_tokens.

Reproduction

convert   safetensors -> GGUF arch "flint" (BF16 tensors written raw, F32 norms / sinks / router), streamed
quant     gufo-quant <bf16> <out> COHERENT 24 \
            --tensor-type =token_embd.weight=Q8_0 --tensor-type =output.weight=Q8_0 \
            --tensor-type attn_q.weight$=Q8_0 --tensor-type attn_k.weight$=Q8_0 --tensor-type attn_v.weight$=Q8_0 \
            --tensor-type attn_output.weight$=Q8_0 --tensor-type _shexp.weight$=Q8_0 \
            --tensor-type =blk.0.ffn_gate.weight=Q8_0 --tensor-type =blk.0.ffn_up.weight=Q8_0 \
            --tensor-type =blk.0.ffn_down.weight=Q8_0 --tensor-type =blk.1.ffn_gate.weight=Q8_0 \
            --tensor-type =blk.1.ffn_up.weight=Q8_0 --tensor-type =blk.1.ffn_down.weight=Q8_0
ppl       windows of n_ctx tokens from the start of the corpus, positions n_ctx/2 .. n_ctx-2 score the next token, no BOS
speed     one warm-up, then the timed prompt and 128 greedy tokens

Credits

Base model: RecursiveSI/flint, Apache-2.0. Converted, quantized and measured by kingjones777; these files inherit Apache-2.0.