myroslavtryhubets/d1-3b-NVFP4

d1-3B — NVFP4

An unofficial, community NVFP4 quantization of LiquidAI/d1-3B (revision 051bcc464b01b9f92942b364d9586b0ef5912432): the language-model weights are 4-bit and the activations of those layers are 4-bit too ("W4A4"). d1-3B is a decision model: you give it a state (text, JSON, images) and typed questions, and it returns calibrated answers in one forward pass with zero output tokens. See the original card for what it is for and how to write questions.

Not affiliated with or endorsed by Liquid AI.

License: LFM Open License v1.0, not Apache. Commercial use is licensed only for organizations whose annual revenue is below US$10 million (section 5; the test applies to you and to entities that control you or are under common control with you). Read the LICENSE file in this repo before using the model at work. Qualified non-profits are exempt from the threshold for non-commercial or research use (section 5(c)). Above the threshold, Liquid AI says to contact sales@liquid.ai for a commercial license. This repo is shared free of charge; it is not legal advice.

Modified files (LFM Open License section 4(b)): model-*.safetensors and model.safetensors.index.json (the weights are quantized), config.json (a quantization_config block was added), input_scales.safetensors, calibration.json and d1_nvfp4.py are new. modeling_d1.py, hybrid.py, lfm2_vl.py, runner.py, prompt.py, api.py, the tokenizer and processor files and the chat template are byte-identical to the original release.

What you get

Measured on one RTX 5060 Ti 16 GB (Blackwell, sm_120), torch 2.14.0+cu130, transformers 5.17.0. Warm calls, CUDA-synchronized, batch 1 unless noted. The states are UltraChat conversation openings (about 125 tokens, one yes/no question) and one 2,688-token state.

	bf16 (original)	this model
Weights in VRAM (whole model, vision tower and embeddings included)	5.83 GiB	2.59 GiB
Peak VRAM in this benchmark	6.40 GiB	3.17 GiB
One question, 125-token state, p50 / p95	22.1 / 27.9 ms	39.6 / 46.2 ms
Three questions over one state, p50 / p95	30.4 / 37.2 ms	42.5 / 50.5 ms
2,688-token state	346 ms	222 ms
64 states packed into one call (system_one_batch)	60.5 states/s	89.5 states/s
Median per request over 20,337 Decision Index requests (mixed lengths, batch 1)	82.1 ms	53.6 ms

Short single requests are slower than bf16: they are bound by the number of small kernels, and every quantized layer adds activation quantization steps (eager PyTorch, no CUDA graphs). The FP4 matrix multiplications pay off for long states and batches. The memory saving does not depend on request shape.

Quality

Decision Index 0.2.1, the 11 benchmarks and 20,337 requests we could build with the official kit (apolinario/decision-index@87d4650); scored with the kit's own scorer, same rows, same questions, through the release's own system_one(). Liquid has not published per-benchmark numbers for these 11, so the reference column is our own bf16 run of the original checkpoint (repeating it on 4,000 requests gave bit-identical answers, so differences below are not run-to-run noise). The last column is a paired bootstrap over cases (1,000 resamples, 95% interval) of the score difference to bf16, using the kit's scorer on each resample.

Benchmark	Metric	bf16	this model (W4A4)	difference to bf16 [95% interval]
BFCL tool selection	case exact acc.	89.3	88.8	-0.41 [-0.95, +0.18]
BANKING77	macro-F1	87.9	86.9	-1.07 [-1.64, -0.46]
CLINC150+OOS	macro-F1	91.2	91.6	+0.42 [+0.09, +0.76]
ContractNLI	macro-F1	80.5	80.5	+0.04 [-0.75, +0.90]
ANLI	macro-F1	62.3	60.5	-1.83 [-2.82, -0.91]
ARC-Challenge	accuracy	88.1	86.9	-1.19 [-2.30, -0.09]
WinoGrande	accuracy	77.7	76.3	-1.42 [-2.76, -0.16]
MuSR	accuracy	56.9	55.6	-1.33 [-3.86, +1.20]
FinEntity	macro-F1	88.1	88.0	-0.16 [-1.05, +0.70]
CRUXEval	accuracy	30.9	30.9	+0.00 [-2.46, +2.46]
PhishNChips	accuracy	53.9	52.8	-1.05 [-1.65, -0.55]
mean absolute difference to bf16			0.81 (max 1.83)

This quantization is not lossless. Five of the eleven benchmarks lose between 1.1 and 1.8 points with intervals that exclude zero, one gains 0.4, and the rest are within noise. On the same requests the chosen option is identical to bf16 in 83% (CRUXEval) to 99% (BFCL) of the answered fields, and the probability of bf16's chosen option moves by 0.02-0.08 on average (per-benchmark table below).

Benchmark	fields	same option as bf16	mean abs change of the probability of bf16's choice
BFCL	4768	98.74%	0.0202
BANKING77	3080	96.56%	0.0392
CLINC150+OOS	5500	95.15%	0.0431
ContractNLI	2091	96.89%	0.0391
ANLI	3200	91.56%	0.0735
ARC-Challenge	1172	95.99%	0.0413
WinoGrande	1267	94.48%	0.0573
MuSR	756	87.17%	0.0731
FinEntity	2129	96.10%	0.0351
CRUXEval	570	82.63%	0.0774
PhishNChips	18000	94.14%	0.0483

Calibration quality of the probabilities (what makes a decision model useful for thresholds) was not re-measured; the probabilities do shift as shown above.

Image path. Images were only smoke-tested, not benchmarked: 12 synthetic images (rendered text, simple drawings, noise, a chart) as the whole state, three questions each. All 36 decisions (threat yes/no, text yes/no, kind of image) match bf16; the largest probability change is 0.03 for the yes/no questions and 0.14 for the confidence of an ambiguous "what kind of image" pick. This says nothing about accuracy on real images.

How it was made

• Weights: llm-compressor 0.14 model-free PTQ, NVFP4, group size 16, FP8-E4M3 block scales, compressed-tensors nvfp4-pack-quantized, with llm-compressor's NVFP4-tuned MSE observer (it searches each 16-value group's scale for the lowest rounding error). 166 Linear layers of the language model are quantized: attention q/k/v/out, the short-convolution in_proj/out_proj, and the MLP w1/w2/w3. q/k/v and w1+w3 share one global weight scale (the pairs an inference engine would fuse). Kept in bf16: the SigLIP2 vision tower, the multimodal projector, token embeddings (tied output head), all norms and the depthwise convolution taps.
• Activations: static per-tensor input global scales (448*6 / max|x|) from the bf16 model run through its own system_one on 420 calibration requests built in the Decision Index question format, from training splits only: 60 each from BANKING77 train (MIT), CLINC150 plus train (CC BY 3.0), SNLI train (CC BY-SA 4.0, posed as an entailment question), ARC-Challenge train (CC BY-SA 4.0), WinoGrande XL train (license not stated in its dataset metadata), and 120 UltraChat test_sft conversations (MIT) with generic yes/no and topic questions. Only per-layer maximum activation values were taken from this text (see calibration.json); no data is redistributed. The test rows used above are not part of it. Layer 4's w2 input has a very large outlier (max about 20,000 times the mean magnitude), which sets a coarse scale for that one layer.

Use

The checkpoint is not a plain from_pretrained checkpoint: the packed 4-bit weights need d1_nvfp4.py from this repo (a short file; it builds the original model from config.json and swaps the Linear layers for NVFP4 ones). Tested with torch 2.14.0+cu130, transformers 5.17.0 and, for images, torchvision 0.29.0. transformers>=5.14 is required by the original code, and trust_remote_code=True runs the release's own code in this repo (unchanged, see above).

pip install -U "huggingface_hub[cli]" "transformers>=5.14" torchvision pillow     # torchvision/pillow only for images
hf download myroslavtryhubets/d1-3b-NVFP4 --local-dir d1-3b-NVFP4

import sys; sys.path.insert(0, "d1-3b-NVFP4")                  # the folder downloaded above
from d1_nvfp4 import load_d1_nvfp4
model = load_d1_nvfp4("d1-3b-NVFP4")                           # W4A4 on Blackwell GPUs
questions = {"refund": {"type": "noul", "instructions": "Is the customer asking for a refund?"}}
print(model.system_one("I was charged twice this month, please refund one of them.", questions))

W4A4 uses torch._scaled_mm with FP4 operands, which needs a GPU with FP4 tensor cores (Blackwell); we tested sm_120 only.

Not tested

• Other GPUs (all numbers are from one RTX 5060 Ti), other torch / transformers versions, vLLM / SGLang / llama.cpp loading (the decision head is custom code; we did not try).
• The rest of the Decision Index panel and Liquid's headline 48.57; the benchmarks above are 11 of the suite, scored against our own bf16 run.
• Image benchmarks (only the 12-image smoke test above), audio, long states beyond about 16k tokens.
• Calibration of the probabilities, thresholds other than the default argmax, and the "benchmarks as decisions" / DecisionBench / Fast Decisions sets from the original card.
• Like the original, this is a classifier that makes mistakes; do not use it as the only control for decisions that matter.

Credits

Model, architecture, system-one API and prompt format by Liquid AI (model, base LFM2.5-VL-3B); Decision Index by its authors. Quantization by @myroslavtryhubets.