Nemotron-3.5-Embed-8B — GGUF

GGUF conversions of nvidia/Nemotron-3.5-Embed-8B-BF16 — NVIDIA's 8.38B-parameter multimodal (text + document-image) multilingual embedding model. Produces 4096-dim L2-normalized embeddings, max context 32,768 tokens, mean pooling. For the smaller sibling (1B, 2048-dim, same text embedding space), see Nemotron-3.5-Embed-1B-GGUF.

Converted with llama.cpp @ 19ab5a5 (2026-10-22), which supports this architecture natively (Mistral3EmbedNemotronModel / Mistral3EmbeddingVisionModel). Quantized with llama-quantize.

Files

File	Size
BF16 (master)	15177 MB
F16	15177 MB
F32	30345 MB
Q8_0	8067 MB
Q6_K	6230 MB
Q5_K_M	5426 MB
Q5_K_S	5291 MB
Q5_0	5291 MB
Q5_1	5733 MB
Q4_K_M	4670 MB
Q4_K_S	4437 MB
Q4_0	4407 MB
Q4_1	4849 MB
IQ4_NL	4452 MB
IQ4_XS	4242 MB
Q3_K_L	4134 MB
Q3_K_M	3826 MB
Q3_K_S	3467 MB
IQ3_M	3587 MB
IQ3_S	3485 MB
IQ3_XS	3322 MB
IQ3_XXS	3075 MB
Q2_K	3030 MB
Q2_K_S	2834 MB
IQ2_M	2743 MB
IQ2_S	2550 MB
IQ2_XS	2451 MB
IQ2_XXS	2241 MB
TQ1_0	1920 MB
TQ2_0	2252 MB
IQ1_M	2000 MB
IQ1_S	1856 MB
Q1_0	1423 MB
Q2_0	2418 MB
mmproj-BF16 (vision encoder + projector)	827 MB
imatrix-wikitext-2 (calibration stats)	10 MB

*mmproj*.gguf — Pixtral vision encoder + projector (BF16), only needed for image inputs.
Quants below Q4 were built using an importance matrix computed on wikitext-2-raw (200 chunks × 512 tokens, BF16 master) — imatrix-wikitext-2.gguf is included for reproducibility.
Embedding models are more quantization-sensitive than chat models: Q8_0 / Q6_K / Q5_K_M are the recommended tiers; the IQ1/IQ2/TQ/Q1/Q2 tiers are provided for completeness and will measurably degrade retrieval quality.

Usage (text embeddings)

llama-server -m Nemotron-3.5-Embed-8B-Q8_0.gguf --embeddings --port 8080

curl http://127.0.0.1:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input": ["query: What is NVIDIA NeMo Framework?"]}'

Pooling (mean) is baked into the GGUF metadata — no --pooling flag needed. Follow the original model card's prompt format: query: prefix for queries, passage: prefix for documents.

Image embeddings: the mmproj file is included and loads into llama-server, but llama.cpp does not yet accept image input on its embedding endpoint for this architecture — text-only usage works today; image support should arrive in a future llama.cpp release.

Conversion (reproducibility)

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && git checkout 19ab5a5
python convert_hf_to_gguf.py <hf-model-dir> --outfile model-BF16.gguf --outtype bf16
python convert_hf_to_gguf.py <hf-model-dir> --mmproj --outfile mmproj.gguf --outtype bf16
./build/bin/llama-quantize model-BF16.gguf model-Q6_K.gguf Q6_K

Credit & license

All credit for the model goes to NVIDIA. The original model is governed by the OpenMDW License 1.1, which also applies to these conversions. This repository only redistributes converted weights; see the original model card for full details, benchmarks, and intended use.